Skip to content

Commit

Permalink
Merge pull request #29 from crunchdao/example-update
Browse files Browse the repository at this point in the history
2.1.0
  • Loading branch information
Caceresenzo authored May 9, 2024
2 parents 4a66e46 + 104c005 commit 1176ae7
Show file tree
Hide file tree
Showing 12 changed files with 404 additions and 398 deletions.
21 changes: 3 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ A small backtesting utility.
- [Exporters](#exporters)
- [Console](#console)
- [Dump](#dump)
- [Influx](#influx)
- [QuantStats](#quantstats)
- [PDF](#pdf)
- [Specific Return](#specific-return)
Expand Down Expand Up @@ -39,7 +38,7 @@ bktest [OPTIONS]
| --- | --- | --- | --- | --- |
| `--start` | `<start date>` | `orders' first date` | `date` (ISO-8601) | The starting date of the backtesting. If the value is before the first ordering day, the value will be discarded. |
| `--end` | `<end date>` | `orders' last date` | `date` (ISO-8601) | The ending date of the backtesting. If the value is after today, the value will be discarded. |
| `--offset-before-trading` | `<days>` | `1` | `int` | Number of day to offset to push the signal before trading it. |
| `--offset-before-trading` | `<days>` | `1` | `int` | Number of day to offset to push each date of the portfolio before trading it. |
| `--offset-before-ending` | `<days>` | `0` | `int` | Number of day to continue the backtest after every orders. |
| `--order-file` | `<file>` | | `path` | The single order file to use. The file must contain symbol, quantity and date information. |
| `--single-file-provider-column-date` | `<column>` | `date` | `string` | Change the date column name to use. |
Expand All @@ -49,13 +48,13 @@ bktest [OPTIONS]
| `--order-files-extension` | `<extension>` | `csv` | `[csv, parquet, json]` | Change the file extension to use when listing for order files. |
| `--initial-cash` | `<amount>` | `100_000` | `number` | Change the initial cash to use for the backtesting. |
| `--quantity-mode` | `<mode>` | `percent` | `[percent, share]` | If the mode is `share`, all quantities will be interpreted as integers. If the mode is `percent`, all values will be multiplied by the current cash value. |
| `--auto-close-others` | | `false` | | Should other position be closed after an ordering? |
| `--weekends` | | `false` | | Enable ordering on weekends. |
| `--holidays` | | `false` | | Enable ordering on holidays. |
| `--symbol-mapping` | `<mapping>` | | `path` (.json) | Specify a custom symbol mapping file enabling vendor-id translation. |
| `--no-caching` | | `false` | | Disable prices caching. |
| `--fee-model` | `<model>` | | `expression` or `constant` | Specify a fee model to use. The value can be a `constant`. Or an expression that allow the usage of the `price` and `quantity` variable. <br /> Example: `abs(price * quantity) * 0.1` |
| `--rfr-file` | `<directory>` | | `path` | The directory of rfr file to use. The file must contain a column with date information and a column with the rfr information in %. |
| `--holiday-provider` | `<name>` | `nyse` | `[legacy, nyse]` | Specify which holiday provider to use. |
| `--rfr-file` | `<directory>` | | `path` | The directory of rfr file to use. The file must contain a column with date information and a column with the rfr information in %. |
| `--rfr-file-column-date` | `<column>` | `date` | `string` | Change the date column name to use. |

### Exporters
Expand Down Expand Up @@ -84,20 +83,6 @@ The dump exporter generate a dump of the portfolio at each day.
| `--dump-output-file` | `<file>` | `dump.csv` | `path` | Specify the output file. |
| `--dump-auto-delete` | | `false` | | Automatically delete the previous dump file if it is present. |

#### Influx

Export the generated data to an Influx database. <br />
Making it easier to plot the values using software like Grafana.

| Option | Value | Default | Format | Description |
| --- | --- | --- | --- | --- |
| `--influx` | | `false` | | Enable the influx exporter. |
| `--influx-host` | `<host>` | `localhost` | `string` | Specify the remote influx address. |
| `--influx-port` | `<port>` | `8086` | `number` | Specify the remote influx port. |
| `--influx-database` | `<database>` | `backtest` | `string` | Specify the influx database to use. |
| `--influx-measurement` | `<measurement>` | `snapshots` |`string` | Specify the table name to use. |
| `--influx-key` | `<key>` | `test` | `string` | Specify the unique key to use. **Previous data with the same key will be deleted!** |

#### QuantStats

Generate a tearsheet from the backtest data.
Expand Down
2 changes: 1 addition & 1 deletion bktest/__version__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
__title__ = 'bktest'
__description__ = 'bktest - A simple backtester by CrunchDAO'
__version__ = '2.0.0'
__version__ = '2.1.0'
__author__ = 'Enzo CACERES'
__author_email__ = '[email protected]'
__url__ = 'https://github.com/crunchdao/backtest'
41 changes: 24 additions & 17 deletions bktest/backtest.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import typing

from .account import Account
from .data.holidays import HolidayProvider, LegacyHolidayProvider
from .data.source.base import DataSource
from .export import Exporter, ExporterCollection
from .fee import ConstantFeeModel, FeeModel
Expand Down Expand Up @@ -149,8 +150,9 @@ def __init__(
mapper: SymbolMapper = None,
fee_model: FeeModel = ConstantFeeModel(0.0),
caching=True,
weekends=False,
holidays=False,
allow_weekends=False,
allow_holidays=False,
holiday_provider: HolidayProvider = LegacyHolidayProvider(),
):
self.order_provider = order_provider
order_dates = order_provider.get_dates()
Expand Down Expand Up @@ -179,8 +181,9 @@ def __init__(
end,
self.price_provider.is_closeable(),
order_dates,
weekends,
holidays
holiday_provider,
allow_weekends,
allow_holidays
)

def update_price(self, date):
Expand Down Expand Up @@ -226,12 +229,13 @@ def order(
def run(self):
self._fire_initialize()

for date, ordered, postponned in self.date_iterator:
for postpone in postponned:
for date, ordered, skips in self.date_iterator:
for skip in skips:
for pod in self.pods:
pod.exporters.fire_skip(postpone.date, postpone.reason, True)
pod.exporters.fire_skip(skip.date, skip.reason, skip.ordered)

self.order(postpone.date, price_date=date)
if skip.ordered:
self.order(skip.date, price_date=date)

self.update_price(date)

Expand Down Expand Up @@ -265,8 +269,9 @@ def __init__(
mapper: SymbolMapper = None,
fee_model: FeeModel = ConstantFeeModel(0.0),
caching=True,
weekends=False,
holidays=False,
allow_weekends=False,
allow_holidays=False,
holiday_provider: HolidayProvider = LegacyHolidayProvider(),
):
self.order_provider = order_provider
order_dates = order_provider.get_dates()
Expand All @@ -287,8 +292,9 @@ def __init__(
end,
self.price_provider.is_closeable(),
order_dates,
weekends,
holidays
holiday_provider,
allow_weekends,
allow_holidays,
)

def update_price(self, date):
Expand Down Expand Up @@ -318,12 +324,13 @@ def order(
def run(self):
self.exporters.fire_initialize()

for date, ordered, postponned in self.date_iterator:
for postpone in postponned:
self.exporters.fire_skip(postpone.date, postpone.reason, True)
for date, ordered, skips in self.date_iterator:
for skip in skips:
self.exporters.fire_skip(skip.date, skip.reason, skip.ordered)

result = self.order(postpone.date, price_date=date)
self.exporters.fire_snapshot(date, self.account, result, postponned=postpone.date)
if skip.ordered:
result = self.order(skip.date, price_date=date)
self.exporters.fire_snapshot(date, self.account, result, postponned=skip.date)

self.update_price(date)
self.exporters.fire_snapshot(date, self.account, None)
Expand Down
79 changes: 54 additions & 25 deletions bktest/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,96 +23,128 @@


@click.group(invoke_without_command=True)
#
@click.option('--start', type=click.DateTime(formats=["%Y-%m-%d"]), default=None, help="Start date.")
@click.option('--end', type=click.DateTime(formats=["%Y-%m-%d"]), default=None, help="End date.")
@click.option('--offset-before-trading', type=int, default=1, show_default=True, help="Number of day to offset to push the signal before trading it.")
#
@click.option('--offset-before-trading', type=int, default=0, show_default=True, help="Number of day to offset to push each date of the portfolio before trading it.")
@click.option('--offset-before-ending', type=int, default=0, show_default=True, help="Number of day to continue the backtest after every orders.")
#
@click.option('--order-file', type=click.Path(exists=True, dir_okay=False), required=True, show_default=True, help="Specify an order file to use.")
@click.option('--order-file-column-date', '--single-file-provider-column-date', type=str, default=constants.DEFAULT_DATE_COLUMN, show_default=True, help="Specify the date column name.")
@click.option('--order-file-column-symbol', '--single-file-provider-column-symbol', type=str, default=constants.DEFAULT_SYMBOL_COLUMN, show_default=True, help="Specify the symbol column name.")
@click.option('--order-file-column-quantity', '--single-file-provider-column-quantity', type=str, default=constants.DEFAULT_QUANTITY_COLUMN, show_default=True, help="Specify the quantity column name.")
#
@click.option('--initial-cash', type=int, default=100_000, show_default=True, help="Specify an initial cash amount.")
@click.option('--quantity-mode', type=click.Choice(['percent', 'share']), default="percent", show_default=True, help="Use percent for weight and share for units.")
@click.option('--auto-close-others', is_flag=True, help="Close the position that hasn't been provided after all of the order.")
@click.option('--auto-close-others', is_flag=True, help="[deprecated] Close the position that hasn't been provided after all of the order.")
@click.option('--weekends', is_flag=True, help="Include weekends?")
@click.option('--holidays', is_flag=True, help="Include holidays?")
@click.option('--symbol-mapping', type=str, required=False, help="Custom symbol mapping file enabling vendor-id translation.")
@click.option('--no-caching', is_flag=True, help="Disable price caching.")
@click.option('--fee-model', "fee_model_value", type=str, help="Specify a fee model. Must be a constant or an expression.")
#
@click.option('holiday_provider_name', '--holiday-provider', type=click.Choice(['legacy', 'nyse']), default="nyse", help="Specify the holiday provider to use.")
#
@click.option('--console', is_flag=True, help="Enable the console exporter.")
@click.option('--console-format', type=click.Choice(['text', 'json']), default="text", show_default=True, help="Console output format.")
@click.option('--console-file', type=click.Choice(['out', 'err']), default="out", show_default=True, help="Console output destination file.")
@click.option('--console-hide-skips', is_flag=True, show_default=True, help="Should the console hide skipped days?")
@click.option('--console-text-no-color', is_flag=True, help="Disable colors in the console output.")
#
@click.option('--dump', is_flag=True, help="Enable the dump exporter.")
@click.option('--dump-output-file', type=str, default="dump.csv", show_default=True, help="Specify the output file.")
@click.option('--dump-auto-delete', is_flag=True, help="Should conflicting files be automatically deleted?")
@click.option('--influx', is_flag=True, help="Enable the influx exporter.")
@click.option('--influx-host', type=str, default="localhost", show_default=True, help="Influx's database host.")
@click.option('--influx-port', type=int, default=8086, show_default=True, help="Influx's database port.")
@click.option('--influx-database', type=str, default="backtest", show_default=True, help="Influx's database name.")
@click.option('--influx-measurement', type=str, default="snapshots", show_default=True, help="Influx's database table.")
@click.option('--influx-key', type=str, default="test", show_default=True, help="Key to use to uniquely identify the exported values.")
#
@click.option('--quantstats', is_flag=True, help="Enable the quantstats exporter.")
@click.option('--quantstats-output-file-html', type=str, default="report.html", show_default=True, help="Specify the output html file.")
@click.option('--quantstats-output-file-csv', type=str, default="report.csv", show_default=True, help="Specify the output csv file.")
@click.option('--quantstats-benchmark-ticker', type=str, default="SPY", show_default=True, help="Specify the symbol to use as a benchmark.")
@click.option('--quantstats-auto-delete', is_flag=True, help="Should conflicting files be automatically deleted?")
#
@click.option('--pdf', is_flag=True, help="Enable the quantstats exporter.")
@click.option('--pdf-template', type=str, default="tearsheet.sketch", show_default=True, help="Specify the template file.")
@click.option('--pdf-output-file', type=str, default="report.pdf", show_default=True, help="Specify the output pdf file.")
@click.option('--pdf-auto-delete', is_flag=True, help="Should aa conflicting file be automatically deleted?")
@click.option('--pdf-debug', is_flag=True, help="Enable renderer debugging.")
@click.option('--pdf-variable', "pdf_variables", nargs=2, multiple=True, type=(str, str), help="Specify custom variables.")
@click.option('--pdf-user-script', "pdf_user_script_paths", multiple=True, type=str, help="Specify custom scripts.")
#
@click.option('--specific-return', type=str, help="Enable the specific return exporter by proving a .parquet.")
@click.option('--specific-return-column-date', type=str, default="date", show_default=True, help="Specify the column name containing the dates.")
@click.option('--specific-return-column-symbol', type=str, default="symbol", show_default=True, help="Specify the column name containing the symbols.")
@click.option('--specific-return-column-value', type=str, default="specific_return", show_default=True, help="Specify the column name containing the value.")
@click.option('--specific-return-output-file-html', type=str, default="sr-report.html", show_default=True, help="Specify the output html file.")
@click.option('--specific-return-output-file-csv', type=str, default="sr-report.csv", show_default=True, help="Specify the output csv file.")
@click.option('--specific-return-auto-delete', is_flag=True, help="Should conflicting files be automatically deleted?")
#
@click.option('--yahoo', is_flag=True, help="Use yahoo finance as the data source.")
#
@click.option('--coinmarketcap', is_flag=True, help="Use coin market cap as the data source.")
@click.option('--coinmarketcap-force-mapping-refresh', is_flag=True, help="Force a mapping refresh.")
@click.option('--coinmarketcap-page-size', default=10_000, help="Specify the query page size when building the mapping.")
#
@click.option('--factset', is_flag=True, help="Use factset prices as the data source.")
@click.option('--factset-username-serial', type=str, envvar="FACTSET_USERNAME_SERIAL", help="Specify the factset username serial to use.")
@click.option('--factset-api-key', type=str, envvar="FACTSET_API_KEY", help="Specify the factset api key to use.")
#
@click.option('--file-parquet', type=str, required=False, help="Use a .parquet file as the data source.")
@click.option('--file-parquet-column-date', type=str, default="date", show_default=True, help="Specify the column name containing the dates.")
@click.option('--file-parquet-column-symbol', type=str, default="symbol", show_default=True, help="Specify the column name containing the symbols.")
@click.option('--file-parquet-column-price', type=str, default="price", show_default=True, help="Specify the column name containing the prices.")
#
@click.pass_context
def cli(ctx: click.Context, **kwargs):
if ctx.invoked_subcommand is None:
main(**kwargs)


def main(
start: datetime.datetime, end: datetime.datetime,
start: datetime.datetime,
end: datetime.datetime,
#
offset_before_trading: int,
offset_before_ending: int,
#
order_file,
order_file_column_date: str,
order_file_column_symbol: str,
order_file_column_quantity: str,
initial_cash, quantity_mode, auto_close_others,
weekends, holidays, symbol_mapping, no_caching,
#
initial_cash,
quantity_mode,
auto_close_others,
weekends,
holidays,
symbol_mapping,
no_caching,
fee_model_value,
#
holiday_provider_name: str,
#
console, console_format, console_file, console_hide_skips, console_text_no_color,
#
dump: str, dump_output_file: str, dump_auto_delete: bool,
influx, influx_host, influx_port, influx_database, influx_measurement, influx_key,
#
quantstats, quantstats_output_file_html, quantstats_output_file_csv, quantstats_benchmark_ticker, quantstats_auto_delete,
#
pdf: bool, pdf_template: str, pdf_output_file: str, pdf_auto_delete: bool, pdf_debug: bool, pdf_variables: typing.Tuple[typing.Tuple[str, str]], pdf_user_script_paths: str,
#
specific_return: str, specific_return_column_date: str, specific_return_column_symbol: str, specific_return_column_value: str, specific_return_output_file_html: str, specific_return_output_file_csv: str, specific_return_auto_delete: bool,
#
yahoo,
#
coinmarketcap, coinmarketcap_force_mapping_refresh, coinmarketcap_page_size,
#
factset: bool, factset_username_serial: str, factset_api_key: str,
#
file_parquet, file_parquet_column_date, file_parquet_column_symbol, file_parquet_column_price,
):
logging.getLogger('matplotlib.font_manager').setLevel(logging.ERROR)

if auto_close_others:
print("[warning] `--auto-close-others` is deprecated and is forced to `true`", file=sys.stderr)

now = datetime.date.today()

quantity_in_decimal = quantity_mode == "percent"
Expand All @@ -138,6 +170,12 @@ def main(
end = now

print(f"[warning] end is after today, using: {now}", file=sys.stderr)

from .data.holidays import LegacyHolidayProvider, SimpleHolidayProvider
holiday_provider = ({
"legacy": lambda: LegacyHolidayProvider(),
"nyse": lambda: SimpleHolidayProvider.nyse()
}[holiday_provider_name])()

data_source = None
if yahoo:
Expand Down Expand Up @@ -232,16 +270,6 @@ def main(
auto_delete=dump_auto_delete,
))

if influx:
from .export import InfluxExporter
exporters.append(InfluxExporter(
host=influx_host,
port=influx_port,
database=influx_database,
measurement=influx_measurement,
key=influx_key
))

if quantstats:
from .export import QuantStatsExporter
exporters.append(QuantStatsExporter(
Expand Down Expand Up @@ -300,14 +328,15 @@ def main(
order_provider=order_provider,
initial_cash=initial_cash,
quantity_in_decimal=quantity_in_decimal,
auto_close_others=auto_close_others,
auto_close_others=True,
data_source=data_source,
mapper=symbol_mapper,
exporters=exporters,
fee_model=fee_model,
caching=not no_caching,
weekends=weekends,
holidays=holidays
allow_weekends=weekends,
allow_holidays=holidays,
holiday_provider=holiday_provider
).run()


Expand Down
Loading

0 comments on commit 1176ae7

Please sign in to comment.