FEATURE: add csvsource #1389

go-dockly · 2023-11-02T17:09:09Z

Add a tick and kline level reader interface for multiple exchanges (binance, bybit) or fin-apps like meta-trader.
The goal is to expose the stream interface on the csvsource so arbitrary tick level simulations can be run against strategy

bbgokarma-bot · 2023-11-02T17:09:13Z

Hi @go-dockly,

This is KarmaBot, and we reward your contributions with tokens sent directly to your wallet to support development.

This pull request may get 3955 BBG.

To receive BBG tokens, please provide your Polygon (can be Ethereum) address as an issue comment in this pull request, following this format:

polygon:0xAb5801a7D398351b8bE11C439e05C5B3259aeC9B

Once this pull request is merged, your BBG tokens will be transferred to your wallet.

--
If you're interested in our project, feel free to join our Telegram group https://t.me/bbgo_intl

Best,
KarmaBot

bbgokarma-bot · 2023-11-02T18:58:19Z

Re-estimated karma: this pull request may get 4072 BBG

bbgokarma-bot · 2023-11-02T21:59:21Z

Re-estimated karma: this pull request may get 4126 BBG

bbgokarma-bot · 2023-11-02T22:37:23Z

Re-estimated karma: this pull request may get 4152 BBG

c9s · 2023-11-02T23:20:30Z

@go-dockly can do you git rebase against the main branch? and maybe squash some fix commits? I saw you have merge-point commits, thanks

c9s · 2023-11-02T23:21:16Z

git checkout feat/csvsource
git fetch upstream
git rebase upstream/main
git push -f origin HEAD

bbgokarma-bot · 2023-11-02T23:26:37Z

Re-estimated karma: this pull request may get 4222 BBG

bbgokarma-bot · 2023-11-03T11:05:45Z

Re-estimated karma: this pull request may get 4228 BBG

pkg/datasource/csvsource/csv_kline_reader_test.go

pkg/datasource/csvsource/csv_tick_reader_test.go

pkg/datasource/csvsource/read_klines_test.go

pkg/datasource/csvsource/stream.go

c9s · 2023-11-18T02:26:49Z

pkg/backtest/exchange.go

@@ -375,12 +376,12 @@ func (e *Exchange) SubscribeMarketData(
 	for symbol := range loadedSymbols {
 		symbols = append(symbols, symbol)
 	}
+	symbols = append(symbols, "FXSUSDT")


debugging code?

I didn't really find a way yet to init the loadedSymbols from config

nor did the backtest execute any trades so I assume the trade service is not properly connected to the backtest exchange. Do you have any ideas what I am missing here?

You have to set it in the config yaml. check pkg/bbgo/config.go in the Backtest struct.

backtest: symbols: - BTCUSDT - ETHUSDT

also you need to set the sync section maybe.

sync: userDataStream: trades: true filledOrders: true sessions: - binance symbols: - BTCUSDT

gonna try that thanks

pkg/exchange/kucoin/symbols.go

c9s · 2023-11-18T02:31:19Z

pkg/service/backtest_sql.go

 type BacktestService struct {
 	DB *sqlx.DB
 }

-func (s *BacktestService) SyncKLineByInterval(ctx context.Context, exchange types.Exchange, symbol string, interval types.Interval, startTime, endTime time.Time) error {
+func NewBacktestService(db *sqlx.DB) IBacktestService {


avoid setting the interface type as the return type in the constructor, you can force the interface type from the caller. e.g.,

var s IBacktestService = NewBacktestService(db)

and where would you take db from?

it's initialized from the bbgo.Environment

pkg/service/backtest_sql.go

c9s · 2023-11-18T02:35:08Z

@go-dockly I saw you're adding more changes to this PR, this could make this PR too large to review as the previous PR issue, could you please narrow down the scope of this PR, and if you need to continue the work, you can create another PR that sets the base branch to this one algo-boyz:feat/csvsource

c9s · 2023-11-18T02:36:07Z

Your original changes was almost ready to be merged, it just needs to be rebased (git rebase)

c9s · 2023-11-20T07:28:25Z

pkg/datasource/csvsource/tick_downloader.go

+	case types.ExchangeBybit:
+		csvContent, err = gunzip(body)
+		if err != nil {
+			return nil, fmt.Errorf("gunzip data %s: %w", exchange, err)


looks like the exchange variable is only used in the log?

switch exchange { case types.ExchangeBybit:

go-dockly · 2023-11-20T20:32:47Z

It wouldn't have worked if we had merged before. Please check the last commit about what I mean addMissingKLines...
I will squash once it's working. I won't add more than what it's now

…to feat/csvsource

c9s · 2023-11-21T02:03:20Z

pkg/bbgo/config.go

@@ -150,7 +151,8 @@ type Backtest struct {
 	Sessions []string                   `json:"sessions" yaml:"sessions"`

 	// sync 1 second interval KLines
-	SyncSecKLines bool `json:"syncSecKLines,omitempty" yaml:"syncSecKLines,omitempty"`
+	SyncSecKLines bool                 `json:"syncSecKLines,omitempty" yaml:"syncSecKLines,omitempty"`
+	CsvSource     *csvsource.CsvConfig `json:"csvConfig,omitempty" yaml:"csvConfig,omitempty"`


it's better to put the Config struct under the bbgo package (avoid importing csvsource, could cause cyclic import in the future)

c9s · 2023-11-21T02:03:47Z

pkg/cmd/backtest.go

@@ -163,7 +162,14 @@ var BacktestCmd = &cobra.Command{

 		environ := bbgo.NewEnvironment()

-		backtestService := service.NewBacktestServiceCSV(outputDirectory, csvsource.SPOT, csvsource.AGGTRADES) // todo make configurable
+		if userConfig.Backtest.CsvSource == nil {


this should be optional

c9s · 2023-11-21T02:06:02Z

pkg/datasource/csvsource/types.go

@@ -20,6 +26,10 @@ type CsvTick struct {
 }

 func (c *CsvTick) ToGlobalTrade() (*types.Trade, error) {
+	var isFutures bool
+	if c.Market == FUTURES {


constants should be in CamelCase (like this)

c9s · 2023-11-21T02:06:45Z

pkg/datasource/csvsource/types.go

@@ -5,8 +5,14 @@ import (
 	"github.com/c9s/bbgo/pkg/types"
 )

+type CsvConfig struct {
+	Market      MarketType `json:"market"`


move the MarketType to the types package

c9s · 2023-11-21T02:08:40Z

pkg/datasource/csvsource/types.go

+func (c *CsvTick) ToGlobalTrade() (*types.Trade, error) {
+	var isFutures bool
+	if c.Market == FUTURES {
+		isFutures = true


it could be just:

isFutures = c.Market == FUTURES

c9s · 2023-11-21T02:09:07Z

pkg/datasource/csvsource/testdata/okex/BTC-USDT-aggtrades-2023-11-18.csv

@@ -0,0 +1,16 @@
+trade_id/���id,side/���׷���,size/����,price/�۸�,created_time/�ɽ�ʱ��


what's the text encoding here?

good question not sure what okex is doing here. It's their raw content but we skip this line anyway on ingest

I guess it's chinese, maybe you need to use UTF-8 to decode it?

c9s · 2023-11-21T02:11:41Z

It wouldn't have worked if we had merged before. Please check the last commit about what I mean addMissingKLines... I will squash once it's working. I won't add more than what it's now

got it, thanks

c9s · 2023-11-21T02:13:05Z

FYI, best practice: rebase while you're git pulling

https://sdq.kastel.kit.edu/wiki/Git_pull_--rebase_vs._--merge

c9s · 2023-11-21T17:28:12Z

doc/topics/back-testing.md

+
+  1: Through csv data source (supported right now are binance, bybit and OkEx)
+
+  2: Alternatively run backtests through [MySQL](../../README.md#configure-mysql-database) or [SQLite3


Would suggest to put mysql as the default backtest engine.

Let's put the csv data source as the second one

c9s · 2023-12-08T07:43:37Z

@go-dockly waiting for you updates ;-)

c9s · 2023-12-14T02:48:31Z

@go-dockly I am trying to rebase your commits, and found you have the "reset main" commits,

please avoid doing this in the pull request, or it could make it harder to do the rebase

you should avoid merge the main branch into your branch, instead you can use rebase

c9s · 2023-12-14T02:53:45Z

@go-dockly this is what the "reset main" causes during the rebase process :/

c9s · 2023-12-14T07:03:31Z

I have rebased the commits in this PR #1454

c9s · 2023-12-14T07:50:52Z

@go-dockly please continue your work from the pull request #1454 (branch algo-boyz/feat/csvsource)

go-dockly and others added 6 commits November 2, 2023 02:16

add more trade stats

f19bbbc

clean up

0c2f4ae

Merge branch 'main' into main

93c619c

fix drawdown

539e35d

Merge remote-tracking branch 'origin'

f7d5429

reset main

0d7990f

go-dockly changed the title ~~FEATURE: csvsource~~ WIP FEATURE: csvsource Nov 2, 2023

go-dockly added 10 commits November 3, 2023 11:59

add more trade stats

afa6cc1

clean up

b756e76

fix drawdown

abb3141

reset main

4259985

add csvsource feature

9865c12

wip stream

cb68de4

wip reviewable

91aba2e

finalise decoder

145a3d1

wip csv source stream

e0ec268

wip csv source stream mock fees

b9150e5

go-dockly force-pushed the feat/csvsource branch from c96eb7a to b9150e5 Compare November 3, 2023 11:05

c9s reviewed Nov 4, 2023

View reviewed changes

pkg/datasource/csvsource/csv_kline_reader_test.go Outdated Show resolved Hide resolved

c9s reviewed Nov 4, 2023

View reviewed changes

pkg/datasource/csvsource/csv_tick_reader_test.go Outdated Show resolved Hide resolved

c9s reviewed Nov 4, 2023

View reviewed changes

pkg/datasource/csvsource/read_klines_test.go Outdated Show resolved Hide resolved

c9s reviewed Nov 4, 2023

View reviewed changes

pkg/datasource/csvsource/stream.go Outdated Show resolved Hide resolved

c9s reviewed Nov 18, 2023

View reviewed changes

pkg/exchange/kucoin/symbols.go Show resolved Hide resolved

c9s reviewed Nov 18, 2023

View reviewed changes

pkg/service/backtest_sql.go Outdated Show resolved Hide resolved

Merge branch 'c9s:main' into feat/csvsource

37723a3

c9s reviewed Nov 20, 2023

View reviewed changes

go-dockly added 3 commits November 20, 2023 21:45

release candidate csvsource backtest

ae1c0d3

Merge branch 'feat/csvsource' of https://github.com/algo-boyz/bbgo in…

bda5b05

…to feat/csvsource

release candidate csvsource backtest address comments

75ef328

c9s reviewed Nov 21, 2023

View reviewed changes

c9s assigned c9s and unassigned c9s Nov 28, 2023

c9s mentioned this pull request Dec 14, 2023

FEATURE: add csvsource backtesting support (rebase pull request #1389) #1454

Open

c9s closed this Dec 14, 2023

		@@ -0,0 +1,16 @@
		trade_id/��id,side/��׷��,size/��,price/�۸�,created_time/�ɽ�ʱ��


		1: Through csv data source (supported right now are binance, bybit and OkEx)

		2: Alternatively run backtests through [MySQL](../../README.md#configure-mysql-database) or [SQLite3

FEATURE: add csvsource #1389

FEATURE: add csvsource #1389

Conversation

go-dockly commented Nov 2, 2023

bbgokarma-bot commented Nov 2, 2023

bbgokarma-bot commented Nov 2, 2023

bbgokarma-bot commented Nov 2, 2023

bbgokarma-bot commented Nov 2, 2023

c9s commented Nov 2, 2023

c9s commented Nov 2, 2023

bbgokarma-bot commented Nov 2, 2023

bbgokarma-bot commented Nov 3, 2023

Choose a reason for hiding this comment

go-dockly Nov 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

c9s Nov 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

c9s commented Nov 18, 2023

c9s commented Nov 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

go-dockly commented Nov 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

c9s commented Nov 21, 2023

c9s commented Nov 21, 2023

Choose a reason for hiding this comment

c9s commented Dec 8, 2023

c9s commented Dec 14, 2023

c9s commented Dec 14, 2023

c9s commented Dec 14, 2023

c9s commented Dec 14, 2023

go-dockly Nov 19, 2023 •

edited

Loading

c9s Nov 18, 2023 •

edited

Loading

go-dockly commented Nov 20, 2023 •

edited

Loading