How can we run parametrized strategies with multiprocessing?

[crazy25000]

Hello,

The idea is to initially load all data into one engine instance and use the instance to run parametrized strategies with multiprocessing.

Using this example:

nautilus_trader/examples/backtest/fx_ema_cross_gbpusd_bars.py

Lines 46 to 53 in f713b55

	if __name__ == "__main__":
	# Create the backtest engine
	engine = BacktestEngine(
	use_data_cache=True, # Pre-cache data for increased performance on repeated runs
	# exec_db_type="redis",
	# exec_db_flush=False,
	# bypass_logging=True
	)

# New import and method
from joblib import Parallel, delayed

all_backtest_runs = []

def run_engine(engine, strategy):
    res = engine.run(strategies=[strategy])
    all_backtest_runs.append(res)
    engine.reset()

I modified it by creating a list of strategies with different params:

            # Added to line 102
            strategies_list = []
            for period in [10, 20, 50, 100]:
                strategy = EMACross(
                    instrument_id=GBPUSD.id,
                    bar_spec=BarSpecification(5, BarAggregation.MINUTE, PriceType.BID),
                    fast_ema_period=period,
                    slow_ema_period=period*2,
                    trade_size=Decimal(1_000_000),
                    order_id_tag=f"00{period}",
                )
                strategies_list.append(strategy)

            # Added to line 115
            with Parallel(n_jobs=2, require="sharedmem") as parallel:
                  parallel(delayed(run_engine)(engine, strategy) for strategy in strategies_list)

I've tried several methods with concurrent futures, multiprocessing, and joblib. But was getting the same error Can't pickle local object '__Pyx_CFunc and makes me wonder if this is because of Cython or something not aware of. Is there a simple, reproducible example available that someone can share?

[limx0]

Hey @crazy25000, good find. I'm thinking this has not be handled (yet) in nautilus trader, but definitely on the near term list of things todo. It looks to be an issue with pickling cython classes. I thought we might get lucky and it would be handled by cloudpickle, but unfortunately it doesn't look like it is - cloudpipe/cloudpickle#186

We've got a couple of options, which we will need to discuss:

1. Implementing __reduce__ / __setstate__ on cython objects so they can be pickled or
2. create some sort of backtest_runner function that is able to reconstruct a fresh backtest from some kwargs.

I'm thinking it's going to be cheaper to do 2., both from a performance and network bandwidth perspective. I'm a big fan of the dask project, which has spent alot of time solving these issues, specifically moving data around and efficiently running tasks, so my first thought would be working within that ecosystem.

I'll wait and see what @cjdsellers thinks, but I'll say this is something I'll take a look at very soon.

[crazy25000]

I'm glad this is a known issue, I wasted a few hours on it and was feeling pretty frustrated 😅. I lost hair along the way :p....+1 for dask, that was the next library I wanted to try, but felt burned out and stopped working on it.

I like the idea of 2 so that we don't have to update cython objects everywhere. Let me know if I can help in any way, I might take a look at dask since I have some experience with it, but Cython threw me off.

[limx0]

Hey @crazy25000 - I've opened #354 after I had a bit of a think about this, would you mind adding your 2c?

[thematz1]

Does Cython's prange with a shared memory solve this issue? I believe things are already converting into C data types so prange may be a viable route to distribute engine runs over multiple cores.

https://cython.readthedocs.io/en/latest/src/userguide/parallelism.html
https://stackoverflow.com/questions/31954459/shared-memory-with-cython-multiprocessing

I'm not very familiar with C/C++ programming concepts but would a pointer be useful to pass the next result of the data producer generator or even pointing to the entire data producer? Does the fact that a point being resolving the memory address null the shared memory idea?

I don't know much about job lib either so I I will check it out to understand what benefits it offers. My sticking point was the sharedmem but a public data type class could resolve this as per the stack overflow post above. Another option I remember from some books I read on Cython is type memory views which could be helpful if going the prange route as the project relies heavily on pandas.

https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html

[crazy25000]

I don't know all the answers, but I did test Cython memory views and prange, but was not successful. It would be good if someone else could also try it, it's entirely possible that I didn't implement it correctly.

Cython is type memory views

[thematz1]

I will give it a try when I have some nights free after work :)