How to run multiple cuda sessions in parallel

[sunhmy]

Hi,

I'm new to OnnxRuntime and trying to increase performance by running 2 GPU sessions in parallel. I'd assume I can do it as simple as:

1. create 2 sessions and 2 cuda streams.
2. assign 1st stream to 1st session by OrtCUDAProviderOptions, same for the 2nd stream to the 2nd session
3. create 2 threads, each for one of the sessions above.

When running it, I do see that lots of the time the 2 GPU streams are running in parallel, however, from profiling the GPU perf, I'm seeing a lot of pthread_mutex_lock causing huge latency. So that now running 2 GPU streams in parallel is as slow as running them sequentially for me...

Below is a screenshot showing those pthread_mutex_lock causing the extra dependency(false dependency?).

1)Are those inserted by OnnxRuntime?
2) Is there any way to get rid of those locks if they are false dependency?
3) Is there any other way to achieve running multiple cuda sessions in parallel?

Thanks!