How to use parallel execution mode on CUDA Execution Provider, while executing onnxruntime session in multiprocessing

[AbhijitManepatil]

I am trying to execute onnx runtime session in multiprocessing on cuda using, onnxruntime.ExecutionMode.ORT_PARALLEL but while executing in parallel on cuda getting the following issue.
[W:onnxruntime:, inference_session.cc:421 RegisterExecutionProvider]
Parallel execution mode does not support the CUDA Execution Provider. So making the execution mode sequential for this session since it uses the CUDA Execution Provider.

[pranavsharma]

Please do not use the parallel executor as it's unreliable. The error message clearly says that parallel execution mode is not supported for CUDA EP.

[AbhijitManepatil]

import onnxruntime
so = onnxruntime.SessionOptions()
so.execution_mode = onnxruntime.ExecutionMode.ORT_PARALLEL
session = onnxruntime.InferenceSession('sample.onnx', sess_options=so)

I have the above scenario, to run InferenceSession in parallel (multiprocessing way) on onnx file, later this session object can be used by processes separately on each process in parallel. (which runs on cuda)

[pranavsharma]

ExecutionMode.ORT_PARALLEL has nothing to do with running an ONNX model in multiple processes. Each session is tied to a single process. If you want to do inferencing in multiple processes, create the session in those processes separately.

[AbhijitManepatil]

Yes @pranavsharma , I was trying to share the session among the processes like below with minimum workable example:

import multiprocessing as mp
import onnxruntime as ort

def worker(cam_name,session):
    session.run("..")
    
if __name__=='__main__':
    mp.set_start_method('spawn')
    session = ort.InferenceSession("mymode.onnx", None)
    CamDevices=["single_face.mp4","single_face1.mp4"]
    processes=[]
    for cam_name in CamDevices:        
        p = mp.Process(target=worker,args=[cam_name,session])
        processes.append(p)
    for p in processes:
        p.start() 
    for p in processes:
       p.join()

but getting following error :
TypeError: cannot pickle 'onnxruntime.capi.onnxruntime_pybind11_state.InferenceSession' object

I was also trying to get solution from #7846,

but still not found a suitable solution.
Can you help to this, how exactly use session among the multiple processes?

[pranavsharma]

InferenceSession is not "pickable". You'll need to create a wrapper class to make it "pickable". I wrote a working example a few months back here #7846 (comment). Make sure you're using the onnxruntime-gpu pkg.

[AbhijitManepatil]

Hi, @pranavsharma thanks, I was able to do multiprocessing with the session object, session.run executed by each process by referring above-mentioned link example, it works for me, but I have some stats to point out here:
For single-process It took around 7 sec. to complete,
For Two processes it took around 12 sec, and for Three processes took 18 sec. to complete, for the same number of iterations(same logic in each process).
How it is exactly then? It should be near to 7 + sec. for all processes even I run three processes together, is it?