Inference performances decrease as the inference thread(also process) increase

[mysjo7789]

On my server, I run the multiple onnx inference processes and I measured the time for a inference each process. As the number of inference process increase, the time to calc by onnx also increase. how can I configure the onnxRuntime not to decrease the performances on each process

[snnn]

By default each process assumes it can use all the CPUs. If you have multiple processes, then the CPUs are over subscribed and the perf is less optimal. If you have a fine control of the number of process, then you can divide the number of CPUs by the number of process to get how many CPUs each process should use. Then you can use ONNX Runtime session options to adjust the number of threads of each process.