Loading small models quickly

[zekronium]

Hi,

I have small models that I load (onnx java) on a serverless platform with limited CPU and Memory.

I have tried reducing the amount of threads, increasing ram and such, but it seems that the creation of the session is still very slow (300-400ms).

I have tried to use multithreading to create sessions but seemingly there are some synchronizations going on.

What would be the fastest configuration both for global Ort::Environment and per session to truly load the models as fast as possible from the ONNX side, since it seems to react very differently when initializing sessions with multiple threads (java threads) versus a single thread.

Is there a right way to do it?

[Craigacp]

Are you creating the sessions from an in memory array, or by loading the models from disk?

[zekronium]

Currently I pass the path to Session and onnx reads it for me.

I have tried reading to buffer and passing it to onnx but in sum the speed was the same no matter which part did it.

[Craigacp]

Ok. And are you currently setting any session options?

[zekronium]

For Ort::Environment I use global threadpools with 1 intra and 1 inter thread, since its a serverless environment with 1 core already. I also disable spinwait for the same reason.

For Session:

Per session thread pool is disabled for same reasons as above and I think its ok since theres not much concurrent inference going on.

Disabled optimizations

Disabled cpu arena allocator since I noticed in the logs, during initialization using multiple java threads it would sync on the allocator to expand the arena

Execution mode set to sequential due to 1 thread

Just tried to reduce anything cpu wise as much as I could

[cbourjau]

In my experience it is the protobuf (de)serialization that is to blame. I'd love to be proven wrong, though! There is little to be (properly) done about this as long as the upstream ONNX project sticks with protobuf, IMHO. For now you may try to use onnxruntime's internal format, but it is not compatible between onnxruntime versions.