[Mobile] Correct Understanding Of Multi-Processing/Multi-threading Concept And General Performance Of onnxruntime-react-native

[jackylu0124]

Describe the issue

I have a few questions on the concept of multi-processing/multi-threading and also on general performance in onnxruntime-react-native's inference session creation and run:

1. My understanding is that React Native modules depend on native code, whether being Java on Anroid or Objective C on iOS, and the function calls would be executed in its own process; and since I see onnxruntime-react-native relies on the onnxruntime Java API for its native code on Android and relies on the onnxruntime C/C++ API (through Objective C interop) for its native code on iOS, would onnxruntime-react-native's functions be executed on a separate process from the main React Native app's process?
2. Are the onnxruntime-react-native functions create() (https://onnxruntime.ai/docs/api/js/interfaces/InferenceSessionFactory.html#create) and run() (https://onnxruntime.ai/docs/api/js/interfaces/InferenceSession.html#run) synchronous/blocking with respect to the main logic that invoke those functions in the React Native app? (By main logic, I mean, for example, the code that handles preprocessing and postprocessing of the tensor data in pure JavaScript and the code that controls the button's reaction in React Native) My understanding is that those onnxruntime-react-native functions are not blocking because I checked the Native Modules files for the onnxruntime-react-native module, and I don't see the functions annotated with @ReactMethod(isBlockingSynchronousMethod = true) on Android and annotated with RCT_EXPORT_BLOCKING_SYNCHRONOUS_METHOD in Objective C (reference: https://reactnative.dev/docs/native-modules-android#synchronous-methods and https://reactnative.dev/docs/native-modules-ios#synchronous-methods), but please correct me if I am wrong.
3. I tried setting different numbers (e.g. 2, 4, 8) for interOpNumThreads and intraOpNumThreads in SessionOptions as well as "sequential" and "parallel" mode for the executionMode, but they have very little impact, if any, on the inference time of models on my Android phone (Samsung 9) in comparison to not passing SessionOptions into the create() function when creating inference sessions. Is this expected?
4. Are the CPU kernels for different operators (e.g. convolution, pooling, batchnorm) actively developed and improved upon across different onnxruntime versions/releases? I have found that model inference time on the most recent release (1.13.1) tend to be faster than the inference time on the 1.12 release. I am wondering if this is a contributing factor, or it may be caused by other factors.

To reproduce

I used implementation "com.microsoft.onnxruntime:onnxruntime-android:latest.integration@aar" in the build.gradle file in the onnxruntime-react-native module in order to be able to use the full build.

Urgency

High

Platform

React Native

OS Version

N/A

ONNX Runtime Installation

Released Package

Compiler Version (if 'Built from Source')

No response

Package Name (if 'Released Package')

onnxruntime-react-native

ONNX Runtime Version or Commit ID

1.13.1

ONNX Runtime API

JavaScript

Architecture

Other / Unknown

Execution Provider

Default CPU

Execution Provider Library Version

No response

[jackylu0124]

Could anyone please provide some clarifications on this? Any insights would be hugely appreciated. Thanks a lot in advance!

[RitheshKumar]

Hi @jackylu0124 where you able to make progress here? I'm also trying to find a solution for the exact problem. I'm also inferring a model on an Android Phone using the ONNX Runtime Java API