How can i get the size of memory allocated by onnxruntime for my infer model?

[EasonChow001]

I want to know how to get the size info of memory allocated by onnxruntime for my infer model. Is there any debug info to print?

[SherlockNoMad]

this is already logged!
https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/core/framework/sequential_executor.cc#L480

You can set the session_log_severity_level in SessionOptions to INFO to see these logs.

[jmren168]

@EasonChow001

Hi,

Have you tried successfully get the memory usage of your infer model? If so, could you please share how to do it. Many thanks.

[skottmckay]

Look in the log for output from 'bfc_arena' when log severity is INFO. This will have info about the allocations made by the arena.

[jmren168]

Hi @skottmckay

Thanks for the reply.

I tried to set log_severity_level to INFO in my python script, but did not find the 'bfc_arena' keyword.

Python Code

import onnxruntime as ort

options = ort.SessionOptions()
options.enable_mem_pattern = True
options.enable_cpu_mem_arena = True
options.enable_mem_reuse = True
options.log_severity_level=1
onnx_session = ort.InferenceSession("model.onnx", sess_options=options, providers=["CPUExecutionProvider"])

Result

2024-02-21 22:28:24.6903627 [I:onnxruntime:, inference_session.cc:271 onnxruntime::InferenceSession::ConstructorCommon::<lambda_6703f6a0539fc3437b33734ced8cf4ee>::operator ()] Flush-to-zero and denormal-as-zero are off
2024-02-21 22:28:24.7044428 [I:onnxruntime:, inference_session.cc:279 onnxruntime::InferenceSession::ConstructorCommon] Creating and using per session threadpools since use_per_session_threads_ is true
2024-02-21 22:28:24.7176562 [I:onnxruntime:, inference_session.cc:297 onnxruntime::InferenceSession::ConstructorCommon] Dynamic block base set to 0
2024-02-21 22:28:24.7290153 [I:onnxruntime:, inference_session.cc:1332 onnxruntime::InferenceSession::Initialize] Initializing session.
2024-02-21 22:28:24.7417433 [I:onnxruntime:, constant_sharing.cc:257 onnxruntime::ConstantSharing::ApplyImpl] Total shared scalar initializer count: 0
2024-02-21 22:28:24.7514683 [I:onnxruntime:, reshape_fusion.cc:53 onnxruntime::ReshapeFusion::ApplyImpl] Total fused reshape node count: 0
2024-02-21 22:28:24.7591362 [I:onnxruntime:, constant_sharing.cc:257 onnxruntime::ConstantSharing::ApplyImpl] Total shared scalar initializer count: 0
2024-02-21 22:28:24.7667394 [I:onnxruntime:, reshape_fusion.cc:53 onnxruntime::ReshapeFusion::ApplyImpl] Total fused reshape node count: 0
2024-02-21 22:28:24.7750669 [I:onnxruntime:, constant_sharing.cc:257 onnxruntime::ConstantSharing::ApplyImpl] Total shared scalar initializer count: 0
2024-02-21 22:28:24.7831177 [I:onnxruntime:, reshape_fusion.cc:53 onnxruntime::ReshapeFusion::ApplyImpl] Total fused reshape node count: 0
2024-02-21 22:28:24.7901420 [I:onnxruntime:, constant_sharing.cc:257 onnxruntime::ConstantSharing::ApplyImpl] Total shared scalar initializer count: 0
2024-02-21 22:28:24.7975769 [I:onnxruntime:, reshape_fusion.cc:53 onnxruntime::ReshapeFusion::ApplyImpl] Total fused reshape node count: 0
2024-02-21 22:28:24.8049844 [I:onnxruntime:, constant_sharing.cc:257 onnxruntime::ConstantSharing::ApplyImpl] Total shared scalar initializer count: 0
2024-02-21 22:28:24.8133635 [I:onnxruntime:, reshape_fusion.cc:53 onnxruntime::ReshapeFusion::ApplyImpl] Total fused reshape node count: 0
2024-02-21 22:28:24.8201895 [I:onnxruntime:, constant_sharing.cc:257 onnxruntime::ConstantSharing::ApplyImpl] Total shared scalar initializer count: 0
2024-02-21 22:28:24.8279259 [I:onnxruntime:, reshape_fusion.cc:53 onnxruntime::ReshapeFusion::ApplyImpl] Total fused reshape node count: 0
2024-02-21 22:28:24.8352884 [I:onnxruntime:, constant_sharing.cc:257 onnxruntime::ConstantSharing::ApplyImpl] Total shared scalar initializer count: 0
2024-02-21 22:28:24.8422112 [I:onnxruntime:, reshape_fusion.cc:53 onnxruntime::ReshapeFusion::ApplyImpl] Total fused reshape node count: 0
2024-02-21 22:28:24.8500708 [I:onnxruntime:, constant_sharing.cc:257 onnxruntime::ConstantSharing::ApplyImpl] Total shared scalar initializer count: 0
2024-02-21 22:28:24.8582207 [I:onnxruntime:, reshape_fusion.cc:53 onnxruntime::ReshapeFusion::ApplyImpl] Total fused reshape node count: 0
2024-02-21 22:28:24.8678195 [I:onnxruntime:, constant_sharing.cc:257 onnxruntime::ConstantSharing::ApplyImpl] Total shared scalar initializer count: 0
2024-02-21 22:28:24.8760125 [I:onnxruntime:, reshape_fusion.cc:53 onnxruntime::ReshapeFusion::ApplyImpl] Total fused reshape node count: 0
2024-02-21 22:28:24.8828220 [I:onnxruntime:, constant_sharing.cc:257 onnxruntime::ConstantSharing::ApplyImpl] Total shared scalar initializer count: 0
2024-02-21 22:28:24.8901235 [I:onnxruntime:, reshape_fusion.cc:53 onnxruntime::ReshapeFusion::ApplyImpl] Total fused reshape node count: 0
2024-02-21 22:28:24.8986790 [I:onnxruntime:, allocation_planner.cc:2376 onnxruntime::IGraphPartitioner::CreateGraphPartitioner] Use DeviceBasedPartition as default
2024-02-21 22:28:24.9067007 [I:onnxruntime:, session_state_utils.cc:201 onnxruntime::session_state_utils::SaveInitializedTensors] Saving initialized tensors.
2024-02-21 22:28:24.9150753 [I:onnxruntime:, session_state_utils.cc:344 onnxruntime::session_state_utils::SaveInitializedTensors] Done saving initialized tensors
2024-02-21 22:28:24.9229284 [I:onnxruntime:, inference_session.cc:1633 onnxruntime::InferenceSession::Initialize] Session successfully initialized.

Any comments are highly appreciated.

[jmren168]

Hi @SherlockNoMad,

It seems that the link has updated, https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/core/framework/sequential_executor.cc#L480.

Do you have any comments? Many thanks.