[BUG] Failed to compile yolov5 onnx format model

[fomiuna]

🐛 Bug

I have downloaded a yolov5 onnx model yolov5s.onnx which can be inferenced by onnxruntime, but I got a failure when compiling it with nnfusion. The compiling environment is as follows.

Component	Version
model	https://github.com/ultralytics/yolov5/releases/download/v7.0/yolov5s.onnx
nnfusion	git clone https://github.com/microsoft/nnfusion.git (main branch)
cuda	10.0.130
onnx/onnxruntime-gpu	1.14.1
ubuntu	18.04

To Reproduce
Run Rammer with library kernels in the compiling environment:

nnfusion yolov5s.onnx -f onnx -fkernel_fusion_level=3 -fblockfusion_level=1 -fconst_folding_backend=CUDA -fwarmup_step=5 -frun_step=5 

and the output is:

[WARNING] 2024-07-01T11:00:08z src/contrib/custom_op/custom_op.h 27	$NNFUSION_HOME was not set, use /root/.nnfusion.
[WARNING] 2024-07-01T11:00:08z src/contrib/custom_op/custom_op.h 27	$NNFUSION_HOME was not set, use /root/.nnfusion.

============================================================================
---- Processing '/root/model/yolo/yolov5-releases-v7.0/yolov5s.onnx'
============================================================================
[INFO] 2024-07-01T11:00:08z src/nnfusion/frontend/onnx_import/onnx.cpp 54	Optimizing ONNX Graph with External Tool (models/pytorch2onnx/ort_run_frozen.py)
[WARNING] 2024-07-01T11:00:08z src/nnfusion/common/util.cpp 47	$NNFUSION_HOME was not set, use /root/.nnfusion.
2024-07-01 11:00:08.796084517 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:541 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met.
ONNX model check passed!
Importing ONNX model into ONNX Runtime...
Execution Providers: ['CPUExecutionProvider']
output0
[3.467e+00 3.309e+00 7.027e+00 8.203e+00 2.408e-05 2.323e-01 3.120e-03
 3.339e-02 2.266e-03 1.929e-02] ...(size= 2142000 end with 0.001398 )
[INFO] 2024-07-01T11:00:09z src/nnfusion/frontend/onnx_import/onnx.cpp 40	Import ONNX Graph Size: [14731163]
... ...
(many INFO log lines from src/nnfusion/frontend/onnx_import/util/graph_convert.cpp)
... ...
[INFO] 2024-07-01T11:00:10z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: /model.24/Sigmoid
[INFO] 2024-07-01T11:00:10z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 538	node /model.24/Sigmoid, output InsertedCast_/model.24/Sigmoid_output_0, shape Shape{1, 3, 80, 80, 85}
[INFO] 2024-07-01T11:00:10z src/nnfusion/frontend/onnx_import/util/graph_convert.cpp 519	convert node: /model.24/Split
[ERROR] 2024-07-01T11:00:10z src/nnfusion/util/errors.hpp 169	Check failed: 'it != std::end(m_attributes)' at /root/nnfusion-main/src/nnfusion/frontend/onnx_import/core/node.cpp:106:
Node (/model.24/Split): unknown attribute 'split'
[ERROR] 2024-07-01T11:00:10z src/nnfusion/util/errors.hpp 169	Check failed: 'axis_length % num_splits == 0' at /root/nnfusion-main/src/nnfusion/frontend/onnx_import/op/split.cpp:61:
(no explanation given)
terminate called after throwing an instance of 'nnfusion::errors::CheckError'
  what():  Check failed: 'axis_length % num_splits == 0' at /root/nnfusion-main/src/nnfusion/frontend/onnx_import/op/split.cpp:61:
(no explanation given)
Aborted (core dumped)

It seems that the operator "split" is not handled correctly during graph converting, is there any suggestion?

Additional context
Besides, I tried to convert yolov5s.pt to yolov5s-export.onnx in following converting environment:

Component	Version
yolov5	git clone https://github.com/ultralytics/yolov5.git (master branch)
model	https://github.com/ultralytics/yolov5/releases/download/v7.0/yolov5s.pt
onnx	1.16.1
python	3.10.9
torch	1.12.1

Convert as follows:

$ cd yolov5/ && cp /path/to/yolov5s.pt .
$ python export.py --weights yolov5s.pt --include torchscript onnx --opset 11
$ mv yolov5s.onnx yolov5s-export.onnx

Again, compiled the result yolov5s-export.onnx using Rammer with library kernel in compiling environment as mentioned before, I got a different error message as follows:

(similar info as that of yolov5s.onnx compiling result)
... ...
(many INFO log lines from src/nnfusion/frontend/onnx_import/util/graph_convert.cpp)
... ...
(many INFO log lines from src/nnfusion/engine/pass/graph/runtime_const_folding_pass.cpp)
... ...
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/batchnorm_inference_folding_pass.cpp 901	batchnorm inference folding Pass ends for Graph: Graph_1
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/cache/manager.cpp 52	Open kernel cache from: /root/.cache/nnfusion/kernel_cache.db
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/cache/manager.cpp 52	Open kernel cache from: /root/.cache/nnfusion/kernel_cache.db
[ERROR] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270	No valid kernel found:images(op type: Parameter, dev type: CUDA_GPU)
[ERROR] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270	No valid kernel found:onnx::Concat_246(op type: Resize, dev type: CUDA_GPU)
[ERROR] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/kernel_selection.cpp 270	No valid kernel found:onnx::Concat_271(op type: Resize, dev type: CUDA_GPU)
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/cache/manager.cpp 52	Open kernel cache from: /root/.cache/nnfusion/kernel_cache.db
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/cache/manager.cpp 52	Open kernel cache from: /root/.cache/nnfusion/kernel_cache.db
[WARNING] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153	Kernel should be emitted before this pass:images
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_140 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_145 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_155 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_150 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_160 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_165 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_172 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_177 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_187 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_182 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_192 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_197 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_203 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_208 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_215 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_220 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_225 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_230 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_235 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_240 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_246 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_251 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_257 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_262 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_269 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_274 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_279 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_284 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_289 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_294 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_301 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_306 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator onnx::MaxPool_232 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator onnx::MaxPool_233 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator onnx::Concat_234 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_315 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_320 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153	Kernel should be emitted before this pass:onnx::Concat_246
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_327 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_332 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_337 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_342 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_348 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_353 is not BlockCudaEmitter, skip in BlockFusion
[WARNING] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 153	Kernel should be emitted before this pass:onnx::Concat_271
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_360 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_365 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_370 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_375 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_381 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_386 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_500 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator onnx::Transpose_341 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_392 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_397 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator onnx::Sigmoid_342 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_402 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_407 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator onnx::Concat_367 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_413 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_475 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_418 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator onnx::Transpose_380 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator onnx::Sigmoid_381 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_424 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_429 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_434 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_439 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator onnx::Concat_406 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_445 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Convolution_450 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator onnx::Transpose_419 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator onnx::Sigmoid_420 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator onnx::Concat_445 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:41z src/nnfusion/engine/pass/graph/blockfusion/blockfusion_optimizer.cpp 184	Operator Result_526 is not BlockCudaEmitter, skip in BlockFusion
[INFO] 2024-07-01T11:02:42z src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp 211	assign thread info-------------------------------
[ERROR] 2024-07-01T11:02:42z src/nnfusion/util/errors.hpp 169	Check failed: '(*gnode)["Kernel_Selection_Result"].is_valid()' at /root/nnfusion-main/src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp:924:
Kernel should be selected before this pass:Resize
terminate called after throwing an instance of 'nnfusion::errors::CheckError'
  what():  Check failed: '(*gnode)["Kernel_Selection_Result"].is_valid()' at /root/nnfusion-main/src/nnfusion/engine/pass/graph/assign_async_info_pass.cpp:924:
Kernel should be selected before this pass:Resize
Aborted (core dumped)

What does it mean "No valid kernel found" here and how to fix it? And why the errors from compiling yolov5s.onnx and yolov5s-export.onnx differ? Appreciate for any help!