[Tracker] Onnx FE Support

[vivekkhandelwal1]

This issue is for the purpose of tracking all the ONNX Frontend Requirements.

Instructions for finding the models/setup:

• Linux server access
• Source location of tests
• IREE EP

Important Links

• e2eshark nightly reports
• CI test lists
• Onnx OP Support [tracking] ONNX Op Support #215
• Main Issues Tracker [Tracker] All the issue related with e2e shark test suite #812
• The torch OP support is tracked by [tracking] TorchToLinalg Op Support #347

ONNX model

Passing Summary

CPU

TOTAL TESTS = 2338

Stage	# Passing	% of Total	% of Attempted
Setup	2318	99.1%	99.1%
IREE Compilation	2084	89.1%	89.9%
Gold Inference	2084	89.1%	100.0%
IREE Inference Invocation	1872	80.1%	89.8%
Inference Comparison (PASS)	1443	61.7%	77.1%

GPU

TOTAL TESTS = 2338

Stage	# Passing	% of Total	% of Attempted
Setup	2315	99.0%	99.0%
IREE Compilation	1836	78.5%	79.3%
Gold Inference	1836	78.5%	100.0%
IREE Inference Invocation	704	30.1%	38.3%
Inference Comparison (PASS)	359	15.4%	51.0%

Fail Summary

CPU and GPU

TOTAL TESTS = 2338

Stage	CPU	GPU
Setup	20	23
IREE Compilation	234	479
Gold Inference	0	0
IREE Inference Invocation	212	1132
Inference Comparison	429	345

Latest Status (Inference Pass/ Compile Pass/Total)

Item	Current (Oct 10)	Target (Oct 14 )
Pre-June-Shark test suite	21/29/32	100%
Vision int8 Models	26/78/78	100%
P0/P1 int8 CNN Models	417/486/486	100%
Hugging Face CNN FP32 Models	215/376/529	100%
Protected Models	14/23/25	100%
MIGraphX Models	17/23/31	100%
Hugging face non-CNN models	744/1127/1218	100%
IREE EP Models	19/19/35	100%
Onnx iree tests	764/1217(63%)	65%
Torch OP	871/1408 (62%)	65%
Total tests	3108/3796/5059

The Onnx lowering Lit Tests

View the op name from the tracker and then take out the lit test corresponding to that op in a seperate file, and run:

torch-mlir-opt --convert-torch-onnx-to-torch --torch-decompose-complex-ops --canonicalize --torch-backend-to-linalg-on-tensors-backend-pipeline test.mlir

Torch Op E2E Tests of torch-mlir

Take out the E2E test from the tracker and run:

python -m projects.pt1.e2e_testing.main -f <test_name> -v --config=onnx

ONNX Op Shark_TestSuite/iree_tests

Compile time Tests - #563
Runtime Tests - #583
To run the test, please follow:
build venv following here and run

iree-compile` iree_tests/onnx/node/generated/TEST_NAME/model.mlir -o test.vmfb --iree-hal-target-backends=llvm-cpu --mlir-print-ir-after-all

Models Shark_TestSuite/e2eshark

The E2EShark Model Tests are tracked through #566

First, follow setup instructions at https://github.com/nod-ai/SHARK-TestSuite/tree/main/e2eshark. No need to do the Turbine setup part as we are looking at onnx mode. Then, run this command (HF_TOKEN needed for llama, gemma model):

HF_TOKEN=your_token python run.py --torchmlirbuild /path/to/torch-mlir/build --ireebuild /path/to/iree-build --cachedir ~/.cache/huggingface --tests pytorch/models/<model_name> -r test-onnx --tolerance .001 .001 --mode onnx --report

[kumardeepakamd]

Owners, kindly add clear steps to reproduce failures and allow ability for contributors to take up a unique issue and work on fix to have more folks join in for this quality push. Great start! Let's do it.

[Shukla-Gaurav]

The IREE-EP related efforts are being tracked here: https://github.com/nod-ai/npu-benchmark/issues/2
Currently, we don't have any numbers related to model-passing rate. Once we have that, I will update that here as well. Thanks!