[tracking] HF CNN fp32 model tests

[jinchen62]

Below is the list of HF CNN fp32 model issues for Full model list. Tests are imported in nod-ai/SHARK-TestSuite#313.

Status on 10/1:

items	tests	model-run	onnx-import	torch-mlir	iree-compile	inference
total-count	529	499	489	418	313	161

Steps:
Build torch-mlir and iree locally

cd SHARK-TestSuite/e2eshark/
python ./run.py  --tolerance 0.001 0.001 -c path/to/torch-mlir/build -i path/to/iree-build --cachedir path/to/cache -f onnx -g models --mode onnx --report -j 12 --torchtolinalg --testsfile model_list.txt

model-run

• ONNXRuntimeError (30 models)

onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /onnxruntime_src/onnxruntime/core/optimizer/initializer.cc:49 onnxruntime::Initializer::Initializer(const onnx::TensorProto&, const onnxruntime::Path&) [ONNXRuntimeError] : 1 : FAIL : GetFileLength for ./patch_embed.proj.weight failed:Invalid fd was supplied: -1

onnx-import

• opset version issue (10 models)
• miss shape inference (303 models)
  torch_mlir.tools.import_onnx misses shape inference llvm/torch-mlir#3593
  Note: Setting opset vertsion to 21

torch-mlir

• onnx.Expand with empty shape result (47 models)
  Failed to legalize operation onnx.Expand #813
• onnx.Resize with cubic mode attr (11 models)
  Failed to legalize operation onnx.Resize. #814
• onnx.If with different result shapes in then & else branches (2 models)
  onnx.If errors when the two branches yield with different shapes #696
• onnx.LSTM with bidirectional direction attr (1 model)
  LSTM #315
• crash (7 models)
• torch.aten.squeeze (113 models)
  Fix onnx.Gather lowering with dynamic shapes llvm/torch-mlir#3675
• onnx.BatchNormalization (61 models)
  [ONNX] Add training mode support for BatchNormalization op llvm/torch-mlir#3597
• onnx.AveragePool (5 models)
• onnx.Conv (10 models)
• onnx.Mul with shape mismatch (3 models)
• onnx.Split crash (6 models)
• onnx.Transpose (280 models)
• killed (8 models)

iree-compile

• 'util.initializer' op failed to inline into combined initializer (13 models)
  'util.initializer' op failed to inline into combined initializer iree-org/iree#18386
• 'linalg.conv_2d_nchw_fchw' op inferred input/output operand dim mismatch (62 models)
  'linalg.conv_2d_nchw_fchw' op inferred input/output operand dims mismatch iree-org/iree#18387
  Missing support for handling negative step values for Onnx.Slice op #824
• 'linalg.generic' op inferred input/output operand dim mismatch (12 models)
  linalg.generic' op inferred input/output operand #1 has shape's dimension #0 to be 12288, but found 512 #825
• 'vector.shape_cast' op source/result number of elements must match (16 models)
  [compile] error: 'vector.shape_cast' op source/result number of elements must match iree-org/iree#18413
• One or more operations with large vector sizes were found (3 models)
• 'stream.async.dispatch' op has invalid Read access range (2 model)
  'stream.async.dispatch' op has invalid Read access range [0 to -96 for -96] of resource iree-org/iree#18468
• runtime error in consteval (1 models)
• expected offsets to be non-negative, but got -1 (1 model)
• crash (1 model)
• 'func.func' op exceeded stack allocation limit (3 models)
  [Comipler] 'func.func' op exceeded stack allocation limit  iree-org/iree#18187
• jit global source initialization order invalid: global "__hoisted_i32" has no value (90 models)
  jit global source initialization order invalid: global "__hoisted_i32" has no value iree-org/iree#18232
• integer value too large (40 models) torch.constant.int with large negative values causes error llvm/torch-mlir#3621
  Note: Use --torchtolinalg flag to lower the IR to linalg through torch-mlir

inference

• mismatch (152 models)
• failed (1 model)

[AmosLewis]

onnx.Conv in #566 has been fixed by disable quantized 1d conv. This should be a different conv error.

[AmosLewis]

Better to add the failed op into #797

[AmosLewis]

onnx.Batchnorm in #566 also has been fixed by bump iree. This should be a different error. Maybe you should check vivek's patch llvm/torch-mlir#3597

[vivekkhandelwal1]

Hi @jinchen62, can you please update this issue, since it seems some of these have been fixed.

[jinchen62]

@vivekkhandelwal1 Nothing was changed by Monday, will verify and update weekly.