Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tracking] E2EShark IREE Tests - Onnx Op - Runtime #583

Open
1 of 51 tasks
PhaneeshB opened this issue Mar 29, 2024 · 0 comments
Open
1 of 51 tasks

[tracking] E2EShark IREE Tests - Onnx Op - Runtime #583

PhaneeshB opened this issue Mar 29, 2024 · 0 comments

Comments

@PhaneeshB
Copy link
Contributor

PhaneeshB commented Mar 29, 2024

Tracking fixes of E2E IREE tests. Below is the list of ops for which OnnxToTorch lowering exists and compiles but it fails at runtime.

To reproduce issue:
build venv following https://github.com/nod-ai/SHARK-TestSuite/tree/main/iree_tests#common-venv-setup-with-deps

cd iree_tests/onnx/node/generated/TEST_NAME/
iree-compile model.mlir --iree-hal-target-backends=llvm-cpu -o test.vmfb 
iree-run-module --module=test.vmfb --device=local-sync --flagfile=test_data_flags.txt

When picking up any failures below, please cross check #215, #450, #566 to see if there is overlap issue.

Failures By Category:

Return Type Mismatch (for ints signed / unsigned)

  • ISSUE : Test case failures in iree-run-module with signed/unsigned/signless types iree-org/iree#16674
    • test_add_uint8
    • test_argmax_default_axis_example
    • test_argmax_default_axis_random
    • test_argmax_keepdims_example
    • test_argmax_keepdims_random
    • test_argmax_negative_axis_keepdims_example
    • test_argmax_negative_axis_keepdims_random
    • test_argmax_no_keepdims_example
    • test_argmax_no_keepdims_random
    • test_argmin_default_axis_example
    • test_argmin_default_axis_random
    • test_argmin_keepdims_example
    • test_argmin_keepdims_random
    • test_argmin_negative_axis_keepdims_example
    • test_argmin_negative_axis_keepdims_random
    • test_argmin_no_keepdims_example
    • test_argmin_no_keepdims_random
    • test_bitshift_left_uint16
    • test_bitshift_left_uint32
    • test_bitshift_left_uint64
    • test_bitshift_left_uint8
    • test_bitshift_right_uint16
    • test_bitshift_right_uint32
    • test_bitshift_right_uint64
    • test_bitshift_right_uint8
    • test_bitwise_and_i16_3d
    • test_bitwise_and_i32_2d
    • test_bitwise_and_ui64_bcast_3v1d
    • test_bitwise_and_ui8_bcast_4v3d
    • test_bitwise_not_2d
    • test_bitwise_not_3d
    • test_bitwise_not_4d
    • test_bitwise_or_i16_4d
    • test_bitwise_or_i32_2d
    • test_bitwise_or_ui64_bcast_3v1d
    • test_bitwise_or_ui8_bcast_4v3d
    • test_bitwise_xor_i16_3d
    • test_bitwise_xor_i32_2d
    • test_bitwise_xor_ui64_bcast_3v1d
    • test_bitwise_xor_ui8_bcast_4v3d
    • test_cast_FLOAT_to_BFLOAT16
    • test_castlike_FLOAT_to_BFLOAT16
    • test_castlike_FLOAT_to_BFLOAT16_expanded
    • test_clip_default_int8_max
    • test_clip_default_int8_max_expanded
    • test_clip_default_int8_min
    • test_clip_default_int8_min_expanded
    • test_convinteger_with_padding
    • test_convinteger_without_padding
    • test_div_uint8
    • test_dynamicquantizelinear
    • test_dynamicquantizelinear_max_adjusted
    • test_dynamicquantizelinear_min_adjusted
    • test_eyelike_without_dtype
    • test_gathernd_example_int32
    • test_max_int16
    • test_max_int32
    • test_max_int64
    • test_max_int8
    • test_max_uint16
    • test_max_uint32
    • test_max_uint64
    • test_max_uint8
    • test_min_int16
    • test_min_int32
    • test_min_int64
    • test_min_int8
    • test_min_uint16
    • test_min_uint32
    • test_min_uint64
    • test_min_uint8
    • test_mod_broadcast
    • test_mod_int64_fmod
    • test_mod_mixed_sign_int64
    • test_mod_uint16
    • test_mod_uint32
    • test_mod_uint8
    • test_mul_uint8
    • test_qlinearconv
    • test_qlinearmatmul_2D_int8_float16
    • test_qlinearmatmul_2D_int8_float32
    • test_qlinearmatmul_3D_int8_float16
    • test_qlinearmatmul_3D_int8_float32
    • test_qlinearmatmul_3D_uint8_float16
    • test_qlinearmatmul_3D_uint8_float32
    • test_quantizelinear
    • test_range_int32_type_negative_delta
    • test_shape
    • test_shape_clip_end
    • test_shape_clip_start
    • test_shape_end_negative_1
    • test_shape_example
    • test_shape_start_1
    • test_sub_uint8
    • test_tril
    • test_tril_neg
    • test_tril_one_row_neg
    • test_tril_out_neg
    • test_tril_out_pos
    • test_tril_pos
    • test_tril_square
    • test_tril_square_neg
    • test_where_long_example

OUT_OF_RANGE

  • test_bernoulli
  • test_cast_FLOAT_to_DOUBLE
  • test_cast_FLOAT16_to_DOUBLE
  • test_castlike_FLOAT_to_DOUBLE
  • test_castlike_FLOAT_to_DOUBLE_expanded
  • test_castlike_FLOAT16_to_DOUBLE
  • test_castlike_FLOAT16_to_DOUBLE_expanded
  • test_eyelike_with_dtype
  • test_max_float64
  • test_min_float64

NUMERICAL ERROR

  • test_cast_DOUBLE_to_FLOAT
  • test_cast_DOUBLE_to_FLOAT16
  • test_castlike_DOUBLE_to_FLOAT
  • test_castlike_DOUBLE_to_FLOAT_expanded
  • test_castlike_DOUBLE_to_FLOAT16
  • test_castlike_DOUBLE_to_FLOAT16_expanded
  • test_constant_pad
  • test_elu_default
  • test_gather_elements_negative_indices
  • test_hardsigmoid
  • test_hardsigmoid_default
  • test_hardsigmoid_example
  • test_hardswish_expanded
  • test_pow
  • test_pow_example
  • test_pow_types_float32_int32
  • test_pow_types_float32_int64
  • test_pow_types_float32_uint32
  • test_pow_types_float32_uint64
  • test_scatter_elements_with_negative_indices
  • test_selu_default

INVALID_ARGUMENT

  • test_cast_BFLOAT16_to_FLOAT
  • test_castlike_BFLOAT16_to_FLOAT
  • test_castlike_BFLOAT16_to_FLOAT_expanded

ERROR CODE 248

  • test_mod_mixed_sign_int16
  • test_mod_mixed_sign_int32
  • test_mod_mixed_sign_int8
  • test_mod_uint64

ERROR CODE 245

  • test_constantofshape_float_ones
  • test_constantofshape_int_shape_zero
  • test_constantofshape_int_zeros
  • test_dropout_default_mask_ratio
  • test_reduce_min_empty_set
  • test_shape_end_1
  • test_shape_start_1_end_2
  • test_shape_start_1_end_negative_1
  • test_shape_start_negative_1
  • test_size
  • test_size_example
  • test_tril_zero
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant