Improve reduction fusion with reshape operators #2698

pfultz2 · 2024-01-30T23:55:19Z

This will rewrite the reshapes to improve fusions for reductions. There is a rewrite_reshapespass that resuses the same logic for fuse_pointwise and fuse_reduce. Right now, it will only work for reshape and multibroadcast. I plan to make it work for any arbitrary reshape/broadcast/transpose operators in the future.

codecov · 2024-01-31T00:46:22Z

Codecov Report

Attention: Patch coverage is 97.38562% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 91.86%. Comparing base (5c2302e) to head (4c3fca8).

Files	Patch %	Lines
src/include/migraphx/rewrite_reshapes.hpp	94.73%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2698      +/-   ##
===========================================
+ Coverage    91.84%   91.86%   +0.02%     
===========================================
  Files          478      479       +1     
  Lines        18179    18284     +105     
===========================================
+ Hits         16696    16797     +101     
- Misses        1483     1487       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

bpickrel · 2024-02-06T18:10:59Z

Would you please add code comments explaining what is being done and why? This code will be very difficult to maintain for anyone other than the author.

migraphx-bot · 2024-02-06T23:20:02Z

Test	Batch	Rate new 895acf	Rate old 5032ef	Diff	Compare
torchvision-resnet50	64	3,016.69	3,058.75	-1.37%	✅
torchvision-resnet50_fp16	64	6,768.24	7,127.23	-5.04%	🔴
torchvision-densenet121	32	2,438.95	2,451.06	-0.49%	✅
torchvision-densenet121_fp16	32	4,112.03	4,109.97	0.05%	✅
torchvision-inceptionv3	32	1,652.15	1,659.74	-0.46%	✅
torchvision-inceptionv3_fp16	32	2,618.34	2,619.60	-0.05%	✅
cadene-inceptionv4	16	777.11	780.84	-0.48%	✅
cadene-resnext64x4	16	745.35	745.77	-0.06%	✅
slim-mobilenet	64	6,710.40	6,714.58	-0.06%	✅
slim-nasnetalarge	64	110.47	175.97	-37.22%	🔴
slim-resnet50v2	64	2,976.83	2,981.85	-0.17%	✅
bert-mrpc-onnx	8	1,068.72	1,070.87	-0.20%	✅
bert-mrpc-tf	1	453.20	446.05	1.60%	✅
pytorch-examples-wlang-gru	1	473.39	377.78	25.31%	🔆
pytorch-examples-wlang-lstm	1	510.23	354.57	43.90%	🔆
torchvision-resnet50_1	1	735.59	792.31	-7.16%	🔴
cadene-dpn92_1	1	427.99	428.08	-0.02%	✅
cadene-resnext101_1	1	364.12	363.76	0.10%	✅
onnx-taau-downsample	1	348.56	349.31	-0.21%	✅
dlrm-criteoterabyte	1	34.70	34.73	-0.09%	✅
dlrm-criteoterabyte_fp16	1	58.41	57.72	1.20%	✅
agentmodel	1	6,113.44	6,816.03	-10.31%	🔴
unet_fp16	2	58.27	58.23	0.06%	✅
resnet50v1_fp16	1	950.43	985.92	-3.60%	🔴
resnet50v1_int8	1	847.59	866.51	-2.18%	✅
bert_base_cased_fp16	64	1,041.10	1,041.04	0.01%	✅
bert_large_uncased_fp16	32	321.74	321.85	-0.03%	✅
bert_large_fp16	1	nan	nan	nan%	❌
distilgpt2_fp16	16	2,027.91	2,026.48	0.07%	✅
yolov5s	1	512.62	514.61	-0.39%	✅
tinyllama	1	44.79	44.75	0.09%	✅
vicuna-fastchat	1	181.25	182.17	-0.50%	✅
whisper-tiny-encoder	1	404.51	403.43	0.27%	✅
whisper-tiny-decoder	1	434.51	420.63	3.30%	🔆

This build is not recommended to merge 🔴

migraphx-bot · 2024-02-06T23:20:04Z

❌bert-mrpc-onnx: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/onnx/onnx_parser.cpp:264: parse_from: PARSE_FROM: Failed reading onnx file: /new-saved-models/huggingface-transformers/bert_mrpc1.onnx

✅ bert-mrpc-tf: PASSED: MIGraphX meets tolerance

✅ pytorch-examples-wlang-gru: PASSED: MIGraphX meets tolerance

✅ pytorch-examples-wlang-lstm: PASSED: MIGraphX meets tolerance

✅ torchvision-resnet50_1: PASSED: MIGraphX meets tolerance

✅ cadene-dpn92_1: PASSED: MIGraphX meets tolerance

❌cadene-resnext101_1: ERROR - check error output

2024-03-18 13:46:22.784261292 [W:onnxruntime:, model.cc:183 Model] ONNX Runtime only guarantees support for models stamped with opset version 7 or above for opset domain 'ai.onnx'. Please upgrade your model to opset 7 or higher. For now, this opset 6 model may run depending upon legacy support of some older opset version operators.
2024-03-18 13:46:22.790914846 [W:onnxruntime:, transpose_optimizer.cc:28 ApplyImpl] Transpose optimizer failed: Unsupported ONNX opset: 6
Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 267, in main
sess = ort.InferenceSession(model_name,
File "/usr/local/lib/python3.8/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 419, in init
self._create_inference_session(providers, provider_options, disabled_optimizers)
File "/usr/local/lib/python3.8/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 463, in _create_inference_session
sess.initialize_session(providers, provider_options, disabled_optimizers)
onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Could not find an implementation for BatchNormalization(6) node with name ''

✅ dlrm-criteoterabyte: PASSED: MIGraphX meets tolerance

✅ agentmodel: PASSED: MIGraphX meets tolerance

❌unet: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 207, in main
model = migraphx.parse_onnx(model_name,
RuntimeError: /src/AMDMIGraphX/src/onnx/onnx_parser.cpp:264: parse_from: PARSE_FROM: Failed reading onnx file: /new-saved-models/unet/model.onnx

✅ resnet50v1: PASSED: MIGraphX meets tolerance

✅ bert_base_cased_fp16: PASSED: MIGraphX meets tolerance

🔴bert_large_uncased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output

❌bert_large: ERROR - check error output

Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 340, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 205, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /src/AMDMIGraphX/src/onnx/onnx_parser.cpp:264: parse_from: PARSE_FROM: Failed reading onnx file: /new-saved-models/bert/model.onnx

✅ yolov5s: PASSED: MIGraphX meets tolerance

✅ tinyllama: PASSED: MIGraphX meets tolerance

✅ vicuna-fastchat: PASSED: MIGraphX meets tolerance

✅ whisper-tiny-encoder: PASSED: MIGraphX meets tolerance

✅ whisper-tiny-decoder: PASSED: MIGraphX meets tolerance

✅ distilgpt2_fp16: PASSED: MIGraphX meets tolerance

test/fuse_reduce.cpp

src/common_dims.cpp

TedThemistokleous

A few questions, also add in unit tests for code coverage warnings

src/include/migraphx/rewrite_reshapes.hpp

TedThemistokleous · 2024-03-04T14:23:06Z

src/include/migraphx/rewrite_reshapes.hpp

+            auto input = ins->inputs().front();
+            if(input->name() == "contiguous")
+                return match_input(input, x_ins);
+            return x_ins == input;


Add simple case where we hit this return.

There is coverage for this line. Did you mean the return above?

Test added.

TedThemistokleous

Still seeing some minor CI fails/coverage

TedThemistokleous

Fix licensing + add more coverage

pfultz2 · 2024-03-18T21:15:11Z

The perf report is a fluke. I dont see a difference in performance between develop and this branch.

pfultz2 added 17 commits December 22, 2023 16:49

Add custom inserter

d6791f3

Format

42739b0

Refactor the reshape rewriting

350554c

Format

59f3ecf

Try to rewrite reshapes for reduce as well

02e9989

Format

f2d0745

Sort axes

ef386c2

Merge branch 'develop' into fuse-reshape-pointwise

bcbe021

Add unit test

9249bdd

Format

2c0f919

Handle base shape

0fea84f

Format

33eea67

Fixes for fusion

9c9fe65

Format

2fc67f7

Fix test

ff95292

Improve testing

b974caa

Format

b563470

pfultz2 requested review from umangyadav and TedThemistokleous January 30, 2024 23:55

TedThemistokleous assigned pfultz2 Jan 31, 2024

TedThemistokleous added the enhancement New feature or request label Jan 31, 2024

Merge branch 'develop' into fuse-reshape-pointwise

4e7a5d7

umangyadav reviewed Feb 9, 2024

View reviewed changes

test/fuse_reduce.cpp Show resolved Hide resolved

umangyadav reviewed Feb 9, 2024

View reviewed changes

src/common_dims.cpp Show resolved Hide resolved

umangyadav reviewed Feb 9, 2024

View reviewed changes

src/common_dims.cpp Show resolved Hide resolved

umangyadav reviewed Feb 9, 2024

View reviewed changes

src/common_dims.cpp Show resolved Hide resolved

TedThemistokleous requested changes Feb 16, 2024

View reviewed changes

pfultz2 added 3 commits February 16, 2024 07:32

Add default

9807e66

Improve test coverage and fix bugs

2b4cd04

Format

147526e

pfultz2 requested a review from causten as a code owner February 16, 2024 20:54

pfultz2 added 3 commits February 28, 2024 20:27

Merge branch 'develop' into fuse-reshape-pointwise

a59086c

Add another test

34da10f

Format

d5a51f2

pfultz2 requested a review from TedThemistokleous March 2, 2024 01:09

Add TODOs

ad0544c

TedThemistokleous reviewed Mar 4, 2024

View reviewed changes

src/include/migraphx/rewrite_reshapes.hpp Show resolved Hide resolved

TedThemistokleous reviewed Mar 4, 2024

View reviewed changes

Remove comment

e40eb2f

pfultz2 requested a review from TedThemistokleous March 18, 2024 14:47

TedThemistokleous approved these changes Mar 18, 2024

View reviewed changes

TedThemistokleous reviewed Mar 18, 2024

View reviewed changes

TedThemistokleous requested changes Mar 18, 2024

View reviewed changes

pfultz2 added 4 commits March 18, 2024 09:44

Add test for contiguous skip

69ec320

Format

7f07a85

Update license

6e1c2d9

Merge branch 'develop' into fuse-reshape-pointwise

cec782d

pfultz2 requested a review from TedThemistokleous March 18, 2024 16:47

pfultz2 added 2 commits March 18, 2024 10:13

Format

96713e1

Add missing header

895acf4

TedThemistokleous approved these changes Mar 18, 2024

View reviewed changes

TedThemistokleous added the high priority A PR with high priority for review and merging. label Mar 18, 2024

Merge branch 'develop' into fuse-reshape-pointwise

4c3fca8

causten merged commit 9077e74 into develop Mar 18, 2024
17 of 18 checks passed

causten deleted the fuse-reshape-pointwise branch March 18, 2024 21:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve reduction fusion with reshape operators #2698

Improve reduction fusion with reshape operators #2698

pfultz2 commented Jan 30, 2024

codecov bot commented Jan 31, 2024 •

edited

Loading

bpickrel commented Feb 6, 2024

migraphx-bot commented Feb 6, 2024 •

edited

Loading

migraphx-bot commented Feb 6, 2024 •

edited

Loading

TedThemistokleous left a comment

TedThemistokleous Mar 4, 2024

pfultz2 Mar 18, 2024

TedThemistokleous Mar 18, 2024

pfultz2 Mar 18, 2024

TedThemistokleous left a comment •

edited

Loading

TedThemistokleous left a comment

pfultz2 commented Mar 18, 2024

Improve reduction fusion with reshape operators #2698

Improve reduction fusion with reshape operators #2698

Conversation

pfultz2 commented Jan 30, 2024

codecov bot commented Jan 31, 2024 • edited Loading

Codecov Report

bpickrel commented Feb 6, 2024

migraphx-bot commented Feb 6, 2024 • edited Loading

migraphx-bot commented Feb 6, 2024 • edited Loading

TedThemistokleous left a comment

Choose a reason for hiding this comment

TedThemistokleous Mar 4, 2024

Choose a reason for hiding this comment

pfultz2 Mar 18, 2024

Choose a reason for hiding this comment

TedThemistokleous Mar 18, 2024

Choose a reason for hiding this comment

pfultz2 Mar 18, 2024

Choose a reason for hiding this comment

TedThemistokleous left a comment • edited Loading

Choose a reason for hiding this comment

TedThemistokleous left a comment

Choose a reason for hiding this comment

pfultz2 commented Mar 18, 2024

codecov bot commented Jan 31, 2024 •

edited

Loading

migraphx-bot commented Feb 6, 2024 •

edited

Loading

migraphx-bot commented Feb 6, 2024 •

edited

Loading

TedThemistokleous left a comment •

edited

Loading