Fuse inputs with mlir #3010

pfultz2 · 2024-04-26T19:33:54Z

This will fuse the inputs but only when using the MIGRAPHX_ENABLE_MLIR_INPUT_FUSION env variable.

codecov · 2024-04-26T20:38:19Z

Codecov Report

Attention: Patch coverage is 98.36066% with 1 line in your changes missing coverage. Please review.

Project coverage is 92.21%. Comparing base (b4c29f0) to head (dd7985f).
Report is 161 commits behind head on develop.

Files with missing lines	Patch %	Lines
src/module.cpp	96.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3010      +/-   ##
===========================================
- Coverage    92.21%   92.21%   -0.01%     
===========================================
  Files          493      493              
  Lines        19725    19730       +5     
===========================================
+ Hits         18190    18194       +4     
- Misses        1535     1536       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

migraphx-bot · 2024-04-26T21:41:08Z

Test	Batch	Rate new dd7985	Rate old 9bee6a	Diff	Compare
torchvision-resnet50	64	1,750.65	1,741.07	0.55%	✅
torchvision-resnet50_fp16	64	4,182.05	4,163.68	0.44%	✅
torchvision-densenet121	32	1,469.41	1,460.87	0.59%	✅
torchvision-densenet121_fp16	32	2,552.38	2,543.03	0.37%	✅
torchvision-inceptionv3	32	889.12	885.97	0.35%	✅
torchvision-inceptionv3_fp16	32	1,492.34	1,487.78	0.31%	✅
cadene-inceptionv4	16	412.01	410.41	0.39%	✅
cadene-resnext64x4	16	419.47	417.50	0.47%	✅
slim-mobilenet	64	4,013.92	3,995.38	0.46%	✅
slim-nasnetalarge	64	100.99	100.60	0.38%	✅
slim-resnet50v2	64	1,680.21	1,672.90	0.44%	✅
bert-mrpc-onnx	8	616.89	612.15	0.77%	✅
bert-mrpc-tf	1	277.80	277.43	0.13%	✅
pytorch-examples-wlang-gru	1	322.05	367.65	-12.40%	🔴
pytorch-examples-wlang-lstm	1	291.17	295.92	-1.61%	✅
torchvision-resnet50_1	1	471.29	472.00	-0.15%	✅
cadene-dpn92_1	1	246.73	247.15	-0.17%	✅
cadene-resnext101_1	1	204.62	203.28	0.66%	✅
onnx-taau-downsample	1	206.30	205.42	0.43%	✅
dlrm-criteoterabyte	1	22.89	22.82	0.32%	✅
dlrm-criteoterabyte_fp16	1	43.82	43.76	0.15%	✅
agentmodel	1	6,095.22	6,124.86	-0.48%	✅
unet_fp16	2	34.29	34.22	0.21%	✅
resnet50v1_fp16	1	592.15	602.05	-1.64%	✅
resnet50v1_int8	1	567.57	572.31	-0.83%	✅
bert_base_cased_fp16	64	646.67	643.28	0.53%	✅
bert_large_uncased_fp16	32	198.91	197.79	0.56%	✅
bert_large_fp16	1	116.85	116.50	0.30%	✅
distilgpt2_fp16	16	1,212.19	1,203.59	0.71%	✅
yolov5s	1	301.49	295.96	1.87%	✅
tinyllama	1	23.31	23.22	0.40%	✅
vicuna-fastchat	1	133.33	132.88	0.34%	✅
whisper-tiny-encoder	1	244.23	243.45	0.32%	✅
whisper-tiny-decoder	1	255.71	255.37	0.14%	✅

This build is not recommended to merge 🔴

migraphx-bot · 2024-04-26T21:41:09Z

✅ bert-mrpc-onnx: PASSED: MIGraphX meets tolerance

✅ bert-mrpc-tf: PASSED: MIGraphX meets tolerance

✅ pytorch-examples-wlang-gru: PASSED: MIGraphX meets tolerance

✅ pytorch-examples-wlang-lstm: PASSED: MIGraphX meets tolerance

✅ torchvision-resnet50_1: PASSED: MIGraphX meets tolerance

✅ cadene-dpn92_1: PASSED: MIGraphX meets tolerance

✅ cadene-resnext101_1: PASSED: MIGraphX meets tolerance

✅ dlrm-criteoterabyte: PASSED: MIGraphX meets tolerance

✅ agentmodel: PASSED: MIGraphX meets tolerance

✅ unet: PASSED: MIGraphX meets tolerance

✅ resnet50v1: PASSED: MIGraphX meets tolerance

✅ bert_base_cased_fp16: PASSED: MIGraphX meets tolerance

🔴bert_large_uncased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output

✅ bert_large: PASSED: MIGraphX meets tolerance

✅ yolov5s: PASSED: MIGraphX meets tolerance

✅ tinyllama: PASSED: MIGraphX meets tolerance

✅ vicuna-fastchat: PASSED: MIGraphX meets tolerance

✅ whisper-tiny-encoder: PASSED: MIGraphX meets tolerance

✅ whisper-tiny-decoder: PASSED: MIGraphX meets tolerance

✅ distilgpt2_fp16: PASSED: MIGraphX meets tolerance

src/module.cpp

bpickrel · 2024-05-08T18:31:10Z

src/param_utils.cpp

@@ -43,5 +46,31 @@ void sort_params(std::vector<instruction_ref>& params)
              }));
 }

+std::vector<instruction_ref>
+find_inputs(const std::unordered_map<instruction_ref, instruction_ref>& map_ins,


This function needs a descriptive comment or no one else will ever be able to use it.

src/targets/gpu/fuse_mlir.cpp

krzysz00

Agreed with Brian about the need for comments and the need for filtering.

(I'm not going to block this because this isn't my neam, but)

src/targets/gpu/fuse_mlir.cpp

Co-authored-by: Umang Yadav <[email protected]>

umangyadav · 2024-06-20T21:13:26Z

src/include/migraphx/module.hpp

@@ -245,6 +245,21 @@ struct MIGRAPHX_EXPORT module
                                     const std::vector<instruction_ref>& splits1,
                                     const std::vector<instruction_ref>& splits2) const;

+    // Fuse the instruction into the module by inserting the instructions and
+    // parameters for any missing inputs.
+    std::vector<instruction_ref>


Need some unit-tests

src/targets/gpu/fuse_mlir.cpp

src/param_utils.cpp

umangyadav · 2024-06-20T21:26:08Z

src/param_utils.cpp

+    std::transform(names.begin(), names.end(), std::back_inserter(result), [](const auto& p) {
+        return p.second;
+    });
+    assert(not sub or result.size() == sub->get_parameter_shapes().size());


If sub == nullptr you can just do early return

If sub == nullptr you can just do early return

Early return where?

just when it starts the body of find_inputs()

Then that will skip getting the parameters. Its meant to be optional. If sub is null then it will assume all parameters come from the submodule.

umangyadav

Looks good overall but need to add unit & verify tests. I am not sure how to do verify tests with ENV flag though.

pfultz2 · 2024-06-20T23:34:25Z

I am not sure how to do verify tests with ENV flag though.

We can add the ENV var to MLIR jenkins job. We already do this to enable MLIR for everything.

pfultz2 added 7 commits April 17, 2024 19:17

Add fuse mthods to module

d39f832

Format

d2d3bae

Add some initial code

af83509

Format

ac47954

Reuse find_inputs

c9407aa

Format

ea41fb9

Enable with env var

61d788c

pfultz2 requested a review from causten as a code owner April 26, 2024 19:33

pfultz2 requested review from manupak, umangyadav, CharlieL7 and TedThemistokleous and removed request for causten, manupak, umangyadav and CharlieL7 April 26, 2024 19:38

Merge branch 'develop' into mlir-fuse-inputs

cad9d3d

Merge branch 'develop' into mlir-fuse-inputs

52a6a0e

manupak requested a review from krzysz00 May 1, 2024 14:55

pfultz2 mentioned this pull request May 8, 2024

Improve handling for adding more than 10 parameters to a submodule #3054

Merged

bpickrel reviewed May 8, 2024

View reviewed changes

src/module.cpp Show resolved Hide resolved

bpickrel reviewed May 8, 2024

View reviewed changes

src/targets/gpu/fuse_mlir.cpp Show resolved Hide resolved

krzysz00 reviewed May 9, 2024

View reviewed changes

src/targets/gpu/fuse_mlir.cpp Show resolved Hide resolved

causten and others added 2 commits June 19, 2024 17:03

Update src/targets/gpu/fuse_mlir.cpp

0e60085

Co-authored-by: Umang Yadav <[email protected]>

Add doc

fd0b7f7

pfultz2 requested a review from a team as a code owner June 20, 2024 02:48

lpaoletti approved these changes Jun 20, 2024

View reviewed changes

umangyadav reviewed Jun 20, 2024

View reviewed changes

src/targets/gpu/fuse_mlir.cpp Show resolved Hide resolved

umangyadav reviewed Jun 20, 2024

View reviewed changes

src/param_utils.cpp Show resolved Hide resolved

umangyadav reviewed Jun 20, 2024

View reviewed changes

pfultz2 added 3 commits June 20, 2024 16:48

Add input fusion to jenkins

9c4d659

Add unit test for fuse module

5eaaed3

Format

1edac2d

umangyadav mentioned this pull request Jun 21, 2024

Handle reshapes while fusing inputs in MLIR #3211

Open

umangyadav approved these changes Jun 21, 2024

View reviewed changes

Merge branch 'develop' into mlir-fuse-inputs

4a825cb

pfultz2 mentioned this pull request Jun 21, 2024

Fuse reductions with MLIR with multi-outputs #3212

Closed

pfultz2 and others added 8 commits June 21, 2024 14:28

Add unit test

1ebdaf1

Format

d035f3b

Add verify test

271ea78

Format

43e76f5

Fix tidy issue

b357f94

Fix parameter name

5e848ba

Merge branch 'develop' into mlir-fuse-inputs

ee50c26

Merge branch 'develop' into mlir-fuse-inputs

5a7f247

CharlieL7 mentioned this pull request Jul 12, 2024

CHange definition of dequantizelinear to match MIGraphX, ONNX ROCm/rocMLIR#1567

Merged

Merge branch 'develop' into mlir-fuse-inputs

dd7985f

umangyadav merged commit ff81caa into develop Jul 16, 2024
44 of 46 checks passed

umangyadav deleted the mlir-fuse-inputs branch July 16, 2024 12:21

TedThemistokleous pushed a commit that referenced this pull request Aug 21, 2024

Fuse inputs with mlir (#3010)

3831ec4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fuse inputs with mlir #3010

Fuse inputs with mlir #3010

pfultz2 commented Apr 26, 2024

codecov bot commented Apr 26, 2024 •

edited

Loading

migraphx-bot commented Apr 26, 2024 •

edited

Loading

migraphx-bot commented Apr 26, 2024 •

edited

Loading

bpickrel May 8, 2024

krzysz00 left a comment

umangyadav Jun 20, 2024

umangyadav Jun 20, 2024

pfultz2 Jun 20, 2024

umangyadav Jun 21, 2024

pfultz2 Jun 21, 2024

umangyadav left a comment

pfultz2 commented Jun 20, 2024

Fuse inputs with mlir #3010

Fuse inputs with mlir #3010

Conversation

pfultz2 commented Apr 26, 2024

codecov bot commented Apr 26, 2024 • edited Loading

Codecov Report

migraphx-bot commented Apr 26, 2024 • edited Loading

migraphx-bot commented Apr 26, 2024 • edited Loading

bpickrel May 8, 2024

Choose a reason for hiding this comment

krzysz00 left a comment

Choose a reason for hiding this comment

umangyadav Jun 20, 2024

Choose a reason for hiding this comment

umangyadav Jun 20, 2024

Choose a reason for hiding this comment

pfultz2 Jun 20, 2024

Choose a reason for hiding this comment

umangyadav Jun 21, 2024

Choose a reason for hiding this comment

pfultz2 Jun 21, 2024

Choose a reason for hiding this comment

umangyadav left a comment

Choose a reason for hiding this comment

pfultz2 commented Jun 20, 2024

codecov bot commented Apr 26, 2024 •

edited

Loading

migraphx-bot commented Apr 26, 2024 •

edited

Loading

migraphx-bot commented Apr 26, 2024 •

edited

Loading