Add MultiHeadAttention

[marko-fabo-htec]

Add support for MultiHeadAttention operator.
Description: https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#com.microsoft.MultiHeadAttention

TODOs:

• add parse tests
• check performance

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.22%. Comparing base (ebf82f6) to head (aef0dd7).
Report is 1 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3650      +/-   ##
===========================================
+ Coverage    92.18%   92.22%   +0.04%     
===========================================
  Files          514      515       +1     
  Lines        21780    21897     +117     
===========================================
+ Hits         20078    20195     +117     
  Misses        1702     1702              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[migraphx-bot]

Test	Batch	Rate new becc9a	Rate old 241e24	Diff	Compare
torchvision-resnet50	64	3,259.39	3,256.24	0.10%	✅
torchvision-resnet50_fp16	64	6,987.35	6,984.56	0.04%	✅
torchvision-densenet121	32	2,432.85	2,434.44	-0.07%	✅
torchvision-densenet121_fp16	32	4,053.80	4,078.22	-0.60%	✅
torchvision-inceptionv3	32	1,627.89	1,629.38	-0.09%	✅
torchvision-inceptionv3_fp16	32	2,744.31	2,742.92	0.05%	✅
cadene-inceptionv4	16	765.09	764.47	0.08%	✅
cadene-resnext64x4	16	809.30	812.01	-0.33%	✅
slim-mobilenet	64	7,465.84	7,468.13	-0.03%	✅
slim-nasnetalarge	64	208.42	208.51	-0.04%	✅
slim-resnet50v2	64	3,441.34	3,440.29	0.03%	✅
bert-mrpc-onnx	8	1,143.33	1,148.20	-0.42%	✅
bert-mrpc-tf	1	467.26	464.32	0.63%	✅
pytorch-examples-wlang-gru	1	435.58	426.54	2.12%	✅
pytorch-examples-wlang-lstm	1	397.72	380.15	4.62%	🔆
torchvision-resnet50_1	1	782.03	775.13	0.89%	✅
cadene-dpn92_1	1	422.77	403.98	4.65%	🔆
cadene-resnext101_1	1	382.66	383.04	-0.10%	✅
onnx-taau-downsample	1	345.85	345.85	-0.00%	✅
dlrm-criteoterabyte	1	33.34	33.33	0.03%	✅
dlrm-criteoterabyte_fp16	1	52.73	52.75	-0.04%	✅
agentmodel	1	8,044.86	8,145.38	-1.23%	✅
unet_fp16	2	58.95	58.93	0.04%	✅
resnet50v1_fp16	1	960.45	935.00	2.72%	✅
resnet50v1_int8	1	1,006.88	1,023.68	-1.64%	✅
bert_base_cased_fp16	64	1,168.92	1,169.83	-0.08%	✅
bert_large_uncased_fp16	32	363.53	363.35	0.05%	✅
bert_large_fp16	1	198.21	195.75	1.25%	✅
distilgpt2_fp16	16	2,199.65	2,196.59	0.14%	✅
yolov5s	1	524.91	534.53	-1.80%	✅
tinyllama	1	43.44	43.39	0.11%	✅
vicuna-fastchat	1	176.38	171.47	2.87%	✅
whisper-tiny-encoder	1	418.38	417.49	0.21%	✅
whisper-tiny-decoder	1	426.08	427.46	-0.32%	✅

Check results before merge 🔆

[migraphx-bot]

     ✅ bert-mrpc-onnx: PASSED: MIGraphX meets tolerance

     ✅ bert-mrpc-tf: PASSED: MIGraphX meets tolerance

     ✅ pytorch-examples-wlang-gru: PASSED: MIGraphX meets tolerance

     ✅ pytorch-examples-wlang-lstm: PASSED: MIGraphX meets tolerance

     ✅ torchvision-resnet50_1: PASSED: MIGraphX meets tolerance

     ✅ cadene-dpn92_1: PASSED: MIGraphX meets tolerance

     ✅ cadene-resnext101_1: PASSED: MIGraphX meets tolerance

     ✅ dlrm-criteoterabyte: PASSED: MIGraphX meets tolerance

     ✅ agentmodel: PASSED: MIGraphX meets tolerance

     ✅ unet: PASSED: MIGraphX meets tolerance

     ✅ resnet50v1: PASSED: MIGraphX meets tolerance

     ✅ bert_base_cased_fp16: PASSED: MIGraphX meets tolerance

🔴bert_large_uncased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output

     ✅ bert_large: PASSED: MIGraphX meets tolerance

     ✅ yolov5s: PASSED: MIGraphX meets tolerance

     ✅ tinyllama: PASSED: MIGraphX meets tolerance

     ✅ vicuna-fastchat: PASSED: MIGraphX meets tolerance

     ✅ whisper-tiny-encoder: PASSED: MIGraphX meets tolerance

     ✅ whisper-tiny-decoder: PASSED: MIGraphX meets tolerance

     ✅ distilgpt2_fp16: PASSED: MIGraphX meets tolerance

[marko-fabo-htec]

benchmark results:

                     |     REF     |     CPU     |     GPU     |
default input format | 0.539452 ms | 0.841959 ms | 0.270837 ms |
cross attention      | 0.440712 ms | 0.753624 ms | 0.285106 ms |
kv packed format     | 1.121980 ms | 1.454690 ms | 0.268043 ms |
qkv packed format    | 1.375370 ms | 1.740890 ms | 0.264313 ms |

[TedThemistokleous]

Add test cases that are missing. Overall valid otherwise for the inputs you've added and you've gated the other optional inputs for now incase they're required by another model. We can add these later