Difference between attention contrib ops

[asyncth]

What is the difference between com.microsoft.Attention and com.microsoft.MultiHeadAttention contrib operators? ContribOperators.md says that both of them are multi-head attention.

[hariharans29]

CC: @tianleiwu

Attention only support SelfAttention whereas MultiHeadAttention supports self and cross attention

[tianleiwu]

Right. There are also some minor difference in past state input. In Attention, each past input is past_key + past_value, while in MultiHeadAttention, past_key and past_value are separated inputs.

[asyncth]

Thanks for help!

[asyncth]

Should one always prefer Attention over MultiHeadAttention for self-attention?

[tianleiwu]

Sometime MultiHeadAttention is a better choice like the following:
(1) past_key and past_value are separated in ONNX model.
(2) there is no bias in QKV projection and qkv can be packed.

Right now, you can choose Attention if it is applicable.

[asyncth]

(2) there is no bias in QKV projection and qkv can be packed.

What do you mean? Doesn't Attention op handle that case just as well?

Also I'm having trouble understanding bias input of MultiHeadAttention, is just a value that is chunked into 3 tensors of shapes (hidden_size), (hidden_size) and (v_hidden_size) and then those tensors are added to query, key and value?

[tianleiwu]

Doesn't Attention op handle that case just as well?

For this case (there is no bias in QKV projection and qkv can be packed), Attention need transpose BS3NH to BSN3H for fp16 fused attention in CUDA, while packed QKV is BSN3H format so MultiHeadAttention might be faster since it saves a transpose.

Also I'm having trouble understanding bias input of MultiHeadAttention, is just a value that is chunked into 3 tensors of shapes (hidden_size), (hidden_size) and (v_hidden_size) and then those tensors are added to query, key and value?

Yes.

[asyncth]

Got it, thanks.