qwen2 MG and HF mismatch

[vlad-karpuhin]

When converting HF QWEN2 checkpoint into megatron, your conversion script reports the differences between both models layers. There are a lot of mismatches.
The root cause of the mismatch seems to be in the scaled dot product attention implementations between MG and HF, with the same inputs and settings, they produce different outputs. Not completely different, but still non negligible.
Did you experience that? if yes, are you aware of any decreased effectiveness of such converted qwen model later?