A question about flops calculation

[CSlearnerZM]

I try to understand the following function get_flops to calculate flops, but I still can't figure out why the parameter attn needs to multiply a constant value 60.

The paper Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM provides a formula for calculating flops similar to get_flops(Page 12). When we don't use the activation-checkpointing, I think attn needs to multiply 12 not 60 according to the formula. Of course, it is also possible that I misunderstood.

Could you help me to solve the above problem? Thanks!!!

[dashstander]

I think you're right! Thanks for bringing this to our attention @CSlearnerZM

[dashstander]

Ok, I have a branch that pretty much ports over the Megatron-DeepSpeed calculation from here