Question about attention sink arising in pretrained models

[kevinli573]

In section 3.3 of the paper, it mentions 160M parameter models were pretrained. For these models, do you know roughly when during the training, e.g., after how many steps/tokens trained, did the attention sink phenomenon started to arise?

Thanks!