You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, I am surprise about the importance of initial token.
Attention value show the correlation of privious token and the current token.
But the initial token has the huge correlation with the far far away position token.
I was trying to figure out the reason.
I guess maybe its related to the masked self attention layer.
Frist, there are a few token to calculate the attention value. Every i+k token will ref to the first token. And we will force the token after current token to 0. So in the begining, the initial token will be very important!
I am not sure this reason if make sense or not.
Thanks a lot!
The text was updated successfully, but these errors were encountered:
Hi, I am surprise about the importance of initial token.
Attention value show the correlation of privious token and the current token.
But the initial token has the huge correlation with the far far away position token.
I was trying to figure out the reason.
I guess maybe its related to the masked self attention layer.
Frist, there are a few token to calculate the attention value. Every i+k token will ref to the first token. And we will force the token after current token to 0. So in the begining, the initial token will be very important!
I am not sure this reason if make sense or not.
Thanks a lot!
The text was updated successfully, but these errors were encountered: