Question on Default init_values in SwinTransformerV2CrBlock #2233
-
|
Hi @rwightman, I have a question regarding the Thank you for your time! |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 3 replies
-
|
@zhaohm14 yes, it's intentional. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for confirming. Could you elaborate on the rationale behind setting the norm's multiplier to zero during initialization? Additionally, I'm curious about how this specific setting might benefit the training process. Any insights or references on this would be greatly appreciated! |
Beta Was this translation helpful? Give feedback.
-
|
@zhaohm14 norm is at the end of the residual path, so the norm's weight is the last scaling layer before merging with shortcut, therefore, it's similar to layer-scale, skip-init, and resnet zero-init-bn which all scale the residual by a single scalar or one-per-channel and typically start with 0 to very small value. |
Beta Was this translation helpful? Give feedback.
-
|
I've got it. Thank you very much! |
Beta Was this translation helpful? Give feedback.
@zhaohm14 norm is at the end of the residual path, so the norm's weight is the last scaling layer before merging with shortcut, therefore, it's similar to layer-scale, skip-init, and resnet zero-init-bn which all scale the residual by a single scalar or one-per-channel and typically start with 0 to very small value.