question about the weight decay

[kamiLight]

Hi, i want to know why no weight decay for position embedding & layer norm in swin models, what this operation means for training process?

Swin-Transformer-Semantic-Segmentation/configs/swin/upernet_swin_base_patch4_window7_512x512_160k_ade20k.py

Lines 26 to 29 in 87e6f90

	optimizer = dict(_delete_=True, type='AdamW', lr=0.00006, betas=(0.9, 0.999), weight_decay=0.01,
	paramwise_cfg=dict(custom_keys={'absolute_pos_embed': dict(decay_mult=0.),
	'relative_position_bias_table': dict(decay_mult=0.),
	'norm': dict(decay_mult=0.)}))

Best wish.