New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Why setting step_up_ratio=0.0? #6

Open

wuyongfa-genius opened this issue Jan 10, 2021 · 1 comment

wuyongfa-genius commented Jan 10, 2021

I just wonder woud it be better to have some iterations during which the learning rate goes up.

Owner

hyz-xmaster commented Jan 10, 2021

Hi, quoting the original SWA paper:

We propose to use a discontinuous schedule that jumps directly from the minimum to maximum learning rates, and does not steadily increase the learning rate as part of the cycle. We use this more abrupt cycle because for our purposes exploration is more important than the accuracy of individual proposals.

We did not try your idea and thus cannot answer your question. It is welcome that you provide some results on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment