🐛 about learning-rate

[hanruisong00]

Hello, after studying your paper, it has been very inspiring for my work. I still have some questions that I would like to consult with you

For the learning rate issue of training, what your code means is that the loss function is the average value of the losses of all experts. If there are four experts, then for each expert, the actual loss is divided by four, which means that when backpropagation is used to calculate the gradient, it will also be divided by four. Do you need to initially set a learning rate that is four times larger than the single model

[o-laurent]

Hi, that's why we love open source!
Thank you very much for this very relevant remark. After a discussion with @alafage, we are confident that you are right, and this may be a difference with Deep Ensembles that we had not anticipated. The first experiment I ran (R18-C10) shows an improved performance for Packed-Ensembles, but this remains to be thoroughly verified. I'll start a branch for this, which raises some technical questions (there may be a more user-friendly way to do this than multiply the lr). Would you like to be a co-author of the first commit with @alafage and me? To do so, I'd just need your email (potentially the "noreply" one from GitHub if you don't want to share your personal address).

[hanruisong00]

Thank you, it is my pleasure, my email is [email protected], if you have any questions or I can help, you can contact me