Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compare SGD and SVRG in the batching tutorials #203

Open
BalzaniEdoardo opened this issue Jul 26, 2024 · 2 comments
Open

Compare SGD and SVRG in the batching tutorials #203

BalzaniEdoardo opened this issue Jul 26, 2024 · 2 comments

Comments

@BalzaniEdoardo
Copy link
Collaborator

BalzaniEdoardo commented Jul 26, 2024

Expand the tutorial on batching comparing SGD and SVRG:

  1. Show that, if you set the stepsize of the right size, the algorithm converge to the correct estimate
  2. Compare with SGD, which won't converge without a decreasing stepsize schedule carefully calibrated.
  3. Compare the iteration number, showing that SVRG requires less epochs to get to same log-likelihood level than SGD

Ideally, when we will have a run_iterator method we should showcase a simplified interface that will look something like:

# add a batcher parameter to GLM init, optional, if set it will try to run the algorithm using the iterator.
model = GLM(solver_name="SVRG", batcher=my_iterator)
model.fit(X, y)  # X could be lazy loaded
@billbrod
Copy link
Member

As part of this, need to show what extra steps are required for SVRG beyond just calling update (and update the update docstring to point users here. @bagibence said this is already shown in a test -- which one was that?

@BalzaniEdoardo
Copy link
Collaborator Author

The docstrings of the update method of SVRG should point to our tutorial on how to construct the external loop. Eventually, no external loop will be needed, since we will implement a run iterator method.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants