Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE REQUEST] Optimizers should run to some threshold, rather than a single step #25

Open
Dando18 opened this issue Oct 4, 2019 · 0 comments
Assignees
Labels
feature-request requesting a new feature
Milestone

Comments

@Dando18
Copy link
Collaborator

Dando18 commented Oct 4, 2019

Is your feature request related to a problem? Please describe.
Optimizers just run a single step when the minimize function is called. This "works", but is counter-intuitive. The minimize function should run to some threshold (iteration count, difference matrix, gradient norm).

Describe the solution you'd like
This will require rewriting the NeuralNetwork::fit function (it will actually make it smaller) and porting some of that code to Optimizer::minimize. This will make distributed optimizers more flexible.

Also the optimizer will have to support modularity for thresholds and metrics. Taking a metrics parameter (i.e. accuracy function) will allow for custom calculation of compute graph metrics (i.e. accuracy). I would rather not calculate accuracy in general as this is Neural Network specific.

Another metric should be the stopping threshold (perhaps taken as a function pointer or std::function<()>). The stopping threshold will allow custom stopping such as when the avg gradient norm is small, the weight difference is small, the iteration count is past some amount, another process (in distributed) has converged, etc...

This update would also require some rewriting of examples, testers, tutorials, and documentation (also old presentations).

@Dando18 Dando18 added the feature-request requesting a new feature label Oct 4, 2019
@Dando18 Dando18 self-assigned this Oct 4, 2019
@Dando18 Dando18 added this to the MagmaDNN-v1.1 milestone Oct 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request requesting a new feature
Projects
None yet
Development

No branches or pull requests

1 participant