You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Optimizers just run a single step when the minimize function is called. This "works", but is counter-intuitive. The minimize function should run to some threshold (iteration count, difference matrix, gradient norm).
Describe the solution you'd like
This will require rewriting the NeuralNetwork::fit function (it will actually make it smaller) and porting some of that code to Optimizer::minimize. This will make distributed optimizers more flexible.
Also the optimizer will have to support modularity for thresholds and metrics. Taking a metrics parameter (i.e. accuracy function) will allow for custom calculation of compute graph metrics (i.e. accuracy). I would rather not calculate accuracy in general as this is Neural Network specific.
Another metric should be the stopping threshold (perhaps taken as a function pointer or std::function<()>). The stopping threshold will allow custom stopping such as when the avg gradient norm is small, the weight difference is small, the iteration count is past some amount, another process (in distributed) has converged, etc...
This update would also require some rewriting of examples, testers, tutorials, and documentation (also old presentations).
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
Optimizers just run a single step when the minimize function is called. This "works", but is counter-intuitive. The minimize function should run to some threshold (iteration count, difference matrix, gradient norm).
Describe the solution you'd like
This will require rewriting the
NeuralNetwork::fit
function (it will actually make it smaller) and porting some of that code toOptimizer::minimize
. This will make distributed optimizers more flexible.Also the optimizer will have to support modularity for thresholds and metrics. Taking a metrics parameter (i.e. accuracy function) will allow for custom calculation of compute graph metrics (i.e. accuracy). I would rather not calculate accuracy in general as this is Neural Network specific.
Another metric should be the stopping threshold (perhaps taken as a function pointer or
std::function<()>
). The stopping threshold will allow custom stopping such as when the avg gradient norm is small, the weight difference is small, the iteration count is past some amount, another process (in distributed) has converged, etc...This update would also require some rewriting of examples, testers, tutorials, and documentation (also old presentations).
The text was updated successfully, but these errors were encountered: