Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add logging callbacks for training statistics #663

Open
1 of 3 tasks
BradLarson opened this issue Sep 2, 2020 · 1 comment
Open
1 of 3 tasks

Add logging callbacks for training statistics #663

BradLarson opened this issue Sep 2, 2020 · 1 comment
Assignees
Labels
good first issue Good for newcomers

Comments

@BradLarson
Copy link
Contributor

BradLarson commented Sep 2, 2020

The training loop abstraction has a visual progress bar for tracking statistics during training. In many cases, it's desirable to log or store training statistics for later analysis, or to send those statistics to different services for visualization.

Further discussion is present in PR #662, but we'd like to have callbacks such as the following:

as well as a general mechanism for writing more.

A TrainingStatistics callback exists now as a means of collecting statistics, but it is only directly called by TrainingProgress. The idea is to have a single statistics gathering function, so as to avoid the overhead of materializing statistics-related tensors multiple times, and to have that report those statistics to all interested parties. For this, we'd need to have one TrainingStatistics instance that acts as a kind of notification center to other callbacks.

We'd want this implementation detail to be hidden from end-users, so that they just need to specify which logging or visualization callbacks they need, and not have to worry about managing the TrainingStatistics instance itself.

@vkuznet
Copy link

vkuznet commented Sep 2, 2020

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

3 participants