-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update log_evaluation_row!
to not special case Spearman correlation coefficient
#64
Comments
I think we could also consider changing |
And the follow-up:
So the history here is, back in the VERY beginning we wanted to track all of our Lighthouse metrics in tensorboard. And (at the time, potentially not still now) it was tricky (and not an internal priority) to do things like natively log line series or histograms or any of the values that Lighthouse was calculating. So instead, we logged the image of the plots generated from those metrics directly. Then, later, we wanted to be able to directly compare spearman correlation across runs from within tb, and since that was a single value, it was easy to use tb-native logging for it---so we did, and decided that any other values like that could be added as needed. (In hindsight, that special casing should have been done outside this package, but c'est la vie!) In reality, spearman is no more special/necessary/useful than any of the other metrics, and callers will likely want to log and be able to track most/all of them if they want any of them. What should the default behavior be, when not designed for tb? Probably some combo of "posting all fields in the metrics" (which currently include both scalar values and line series) and/or posting the image. (In the (not yet public) LWB we do both image and scalars, and should---but don't yet---log the line series as well) |
Makes sense! Yeah, I think we should just log everything here (scalars, curves, etc), dispatching to |
okay, i'm not "allowed" to comment below this point, but I think we should specialize the below implementation of
log_evaluation_row!
on LearnLogger so that special-casing spearman correlation doesn't have to happen by default---that is a very tb-specific logged item. E.g.,or, better:
and
Originally posted by @hannahilea in #60 (comment)
The text was updated successfully, but these errors were encountered: