Make logging work in case of multiple child processes (e.g. via currently used joblib.Parallel) #1495
Labels
Comments
yarikoptic
added a commit
that referenced
this issue
Sep 5, 2024
joblib does not setup automagically some kind of logging for Parallel. Filed dedicated to possibly see it implemented - #1495 For the sake of current use case (e.g. troubleshooting #1494) it should largely suffice to return and log information about exception which was raised while loading metadata. This is what is done in this PR and while using buggy hdmf we do get nice logging in the log file at DEBUG level. No attempts were made to reduce possibly a flood of duplicate log messages since per file metadata would have unique values
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Encountered while troubleshooting
that apparently joblib.Parallel nohow takes care about configuring logging for subprocesses so there is really no logs there and that is an outstanding requested feature for joblib:
lokybackend joblib/joblib#1017If joblib does not implement it, we might want/need to e.g. via approach described in https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes . For now, in our single use in organize, I would just RF to return desired for logging side effects (error).
The text was updated successfully, but these errors were encountered: