Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Less frequent progress updates for percentage complete when processing logging #457

Open
carlhiggs opened this issue Jun 20, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@carlhiggs
Copy link
Collaborator

carlhiggs commented Jun 20, 2024

Currently, when running the workflow analysis wrapper function, this writes progress to the log file with an update very frequently, perhaps multiple times per second. If these were printed to screen that would maybe be fine --- like realtime feedback. But printed to a log file it is more than excessive (although, it can be helpful, for tracking pogress).

For example, for very large study regions, the updates cause the log files to be extremely long and take up unnecessary disk space, and unnecessary time making disk writes
image

While an ideal solution might be to implement proper logging functionality, a quick solution that would reduce the size of logging outputs while still allowing tracking of progress would be to reduce the frequency of updates, e.g. so only increments of 1% complete are reported.

Example of how to do this are described here:
https://stackoverflow.com/questions/47995958/python-tqdm-package-how-to-configure-for-less-frequent-status-bar-updates

This isn't a high priority, but it would be nice to have.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant