Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix bugs, implement requested enhancements #12

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix bugs, implement requested enhancements #12

wants to merge 1 commit into from

Conversation

gold2718
Copy link
Collaborator

@gold2718 gold2718 commented May 3, 2024 •
edited
Loading

This PR fixes some bugs and adds some new requested features:

  • Speed up ice cataloging by using filename (with option to use data) Implement copy for non-NetCDF files
  • Implement --compress-only which compresses but does not concatenate history files (Provide a no-concatenation option #7)
  • Implement --keep-monthly which does not concatenate monthly history files when doing --yearly compression (Do not always concatenate all files #5)
  • Improve error processing
  • Update tests and add new regression tests
  • Fix date retrieval from ocean files
  • Add date range to --merge-all output files (Add time period to output file #4)
  • Implemented a progress bar for file catalog step
  • Improvements in error reporting
  • Improvements in documentation (--help screen)
  • Add compression or copying of restart files (using rest:rest component) (Include restart and / or initial data files? #3)

fixes #3
fixes #4
fixes #5
fixes #7

Improvements in error reporting
6ba8926 
Speed up ice cataloging by using filename (with option to use data)
Implement copy for non-NetCDF files
Implement --compress-only
Improve error processing
Update tests and add new regression tests
Fix date retrieval from ocean files
Add date range to --merge-all output files
Implemented a progress bar for file catalog step
@gold2718 gold2718 added bug Something isn't working enhancement New feature or request labels May 3, 2024
@gold2718 gold2718 self-assigned this May 3, 2024
@gold2718
Copy link
Collaborator Author

gold2718 commented May 3, 2024

@matsbn, To avoid having to review everything, please at least have a look at the new --compress-only option which compresses files (or just copies non-NetCDF files) but does no concatenation.
Note that for compress only, the default compression setting (2) does not seem to shrink the file size. However, setting --compress to a high number (e.g., 9) does shrink each file.

@gold2718
Copy link
Collaborator Author

gold2718 commented May 3, 2024

@DirkOlivie, please have a look at the new --keep-monthly feature. Is this the expected behavior? Note that the monthly files so not seem to shrink (and sometimes grow a bit) with the default --compress setting of 2. Compression does seem to perform well using a higher setting such as 9.

@gold2718
Copy link
Collaborator Author

gold2718 commented May 3, 2024

@lisesg,
Please have a look at the new --merge-all output filenames to verify that they are what you expect.

lisesg
lisesg approved these changes May 3, 2024
View reviewed changes
Copy link

@lisesg lisesg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lisesg lisesg lisesg approved these changes

@matsbn matsbn Awaiting requested review from matsbn

@DirkOlivie DirkOlivie Awaiting requested review from DirkOlivie

At least 1 approving review is required to merge this pull request.

Assignees

@gold2718 gold2718

Labels
bug Something isn't working enhancement New feature or request
Projects
NorESM Development
Status: Todo
Milestone
No milestone
2 participants
@gold2718 @lisesg