Use custom tokenization function for NetCDFDataProxy objects #6231
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bouweandela
added
the
benchmark_this
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #6231 +/- ##
=======================================
Coverage 89.83% 89.83%
=======================================
Files 88 88
Lines 23315 23317 +2
Branches 4338 4338
=======================================
+ Hits 20945 20947 +2
Misses 1644 1644
Partials 726 726 ☔ View full report in Codecov by Sentry. |
⏱️ Performance Benchmark Report: 3fc5893Performance shiftsFull benchmark resultsGenerated by GHA run |
Looks really neat + I don't doubt you're right this is worth it for some cases. |
bouweandela
added
benchmark_this
⏱️ Performance Benchmark Report: 23e9462Performance shiftsFull benchmark resultsGenerated by GHA run |
|
The benchmark I added just shows: On my laptop the timings are a bit more positive, but maybe this isn't really worth the effort. Shall I just close this? |
|
As discussed at the peloton meeting, this is probably useful after all. The benchmark is likely slower in CI than on my laptop because it has different hardware and therefore spends more time on I/O, reducing the relative speedup of this change on that particular machine. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🚀 Pull Request
Description
Use custom tokenization function for NetCDFDataProxy objects. This speeds up loading small NetCDF files. With the multiple_cubes.nc file from #6223, the speedup is 10 to 20 %, though in the benchmarks below it is not really significant.
Consult Iris pull request check list
Add any of the below labels to trigger actions on this PR: