-
Notifications
You must be signed in to change notification settings - Fork 290
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make NetCDF file cache handling compatible with dask distributed #2822
base: main
Are you sure you want to change the base?
Commits on Jun 14, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 7f6a8d4 - Browse repository at this point
Copy the full SHA 7f6a8d4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6d31c20 - Browse repository at this point
Copy the full SHA 6d31c20View commit details -
Start utility function for distributed friendly
Start work on a utility function to get a dask array from a dataset variable in a way that is friendly to dask.distributed.
Configuration menu - View commit details
-
Copy full SHA for 1e26d1a - Browse repository at this point
Copy the full SHA 1e26d1aView commit details -
Parameterise test and simplify implementation
For the distributed-friendly dask array helper, parameterise the test to cover more cases. Simplify the implementation.
Configuration menu - View commit details
-
Copy full SHA for be40c5b - Browse repository at this point
Copy the full SHA be40c5bView commit details -
Force shape and dtype. First working prototype.
We need to force the shape and the dtype when getting the dask-distributed-friendly xarray-dataarray. Seems to have a first working prototype now.
Configuration menu - View commit details
-
Copy full SHA for cbd00f0 - Browse repository at this point
Copy the full SHA cbd00f0View commit details
Commits on Jun 20, 2024
-
Add group support and speed up tests
Add group support for getting a dask distributed friendly dask array. Speed up the related tests by sharing the dask distributed client setup and breakdown.
Configuration menu - View commit details
-
Copy full SHA for af4ee66 - Browse repository at this point
Copy the full SHA af4ee66View commit details -
Add partial backward-compatibility fol file handle
Add partial backward compatibility for accessing the file handle attribute when using caching with a NetCDF4FileHandler base class. Backward incompatibility is not 100%. Deleting the FileHandler closes the manager and therefore the ``file_handle`` property, however, when accessing the ``file_handle`` property after deleting the ``FileHandler``, it is reopened. Therefore, calling `__del__()`` manually and then accessing ``fh.file_handle`` will now return an open file (was a closed file). This should not happen in any sane use scenario.
Configuration menu - View commit details
-
Copy full SHA for dad3b14 - Browse repository at this point
Copy the full SHA dad3b14View commit details -
Respect auto_maskandscale with new caching
With the new dask-distributed-friendly caching, make sure we are respecting auto_maskandscale and are not applying scale factors twice.
Configuration menu - View commit details
-
Copy full SHA for fc58ca4 - Browse repository at this point
Copy the full SHA fc58ca4View commit details -
Remove a dead code except block that should never be reached.
Configuration menu - View commit details
-
Copy full SHA for 09c821a - Browse repository at this point
Copy the full SHA 09c821aView commit details -
Migrate TestNetCDF4FileHandler from unittest.TestCase to a regular class. Use a pytest fixture for the temporary NetCDF file.
Configuration menu - View commit details
-
Copy full SHA for 4f9c5ed - Browse repository at this point
Copy the full SHA 4f9c5edView commit details -
Broaden test match string for test_filenotfound
Broaden the string that is matched against in TestNetCDF4FileHandler.test_filenotfound. On Linux and MacOS the expected failure gives "No such file or directory". On Windows it gives "Invalid file format".
Configuration menu - View commit details
-
Copy full SHA for ec76fa6 - Browse repository at this point
Copy the full SHA ec76fa6View commit details
Commits on Jul 24, 2024
-
fix docstring example spelling
Fix the spelling in the docstring example using netCDF4. Co-authored-by: David Hoese <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 06d8811 - Browse repository at this point
Copy the full SHA 06d8811View commit details -
Prevent unexpected type promotion in unit test
Add a workaround to prevent an unexpected type promotion in the unit test for dask distributed friendly dask arrays.
Configuration menu - View commit details
-
Copy full SHA for aaf91b9 - Browse repository at this point
Copy the full SHA aaf91b9View commit details -
Use block info getting a dd-friendly da
When getting a dask-distributed friendly dask array from a NetCDF file using the CachingFileManager, use the information provided in bloc_info on the array location in case we are reading not the entire variable.
Configuration menu - View commit details
-
Copy full SHA for a2ad42f - Browse repository at this point
Copy the full SHA a2ad42fView commit details
Commits on Jul 25, 2024
-
Rename to serialisable and remove group argument
Rename get_distributed_friendly_dask_array to get_serialisable_dask_array and remove the group argument, moving the responsibility for handlings groups to the caller.
Configuration menu - View commit details
-
Copy full SHA for 9126bbe - Browse repository at this point
Copy the full SHA 9126bbeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5e576f9 - Browse repository at this point
Copy the full SHA 5e576f9View commit details -
Pytroll uses US spelling. Rename serializable to serialisable. Remove removed keyword argument from call.
Configuration menu - View commit details
-
Copy full SHA for 63e7507 - Browse repository at this point
Copy the full SHA 63e7507View commit details -
Ensure that the meta we pass to map_blocks also has the right dtype. Not sure if this is necessary when map_blocks already has the right dtype, but it can't hurt.
Configuration menu - View commit details
-
Copy full SHA for ea04595 - Browse repository at this point
Copy the full SHA ea04595View commit details -
Merge branch 'main' into bugfix-2815
Fixing three merge conflicts.
Configuration menu - View commit details
-
Copy full SHA for 523671a - Browse repository at this point
Copy the full SHA 523671aView commit details -
Configuration menu - View commit details
-
Copy full SHA for fde3896 - Browse repository at this point
Copy the full SHA fde3896View commit details
Commits on Jul 26, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 5b137e8 - Browse repository at this point
Copy the full SHA 5b137e8View commit details -
Use cache already in scene creation
When caching, make sure we use the CachingFileManager already upon scene creation and not only by the time we are loading.
Configuration menu - View commit details
-
Copy full SHA for c2b1533 - Browse repository at this point
Copy the full SHA c2b1533View commit details -
Use helper function rather than subclass
Don't subclass netCDF4.Dataset, rather just return an instance from a helper function. Seems good enough and gets rid of the weird error messages upon exit.
Configuration menu - View commit details
-
Copy full SHA for 9fce5a7 - Browse repository at this point
Copy the full SHA 9fce5a7View commit details -
restore non-cached group retrieval
Some readers read entire groups; this needs xarray kwargs to be set even if caching is used.
Configuration menu - View commit details
-
Copy full SHA for 4993b65 - Browse repository at this point
Copy the full SHA 4993b65View commit details
Commits on Aug 23, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 7c173e7 - Browse repository at this point
Copy the full SHA 7c173e7View commit details