Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adapt reader mviri_l1b_fiduceo_nc #2802

Open
wants to merge 29 commits into
base: main
Choose a base branch
from
Open

Conversation

bkremmli
Copy link

@bkremmli bkremmli commented May 17, 2024

This PR fixes the mviri_l1b_fiduceo_nc reader when being used with a new xarray version (2024.3.0). When using the original reader, a ValueError about not being able to decode the times is thrown. The file is now opened without decoding. The decoding is now done in DatasetWrapper()._decode_cf(). The time is decoded separatly from the other data values. FillValues for the time are recognized to replace time values with NaT and time is decoded using the offset values included within the attributes.

Also, opening the dataset using chunks is deactivated because the input files contains dimensions of the same name which cannot be processed by xarray at the moment. The chunking as well as renaming the dimensions is also performed in DatsetWrapper().

  • [x ] Tests added
  • Fully documented --> this change should not be visible to users
  • [x ] Add your name to AUTHORS.md if not there already

Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this feature! I'm no expert here, so problably @sfinkens needs to have a look, but I wrote some comments already :)

satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
@mraspaud mraspaud requested a review from sfinkens May 17, 2024 10:09
Copy link
Member

@sfinkens sfinkens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work @bkremmli! Thanks for updating the tests, so that the problem is triggered.

satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
@sfinkens
Copy link
Member

Also I noticed that space pixels now have some finite values (instead of NaN), because decode_cf=False. You can use decode_cf=True together with time.encoding["add_offset/_FillValue"], see sfinkens@7045a87

Copy link

codecov bot commented May 24, 2024

Codecov Report

Attention: Patch coverage is 98.57143% with 1 line in your changes missing coverage. Please review.

Project coverage is 96.03%. Comparing base (084b114) to head (11cf080).
Report is 131 commits behind head on main.

Files with missing lines Patch % Lines
satpy/readers/mviri_l1b_fiduceo_nc.py 96.15% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2802      +/-   ##
==========================================
- Coverage   96.06%   96.03%   -0.04%     
==========================================
  Files         370      373       +3     
  Lines       54320    54464     +144     
==========================================
+ Hits        52185    52305     +120     
- Misses       2135     2159      +24     
Flag Coverage Δ
behaviourtests 3.98% <0.00%> (-0.02%) ⬇️
unittests 96.13% <98.57%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@sfinkens sfinkens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, almost there!

I think in the tests you need to write the dataset to disk and read it back in. Otherwise the dask error with duplicate dimensions is not triggered.

satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
@bkremmli
Copy link
Author

bkremmli commented Jun 4, 2024

The test for the duplicate dimensions still needs to be added.

…ing dataset; also rename dimension "channel"
…; moved decoding into separate method of DatasetWrapper
satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
- include u_independent_toa_bidirectional_reflectance in tests
- patch decode_cf/fix_duplicate_dimensions in test_reassign_coords()
Copy link
Member

@sfinkens sfinkens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks very good, thanks! I'm just nitpicking with minor things now 🙂

satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/readers/mviri_l1b_fiduceo_nc.py Show resolved Hide resolved
satpy/readers/mviri_l1b_fiduceo_nc.py Show resolved Hide resolved
satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
satpy/readers/mviri_l1b_fiduceo_nc.py Show resolved Hide resolved
Copy link
Member

@sfinkens sfinkens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

satpy/readers/mviri_l1b_fiduceo_nc.py Show resolved Hide resolved
@sfinkens
Copy link
Member

Let's see if we can make CodeScene happy

Copy link
Member

@sfinkens sfinkens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job!

satpy/tests/reader_tests/test_mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants