Add combined GRIB reader for both SEVIRI and FCI L2 products #2717

dnaviap · 2024-01-11T12:55:06Z

Add combined GRIB reader that can be used for both SEVIRI and FCI L2 products.

Tests added
Fully documented
Add @dnaviap to AUTHORS.md

Feature/eum l2 grib reader

strandgren · 2024-01-11T16:20:27Z

Thanks for working on this @dnaviap :) Before reviewing in more detail, I think we need some refactoring of the files:

Instead of having a eum_l2_grib.yaml files I suggest to keep the seviri_l2_grib.yaml file (which is still there) and add the FCI cloud mask dataset in a new fci_l2_grib.yaml file. That way we don't change the user interface to the reader. That's also how it's being done for the BUFR reader (Changes to Eumetsat L2 BUFR reader #2603)
eum_l2_grib.py should be compatible with both FCI and SEVIRI data, meaning that seviri_l2_grib.py should become obsolete and should therefore be removed.
Similarly, the new test file test_eum_l2_grib.py should make test_seviri_l2_grib.py obsolete, meaning it can be removed as well.

…ing the user interface to the reader

…and SEVIRI data

Feature/eum l2 grib refactor

dnaviap · 2024-01-12T15:51:20Z

The refactoring is complete and tested.

strandgren

Thanks for the refactoring! :) See a couple of comments inline.

Can you please also modify the doc-string of the _scale_earth_axis method to read:

    def _scale_earth_axis(data):
        """Scale Earth axis data to make sure the value matched the expected unit [m].
        
        The earthMinorAxis value stored in the MPEF aerosol over sea product prior to December 12, 2022 has the wrong unit and this method provides a flexible work-around by making sure that all earth axis values are scaled such
        that they are on the order of millions of meters as expected by the reader. 
        
        """

That was a fix implemented by me a while ago when we noticed some issues in one of the products, and although it has been fixed in the product, the archived files will still have the issue meaning that the fix is still useful to have in the reader.

satpy/readers/eum_l2_grib.py

satpy/readers/fci_base.py

satpy/tests/reader_tests/test_eum_l2_grib.py

dnaviap · 2024-05-13T08:52:46Z

Thanks for the refactoring! :) See a couple of comments inline.

Can you please also modify the doc-string of the _scale_earth_axis method to read:
    def _scale_earth_axis(data):
        """Scale Earth axis data to make sure the value matched the expected unit [m].
        
        The earthMinorAxis value stored in the MPEF aerosol over sea product prior to December 12, 2022 has the wrong unit and this method provides a flexible work-around by making sure that all earth axis values are scaled such
        that they are on the order of millions of meters as expected by the reader. 
        
        """
That was a fix implemented by me a while ago when we noticed some issues in one of the products, and although it has been fixed in the product, the archived files will still have the issue meaning that the fix is still useful to have in the reader.

strandgren

Thanks for the fixes and the refactoring @dnaviap!

I left a couple of minor comments in-line. Could you please also merge the main branch into this branch and resolve the merge conflict such that all unit tests etc. run?

satpy/readers/eum_l2_grib.py

satpy/tests/reader_tests/test_eum_l2_grib.py

…issues.

o HEAD # Conflicts: # AUTHORS.md # satpy/readers/eum_l2_grib.py

Fix open issues in combined EUM GRIB2 reader

codecov · 2024-06-24T08:56:14Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.95%. Comparing base (fc95ee1) to head (12f1860).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2717      +/-   ##
==========================================
+ Coverage   95.94%   95.95%   +0.01%     
==========================================
  Files         366      368       +2     
  Lines       53584    53671      +87     
==========================================
+ Hits        51412    51501      +89     
+ Misses       2172     2170       -2

Flag	Coverage Δ
behaviourtests	`4.03% <0.00%> (-0.01%)`	⬇️
unittests	`96.05% <100.00%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Add fci base test

coveralls · 2024-07-01T12:39:21Z

Pull Request Test Coverage Report for Build 9743868855

Details

168 of 168 (100.0%) changed or added relevant lines in 5 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.01%) to 96.056%

Totals
Change from base Build 9702977764:	0.01%
Covered Lines:	51732
Relevant Lines:	53856

💛 - Coveralls

mraspaud

Nice work, just a couple of comments and questions

mraspaud · 2024-07-19T10:17:48Z

satpy/tests/reader_tests/test_fci_base.py

+from satpy.tests.utils import make_dataid
+
+
+class TestCalculateAreaExtent(unittest.TestCase):


We try to go over to pytest nowadays, rather that using unittest...

Suggested change

class TestCalculateAreaExtent(unittest.TestCase):

class TestCalculateAreaExtent:

mraspaud · 2024-07-19T10:18:34Z

satpy/tests/reader_tests/test_eum_l2_grib.py

+class TestEUML2GribFileHandler(unittest.TestCase):
+    """Test the EUML2GribFileHandler reader."""
+
+    @mock.patch("satpy.readers.eum_l2_grib.ec")
+    def setUp(self, ec_):
+        """Set up the test by creating a mocked eccodes library."""
+        fake_gid_generator = (i for i in FAKE_GID)
+        ec_.codes_grib_new_from_file.side_effect = lambda fh: next(fake_gid_generator)
+        self.ec_ = ec_


We try to go over to pytest nowadays, rather that using unittest. Could you convert this to pytest style instead?

mraspaud · 2024-07-19T10:20:32Z

satpy/tests/reader_tests/test_eum_l2_grib.py

+    @mock.patch("satpy.readers.eum_l2_grib.xr")
+    @mock.patch("satpy.readers.eum_l2_grib.da")


I see a bit of mocking. In our experience this can lead to problems down the road, and nowadays, we prefer to write a synthetic testing file to tmp_path and work with that instead. Do you think this would be doable for this format?

David Navia and others added 6 commits January 11, 2024 09:57

Add common functionality for FCI data readers

28f44c7

Add reader for both SEVIRI and FCI L2 products in GRIB2 format

d3ccedc

Add EUM L2 GRIB-reader test package

de65b63

Add my name to AUTHORS.md

0474fd0

Merge pull request #1 from dnaviap/feature/eum_l2_grib_reader

1150b44

Feature/eum l2 grib reader

Merge branch 'pytroll:main' into main

d8043db

dnaviap requested review from djhoese and mraspaud as code owners January 11, 2024 12:55

David Navia and others added 5 commits January 12, 2024 16:33

Delete eum_l2_grib.yaml and update seviri_l2_grib.yaml to avoid chang…

767aeab

…ing the user interface to the reader

Add fci_l2_grib.yaml reader

a53ddea

Delete seviri_l2_grib.py since eum_l2_grib.py is compatible with FCI …

f94c4f7

…and SEVIRI data

Delete obsolete test_seviri_l2_grib.py

66946ad

Merge pull request #2 from dnaviap/feature/eum_l2_grib_refactor

e7009de

Feature/eum l2 grib refactor

strandgren reviewed Jan 15, 2024

View reviewed changes

satpy/readers/eum_l2_grib.py Outdated Show resolved Hide resolved

satpy/readers/fci_base.py Outdated Show resolved Hide resolved

satpy/tests/reader_tests/test_eum_l2_grib.py Outdated Show resolved Hide resolved

David Navia added 3 commits May 13, 2024 08:47

Refactor duplicate code in tests

5bbb421

Correct for RSS data

486b3a6

Modify fci_base doc-string

c6322fa

dnaviap requested review from sfinkens and adybbroe as code owners May 13, 2024 08:47

dnaviap closed this May 13, 2024

dnaviap reopened this May 13, 2024

strandgren reviewed May 27, 2024

View reviewed changes

satpy/readers/eum_l2_grib.py Outdated Show resolved Hide resolved

satpy/readers/eum_l2_grib.py Outdated Show resolved Hide resolved

satpy/tests/reader_tests/test_eum_l2_grib.py Outdated Show resolved Hide resolved

strandgren and others added 4 commits June 13, 2024 14:05

Fix end_time computation, optimize SEVIRI imports and fix code style …

ed5213b

…issues.

Merge with remote main

de9a73e

o HEAD # Conflicts: # AUTHORS.md # satpy/readers/eum_l2_grib.py

Add tests for end_time.

164bcba

Merge pull request #3 from strandgren/feature_combined_eum_grib_reader

98fcca2

Fix open issues in combined EUM GRIB2 reader

dnaviap requested a review from sjoro as a code owner June 24, 2024 08:43

dnaviap and others added 5 commits June 24, 2024 11:02

Merge branch 'pytroll:main' into main

241bbc0

Add fci base test

c3efc55

Merge pull request #4 from dnaviap/feature/add-fci-base-test

6a7ba7f

Add fci base test

Merge branch 'pytroll:main' into main

291d288

Merge branch 'pytroll:main' into main

12f1860

mraspaud reviewed Jul 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add combined GRIB reader for both SEVIRI and FCI L2 products #2717

Add combined GRIB reader for both SEVIRI and FCI L2 products #2717

dnaviap commented Jan 11, 2024

strandgren commented Jan 11, 2024

dnaviap commented Jan 12, 2024

strandgren left a comment

dnaviap commented May 13, 2024

strandgren left a comment

codecov bot commented Jun 24, 2024 •

edited

Loading

coveralls commented Jul 1, 2024 •

edited

Loading

mraspaud left a comment

mraspaud Jul 19, 2024

mraspaud Jul 19, 2024

mraspaud Jul 19, 2024

		from satpy.tests.utils import make_dataid


		class TestCalculateAreaExtent(unittest.TestCase):

	class TestCalculateAreaExtent(unittest.TestCase):
	class TestCalculateAreaExtent:

		@mock.patch("satpy.readers.eum_l2_grib.xr")
		@mock.patch("satpy.readers.eum_l2_grib.da")

Add combined GRIB reader for both SEVIRI and FCI L2 products #2717

Are you sure you want to change the base?

Add combined GRIB reader for both SEVIRI and FCI L2 products #2717

Conversation

dnaviap commented Jan 11, 2024

strandgren commented Jan 11, 2024

dnaviap commented Jan 12, 2024

strandgren left a comment

Choose a reason for hiding this comment

dnaviap commented May 13, 2024

strandgren left a comment

Choose a reason for hiding this comment

codecov bot commented Jun 24, 2024 • edited Loading

Codecov Report

coveralls commented Jul 1, 2024 • edited Loading

Pull Request Test Coverage Report for Build 9743868855

Details

💛 - Coveralls

mraspaud left a comment

Choose a reason for hiding this comment

mraspaud Jul 19, 2024

Choose a reason for hiding this comment

mraspaud Jul 19, 2024

Choose a reason for hiding this comment

mraspaud Jul 19, 2024

Choose a reason for hiding this comment

codecov bot commented Jun 24, 2024 •

edited

Loading

coveralls commented Jul 1, 2024 •

edited

Loading