Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Rayleigh correction to use the same datatype as the input data #2954

Merged
merged 8 commits into from
Oct 25, 2024

Conversation

pnuu
Copy link
Member

@pnuu pnuu commented Oct 24, 2024

This is workaround for the Pyspectral Rayleigh correction that always uses float64 datatype.

This includes also few additional dtype checks for other modifiers and enhancements. Those changes are here because I originally started this to hunt a data upcasting in true color composite, but didn't end up finding the final issue and didn't want to discard all the changes.

I'll create an issue to Pyspectral to fix the data upcasting. Hopefully tomorrow. Or a direct fix if I see a simple solution.

  • Tests added

Copy link

codecov bot commented Oct 24, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.09%. Comparing base (118fc93) to head (3f1076a).
Report is 7 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2954   +/-   ##
=======================================
  Coverage   96.09%   96.09%           
=======================================
  Files         377      377           
  Lines       55110    55125   +15     
=======================================
+ Hits        52960    52975   +15     
  Misses       2150     2150           
Flag Coverage Δ
behaviourtests 3.94% <0.00%> (-0.01%) ⬇️
unittests 96.19% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@coveralls
Copy link

coveralls commented Oct 24, 2024

Pull Request Test Coverage Report for Build 11517663375

Details

  • 61 of 61 (100.0%) changed or added relevant lines in 3 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.003%) to 96.206%

Totals Coverage Status
Change from base Build 11517264487: 0.003%
Covered Lines: 53219
Relevant Lines: 55318

💛 - Coveralls

@djhoese
Copy link
Member

djhoese commented Oct 24, 2024

@pnuu so is the plan for this to be closed and not used? Theoretically pyspectral should have all the changes for dtype stuff, right? Also I really thought I got pyspectral working and retaining dtypes. Shoot.

@pnuu
Copy link
Member Author

pnuu commented Oct 24, 2024

Well, maybe the tests I've added should stay. I'll have a look at how much needs to be done to Pyspectral so that it doesn't upcast stuff, and whether there are still something that is needed on Satpy. Like a kwarg or something.

@pnuu
Copy link
Member Author

pnuu commented Oct 25, 2024

I'm converting this to a draft while I inspect the Pyspectral side of things.

@pnuu pnuu marked this pull request as draft October 25, 2024 08:27
@pnuu pnuu marked this pull request as ready for review October 25, 2024 10:03
@pnuu
Copy link
Member Author

pnuu commented Oct 25, 2024

Right. There was nothing wrong in Pyspectral, it was just the clipping of input value of reduce_strength that produced int64 for reduce_strength=1 and thus promoted the float32 data to float64.

@pnuu
Copy link
Member Author

pnuu commented Oct 25, 2024

So in short: this PR makes sure the inputs to Rayleigh correction has proper data type to keep the data from being upcast. And adds the unrelated checks to other tests.

Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, please just add some assert for the computed dtype in some places also

satpy/modifiers/atmosphere.py Show resolved Hide resolved
satpy/tests/enhancement_tests/test_enhancements.py Outdated Show resolved Hide resolved
satpy/tests/test_composites.py Show resolved Hide resolved
satpy/tests/test_composites.py Outdated Show resolved Hide resolved
satpy/tests/test_composites.py Show resolved Hide resolved
Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mraspaud
Copy link
Member

@djhoese do you want to look at this? otherwise it's good to merge

Copy link
Member

@djhoese djhoese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pnuu pnuu merged commit 5f4e4c1 into pytroll:main Oct 25, 2024
18 checks passed
@pnuu pnuu deleted the bugfix-rayleigh-reflectance-dtype branch October 25, 2024 13:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants