Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adapt reader mviri_l1b_fiduceo_nc #2802

Merged
merged 37 commits into from
Oct 17, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
f27a91d
fix for file reading; includes removing chunk reading and decoding ti…
bkremmli May 17, 2024
a08d9bc
Merge branch 'read_error' of https://github.com/bkremmli/satpy into r…
bkremmli May 17, 2024
2dbae1a
remove import datetime
bkremmli May 17, 2024
eb0382a
add bkremmli to AUTHORS.md
bkremmli May 17, 2024
7d6608a
correct for failures from hook id ruff
bkremmli May 17, 2024
ec22136
minor adaptations from PR comments
bkremmli May 17, 2024
5beedea
Update satpy/readers/mviri_l1b_fiduceo_nc.py
bkremmli May 17, 2024
21679c6
perform chunking after open_dataset and use decode_cf = False
bkremmli May 24, 2024
a7cb10d
Merge branch 'read_error' of https://github.com/bkremmli/satpy into r…
bkremmli May 24, 2024
59880ce
decode times separatly from other variables, adds TestInterpolator
bkremmli May 28, 2024
fb93f00
fixes _decode_cf() and tests
bkremmli Jun 4, 2024
951c9b0
adds test_fix_duplicate_dimensions and removes leftover dimensions "s…
bkremmli Jun 5, 2024
a379a08
Update mviri_l1b_fiduceo_nc.py
bkremmli Jun 5, 2024
5ec0cf9
adds support for filenames of MVIRI FCDR L1.5 release 2
bkremmli Jun 6, 2024
73acfb7
Merge pull request #1 from bkremmli/mviri_release2
bkremmli Jun 6, 2024
01856fd
Merge branch 'pytroll:main' into read_error
bkremmli Sep 3, 2024
c1cd334
sync/merge with fork diffs
bkremmli Sep 3, 2024
abf916e
adapt changes after xarray release 2024.7.0: include chunks with open…
bkremmli Sep 3, 2024
5822e50
removed chunks part from test_fix_duplicate_dimensions; adapted tests…
bkremmli Sep 3, 2024
f41e068
moved code _get_projection_longitude()
bkremmli Sep 4, 2024
6b079b1
fix test_fix_duplicate_dimensions
bkremmli Sep 4, 2024
fd099cd
- suppress warnings for renaming duplicate dimensions
bkremmli Sep 16, 2024
7c02d2d
define fill_val in test
bkremmli Sep 17, 2024
e90d7d0
ancillary changes like comments, etc.
bkremmli Sep 23, 2024
51d8e77
change fill_val definition in test, changes for Code Scene status
bkremmli Sep 24, 2024
89d9931
change fill_val to int64
bkremmli Sep 24, 2024
7d0ecef
include comments, create fixture_time_fake_dataset()
bkremmli Sep 24, 2024
11cf080
adapt fill_val definition
bkremmli Sep 24, 2024
03ec1fe
adapt fill_val definition
bkremmli Sep 24, 2024
bcae3f7
lower fill_val for tests for windows compatibility
bkremmli Sep 25, 2024
89bb691
Merge remote-tracking branch 'origin/main' into read_error
bkremmli Sep 27, 2024
672cd9c
Merge pull request #2 from pytroll/main
bkremmli Sep 27, 2024
c742238
Merge pull request #3 from pytroll/main
bkremmli Oct 1, 2024
38c8e9b
add test for get_projection_longitude,
bkremmli Oct 1, 2024
2042b1a
Merge remote-tracking branch 'origin/read_error' into read_error
bkremmli Oct 1, 2024
d88d3d3
Fix test miss
sfinkens Oct 1, 2024
484b723
revert to smaller fill_val
bkremmli Oct 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AUTHORS.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ The following people have made contributions to this project:
- [Johannes Johansson (JohannesSMHI)](https://github.com/JohannesSMHI)
- [Sauli Joro (sjoro)](https://github.com/sjoro)
- [Janne Kotro (jkotro)](https://github.com/jkotro)
- [Beke Kremmling (bkremmli)](https://github.com/bkremmli) - Deutscher Wetterdienst
- [Ralph Kuehn (ralphk11)](https://github.com/ralphk11)
- [Panu Lahtinen (pnuu)](https://github.com/pnuu)
- [Jussi Leinonen (jleinonen)](https://github.com/jleinonen) - meteoswiss
Expand Down
24 changes: 18 additions & 6 deletions satpy/readers/mviri_l1b_fiduceo_nc.py
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,7 @@
import warnings

import dask.array as da
import numpy
bkremmli marked this conversation as resolved.
Show resolved Hide resolved
import numpy as np
import xarray as xr

Expand Down Expand Up @@ -513,11 +514,19 @@ def get_time(self):
"""Get time coordinate.

Variable is sometimes named "time" and sometimes "time_ir_wv".
FillValues in time are set to NaT.
"""
try:
return self["time_ir_wv"]
time = self["time_ir_wv"]
except KeyError:
return self["time"]
time = self["time"]

timeOffset = time.attrs["add_offset"]
condition = time == time.attrs["_FillValue"]

time = xr.where(condition, numpy.nan, time + timeOffset)
time = (time*1e9).astype("datetime64[ns]")
bkremmli marked this conversation as resolved.
Show resolved Hide resolved
return time

def get_xy_coords(self, resolution):
"""Get x and y coordinates for the given resolution."""
Expand Down Expand Up @@ -555,11 +564,14 @@ def __init__(self, filename, filename_info, filetype_info, # noqa: D417
self.mask_bad_quality = mask_bad_quality
nc_raw = xr.open_dataset(
filename,
chunks={"x": CHUNK_SIZE,
"y": CHUNK_SIZE,
"x_ir_wv": CHUNK_SIZE,
"y_ir_wv": CHUNK_SIZE}
# chunks={"x": CHUNK_SIZE,
# "y": CHUNK_SIZE,
# "x_ir_wv": CHUNK_SIZE,
# "y_ir_wv": CHUNK_SIZE},
bkremmli marked this conversation as resolved.
Show resolved Hide resolved
decode_times=False,
decode_cf=False
)

self.nc = DatasetWrapper(nc_raw)

# Projection longitude is not provided in the file, read it from the
Expand Down
19 changes: 14 additions & 5 deletions satpy/tests/reader_tests/test_mviri_l1b_fiduceo_nc.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,8 @@
{"sun_earth_distance_correction_applied": True,
"sun_earth_distance_correction_factor": 1.}
)
acq_time_vis_exp = [np.datetime64("1970-01-01 00:30").astype("datetime64[ns]"),
np.datetime64("1970-01-01 00:30").astype("datetime64[ns]"),
acq_time_vis_exp = [np.datetime64("NaT").astype("datetime64[ns]"),
np.datetime64("NaT").astype("datetime64[ns]"),
np.datetime64("1970-01-01 02:30").astype("datetime64[ns]"),
np.datetime64("1970-01-01 02:30").astype("datetime64[ns]")]
vis_counts_exp = xr.DataArray(
Expand Down Expand Up @@ -124,7 +124,7 @@
},
attrs=attrs_exp
)
acq_time_ir_wv_exp = [np.datetime64("1970-01-01 00:30").astype("datetime64[ns]"),
acq_time_ir_wv_exp = [np.datetime64("NaT").astype("datetime64[ns]"),
np.datetime64("1970-01-01 02:30").astype("datetime64[ns]")]
wv_counts_exp = xr.DataArray(
np.array(
Expand Down Expand Up @@ -272,51 +272,60 @@
dtype=np.uint8
)
)
time = np.arange(4) * 60 * 60 * 1e9
time = time.astype("datetime64[ns]").reshape(2, 2)
time = np.arange(4) * 60 * 60
timeFillValue=4294967295
timeAddOffset=0
bkremmli marked this conversation as resolved.
Show resolved Hide resolved
time[0] = timeFillValue
time[1] = timeFillValue
time = time.reshape(2,2)

ds = xr.Dataset(
data_vars={
"count_vis": (("y", "x"), count_vis),
"count_wv": (("y_ir_wv", "x_ir_wv"), count_wv),
"count_ir": (("y_ir_wv", "x_ir_wv"), count_ir),
"toa_bidirectional_reflectance_vis": vis_refl_exp / 100,
"u_independent_toa_bidirectional_reflectance": u_vis_refl_exp / 100,
"quality_pixel_bitmask": (("y", "x"), mask),
"solar_zenith_angle": (("y_tie", "x_tie"), sza),
"time_ir_wv": (("y_ir_wv", "x_ir_wv"), time),
"a_ir": -5.0,
"b_ir": 1.0,
"bt_a_ir": 10.0,
"bt_b_ir": -1000.0,
"a_wv": -0.5,
"b_wv": 0.05,
"bt_a_wv": 10.0,
"bt_b_wv": -2000.0,
"years_since_launch": 20.0,
"a0_vis": 1.0,
"a1_vis": 0.01,
"a2_vis": -0.0001,
"mean_count_space_vis": 1.0,
"distance_sun_earth": 1.0,
"solar_irradiance_vis": 650.0,
"sub_satellite_longitude_start": 57.1,
"sub_satellite_longitude_end": np.nan,
"sub_satellite_latitude_start": np.nan,
"sub_satellite_latitude_end": 0.1,
},
coords={
"y": [1, 2, 3, 4],
"x": [1, 2, 3, 4],
"y_ir_wv": [1, 2],
"x_ir_wv": [1, 2],
"y_tie": [1, 2],
"x_tie": [1, 2]

},
attrs={"foo": "bar"}
)
ds["count_ir"].attrs["ancillary_variables"] = "a_ir b_ir"
ds["count_wv"].attrs["ancillary_variables"] = "a_wv b_wv"

ds["time_ir_wv"].attrs["_FillValue"] = timeFillValue
ds["time_ir_wv"].attrs["add_offset"] = timeAddOffset

Check warning on line 328 in satpy/tests/reader_tests/test_mviri_l1b_fiduceo_nc.py

View check run for this annotation

CodeScene Delta Analysis / CodeScene Cloud Delta Analysis (main)

❌ New issue: Large Method

fixture_fake_dataset has 72 lines, threshold = 70. Large functions with many lines of code are generally harder to understand and lower the code health. Avoid adding more lines to this function.
return ds


Expand Down