About dask usage

[ArthurVincentCS]

Hi,

First, thank you for this library—it’s functional, well-documented, and I was able to find answers to my questions by exploring the code a bit.

I’m interested in using EOReader with Dask, initially for Sentinel-2 L2A data. As mentioned in the documentation (and within the code), I set the environment variable EOREADER_USE_DASK to 1. Then, I proceed to load a Sentinel-2 product. I expected the code to execute very quickly due to Dask’s lazy evaluation, but that wasn’t the case. Additionally, the output data was in NumPy format rather than Dask arrays.

path = "SENTINEL2A_20191119-104916-971_L2A_T31TCH_C_V2-2"
output = os.path.abspath("outputs")
prod = Reader().open(path, output_path=output)
band_list = [GREEN, RED, SWIR_1, BLUE, NARROW_NIR, ALL_CLOUDS, CLOUDS, NDSI, SHADOWS]
ok_bands = to_str([band for band in band_list if prod.has_band(band)])
bands = prod.load(ok_bands, pixel_size=20., **{TO_REFLECTANCE: False}, chunks={'band': 1, 'x': 1024, 'y': 1024})
print(type(bands[GREEN]))

From what I understand, data remains in Dask array format until self._collocate_bands() is called, which in turn uses RasterArray.reproject_match, a function that doesn’t appear to support Dask arrays.

is it right or I'm missing something ?

I fully understand the need to align all input data onto a common grid. However, it would be great to leverage Dask throughout the process in EOReader. If dask capability is a key feature for you too, it seems that there are other options for data reprojection that could be explored: https://discourse.pangeo.io/t/can-a-reprojection-change-of-crs-operation-be-done-lazily-using-rioxarray/4468.

Arthur.

[remi-braun]

Hi Arthur,

Thanks for the kind feedback!

Sadly Dask support is still in its very early stage.

I bet there are multiple functions that are not daskified, and reproject functions from rioxarray are some of them.
See #58 and #115 for example :)

I will go towards a better support in the following releases.
In fact, I'm just currently learning about proper handling of this library, so it might take a little time to have a full support.
Any help would be appreciated 🙏

[remi-braun]

@ArthurVincentCS
I found usecases without any possibility to use Dask with the current state of geospatial opensource ecosystem: for example, reprojecting witth RPCs: #115 (comment)