FixedScaleOffset for handling NaN inputs

[mullenkamp]

Hi,

I was wondering if you all would be interested in a FixedScaleOffset that can handle np.nan inputs? In the style of HDF/netcdf, having a fill value to replace np.nan with an appropriate integer. This could be either user defined or determined automatically based on the astype integer size (assign it the smallest possible integer value).
I can either modify the existing FixedScaleOffset class, or I could create another class. It's a very simple change, though there may be concerns of more memory usage due to boolean masking.

Also, is there any reason why dtype shouldn't always be a float and astype shouldn't always be an integer?

Thanks

[martindurant]

is there any reason why dtype shouldn't always be a float and astype shouldn't always be an integer

I can certainly imagine an offset for integers, but a scale less so. Of course, you also get to specify the bitsize of each, and it can be important whether you save as uint8 and load into float32 or something bigger.

[mps01060]

We have a lot of use for this, especially to move away from using attributes "add_offset" and "scale_factor" in xarray, and instead using the zarr encoding directly. I’ve made some preliminary changes to address this issue in my fork. You can check them out in this branch: fixedscaleoffset-nans.

I added to the tests, except the test_backwards_compatibility. This appears to create files that do not clean up after each run, which will interfere with the nan/fill_value cases because the codecs get run on all the previous datasets saved. This causes an issue when a scale/offset is used on a "old" dataset that has values that do not make sense to use with the current codec's offset/scale. I am probably overlooking a way to handle this:

https://github.com/zarr-developers/numcodecs/blob/main/numcodecs/tests/common.py#L259-L276

    # save fixture data
    for i, arr in enumerate(arrays):
        arr_fn = os.path.join(fixture_dir, f'array.{i:02d}.npy')
        if not os.path.exists(arr_fn):  # pragma: no cover
            np.save(arr_fn, arr)

    # load fixture data
    for arr_fn in glob(os.path.join(fixture_dir, 'array.*.npy')):
        # setup
        i = int(arr_fn.split('.')[-2])
        arr = np.load(arr_fn, allow_pickle=True)
        arr_bytes = arr.tobytes(order='A')
        if arr.flags.f_contiguous:
            order = 'F'
        else:
            order = 'C'

        for j, codec in enumerate(codecs):

Would this be something worth going forward with? Thank you for the help/suggestions.

[mps01060]

I had some confusion on my previous post. I've added some fixtures for the case where a fill_value is present. Would this be worth a PR, or should this "fill_value" FixedScaleOffset be added as a different filter?