to_dataset_dict() - raises exception when trying to load Zarr files from object store #426

agstephens · 2020-11-19T08:55:04Z

agstephens
Nov 19, 2020

I have just built an intake-esm catalog for our CMIP6 Zarr holdings in our own in-house object store on JASMIN (in the UK).

intake-esm is working well in terms of searching/filtering, but when I call col.to_dataset_dict(...) I get an error. The only difference I can see between my own catalog and other online examples is that:

in other examples, zarr paths all begin with "s3://" or "gc://"
in our private object store, the URLs begin with "http://"

I have tested that the zarr files can be loaded properly with xarray and zarr (without using intake) and this works fine:

fsmap = fsspec.get_mapper(zarr_path)
ds = xr.open_zarr(fsmap, consolidated=True, use_cftime=True)

An example zarr_path is:

http://cmip6-zarr-o.s3.jc.rl.ac.uk/CMIP6.DCPP.IPSL.IPSL-CM6A-LR/dcppC-ipv-pos.r1i1p1f1.Amon.rsus.gr.v20190110.zarr

In my catalog, I specify the asset as:

  "assets": {
    "column_name": "zarr_path",
    "format": "zarr"
  },

However, when I use col.to_dataset_dict(zarr_kwargs={"consolidated": True, "use_cftime": True}) I get the following error:

Traceback (most recent call last):
  File "/home/users/astephen/cmip6/cmip6-object-store/venv/lib/python3.7/site-packages/intake_esm/merge_util.py", line 273, in _open_asset
    ds = xr.open_zarr(path, **zarr_kwargs)
  File "/home/users/astephen/cmip6/cmip6-object-store/venv/lib/python3.7/site-packages/xarray/backends/zarr.py", line 667, in open_zarr
    chunk_store=chunk_store,
  File "/home/users/astephen/cmip6/cmip6-object-store/venv/lib/python3.7/site-packages/xarray/backends/zarr.py", line 290, in open_group
    zarr_group = zarr.open_consolidated(store, **open_kwargs)
  File "/apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14/envs/jaspy3.7-m3-4.6.14-r20200606/lib/python3.7/site-packages/zarr/convenience.py", line 1179, in open_consolidated
    meta_store = ConsolidatedMetadataStore(store, metadata_key=metadata_key)
  File "/apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14/envs/jaspy3.7-m3-4.6.14-r20200606/lib/python3.7/site-packages/zarr/storage.py", line 2501, in __init__
    meta = json_loads(store[metadata_key])
  File "/apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14/envs/jaspy3.7-m3-4.6.14-r20200606/lib/python3.7/site-packages/zarr/storage.py", line 791, in __getitem__
    raise KeyError(key)
KeyError: '.zmetadata'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./read_intake_cat.py", line 19, in <module>
    dset_dict = col.to_dataset_dict(zarr_kwargs={"consolidated": True, "use_cftime": True}, storage_options={'anon':True})
  File "/home/users/astephen/cmip6/cmip6-object-store/venv/lib/python3.7/site-packages/intake_esm/core.py", line 927, in to_dataset_dict
    key, ds = task.result()
  File "/apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14/envs/jaspy3.7-m3-4.6.14-r20200606/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14/envs/jaspy3.7-m3-4.6.14-r20200606/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14/envs/jaspy3.7-m3-4.6.14-r20200606/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/users/astephen/cmip6/cmip6-object-store/venv/lib/python3.7/site-packages/intake_esm/core.py", line 913, in _load_source
    return key, source.to_dask()
  File "/home/users/astephen/cmip6/cmip6-object-store/venv/lib/python3.7/site-packages/intake_esm/source.py", line 246, in to_dask
    self._load_metadata()
  File "/apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14/envs/jaspy3.7-m3-4.6.14-r20200606/lib/python3.7/site-packages/intake/source/base.py", line 126, in _load_metadata
    self._schema = self._get_schema()
  File "/home/users/astephen/cmip6/cmip6-object-store/venv/lib/python3.7/site-packages/intake_esm/source.py", line 175, in _get_schema
    self._open_dataset()
  File "/home/users/astephen/cmip6/cmip6-object-store/venv/lib/python3.7/site-packages/intake_esm/source.py", line 227, in _open_dataset
    datasets = dask.compute(*datasets)
  File "/apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14/envs/jaspy3.7-m3-4.6.14-r20200606/lib/python3.7/site-packages/dask/base.py", line 444, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14/envs/jaspy3.7-m3-4.6.14-r20200606/lib/python3.7/site-packages/dask/threaded.py", line 84, in get
    **kwargs
  File "/apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14/envs/jaspy3.7-m3-4.6.14-r20200606/lib/python3.7/site-packages/dask/local.py", line 486, in get_async
    raise_exception(exc, tb)
  File "/apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14/envs/jaspy3.7-m3-4.6.14-r20200606/lib/python3.7/site-packages/dask/local.py", line 316, in reraise
    raise exc
  File "/apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14/envs/jaspy3.7-m3-4.6.14-r20200606/lib/python3.7/site-packages/dask/local.py", line 222, in execute_task
    result = _execute_task(task, data)
  File "/apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14/envs/jaspy3.7-m3-4.6.14-r20200606/lib/python3.7/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/users/astephen/cmip6/cmip6-object-store/venv/lib/python3.7/site-packages/intake_esm/source.py", line 211, in read_dataset
    requested_variables=self.requested_variables,
  File "/home/users/astephen/cmip6/cmip6-object-store/venv/lib/python3.7/site-packages/intake_esm/merge_util.py", line 291, in _open_asset
    raise IOError(message) from exc
OSError: 
            Failed to open zarr store.

            *** Arguments passed to xarray.open_zarr() ***:

            - store: http://cmip6-zarr-o.s3.jc.rl.ac.uk/CMIP6.DCPP.IPSL.IPSL-CM6A-LR/dcppC-ipv-NexTrop-pos.r1i1p1f1.Amon.tauu.gr.v20190110.zarr
            - kwargs: {'consolidated': True, 'use_cftime': True}

            *** fsspec options used ***:

            - root: http://cmip6-zarr-o.s3.jc.rl.ac.uk/CMIP6.DCPP.IPSL.IPSL-CM6A-LR/dcppC-ipv-NexTrop-pos.r1i1p1f1.Amon.tauu.gr.v20190110.zarr
            - protocol: http

            ********************************************

Do you have any tips on what I might need to do to get this working? Thanks

andersy005 · 2020-11-19T12:05:52Z

andersy005
Nov 19, 2020
Maintainer

@agstephens,

The KeyError: '.zmetadata' is suggesting that the http://cmip6-zarr-o.s3.jc.rl.ac.uk/CMIP6.DCPP.IPSL.IPSL-CM6A-LR/dcppC-ipv-NexTrop-pos.r1i1p1f1.Amon.tauu.gr.v20190110.zarr store may have been written without consolidating the metadata

  File "/apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14/envs/jaspy3.7-m3-4.6.14-r20200606/lib/python3.7/site-packages/zarr/storage.py", line 2501, in __init__
    meta = json_loads(store[metadata_key])
  File "/apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14/envs/jaspy3.7-m3-4.6.14-r20200606/lib/python3.7/site-packages/zarr/storage.py", line 791, in __getitem__
    raise KeyError(key)
KeyError: '.zmetadata'

Can you confirm that the following works for this particular zarr store:

import xarray as xr
import fsspec

zarr_path = 'http://cmip6-zarr-o.s3.jc.rl.ac.uk/CMIP6.DCPP.IPSL.IPSL-CM6A-LR/dcppC-ipv-NexTrop-pos.r1i1p1f1.Amon.tauu.gr.v20190110.zarr'

fsmap = fsspec.get_mapper(zarr_path)
ds = xr.open_zarr(fsmap, consolidated=True, use_cftime=True)

Also, does this work ??

col.to_dataset_dict(zarr_kwargs={"consolidated": False, "use_cftime": True})

0 replies

agstephens · 2020-11-19T13:32:03Z

agstephens
Nov 19, 2020
Author

@andersy005 Thanks for your response.

Here are some responses:

The consolidated metadata (.zmetadata) file does exist:

$ curl http://cmip6-zarr-o.s3.jc.rl.ac.uk/CMIP6.DCPP.IPSL.IPSL-CM6A-LR/dcppC-ipv-NexTrop-pos.r1i1p1f1.Amon.tauu.gr.v20190110.zarr/.zmetadata
{
    "metadata": {
        ".zattrs": {
            "CMIP6_CV_version": "cv=6.2.3.5-2-g63b123e",
            "Conventions": "CF-1.7 CMIP-6.2",
            "EXPID": "dcppC-ipv-NexTrop-pos",
...

I confirm that your code works just fine:

import xarray as xr
import fsspec

zarr_path = 'http://cmip6-zarr-o.s3.jc.rl.ac.uk/CMIP6.DCPP.IPSL.IPSL-CM6A-LR/dcppC-ipv-NexTrop-pos.r1i1p1f1.Amon.tauu.gr.v20190110.zarr'

fsmap = fsspec.get_mapper(zarr_path)
ds = xr.open_zarr(fsmap, consolidated=True, use_cftime=True)
print(ds)

yields:

<xarray.Dataset>
Dimensions:      (axis_nbounds: 2, lat: 143, lon: 144, time: 120)
Coordinates:
  * lat          (lat) float32 -90.0 -88.73239 -87.46479 ... 88.73239 90.0
  * lon          (lon) float32 0.0 2.5 5.0 7.5 10.0 ... 350.0 352.5 355.0 357.5
...

Running col.to_dataset_dict(...) fails, with exception I listed above:

>>> dset_dict = col.to_dataset_dict(zarr_kwargs={"consolidated": False, "use_cftime": True})
Traceback (most recent call last):
  File "/home/users/astephen/cmip6/cmip6-object-store/venv/lib/python3.7/site-packages/intake_esm/merge_util.py", line 273, in _open_asset
    ds = xr.open_zarr(path, **zarr_kwargs)
  File "/home/users/astephen/cmip6/cmip6-object-store/venv/lib/python3.7/site-packages/xarray/backends/zarr.py", line 667, in open_zarr
    chunk_store=chunk_store,
...

It returns that it cannot find .zmetadata but I suspect it is something deeper, i.e. that it is not accessing the zarr file properly.

One of my colleagues mentioned that he is using some settings for accessing our Zarr store. Might any of these be required in the storage_options?

        "backend": "s3aioFileObject",
        "api": "S3v4"

0 replies

agstephens · 2020-11-19T13:34:32Z

agstephens
Nov 19, 2020
Author

@andersy005 : following my last point, I get the same result if I try:

dset_dict = col.to_dataset_dict(zarr_kwargs={"consolidated": False, "use_cftime": True},
    storage_options={"backend": "s3aioFileObject", "api": "S3v4"})

0 replies

andersy005 · 2020-11-24T19:49:03Z

andersy005
Nov 24, 2020
Maintainer

One of my colleagues mentioned that he is using some settings for accessing our Zarr store. Might any of these be required in the storage_options?
"backend": "s3aioFileObject",
"api": "S3v4"

@agstephens, are these settings being used when you run:

fsmap = fsspec.get_mapper(zarr_path)
ds = xr.open_zarr(fsmap, consolidated=True, use_cftime=True)

or are xarray/fsspec/zarr able to read your zarr store without any additional settings?

0 replies

agstephens · 2020-11-26T08:27:51Z

agstephens
Nov 26, 2020
Author

Hi @andersy005. No, those settings are not used when I run xr.open_zarr directly. It is quite simply:

fsmap = fsspec.get_mapper(zarr_path)
ds = xr.open_zarr(fsmap, consolidated=True, use_cftime=True)

No extra info is needed.

0 replies

andersy005 · 2020-11-27T14:09:14Z

andersy005
Nov 27, 2020
Maintainer

@agstephens,

Thank you for the clarification.... I tried creating a minimal example to see if there's any bug in intake-esm, but I'm having trouble reproducing the issue:

In [1]: import intake

In [2]: col = intake.open_esm_datastore("test.json")

In [3]: col.df
Out[3]: 
                              path variable                                   long_name
0  http://localhost:8000/test.zarr      air  4xDaily Air temperature at sigma level 995
1  http://localhost:8000/test.zarr     Tair                                        test

In [4]: col.to_dataset_dict()

--> The keys in the returned dictionary of datasets are constructed as follows:
        'path.variable.long_name'
Out[4]: ██████████████████████████████████████████████████████████████████████████████████████████████| 100.00% [2/2 00:00<00:00]
{'http://localhost:8000/test.zarr.air.4xDaily Air temperature at sigma level 995': <xarray.Dataset>
 Dimensions:  (lat: 25, lon: 53, time: 2920)
 Coordinates:
   * lat      (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0
   * lon      (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0
   * time     (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00
 Data variables:
     air      (time, lat, lon) float32 dask.array<chunksize=(730, 13, 27), meta=np.ndarray>
 Attributes:
     Conventions:             COARDS
     description:             Data is from NMC initialized reanalysis\n(4x/day...
     platform:                Model
     references:              http://www.esrl.noaa.gov/psd/data/gridded/data.n...
     title:                   4x daily NMC reanalysis (1948)
     intake_esm_varname:      None
     intake_esm_dataset_key:  http://localhost:8000/test.zarr.air.4xDaily Air ...,
 'http://localhost:8000/test.zarr.Tair.test': <xarray.Dataset>
 Dimensions:  (lat: 25, lon: 53, time: 2920)
 Coordinates:
   * lat      (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0
   * lon      (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0
   * time     (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00
 Data variables:
     air      (time, lat, lon) float32 dask.array<chunksize=(730, 13, 27), meta=np.ndarray>
 Attributes:
     Conventions:             COARDS
     description:             Data is from NMC initialized reanalysis\n(4x/day...
     platform:                Model
     references:              http://www.esrl.noaa.gov/psd/data/gridded/data.n...
     title:                   4x daily NMC reanalysis (1948)
     intake_esm_varname:      None
     intake_esm_dataset_key:  http://localhost:8000/test.zarr.Tair.test}

As you can see everything is working... What's the output of

import intake_esm
intake_esm.show_versions()

I'm running out of ideas about how to diagnose the issue :)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

to_dataset_dict() - raises exception when trying to load Zarr files from object store #426

{{title}}

Replies: 6 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

to_dataset_dict() - raises exception when trying to load Zarr files from object store #426

agstephens Nov 19, 2020

Replies: 6 comments

andersy005 Nov 19, 2020 Maintainer

agstephens Nov 19, 2020 Author

agstephens Nov 19, 2020 Author

andersy005 Nov 24, 2020 Maintainer

agstephens Nov 26, 2020 Author

andersy005 Nov 27, 2020 Maintainer

agstephens
Nov 19, 2020

andersy005
Nov 19, 2020
Maintainer

agstephens
Nov 19, 2020
Author

agstephens
Nov 19, 2020
Author

andersy005
Nov 24, 2020
Maintainer

agstephens
Nov 26, 2020
Author

andersy005
Nov 27, 2020
Maintainer