intake-esm-s3 example fails

[anantharajvg]

https://github.com/aradhakrishnanGFDL/gfdl-aws-analysis/blob/community/examples/intake-esm-s3-nc-simple-access.ipynb

Fails at Line 19 with the error:

ESMDataSourceError: Failed to load dataset with key='CMIP6.NOAA-GFDL.GFDL-ESM4.historical.Amon'
You can use cat['CMIP6.NOAA-GFDL.GFDL-ESM4.historical.Amon'].df to inspect the assets/files for this key.

[aradhakrishnanGFDL]

Thanks for reporting this. With an older version ( version 2020.12.18 ) of intake-esm, it seems fine. This is related to an issue I raised here intake/intake-esm#606. There is a suggested fix merged with the main branch of intake-esm now.

With the newer release of intake-esm, the advice from intake-esm dev is to use the following:

dset_dict = cat.to_dataset_dict(xarray_open_kwargs={'engine':'h5netcdf',
                                                      'chunks':{}},
                                                      storage_options={'anon':True},
) 

instead of dset_dict = cat.to_dataset_dict(cdf_kwargs={'chunks': {'time': 1}},storage_options={'anon':True})

[mgrover1]

@aradhakrishnanGFDL and @anantharajvg - We cut a new release of intake-esm today (v2023.6.14). Please try updating via conda-forge or pip, which should resolve this issue.