Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regional mom6-om2 recipe fails to load intake catalogue #475

Open
Rush74 opened this issue Dec 23, 2024 · 8 comments
Open

Regional mom6-om2 recipe fails to load intake catalogue #475

Rush74 opened this issue Dec 23, 2024 · 8 comments
Labels
❓ question Further information is requested

Comments

@Rush74
Copy link

Rush74 commented Dec 23, 2024

Hi all.

I'm following the steps outlined in https://github.com/COSIMA/cosima-recipes/blob/main/Recipes/regional-mom6-forced-by-access-om2.ipynb

I'm using a virtual python environment built using NRI conda/analysis3 with regional-mom6 installed on top of that via pip.

The command

catalog = intake.cat.access_nri

generates the following error.

FileNotFoundError                         Traceback (most recent call last)
Cell In[5], line 1
----> 1 catalog = intake.cat.access_nri

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/base.py:427](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/base.py#line=426), in Catalog.__getattr__(self, item)
    424 if not item.startswith("_"):
    425     # Fall back to __getitem__.
    426     try:
--> 427         return self[item]  # triggers reload_on_change
    428     except KeyError as e:
    429         raise AttributeError(item) from e

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/base.py:472](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/base.py#line=471), in Catalog.__getitem__(self, key)
    463 """Return a catalog entry by name.
    464 
    465 Can also use attribute syntax, like ``cat.entry_name``, or
   (...)
    468 cat['name1', 'name2']
    469 """
    470 if not isinstance(key, list) and key in self:
    471     # triggers reload_on_change
--> 472     s = self._get_entry(key)
    473     if s.container == "catalog":
    474         s.name = key

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/utils.py:43](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/utils.py#line=42), in reload_on_change.<locals>.wrapper(self, *args, **kwargs)
     40 @functools.wraps(f)
     41 def wrapper(self, *args, **kwargs):
     42     self.reload()
---> 43     return f(self, *args, **kwargs)

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/base.py:355](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/base.py#line=354), in Catalog._get_entry(self, name)
    353 ups = [up for name, up in self.user_parameters.items() if name not in up_names]
    354 entry._user_parameters = ups + (entry._user_parameters or [])
--> 355 return entry()

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/local.py:917](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/local.py#line=916), in EntrypointEntry.__call__(self, **kwargs)
    915 def __call__(self, **kwargs):
    916     """Instantiate the DataSource for the given parameters"""
--> 917     source = self._entrypoint.load()
    918     if kwargs:
    919         source = source.configure_new(**kwargs)

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/importlib_metadata/__init__.py:184](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/importlib_metadata/__init__.py#line=183), in EntryPoint.load(self)
    179 """Load the entry point from its definition. If only a module
    180 is indicated by the value, return that module. Otherwise,
    181 return the named object.
    182 """
    183 match = cast(Match, self.pattern.match(self.value))
--> 184 module = import_module(match.group('module'))
    185 attrs = filter(None, (match.group('attr') or '').split('.'))
    186 return functools.reduce(getattr, attrs, module)

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/importlib/__init__.py:126](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/importlib/__init__.py#line=125), in import_module(name, package)
    124             break
    125         level += 1
--> 126 return _bootstrap._gcd_import(name[level:], package, level)

File <frozen importlib._bootstrap>:1050, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:1027, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:1006, in _find_and_load_unlocked(name, import_)

File <frozen importlib._bootstrap>:688, in _load_unlocked(spec)

File <frozen importlib._bootstrap_external>:883, in exec_module(self, module)

File <frozen importlib._bootstrap>:241, in _call_with_frames_removed(f, *args, **kwds)

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/access_nri_intake/data/__init__.py:9](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/access_nri_intake/data/__init__.py#line=8)
      6 import intake
      8 _here = os.path.abspath(os.path.dirname(__file__))
----> 9 data = intake.open_catalog(os.path.join(_here, "catalog.yaml")).access_nri

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/base.py:427](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/base.py#line=426), in Catalog.__getattr__(self, item)
    424 if not item.startswith("_"):
    425     # Fall back to __getitem__.
    426     try:
--> 427         return self[item]  # triggers reload_on_change
    428     except KeyError as e:
    429         raise AttributeError(item) from e

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/base.py:472](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/base.py#line=471), in Catalog.__getitem__(self, key)
    463 """Return a catalog entry by name.
    464 
    465 Can also use attribute syntax, like ``cat.entry_name``, or
   (...)
    468 cat['name1', 'name2']
    469 """
    470 if not isinstance(key, list) and key in self:
    471     # triggers reload_on_change
--> 472     s = self._get_entry(key)
    473     if s.container == "catalog":
    474         s.name = key

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/utils.py:43](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/utils.py#line=42), in reload_on_change.<locals>.wrapper(self, *args, **kwargs)
     40 @functools.wraps(f)
     41 def wrapper(self, *args, **kwargs):
     42     self.reload()
---> 43     return f(self, *args, **kwargs)

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/base.py:355](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/base.py#line=354), in Catalog._get_entry(self, name)
    353 ups = [up for name, up in self.user_parameters.items() if name not in up_names]
    354 entry._user_parameters = ups + (entry._user_parameters or [])
--> 355 return entry()

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/entry.py:60](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/entry.py#line=59), in CatalogEntry.__call__(self, persist, **kwargs)
     58 def __call__(self, persist=None, **kwargs):
     59     """Instantiate DataSource with given user arguments"""
---> 60     s = self.get(**kwargs)
     61     s._entry = self
     62     s._passed_kwargs = list(kwargs)

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/local.py:313](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/local.py#line=312), in LocalCatalogEntry.get(self, **user_parameters)
    310     return self._default_source
    312 plugin, open_args = self._create_open_args(user_parameters)
--> 313 data_source = plugin(**open_args)
    314 data_source.catalog_object = self._catalog
    315 data_source.name = self.name

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake_dataframe_catalog/core.py:112](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake_dataframe_catalog/core.py#line=111), in DfFileCatalog.__init__(self, path, yaml_column, name_column, mode, columns_with_iterables, storage_options, read_kwargs, **intake_kwargs)
    109 if self.mode in ["w", "x", "a", "r+"]:
    110     self._allow_write = True
--> 112 super().__init__(storage_options=self.storage_options, **self._intake_kwargs)

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/base.py:128](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/base.py#line=127), in Catalog.__init__(self, entries, name, description, metadata, ttl, getenv, getshell, persist_mode, storage_options, user_parameters)
    126 self.updated = time.time()
    127 self._entries = entries if entries is not None else self._make_entries_container()
--> 128 self.force_reload()

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/base.py:186](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake/catalog/base.py#line=185), in Catalog.force_reload(self)
    184 """Imperative reload data now"""
    185 self.updated = time.time()
--> 186 self._load()

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake_dataframe_catalog/core.py:126](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/intake_dataframe_catalog/core.py#line=125), in DfFileCatalog._load(self)
    123         pass
    124         # self._df.to_csv(fobj)
    125 else:
--> 126     with fsspec.open(self.path, **self.storage_options) as fobj:
    127         self._df = pd.read_csv(fobj, **self._read_kwargs)
    128     if self.yaml_column not in self.df.columns:

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/fsspec/core.py:105](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/fsspec/core.py#line=104), in OpenFile.__enter__(self)
    102 mode = self.mode.replace("t", "").replace("b", "") + "b"
    104 try:
--> 105     f = self.fs.open(self.path, mode=mode)
    106 except FileNotFoundError as e:
    107     if has_magic(self.path):

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/fsspec/spec.py:1303](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/fsspec/spec.py#line=1302), in AbstractFileSystem.open(self, path, mode, block_size, cache_options, compression, **kwargs)
   1301 else:
   1302     ac = kwargs.pop("autocommit", not self._intrans)
-> 1303     f = self._open(
   1304         path,
   1305         mode=mode,
   1306         block_size=block_size,
   1307         autocommit=ac,
   1308         cache_options=cache_options,
   1309         **kwargs,
   1310     )
   1311     if compression is not None:
   1312         from fsspec.compression import compr

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/fsspec/implementations/local.py:191](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/fsspec/implementations/local.py#line=190), in LocalFileSystem._open(self, path, mode, block_size, **kwargs)
    189 if self.auto_mkdir and "w" in mode:
    190     self.makedirs(self._parent(path), exist_ok=True)
--> 191 return LocalFileOpener(path, mode, fs=self, **kwargs)

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/fsspec/implementations/local.py:355](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/fsspec/implementations/local.py#line=354), in LocalFileOpener.__init__(self, path, mode, autocommit, fs, compression, **kwargs)
    353 self.compression = get_compression(path, compression)
    354 self.blocksize = io.DEFAULT_BUFFER_SIZE
--> 355 self._open()

File [/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/fsspec/implementations/local.py:360](https://are.nci.org.au/g/data/hh5/public/apps/miniconda3/envs/analysis3-24.04/lib/python3.10/site-packages/fsspec/implementations/local.py#line=359), in LocalFileOpener._open(self)
    358 if self.f is None or self.f.closed:
    359     if self.autocommit or "w" not in self.mode:
--> 360         self.f = open(self.path, mode=self.mode)
    361         if self.compression:
    362             compress = compr[self.compression]

FileNotFoundError: [Errno 2] No such file or directory: '[/g/data/xp65/public/apps/access-nri-intake-catalog/v0.1.0/metacatalog.csv](https://are.nci.org.au/g/data/xp65/public/apps/access-nri-intake-catalog/v0.1.0/metacatalog.csv)'

Is this error

  1. A simple matter of me not having xp65 membership? (in which case the notebook can be amended to specify this), or
  2. Is my python environment conflicting with the access-nri-intake-catalog?
@navidcy
Copy link
Collaborator

navidcy commented Dec 23, 2024

Hi @Rush74, I think you need membership to xp65.

Note, the notebook you point out was tested with the conda analysis environment on hh5; regional-mom6 is installed in that environment. If after getting membership you still get the same error then I'd try with the hh5 env first to ensure that's also broken? (That's what I would do but I'm only suggesting this because personally I'm not familiar with python environments and conda etc and I always get confused what package is used from where.)

@navidcy
Copy link
Collaborator

navidcy commented Dec 23, 2024

in which case the notebook can be amended to specify this

The projects required to access the data in the recipes are mentioned in the cookbook's README. Do you think we need to include them in each recipe?

@navidcy navidcy added the ❓ question Further information is requested label Dec 23, 2024
@Rush74
Copy link
Author

Rush74 commented Dec 23, 2024

Ok - it works fine with xp65.

Re: list of projects required in the README. I just jumped straight into the recipe, someone shared the link with me directly.

I guess it depends on how idiot proof you want to make individual recipes ;)

@navidcy
Copy link
Collaborator

navidcy commented Dec 23, 2024

OK, I'm closing this (but feel free to reopen if there is still an issue).
I opened another issue to discuss whether we need to add reference about required projects in each recipe.

@navidcy navidcy closed this as completed Dec 23, 2024
@Rush74
Copy link
Author

Rush74 commented Dec 23, 2024

Ok I have to download the mom5 tools and alter the default value of expt.toolpath_dir

The following step fails:

expt.FRE_tools(layout = (10, 10)) ## Here the tuple defines the processor layout

Running GFDL's FRE Tools. The following information is all printed by the FRE tools themselves
OUTPUT FROM MAKE SOLO MOSAIC:

CompletedProcess(args='/home/157/ahg157/repos/mom5/src/tools/make_solo_mosaic/make_solo_mosaic --num_tiles 1 --dir . --mosaic_name ocean_mosaic --tile_file hgrid.nc', returncode=126)
OUTPUT FROM QUICK MOSAIC:

CompletedProcess(args='/home/157/ahg157/repos/mom5/src/tools/make_quick_mosaic/make_quick_mosaic --input_mosaic ocean_mosaic.nc --mosaic_name grid_spec --ocean_topog bathymetry.nc', returncode=126)
OUTPUT FROM CHECK MASK:

 CompletedProcess(args='/home/157/ahg157/repos/mom5/src/tools/check_mask/check_mask --grid_file ocean_mosaic.nc --ocean_topog bathymetry.nc --layout 10,10 --halo 4', returncode=126)

[/bin/sh](https://are.nci.org.au/bin/sh): [/home/157/ahg157/repos/mom5/src/tools/make_solo_mosaic/make_solo_mosaic](https://are.nci.org.au/home/157/ahg157/repos/mom5/src/tools/make_solo_mosaic/make_solo_mosaic): Permission denied
[/bin/sh](https://are.nci.org.au/bin/sh): [/home/157/ahg157/repos/mom5/src/tools/make_quick_mosaic/make_quick_mosaic](https://are.nci.org.au/home/157/ahg157/repos/mom5/src/tools/make_quick_mosaic/make_quick_mosaic): Permission denied
[/bin/sh](https://are.nci.org.au/bin/sh): [/home/157/ahg157/repos/mom5/src/tools/check_mask/check_mask](https://are.nci.org.au/home/157/ahg157/repos/mom5/src/tools/check_mask/check_mask): Permission denied

because

expt.toolpath_dir

PosixPath('/home/157/ahg157/repos/mom5/src/tools')

I will download the mom5 tools from
https://github.com/mom-ocean/MOM5/tree/master
and then change the value of exit.toolpath_dir.

Although it looks like I have to build the tools as well. Are their recommendation modules/environments to build these tools on gadi?

@navidcy
Copy link
Collaborator

navidcy commented Dec 23, 2024

@angus-g seems like @Rush74 is trying to access an executable in your user account? Could you give them read access?

@navidcy navidcy reopened this Dec 23, 2024
@navidcy
Copy link
Collaborator

navidcy commented Dec 23, 2024

Btw @Rush74 the workflow design definitely has room for improvement. Any suggestions/PRs welcome!! 🙏

@marc-white
Copy link
Contributor

Just a heads-up, with CLEX ending, the hh5 conda environments will no longer be updated/supported. The environments have been duplicated across to xp65.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
❓ question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants