Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bringing our environment to modern standards (also with removal of NCL) #3812

Open
valeriupredoi opened this issue Nov 14, 2024 · 7 comments · May be fixed by #3853
Open

Bringing our environment to modern standards (also with removal of NCL) #3812

valeriupredoi opened this issue Nov 14, 2024 · 7 comments · May be fixed by #3853
Assignees
@bouweandela @schlunma
Labels
installation

Comments

@valeriupredoi
Copy link
Contributor

This is an umbrella issue that is motivated by our dependencies environment being fairly patchy and out of date; there are some outstaning issues at play:

  • the need for NCL to be moved to an "optional dependency": ncl is lagging behind with every Python and co upgrade, specifically for us, it makes it impossible to:
    • move to support Python 3.13 via Support Python 3.13 #3805
    • move to support recent versions of esmpy>=8.6
    • start supporting numpy>=2
    • altogether get rid of importlib_metadata <8 (predicated by older esmpy, predicated, in turn, by NCL)
  • unpin curl, hopefully with a correct curl=8.11, caused by the issues it has with Julia installing packages, see Unpin curl #3758
  • unpin cartopy, see Unpin cartopy #3769 -> this is done in Update environment: pin iris>=3.11, unpin cartopy and allow for numpy >=2 #3811 - it is safe to do that since we pin iris there
  • unpin pandas - that will be done when we release ESMValCore v2.12 since the fix for pandas is there now
  • unpin numpy<2 ie upper bound - iris 3.11 is safe for numpy>=2 - we should free it too, and consider issues with diagnostics not being numpy2 compatible as they come, probably during the release process for v2.12; unpinning upper bound for `numpy done in Update environment: pin iris>=3.11, unpin cartopy and allow for numpy >=2 #3811
  • unpin r-base <4.3 - I can't remember why we are doing that, I need to look into it
@valeriupredoi valeriupredoi mentioned this issue Nov 14, 2024
Merged
4 tasks
@valeriupredoi
Copy link
Contributor Author

valeriupredoi commented Jan 7, 2025 •
edited
Loading

@ESMValGroup/technical-lead-development-team this is a big red flag and something looming above us:

  • currently we are pinning curl so that we avoid an issue with Julia installing packages; this has now been resolved at Julia end, but we are awaiting a new Julia release to have the fix in...
  • however, that pinned curl is keeping us from a fairly catastrophic environment solving (or not solving), see the tests where curl pin is removed (curl removed altogether) - solving for Python 3.11 takes about 10 minutes, and for Python 3.12 it actually fails after about 15 minutes - this is unacceptable! NB: I am still to investigate the reason why the env solving fails for 3.12, but...
  • look at the tests where I remove NCL and its dependencies - all good and in sub 3 minutes times
  • we really-really need to remove NCL from our main dependencies

🍺

@valeriupredoi
Copy link
Contributor Author

valeriupredoi commented Jan 7, 2025 •
edited
Loading

the Python 3.12 solver issue from above appears to be the solver really getting stuck in a corner case, see eg mamba-org/mamba#3290 I tried to reproduce the issue locally, but my ye olde machine promptly ran out of memory ie the solver was using more than 4GB of my free memory, again, not acceptable

@valeriupredoi valeriupredoi changed the title Bringing our environment to modern standards Bringing our environment to modern standards (also with removal of NCL) Jan 7, 2025
@valeriupredoi valeriupredoi linked a pull request Jan 7, 2025 that will close this issue
Draft
6 tasks
@bouweandela
Copy link
Member

bouweandela commented Jan 8, 2025 •
edited
Loading

We could consider switching from mamba to pixi. That would have the advantage that we can define "extra"s as available for PyPI packages, but then for conda-forge packages, and multiple environments, such as the Linux (environment.yml) and OSX (environment_osx.yml) one, all in the pyproject.toml file. I had a go at that for ESMValCore a few months ago (see here) and then I ran into prefix-dev/pixi#2458, but that seems to be kind of solved now.

@valeriupredoi
Copy link
Contributor Author

thanks bud! I'll look into that, though I am bit weary of going for something that (I bet) not many people have heard of - let's not try that at least for the upcoming release, there are already quite a few major bits that we bring in ie Python 3.12 (hopefully 3.13 too) etc, but will defo look at it after the next release 🍺

@valeriupredoi
Copy link
Contributor Author

love the title of that issue BTW 🤣

@rbeucher
Copy link
Contributor

rbeucher commented Jan 8, 2025

I'll take a look at Pixi, but it seems to be primarily aimed at developers.

@bouweandela
Copy link
Member

This discussion seems to be mostly about our environment.yml files, which are aimed at developers. People who only use the tool can already install it without NCL using e.g. conda install esmvaltool-python esmvaltool-r

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bouweandela bouweandela

@schlunma schlunma

Labels
installation
Projects
None yet
Milestone
No milestone
4 participants
@rbeucher @bouweandela @valeriupredoi @schlunma