Customizing Your DESI Python Environment

[AlexGKim]

I am following the instructions at https://desi.lbl.gov/trac/wiki/Pipeline/GettingStarted/NERSC#CustomizingYourDESIPythonEnvironment. I am running into an error here with regards to PackagesNotFound.

(base) akim@cori04:~> conda create -p ${HOME}/mydesi --copy --file ${DESICONDA}/pkg_list.txt
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

• openssl==1.0.2r=h14c3975_0
• sortedcontainers==2.1.0=py_0
• locket==0.2.0=py_2
• lz4-c==1.8.3=he1b5a44_1001
• msgpack-python==0.6.1=py36h6bb024c_0
• heapdict==1.0.0=py36_1000
• ca-certificates==2019.6.16=hecc5488_0

Current channels:

• https://repo.anaconda.com/pkgs/main/linux-64
• https://repo.anaconda.com/pkgs/main/noarch
• https://repo.anaconda.com/pkgs/r/linux-64
• https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

[tskisner]

Right. The desiconda version has not been updated in so long that some of the package versions used are no longer available from the upstream repositories. The desiconda install should be updated, or you can copy the package list and remove those lines containing packages that were not found. These are all low-level packages and the correct versions should be pulled in by the higher level packages in the list.

[AlexGKim]

Indeed I am able to proceed after removing the above packages from the list. Thanks Ted.

[sbailey]

Our desiconda installation is less than a year old (August 2019), which isn't a good sign about the shelf life of our software releases and instructions for how users can spawn their own custom environments from them.

When installing desiconda, presumably the now-missing packages were downloaded. Is there any option to cache them so that they remain on disk in a place that new user environments could pick them up even if they aren't available from anaconda/condaforge anymore?

I'd like to make a new desiconda installation "soon" to update fitsio and astropy and remove jupyter lab stuff, so if there is some way we can do that to enable the next release to last longer than 6 months that would be helpful.

conda create ... --file ${DESICONDA}/pkg_list.txt requires those exact package versions to be available. Is there some intermediate version that requires those packages, but is more agnostic about the versions? Otherwise we could update the instructions to specify the core external packages that we need (numpy scipy astropy numba etc.) and let conda pickup the rest, with the warning that they may not be exactly the versions in the original base environment.

Containers is another solution for long term reproducibility, but that brings in other overhead for users. I'm inclined to use those for big data releases, but not for daily use of software releases that are less than a year old.

[tskisner]

We could fairly easily install a different package list that included only our "core" high-level dependencies with versions (numpy, astropy, etc). And then just let conda figure out the low-level dependencies. That should probably be fine for creating new environments up to ~1 year later.

I think that the only way to support running very old stacks is to have everything in a container.

[weaverba137]

A very similar problem was recently reported on the DESI Trac ticket system. Did this ever get resolved? I seem to recall discussion of this more recently than March, perhaps on the desi-data mailing list.

[weaverba137]

PS, the ticket on Trac involved a very old version of desiconda, but that still leaves open the question of whether there was a formal resolution to this issue.