Merge spack develop as of 2023/07/10 into jcsda_emc_spack_stack #295

climbfuji · 2023-07-10T14:08:33Z

Description

The title says it all. This PR needs to be tested with JCSDA/spack-stack#672.

There are still a few places in the merged code that have tags # DH* 20230710 for double checking, and also there is a list of files for which I had to resolve conflicts: JCSDA/spack-stack#672 (comment)

Lastly, spack from this version on somehow uses the @= syntax in certain places, for example in the output of spack compiler find on macOS. I had to update the meta modules Python script accordingly (8635085), and then black forced me to reformat it almost entirely (98957cf).

Notes from Alex:

I adopted the fix Fix issue preventing setup-env.sh to be sourced spack/spack#38895 but I wasn't able to push after cherry-picking the commit, so I cheated and modified the file through the web interface as a separate commit.

Issue(s) addressed

n/a

Dependencies

n/a

Impact

See JCSDA/spack-stack#672

Checklist

I have performed a self-review of my own code
I have made corresponding changes to the documentation: update inline comments where applicable, and added a note in the spack-stack PR on where I had to resolve conflicts / what I need people to look at (Merge spack develop as of 2023/07/10 into jcsda_emc_spack_stack spack-stack#672 (comment))
I have run the unit tests before creating the PR

* py-networkx: add 3.1 * Update var/spack/repos/builtin/packages/py-networkx/package.py Co-authored-by: Adam J. Stewart <[email protected]> * Add default variant --------- Co-authored-by: Adam J. Stewart <[email protected]>

…` bug (spack#38297) * hpx: Add conflict with some GCC versions and +rocm due to valarray bug https://gcc.gnu.org/bugzilla/show_bug.cgi?id=103022 * remove conflict for %[email protected] since there is none --------- Co-authored-by: eugeneswalker <[email protected]>

No changes to the build system, no changes to `package.py` needed. Changelog: qt/qtbase@v5.15.9-lts-lgpl...v5.15.10-lts-lgpl Main change taking up space: - bundled 3rdparty/pcre2 updated from 10.39 to 10.40 (spack now includes 10.42, and we don't put specific version requirements in `package.py`)

* py-platformdirs: add 3.5.3 * py-typing-extensions: add 4.6.3

…ck#38038) * Add libxml2 dependency and increase version to v1.20 * Update var/spack/repos/builtin/packages/openturns/package.py Co-authored-by: Alec Scott <[email protected]> * Add v19 * Add libxml2 variant * Fix variant specification --------- Co-authored-by: Marc Schouler <[email protected]> Co-authored-by: Alec Scott <[email protected]>

It installs the LFortran runtime library and LFortran can compile codes to binaries. The interactive mode does not work yet with LLVM > 11, that has to be fixed upstream. Co-authored-by: Wileam Y. Phan <[email protected]>

@spackbot

* py-notebook: add 6.5.4 * [@spackbot] updating style on behalf of manuelakuhn * Update var/spack/repos/builtin/packages/py-notebook/package.py Co-authored-by: Adam J. Stewart <[email protected]> * Fix version of py-nbclassic dependency --------- Co-authored-by: Adam J. Stewart <[email protected]>

Also remove the custom `autoreconf` method as it does not seem to be necessary and potentially hides useful output provided by the default.

* zen4 support for vasp * encourage openmp support in fftw/blas when +openmp * enable gamma and non-collinear builds

@AndrewGaspar

* Add rust v1.70.0 and simplify package logic by moving bootstrap to dedicated package * Fix formatting of rust-bootstrap package file * Re-enable Rust as extendable * Add nightly version to rust and rust-bootstrap * Manually inject openssl certs into environment * Add master and beta versions to rust * Add additional documentation for using rust development releases * Remove @AndrewGaspar as maintainer

* py-bcbio-gff: adding new version 0.7.0 * emblmygff3: adding new package * emblmygff3: adding py-setuptools dep

Co-authored-by: jmcarcell <[email protected]>

…ually exclusive (spack#38533)

)

* Add recipe for iterative-stats * Fix branch name and remove comment * Add git link * Add package maintainer * Enforce multiple requested changes * Update var/spack/repos/builtin/packages/py-iterative-stats/package.py Co-authored-by: Adam J. Stewart <[email protected]> * Update checksum * Fix openturns dependency specification * Add python variant spec to openturns --------- Co-authored-by: Marc Schouler <[email protected]> Co-authored-by: Adam J. Stewart <[email protected]>

@spackbot

* py-pytest: add 7.3.2 * [@spackbot] updating style on behalf of manuelakuhn * Swap py-importlib-metadata dependency order * Restrict python version for older versions

* libEnsemble: add v0.10.0 * Make new deps required * Fixes to deps * Update var/spack/repos/builtin/packages/py-libensemble/package.py Co-authored-by: Adam J. Stewart <[email protected]> * Fix build, run * Reorder required deps --------- Co-authored-by: Adam J. Stewart <[email protected]>

Co-authored-by: greenc-FNAL <[email protected]>

…38403) Remove `--enable-mca-no-build=btl-uct` check for ucx spec as it is already fixed in the openmpi open-mpi/ompi#6666

.github/workflows/valid-style.yml

var/spack/repos/builtin/packages/bufr/package.py

srherbener

I tested on my mac arm64 laptop. spack-stack built successfully and jedi-bundle built successfully (but I had to uninstall git from homebrew and use /usr/bin/git instead due to _iconv symbol not found issue).

I tried running ctest and ran into the _iconv symbol not found issue as well.

climbfuji · 2023-07-21T19:32:16Z

I tested on my mac arm64 laptop. spack-stack built successfully and jedi-bundle built successfully (but I had to uninstall git from homebrew and use /usr/bin/git instead due to _iconv symbol not found issue).

I tried running ctest and ran into the _iconv symbol not found issue as well.

Yes, very annoying. The ctest problem comes from cmake being installed via homebrew. I am testing a minimal homebrew set up right now.

climbfuji · 2023-07-21T23:21:47Z

I am finding bugs over bugs in this particular version of spack, very unhappy with it. I think we need to rethink our strategy and only update from spack when they create official releases, plus cherry-pick individual updates in between (as long as we do merge commits in the spack fork for updates from upstream) if needed.

AlexanderRichert-NOAA · 2023-07-21T23:36:13Z

Darn. What kinds of bugs?

Have you tested with v0.20.1? That one's pretty recent so it should have most or all of the package changes we might want (bufr, netcdf-c, ...).

climbfuji · 2023-07-21T23:41:56Z

spack#39046
spack#39049

--> these are major issues breaking all macOS builds

We have one error from the merge in cmake. This is from the attempt to make everything static (which I am opposed to):

        if self.spec["openssl"].satisfies("~shared"):
            args.append(
                "-DOPENSSL_CRYPTO_LIBRARY={0};{1}".format(
                    find_libraries("libcrypto", self.spec["openssl"].prefix.lib, shared=False),
                    find_libraries(
                        "libz",
                        self.spec["zlib"].prefix.lib,
                        shared=self.spec["zlib"].satisfies("+shared"),
                    ),
                )
            )

I need to remove this, because openssl is no longer a dependency of cmake (see rest of the package). Done in 2796f78

AlexanderRichert-NOAA · 2023-07-21T23:48:20Z

I'm pretty sure the openssl dependency comes in through curl. It should be fixable by changing the if condition to if self.spec.satisfies("^openssl~shared"):

climbfuji · 2023-07-21T23:49:39Z

I'm pretty sure the openssl dependency comes in through curl. It should be fixable by changing the if condition to if self.spec.satisfies("^openssl~shared"):

Note that the entire block that I just removed is not in spack develop. Is it really needed?

AlexanderRichert-NOAA · 2023-07-21T23:51:59Z

It's a change I made as part of the batch of NCO-related changes, and intend to push to main spack once my PR for the openssl package itself is merged.

climbfuji · 2023-07-21T23:52:45Z

Ok, I’ll put it back in with your suggestion.

…

On Jul 21, 2023, at 5:52 PM, Alex Richert ***@***.***> wrote: It's a change I made as part of the batch of NCO-related changes, and intend to push to main spack once my PR for the openssl package itself is merged. — Reply to this email directly, view it on GitHub <#295 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5C2RPXZSBTSH4BDWASDI3XRMI2TANCNFSM6AAAAAA2ETHNHY>. You are receiving this because you were assigned.

…ge.py, openssl is no longer a dependency for make

climbfuji · 2023-07-22T01:45:41Z

Ok, I’ll put it back in with your suggestion.
…
On Jul 21, 2023, at 5:52 PM, Alex Richert @.***> wrote: It's a change I made as part of the batch of NCO-related changes, and intend to push to main spack once my PR for the openssl package itself is merged. — Reply to this email directly, view it on GitHub <#295 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5C2RPXZSBTSH4BDWASDI3XRMI2TANCNFSM6AAAAAA2ETHNHY. You are receiving this because you were assigned.

Done. Here is another spack bug:
spack#39055

…_20230710

srherbener

Mac and JEDI testing look good

manuelakuhn and others added 30 commits June 22, 2023 03:09

py-networkx: add 3.1 (spack#38377)

103370d

* py-networkx: add 3.1 * Update var/spack/repos/builtin/packages/py-networkx/package.py Co-authored-by: Adam J. Stewart <[email protected]> * Add default variant --------- Co-authored-by: Adam J. Stewart <[email protected]>

py-pyproj: add v3.6.0 (spack#38399)

f35fcee

py-patsy: add 0.5.3 (spack#38433)

3e9e01e

py-prompt-toolkit: add 3.0.38 (spack#38472)

082934f

py-platformdirs: add 3.5.3 (spack#38435)

f935f36

* py-platformdirs: add 3.5.3 * py-typing-extensions: add 4.6.3

py-numexpr: add 2.8.4 (spack#38416)

fde33e6

Add LFortran (spack#30901)

a889669

It installs the LFortran runtime library and LFortran can compile codes to binaries. The interactive mode does not work yet with LLVM > 11, that has to be fixed upstream. Co-authored-by: Wileam Y. Phan <[email protected]>

freefem: add missing dependencies (spack#38290)

590d3ba

Also remove the custom `autoreconf` method as it does not seem to be necessary and potentially hides useful output provided by the default.

VASP package: Zen4 Support (spack#36800)

77817a0

* zen4 support for vasp * encourage openmp support in fftw/blas when +openmp * enable gamma and non-collinear builds

py-packaging: add 23.1 (spack#38417)

ebd4113

Emblmygff3 (spack#38515)

c96f93b

* py-bcbio-gff: adding new version 0.7.0 * emblmygff3: adding new package * emblmygff3: adding py-setuptools dep

Add 1606 (spack#38511)

ff6ac42

py-python-gitlab: add 3.15.0 (spack#38524)

88fc8ae

py-qmtest: add patch for the removed bdist_wininst class (spack#38253)

3d342ac

Co-authored-by: jmcarcell <[email protected]>

py-pytz: add 2023.3 (spack#38525)

a076548

spack buildcache sync (--manifest-glob <glob> | <src> <dest>) are mut…

cd67b2a

…ually exclusive (spack#38533)

Add numa variant in ucx package (spack#38423)

b49d098

zlib-ng: add latest, default to drop in replacement of zlib (spack#37370

e76b039

)

fix version dependency logic (spack#38535)

42ed4d8

Added version 0.33 (spack#38534)

f73c8f2

py-pytest: add 7.3.2 (spack#38522)

0470fe5

* py-pytest: add 7.3.2 * [@spackbot] updating style on behalf of manuelakuhn * Swap py-importlib-metadata dependency order * Restrict python version for older versions

range-v3: remove overbroad boost dependence (spack#38539)

09d9b48

Co-authored-by: greenc-FNAL <[email protected]>

Fixes openmpi configure args for multinode execution on slurm (spack#…

d99892e

…38403) Remove `--enable-mca-no-build=btl-uct` check for ucx spec as it is already fixed in the openmpi open-mpi/ompi#6666

climbfuji requested review from AlexanderRichert-NOAA, ulmononian, srherbener, Hang-Lei-NOAA and natalie-perlin and removed request for AlexanderRichert-NOAA July 20, 2023 03:29

AlexanderRichert-NOAA reviewed Jul 20, 2023

View reviewed changes

.github/workflows/valid-style.yml Show resolved Hide resolved

AlexanderRichert-NOAA reviewed Jul 20, 2023

View reviewed changes

var/spack/repos/builtin/packages/bufr/package.py Show resolved Hide resolved

srherbener reviewed Jul 21, 2023

View reviewed changes

climbfuji mentioned this pull request Jul 21, 2023

Ignore errors when creating/using source and binary caches JCSDA/spack-stack#684

Merged

3 tasks

Update static logic from var/spack/repos/builtin/packages/cmake/packa…

a3acfa3

…ge.py, openssl is no longer a dependency for make

climbfuji force-pushed the feature/merge_spack_develop_20230710 branch from 2796f78 to a3acfa3 Compare July 22, 2023 00:17

climbfuji added the bug Something is not working label Jul 22, 2023

AlexanderRichert-NOAA added 2 commits July 31, 2023 13:32

Merge branch 'jcsda_emc_spack_stack' into feature/merge_spack_develop…

7d7749c

…_20230710

Update cray_backend.py

9558f26

srherbener approved these changes Aug 7, 2023

View reviewed changes

ulmononian approved these changes Aug 7, 2023

View reviewed changes

AlexanderRichert-NOAA added 2 commits August 7, 2023 14:56

add python path fix to bufr

3f3aa83

import os in bufr

113ba63

AlexanderRichert-NOAA merged commit d005394 into JCSDA:jcsda_emc_spack_stack Aug 8, 2023
12 of 13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge spack develop as of 2023/07/10 into jcsda_emc_spack_stack #295

Merge spack develop as of 2023/07/10 into jcsda_emc_spack_stack #295

climbfuji commented Jul 10, 2023 •

edited by AlexanderRichert-NOAA

Loading

srherbener left a comment

climbfuji commented Jul 21, 2023

climbfuji commented Jul 21, 2023

AlexanderRichert-NOAA commented Jul 21, 2023

climbfuji commented Jul 21, 2023 •

edited

Loading

AlexanderRichert-NOAA commented Jul 21, 2023

climbfuji commented Jul 21, 2023

AlexanderRichert-NOAA commented Jul 21, 2023

climbfuji commented Jul 21, 2023 via email

climbfuji commented Jul 22, 2023

srherbener left a comment

Merge spack develop as of 2023/07/10 into jcsda_emc_spack_stack #295

Merge spack develop as of 2023/07/10 into jcsda_emc_spack_stack #295

Conversation

climbfuji commented Jul 10, 2023 • edited by AlexanderRichert-NOAA Loading

Description

Issue(s) addressed

Dependencies

Impact

Checklist

srherbener left a comment

Choose a reason for hiding this comment

climbfuji commented Jul 21, 2023

climbfuji commented Jul 21, 2023

AlexanderRichert-NOAA commented Jul 21, 2023

climbfuji commented Jul 21, 2023 • edited Loading

AlexanderRichert-NOAA commented Jul 21, 2023

climbfuji commented Jul 21, 2023

AlexanderRichert-NOAA commented Jul 21, 2023

climbfuji commented Jul 21, 2023 via email

climbfuji commented Jul 22, 2023

srherbener left a comment

Choose a reason for hiding this comment

climbfuji commented Jul 10, 2023 •

edited by AlexanderRichert-NOAA

Loading

climbfuji commented Jul 21, 2023 •

edited

Loading