From b9ee40ec194653940bdbca7220fbb4828006836c Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 7 Mar 2022 17:29:52 +0100 Subject: [PATCH 001/159] Initial commit, already contains the structure to work on the first tutorial. --- docs/2022-CSC_and_LO/index.md | 42 +++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 docs/2022-CSC_and_LO/index.md diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md new file mode 100644 index 00000000..9af354b2 --- /dev/null +++ b/docs/2022-CSC_and_LO/index.md @@ -0,0 +1,42 @@ +# EasyBuild tutorial for LUST + +Overview page of the introductory tutorial on [EasyBuild](https://easybuild.io) for the CSC +and Local Organizations who want to contribute to the LUMI EasyBuild setup. + +This tutorial is based extensively on the more generic EasyBuild tutorials build by +the EasyBuild community and maintained on +[the EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). +See that site for [credits to the contributors of those tutorials](https://easybuilders.github.io/easybuild-tutorial/#contributors). + +## Contents + +TODO + + + + + + +## OLD TEXT + +- [Part I: **Introduction to EasyBuild**](part1_intro.md) *(Tue March 9th 2021, 9am-12 CET)* + * [What is EasyBuild?](what_is_easybuild.md) + * [Terminology](terminology.md) + * [Installation](installation.md) *(hands-on)* + * [Configuration](configuration.md) *(hands-on)* + * [Basic usage](basic_usage.md) *(hands-on)* +- [Part II: **Using EasyBuild**](part2_using.md) *(Tue March 23rd 2021, 9am-12 CET)* + * [Troubleshooting](troubleshooting.md) *(hands-on)* + * [Creating easyconfig files](creating_easyconfig_files.md) *(hands-on)* + * [Implementing easyblocks](implementing_easyblocks.md) *(hands-on)* +- [Part III: **Advanced topics**](part3_advanced.md) *(Tue March 30th 2021, 9am-12 CEST)* + * [Using EasyBuild as a library](easybuild_library.md) *(hands-on)* + * [Using hooks to customise EasyBuild](hooks.md) *(hands-on)* + * [Submitting installations as Slurm jobs](slurm_jobs.md) *(hands-on)* + * [Module naming schemes (incl. hierarchical)](module_naming_schemes.md) *(hands-on)* + * [GitHub integration to facilitate contributing to EasyBuild](github_integration.md) *(hands-on)* +- [Part IV: **EasyBuild on Cray systems**](part4_cray.md) *(Friday June 18th 2021, 09-12 CEST)* + * [Introduction to Cray Programming Environment](cray/introduction.md) *(hands-on)* + * [Cray External Modules](cray/external_modules.md) *(hands-on)* + * [Cray Custom Toolchains](cray/custom_toolchains.md) *(hands-on)* + * [EasyBuild at CSCS](cray/easybuild_at_cscs.md) *(hands-on)* From 183ef3bf22e7f91802714dd2b25457f41acb6537 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 7 Mar 2022 18:30:05 +0100 Subject: [PATCH 002/159] Updated the "What is EasyBuild" page of the tutorial for LUMI." --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 12 + .../2022-CSC_and_LO/1_01_what_is_easybuild.md | 321 ++++++++++++++++++ docs/2022-CSC_and_LO/index.md | 10 + 3 files changed, 343 insertions(+) create mode 100644 docs/2022-CSC_and_LO/1_00_part1_intro.md create mode 100644 docs/2022-CSC_and_LO/1_01_what_is_easybuild.md diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md new file mode 100644 index 00000000..0bfd0ad5 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -0,0 +1,12 @@ +# Part I: Introduction to EasyBuild on Cray systems + +*[[back to start page]](index.md)* + +*(Tue March 4th 2021, 9am-12 CET)* + +- [What is EasyBuild?](1_01_what_is_easybuild.md) + +- [Terminology](terminology.md) +- [Installation](installation.md) *(hands-on)* +- [Configuration](configuration.md) *(hands-on)* +- [Basic usage](basic_usage.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md new file mode 100644 index 00000000..678c8c4a --- /dev/null +++ b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md @@ -0,0 +1,321 @@ +# What is EasyBuild? + +*[[back: Overview]](index.md)* + +

EasyBuild logo

+ +**[EasyBuild](http://easybuild.io)** is a **software build and installation framework** +that manages (scientific) software in an efficient way, without compromising on performance. +Easybuild is especially tailored towards [**High Performance Computing (HPC)**](https://en.wikipedia.org/wiki/Supercomputer) systems, +and provides a **command-line interface** to facilitate the installation and management of a scientific software stack. + +--- + + + + + +### *Elevator pitch* + +EasyBuild intends to relieve HPC support teams from manually managing +software installations while at the same time **providing a consistent and well performing +scientific software stack** to end users of HPC systems. + +It serves as a **uniform interface for installing scientific software** +and saves valuable time (and frustration) by the automation of tedious, boring and repetitive tasks. + +In addition, EasyBuild can **empower scientific researchers to self-manage their software stack**, +and it can serve as a tool that can be leveraged for **building optimized container images**. + +The project has grown to become a **platform for collaboration** among HPC sites worldwide. + +--- + +On LUMI we pay a lot of attention to empowering users to create their custom software stack which +is best suited for their project without being confronted with hundreds of modules that are not +useful to them. It also enables them to easily customize the build instructions to create a +project-specific build or even to install software with custom patches to adapt to their needs. +For this we offer both [EasyBuild](https://easybuild.io/) and [Spack](https://spack.io/) +as tools but obviously this tutorial concentrates on EasyBuild alone. + +EasyBuild can also be used to [generate container recipes](https://docs.easybuild.io/en/latest/Containers.html) +that will then use EasyBuild when building +the container to install platform-optimised software in the container. However, as that functionality +is still under development on LUMI it is not yet covered in this LUMI-specific version of the tutorial. + + +--- + +### *Key features* + +EasyBuild is capable of **fully autonomously installing (scientific) software**, +including making sure that all necessary dependencies are installed, +and automatically generating environment module files. + +***No*** **admin privileges are required**: it is sufficient to have write permissions +to the preferred software installation prefix. + +EasyBuild is [**highly configurable**](https://docs.easybuild.io/en/latest/Configuration.html) via configuration files, +environment variables, and command line options. Various elements of the functionality can be +[**dynamically extended**](https://docs.easybuild.io/en/latest/Including_additional_Python_modules.html) via plugins, +and [**hooks**](https://docs.easybuild.io/en/latest/Hooks.html) are available for further site-specific customizations. + +The installation procedure executed by EasyBuild is thoroughly +[**logged**](https://docs.easybuild.io/en/latest/Logfiles.html), and is fully transparent via support for +[**dry runs**](https://docs.easybuild.io/en/latest/Extended_dry_run.html) and +[**tracing**](https://docs.easybuild.io/en/latest/Tracing_progress.html) the software installation procedure +as it is performed. + +EasyBuild supports using a **custom module naming scheme**, allows for +*hierarchical* module naming schemes, and **integrates with various other tools** ranging from resource managers ([Slurm](https://slurm.schedmd.com) and [GC3Pie](https://github.com/gc3pie/gc3pie)), +container tools ([Singularity](https://github.com/hpcng/singularity) and [Docker](https://www.docker.com)), +packaging tools ([FPM](https://fpm.readthedocs.io)), and so on. + +--- + +### *Focus points* + +EasyBuild was created specifically for **installing scientific software on HPC systems**, +which is reflected in some of the design choices that were made. + + +#### Performance + +EasyBuild strongly prefers to **build software from source code**, whenever possible. + +This is important to ensure that the binaries that are installed can maximally exploit +the capabilities of the system architecture on which the software will be run. + +For that same reason, EasyBuild out-of-the-box **optimizes software for the processor architecture of the build host** +by default, via compiler options like ``-march=native`` (GCC), ``-xHost`` (Intel compilers), etc. +This behaviour [can be changed via the ``--optarch`` configuration setting](https://docs.easybuild.io/en/latest/Controlling_compiler_optimization_flags.html). +When using the HPE Cray Programming Environment, the target modules are used instead. +In principle it is possible to cross-compile with EasyBuild, but this comes with some pitfalls. +By default EasyBuild tries to run tests if the build process of a package provides that +functionality, or can run simple tests of its own, and of course these tests will fail if +the host CPU does not support the full instruction set of the target CPU. Secondly, more +and more software has broken build processes where developers thought that it was a good +idea to hard-code options such as ``-march=native`` or ``--xHost`` in a way that EasyBuild +cannot easily overwrite those. + + +#### Reproducibility + +In addition to performance, **reproducibility of installations** is a core aspect of EasyBuild. + +TODO correct link in the next lines. + +Most software installations performed with EasyBuild use a **particular compiler toolchain**, +with which we aim to be in control over the build environment and avoid relying on tools and libraries +provided by the operating system. For similar reasons, we try to **provide all required dependencies through EasyBuild** as well, +with a few notable exceptions, like ``OpenSSL`` for security reasons, and Infiniband and GPU drivers which +are too closely intertwined with the operating system. + +For both toolchains and dependencies, **fixed software versions** are specified in the +easyconfig files. That way, easyconfig files can easily be shared with others: +if they worked for you it is very likely that they will work for others too, because the vast majority of the +software stack is controlled by EasyBuild. + +As we shall see when discussing the HPE Cray Programming Environment, the reproducibility is a bit more limited +in that environment due to the way the programming environment selects shared libraries at runtime. + + +#### Community effort + +In a number of different ways, we try to encourage EasyBuild users to **collaborate** and help each other out. + +We actively recommend people to report problems and bugs, to submit ideas for additional features and improvements, +and to [**contribute back**](https://docs.easybuild.io/en/latest/Contributing.html) when possible, be it +by opening pull requests to the [GitHub repositories](https://github.com/easybuilders) or the [documentation](https://docs.easybuild.io). + +The EasyBuild community also provides packages for two standard toolchains, called the +[**common toolchains**](https://easybuild.readthedocs.io/en/latest/Common-toolchains.html). +The so-called ``foss`` is based on free and open software (GNU compilers, FlexiBlas with OpenBLAS backend, LaPACK, FFTW and Open MPI) +while the ``intel`` toolchain uses the full stack of Intel compilers, libraries and MPI. Currently these toolchains +cannot be supported on LUMI though as getting Open MPI to run on LUMI is challenging and as the Intel libraries and +some MPI versions also produce slow or incorrect results on AMD CPUs (and it is not clear if Intel MPI is compatible +with SlingShot 11). +Focusing the efforts on those two popular common toolchains makes it possible to support a lot of build recipes that are +collected in the [central repository](https://github.com/easybuilders/easybuild-easyconfigs). +They can still serve as a basis to develop build recipes for the HPE Cray Programming Environment. + +Last but not least, EasyBuild provides various [**GitHub integration features**](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) +that greatly facilitate the contribution process: opening, updating, and testing pull requests, +reviewing incoming contributions, and much more can all be done directly from the EasyBuild +command line. This not only saves time, effort, brain cycles, and mouse clicks for contributors, +but it also makes the review process for *maintainers* significantly easier. +All together this leads to improved stability and consistency. + +--- + +### *What EasyBuild is* ***not*** + +EasyBuild is ***not*** **YABT (Yet Another Build Tool)**: it does *not* replace established build +tools like CMake or ``make``, it wraps around them. +If the installation procedure of a software package involves running some unholy trinity of tools while whispering +the correct magic incantations, EasyBuild automates this process for you. + +It is ***not*** **a replacement for traditional Linux package managers** like ``yum``, ``dnf`` or ``apt``. +EasyBuild relies on certain tools and libraries provided by the operating system. This includes glibc, OpenSSL, +drivers for Infiniband and GPUs, and so on. It is required that these tools are installed and managed by other means. +This is typically done via the package management tool that comes with your Linux distribution. + +Finally, EasyBuild is ***not*** **a magic solution to all your (software installation) problems**. +You may, and probably still will occasionally, run into compiler errors unless somebody has already taken care of the problem for you. +This is even more likely to happen in the Cray environment as builds are not often tested with the Cray compilers +or even with the AMD compilers. + +--- + +### *Requirements* + +Since EasyBuild is primarily intended for installing (scientific) software on HPC systems, the main target is +systems running some flavor of **GNU/Linux**. EasyBuild also works on *macOS*, +but significant effort would be required to use it extensively there. + + + +EasyBuild is implemented in [**Python**](https://www.python.org), and is compatible with Python 2.7 and 3.5+. +Since [Python 2 is no longer maintained](https://blog.python.org/2020/04/python-2718-last-release-of-python-2.html), +it is recommended to use a recent version of Python 3 for running EasyBuild. + +An [**environment modules**](https://en.wikipedia.org/wiki/Environment_Modules_(software)) tool is required for running EasyBuild. +The modern Lua-based implementation **[Lmod](https://lmod.readthedocs.io)** is strongly recommended, +but other (Tcl-based) implementations are also supported. +Both Tcl and Lua are supported as syntax for environment module files. +On LUMI Lmod is used as the module tool. + +--- + +### *Development* + + +EasyBuild is **actively developed** by a worldwide community under the +[**GPLv2 open source license**](https://www.gnu.org/licenses/old-licenses/gpl-2.0.html) +via the [**``easybuilders``** organisation on GitHub](https://github.com/easybuilders), in the following repositories: + +- [easybuild-framework](https://github.com/easybuilders/easybuild-framework) +- [easybuild-easyblocks](https://github.com/easybuilders/easybuild-easyblocks) +- [easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs) +- [easybuild](https://github.com/easybuilders/easybuild) (website, documentation) +- [easybuild-tutorial](https://github.com/easybuilders/easybuild-tutorial) (the generic EasyBuild tutorials) + +On LUMI the generic EasyBuild installation is used but extended and adapted using LUMI-specifics +plugins and hooks maintained in the [LUMI software stack repository](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack). + +Stable versions of EasyBuild have been released roughly every 6-8 weeks since 2012, +and are published via the *Python Package Index (PyPI)*, +see [https://pypi.org/project/easybuild](https://pypi.org/project/easybuild/). + +**Comprehensive testing** practices are applied throughout the +development cycle, with extensive suites of unit and integration tests being run in a CI environment (GitHub Actions), +consistent testing of incoming contributions (pull requests), and thorough regression testing before every release. + +--- + +### *A worldwide community* + +EasyBuild has grown into a **community project** since it was first released publicy in 2012, +and is used and developed by dozens (hundreds?) of HPC centres and consortia worldwide. + +Since 2016 the EasyBuild community has gathered for a yearly [**EasyBuild User Meeting**](https://easybuild.io/eum), +which is an open and highly interactive event that provides a great opportunity to meet fellow EasyBuild users, +developers, and maintainers, and to discuss related topics, learn about new aspects of the tool, and how other HPC +sites leverage it, and integrate it with other tools like [Jenkins](https://www.youtube.com/watch?v=bzW45cfRUMU), +[Ansible](https://www.youtube.com/watch?v=Qr2udsAjkG0), [CernVM-FS](https://www.youtube.com/watch?v=_0j5Shuf2uE), +[ReFrame](https://www.youtube.com/watch?v=KN9QwQIrso4) etc. + +HPC sites, consortia, and companies that use EasyBuild include (but are not limited to): + +VSC logo +JSC logo +HPCNow! logo +Fred Hutch logo +Vienna BioCenter logo +CÉCI logo +Compute Canada logo +UiO logo +BEAR logo +SNIC logo +National University of Singapore logo +IT4Innovations logo +Texas A&M logo +TU Dresden logo +University of Melbourne logo +University of Luxembourg logo +EPCC logo +Cyprus Institute logo +University of Basel logo +Microsoft logo + +--- + +### *Getting help* + +#### *Documentation* + +If you have any questions about EasyBuild, make sure to check the EasyBuild documentation first, +which is available at [**https://docs.easybuild.io**](https://docs.easybuild.io). + +It is not unlikely that an answer to your question is available there already. +The [search box](https://docs.easybuild.io/en/latest/search.html) on the top left may be helpful. + +#### GitHub + + + +You can open an issue in one of the EasyBuild repositories in [the `easybuilders` GitHub organisation](https://github.com/easybuilders/) to +ask questions, report problems or share suggestions and ideas for additional +improvements. + +Please try to post the issue in the most suitable repository. If you are +not sure which repository is most suited just go ahead and open the issue anyway, +the EasyBuild maintainers can move your issue if deemed appropriate. + +Also here you can use the search box on the top left, to see if somebody else +has already asked a similar question or reported a similar problem. + +#### Mailing list + +Feel free to ask any questions or provide feedback via the EasyBuild mailing list +`easybuild@lists.ugent.be`. + +Keep in mind that **you must be subscribed to the mailing list in order to +post messages there!** You can subscribe via [this link](https://lists.ugent.be/wws/subscribe/easybuild). + +In 2016-2017 over 1,000 messages were posted to the EasyBuild mailing list. +More recently this has decreased to about 700 messages per year, mostly due +to the EasyBuild Slack channel. + +#### Slack + +Slack logo + +The [EasyBuild Slack](https://easybuild.slack.com/) is a quick and easy way +to get in touch with the EasyBuild community. +Over 400 people have joined it so far, and there is usually somebody awake in there +who may be able to help you out. + +**To join the EasyBuild Slack request an invitation via +[https://easybuild-slack.herokuapp.com](https://easybuild-slack.herokuapp.com/).** + +If you are not a big fan of Slack, you can also consider joining the `#easybuild` +IRC channel (available on the [Freenode IRC network](https://freenode.net/)). +A bot relays messages between this IRC channel and the `#general` channel in the EasyBuild Slack. + + +#### Conference calls + +Every other week on Wednesday an EasyBuild conference call is set up to discuss recent +developments, provide an outlook to the next EasyBuild release, and +provide answers to questions that can be raised by the people attending the call. + +These conference calls are open to anyone interested in the EasyBuild project, +and have been organised since November 2013. + +More information, including the dates and times of the next calls, is available +[on the EasyBuild Wiki](https://github.com/easybuilders/easybuild/wiki/Conference-calls). + +--- + +*[[next: Terminology]](terminology.md)* diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md index 9af354b2..5e6b8b53 100644 --- a/docs/2022-CSC_and_LO/index.md +++ b/docs/2022-CSC_and_LO/index.md @@ -10,6 +10,15 @@ See that site for [credits to the contributors of those tutorials](https://easyb ## Contents +- [Part I: **Introduction to EasyBuild**](1_00_part1_intro.md) + - [What is EasyBuild?](1_01_what_is_easybuild.md) + - [Terminology](terminology.md) + - [Installation](installation.md) *(hands-on)* + - [Configuration](configuration.md) *(hands-on)* + - [Basic usage](basic_usage.md) *(hands-on)* + + + TODO @@ -17,6 +26,7 @@ TODO + ## OLD TEXT - [Part I: **Introduction to EasyBuild**](part1_intro.md) *(Tue March 9th 2021, 9am-12 CET)* From 6541e8c506e49b5586be7cc0d6f4db592af0b7f7 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 9 Mar 2022 10:58:10 +0100 Subject: [PATCH 003/159] Updated the structure, initial updates of terminology and a new page about the Cray PE and Lmod. --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 4 +- .../2022-CSC_and_LO/1_01_what_is_easybuild.md | 34 +- docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md | 71 ++++ docs/2022-CSC_and_LO/1_03_terminology.md | 348 ++++++++++++++++++ docs/2022-CSC_and_LO/index.md | 5 +- 5 files changed, 441 insertions(+), 21 deletions(-) create mode 100644 docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md create mode 100644 docs/2022-CSC_and_LO/1_03_terminology.md diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md index 0bfd0ad5..e381f45d 100644 --- a/docs/2022-CSC_and_LO/1_00_part1_intro.md +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -5,8 +5,8 @@ *(Tue March 4th 2021, 9am-12 CET)* - [What is EasyBuild?](1_01_what_is_easybuild.md) - -- [Terminology](terminology.md) +- [Terminology](1_02_terminology.md) + - [Installation](installation.md) *(hands-on)* - [Configuration](configuration.md) *(hands-on)* - [Basic usage](basic_usage.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md index 678c8c4a..5ccdb1db 100644 --- a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md @@ -15,7 +15,7 @@ and provides a **command-line interface** to facilitate the installation and man -### *Elevator pitch* +## Elevator pitch EasyBuild intends to relieve HPC support teams from manually managing software installations while at the same time **providing a consistent and well performing @@ -46,7 +46,7 @@ is still under development on LUMI it is not yet covered in this LUMI-specific v --- -### *Key features* +## Key features EasyBuild is capable of **fully autonomously installing (scientific) software**, including making sure that all necessary dependencies are installed, @@ -73,13 +73,13 @@ packaging tools ([FPM](https://fpm.readthedocs.io)), and so on. --- -### *Focus points* +## Focus points EasyBuild was created specifically for **installing scientific software on HPC systems**, which is reflected in some of the design choices that were made. -#### Performance +### Performance EasyBuild strongly prefers to **build software from source code**, whenever possible. @@ -99,7 +99,7 @@ idea to hard-code options such as ``-march=native`` or ``--xHost`` in a way that cannot easily overwrite those. -#### Reproducibility +### Reproducibility In addition to performance, **reproducibility of installations** is a core aspect of EasyBuild. @@ -120,7 +120,7 @@ As we shall see when discussing the HPE Cray Programming Environment, the reprod in that environment due to the way the programming environment selects shared libraries at runtime. -#### Community effort +### Community effort In a number of different ways, we try to encourage EasyBuild users to **collaborate** and help each other out. @@ -148,7 +148,7 @@ All together this leads to improved stability and consistency. --- -### *What EasyBuild is* ***not*** +## What EasyBuild is ***not*** EasyBuild is ***not*** **YABT (Yet Another Build Tool)**: it does *not* replace established build tools like CMake or ``make``, it wraps around them. @@ -167,7 +167,7 @@ or even with the AMD compilers. --- -### *Requirements* +## Requirements Since EasyBuild is primarily intended for installing (scientific) software on HPC systems, the main target is systems running some flavor of **GNU/Linux**. EasyBuild also works on *macOS*, @@ -187,7 +187,7 @@ On LUMI Lmod is used as the module tool. --- -### *Development* +## Development EasyBuild is **actively developed** by a worldwide community under the @@ -213,7 +213,7 @@ consistent testing of incoming contributions (pull requests), and thorough regre --- -### *A worldwide community* +## A worldwide community EasyBuild has grown into a **community project** since it was first released publicy in 2012, and is used and developed by dozens (hundreds?) of HPC centres and consortia worldwide. @@ -250,9 +250,9 @@ HPC sites, consortia, and companies that use EasyBuild include (but are not limi --- -### *Getting help* +## Getting help -#### *Documentation* +### *Documentation* If you have any questions about EasyBuild, make sure to check the EasyBuild documentation first, which is available at [**https://docs.easybuild.io**](https://docs.easybuild.io). @@ -260,7 +260,7 @@ which is available at [**https://docs.easybuild.io**](https://docs.easybuild.io) It is not unlikely that an answer to your question is available there already. The [search box](https://docs.easybuild.io/en/latest/search.html) on the top left may be helpful. -#### GitHub +### GitHub @@ -275,7 +275,7 @@ the EasyBuild maintainers can move your issue if deemed appropriate. Also here you can use the search box on the top left, to see if somebody else has already asked a similar question or reported a similar problem. -#### Mailing list +### Mailing list Feel free to ask any questions or provide feedback via the EasyBuild mailing list `easybuild@lists.ugent.be`. @@ -287,7 +287,7 @@ In 2016-2017 over 1,000 messages were posted to the EasyBuild mailing list. More recently this has decreased to about 700 messages per year, mostly due to the EasyBuild Slack channel. -#### Slack +### Slack Slack logo @@ -304,7 +304,7 @@ IRC channel (available on the [Freenode IRC network](https://freenode.net/)). A bot relays messages between this IRC channel and the `#general` channel in the EasyBuild Slack. -#### Conference calls +### Conference calls Every other week on Wednesday an EasyBuild conference call is set up to discuss recent developments, provide an outlook to the next EasyBuild release, and @@ -318,4 +318,4 @@ More information, including the dates and times of the next calls, is available --- -*[[next: Terminology]](terminology.md)* +*[[next: The HPE Cray Programming Environment and Lmod]](1_02_CPE_and_Lmod.md)* diff --git a/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md b/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md new file mode 100644 index 00000000..01812057 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md @@ -0,0 +1,71 @@ +# The HPE Cray Programming Environment and Lmod + +*[[back: What is EasyBuild?]](1_01_what_is_easybuild.md)* + +--- + +On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated +as Cray PE). The environment provides several tools, including compilers, communication libraries, +optmized math libraries and various other libraries, analyzers and debuggers. + +The Cray PE is made available through *environment modules* tha allow to select particular versions of +tools and to configure the environment in a flexible way. + +--- + +## Modules + +*Module* is a massively overloaded term in (scientific) software and IT in general +(kernel modules, Python modules, and so on). +In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. + +[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept +on HPC systems: it is a way to specify changes that should be made to one or more +[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a +[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file +is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or +[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, +and specifies which environment variables should be updated, and how (append, +prepend, (re)define, undefine, etc.) upon loading the environment module. +Unloading the environment module will restore the shell environment to its previous state. + +Environment module files are processed via a **modules tool**, of which there +are several conceptually similar yet slightly different implementations. +The oldest module tool still in use today is Environment Modules 3.2, implemented in C and +supporting module files written in Tcl. After a gap in developement, Xavier Delaruelle of CEA +developed [Environment Modules 4 and 5](https://sourceforge.net/projects/modules/) which is +fully implemented on Tcl. An alternative module tool is [Lmod](https://lmod.readthedocs.io), +developed by Robert McLay at TACC and implemented in LUA. This tool supports natively LUA +module files but also offers a high degree of compatibility with Tcl-based module files +developed for Environment Modules fia a translation layer and some API translation. + +The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no +packages or official support for Environment Modules 4 or 5. On LUMI, Lmod was selected as +the module tool. At the user level, Lmod and the various versions of Emvironment Modules +have very simmilar commands for managing the environment, but with different options. +The commands for searching for modules are very different though so if you are not familiar +with Lmod and its commands for users, it is worthwile to read the +[LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). + +--- + +## Lmod features + + +--- + +## Cray PE components + + +--- + +## Configuring the Cray PE through modules + +--- + +## Further reading + + +--- + +*[[next: Terminology]](1_03_terminology.md)* diff --git a/docs/2022-CSC_and_LO/1_03_terminology.md b/docs/2022-CSC_and_LO/1_03_terminology.md new file mode 100644 index 00000000..ec085b0c --- /dev/null +++ b/docs/2022-CSC_and_LO/1_03_terminology.md @@ -0,0 +1,348 @@ +# Terminology + +*[[back: The HPE Cray Programming Environment and Lmod]](1_02_CPE_and_Lmod.md)* + +--- + +Over the years, we have come up with some terminology specific to EasyBuild +to refer to particular components, which we use alongside established terminology relevant to the context +of building and installing software. + +It is important to be familiar with these terms, so we'll briefly cover them one by one. + +--- + +## *Toolchains* + +A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, +which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. + +We refer to the different parts of a toolchain as **toolchain components**. + +The *compiler component* typically consists of [C](https://en.wikipedia.org/wiki/C_(programming_language)), +[C++](https://en.wikipedia.org/wiki/C%2B%2B), and [Fortran](https://en.wikipedia.org/wiki/Fortran) +compilers in the context of HPC, but additional compilers (for example, +a [CUDA](https://developer.nvidia.com/cuda-zone) compiler for +[GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units) software) +can also be included. + +Additional toolchain components are usually special-purpose libraries: + +- an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/)); + +- libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), + [LAPACK](http://performance.netlib.org/lapack/)); + + These libraries are included because they tend to provide a common API, so mixing them in a single + executable would cause name conflicts when linking. + +- a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); + +A toolchain that includes all of these libraries is referred to as a **full toolchain**, while +a **subtoolchain** is a toolchain that is missing one or more of these libraries. +A **compiler-only toolchain** only consists of compilers (no additional libraries). + +Many EasBuild toolchains are organised in a hierarchy. Each toolchain can use components compiled +with itself or with one of the subtoolchains. + + +### System toolchain + +The **`system` toolchain** is a special case which corresponds to using the compilers and libraries +*provided by the operating system*, rather than using toolchain components that were installed using EasyBuild. + +It used sparingly, mostly to install software where no actual compilation is done or +to build a set of toolchain compilers and its dependencies, since the versions of the system compilers +and libraries are beyond the control of EasyBuild, which could affect the reproducibility of the installation. + +On LUMI however it is used a bit more and it takes some of the functions of the ``GCCcore`` toolchain in +other EasyBuild toolchain hierarchies. + + +### Common toolchains + +The `foss` and `intel` toolchains are also known as the *common toolchains*, +because they are widely adopted by the EasyBuild community. + +The `foss` toolchain consists of all open source components (hence the name: +"FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), +[ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). + +The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version +controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. + +Roughly every 6 months, a new version of these common toolchains is agreed upon +in the EasyBuild community, after extensive testing. + +More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). + +--- + +## *Modules* + +*Module* is a massively overloaded term in (scientific) software and IT in general +(kernel modules, Python modules, and so on). +In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. + +[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept +on HPC systems: it is a way to specify changes that should be made to one or more +[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a +[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file +is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or +[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, +and specifies which environment variables should be updated, and how (append, +prepend, (re)define, undefine, etc.) upon loading the environment module. +Unloading the environment module will restore the shell environment to its previous state. + +Environment module files are processed via a **modules tool**, of which there +are several conceptually similar yet slightly different implementations. +The Tcl-based [Environment Modules](https://sourceforge.net/projects/modules/) implementation, and +[Lmod](https://lmod.readthedocs.io), a more recent Lua-based implementation (which also supports module +files written in Tcl syntax), are the most commonly used ones. + +**Environment module files are automatically generated for each software installation** by EasyBuild, +and *loading* a module results in changes being made to the environment of the current shell +session such that the corresponding software installation can be used. + +--- + +## *EasyBuild framework* + +The EasyBuild *framework* consists of a set of Python modules organised in packages (``easybuild.framework``, +``easybuild.toolchains``, ``easybuild.tools``, etc.) that collectively form **the core of EasyBuild**, +and is developed in the [``easybuild-framework`` repository on GitHub](https://github.com/easybuilders/easybuild-framework). + +It implements the **common functionality that you need when building software from source**, +providing functions for unpacking source files, applying patch files, collecting the output produced +by shell commands that are being run and checking their exit code, generating environment module files, etc. + +The EasyBuild framework does *not* implement any specific installation procedure, it only provides +the necessary functionality to facilitate this. + + +--- + +## *Easyblocks* + +An *easyblock* is **a Python module that implements a specific software installation procedure**, +and can be viewed as a plugin to the EasyBuild framework. Easyblocks can be either *generic* or *software-specific*. + +A **generic easyblock** implements an installation procedure that can be used for +multiple different software packages. Commonly used examples include the ``ConfigureMake`` easyblock +which implements the ubiquitous ``configure``-``make``-``make install`` procedure, and the +``PythonPackage`` easyblock that can be used to install a Python package. + +A **software-specific easyblock** implements an installation procedure that is specific to a particular software packages. +Infamous examples include the easyblocks we have for ``GCC``, ``OpenFOAM``, ``TensorFlow``, ``WRF``, ... + +The installation procedure performed by an easyblock can be controlled by defining +**easyconfig parameters** in an easyconfig file. + +A collection of (generic and software-specific) easyblocks is developed by the EasyBuild community +in the [``easybuild-easyblocks`` repository on GitHub](https://github.com/easybuilders/easybuild-easyblocks). + +--- + +## *Easyconfig parameters* + +An **easyconfig parameter** specifies a particular aspect of a software installation that should be performed by +EasyBuild. + +Some easyconfig parameters are **mandatory**. The following parameters *must* be defined in *every* easyconfig file: + +* ``name`` and ``version``, which specify the name and version of the software to install; +* ``homepage`` and ``description``, which provide key metadata for the software; +* ``toolchain``, which specifies the compiler toolchain to use to install the software (see + ``toolchains`` tab); + +Other easyconfig parameters are **optional**: they can be used to provide required information, +or to control specific aspects of the installation procedure performed by the easyblock. + +Some commonly used optional easyconfig parameters include: + +* ``easyblock``, which specifies which (generic) easyblock should be used for the installation; +* ``sources`` and ``source_urls``, which specify the list of source files and where to download them; +* ``dependencies`` and ``builddependencies``, which specify the list of (build) dependencies; +* ``configopts``, ``buildopts``, and ``installopts``, which specify options for the configuration/build/install commands, respectively; + +If no value is specified for an optional easyconfig parameter, the corresponding default value will be used. + +There are two groups of easyconfig parameters. *General* easyconfig parameters can be defined for any software +package, and (usually) control a specific aspect of the installation. *Custom* easyconfig parameters are +only supported by certain easyblocks, and only make sense for particular (types of) software. + +--- + +## *Easyconfig files* + +*Easyconfig files* (or *easyconfigs* for short), are **simple text files written in Python syntax +that specify what EasyBuild should install**. +Each easyconfig file defines the set of **easyconfig parameters** that collectively form a complete specification +for a particular software installation. + +The **filename** of an easyconfig file usually ends with the ``.eb`` extension. +In some contexts the filename is expected to be determined by the value of a handful of key +easyconfig parameters: ``name``, ``version``, ``toolchain`` and ``versionsuffix``. The general format for +the filename of an easyconfig file is: ``--.eb``, +where the toolchain part is omitted when a ``system`` toolchain is used, and the `` can be empty. + +The filename of easyconfig files is particularly relevant when EasyBuild is searching for easyconfig files to resolve dependencies, +since it does this purely based on filenames: interpreting the contents of every (potential) easyconfig file +it encounters would be too expensive. + +In the [``easybuild-easyconfigs`` repository on GitHub](https://github.com/easybuilders/easybuild-easyconfigs), +the EasyBuild community maintains a large (and growing) collection of easyconfig files, for a wide range of +(scientific) software. + +--- + +## *Easystack files* + +[**Easystack files**](https://docs.easybuild.io/en/latest/Easystack-files.html) +are a new concept in EasyBuild, providing a way to define a *software stack* +that should be installed by EasyBuild. + +They are written in [YAML syntax](https://yaml.org/), and include a list of *software specifications* +which correspond to a list of easyconfig files, with support for providing specific EasyBuild +configuration options for particular software packages, and including or excluding specific software +packages based on labels. + +The support for using easystack files is currently marked as *experimental*, +which means it is subject to change in future EasyBuild releases, and may be prone to errors. + +--- + +## *Extensions* + +*Extensions* is the collective term we use for **additional software packages that can be installed +on top of another software package**. Common examples are *Python packages*, *R libraries*, and *Perl modules*. + +As you can tell the common terminology here is a bit messy, so we came up with a unifying term... + +Extensions can be installed in different ways: + +* *stand-alone*, as a separate installation on top of one or more other installations; +* as a part of a *bundle* of extensions that collectively form a separate installation; +* or as an actual *extension* to a specific installation to yield a "batteries included" + type of installation (for examples by adding a bunch of Python packages from PyPI into + a Python installation); + +--- + +## *Dependencies* + +A *dependency* is a common term in the context of software. It refers to **a software +package that is either strictly required by other software, or that can be leveraged to +enhance other software** (for example to support specific features). + +There are three main types of dependencies for computer software: + +* a **build dependency** is only required when building/installing a software package; + once the software package is installed, it is no longer needed to *use* that software + (examples: `CMake`, `pkg-config`); +* a **run-time dependency** (often referred to simply as *dependency*) is a software package that is + required to *use* (or *run*) another software package (example: `Python`); +* a **link-time dependency** is somewhere in between a build and runtime dependency: + it is only needed when *linking* a software package; it can become either a build or runtime + dependency, depending on exactly how the software is installed (example: `OpenBLAS`); + +The distinction between link-time and run-time dependencies is mostly irrelevant for this tutorial, +but we will discriminate build-only dependencies. + +--- + +## *Toolchains* + +A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, +which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. + +We refer to the different parts of a toolchain as **toolchain components**. + +The *compiler component* typically consists of [C](https://en.wikipedia.org/wiki/C_(programming_language)), +[C++](https://en.wikipedia.org/wiki/C%2B%2B), and [Fortran](https://en.wikipedia.org/wiki/Fortran) +compilers in the context of HPC, but additional compilers (for example, +a [CUDA](https://developer.nvidia.com/cuda-zone) compiler for +[GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units) software) +can also be included. + +Additional toolchain components are usually special-purpose libraries: + +* an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/)); +* libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), + [LAPACK](http://performance.netlib.org/lapack/)); +* a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); + +A toolchain that includes all of these libraries is referred to as a **full toolchain**, while +a **subtoolchain** is a toolchain that is missing one or more of these libraries. +A **compiler-only toolchain** only consists of compilers (no additional libraries). + +### System toolchain + +The **`system` toolchain** is a special case which corresponds to using the compilers and libraries +*provided by the operating system*, rather than using toolchain components that were installed using EasyBuild. + +It used sparingly, mostly to install software where no actual compilation is done or +to build a set of toolchain compilers and its dependencies, since the versions of the system compilers +and libraries are beyond the control of EasyBuild, which could affect the reproducibility of the installation. + +### Common toolchains + +The `foss` and `intel` toolchains are also known as the *common toolchains*, +because they are widely adopted by the EasyBuild community. + +The `foss` toolchain consists of all open source components (hence the name: +"FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), +[ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). + +The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version +controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. + +Roughly every 6 months, a new version of these common toolchains is agreed upon +in the EasyBuild community, after extensive testing. + +More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). + +--- + +## *Modules* + +*Module* is a massively overloaded term in (scientific) software and IT in general +(kernel modules, Python modules, and so on). +In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. + +[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept +on HPC systems: it is a way to specify changes that should be made to one or more +[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a +[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file +is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or +[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, +and specifies which environment variables should be updated, and how (append, +prepend, (re)define, undefine, etc.) upon loading the environment module. +Unloading the environment module will restore the shell environment to its previous state. + +Environment module files are processed via a **modules tool**, of which there +are several conceptually similar yet slightly different implementations. +The Tcl-based [Environment Modules](https://sourceforge.net/projects/modules/) implementation, and +[Lmod](https://lmod.readthedocs.io), a more recent Lua-based implementation (which also supports module +files written in Tcl syntax), are the most commonly used ones. + +**Environment module files are automatically generated for each software installation** by EasyBuild, +and *loading* a module results in changes being made to the environment of the current shell +session such that the corresponding software installation can be used. + +--- + +## *Bringing it all together* + +The EasyBuild **framework** leverages **easyblocks** to automatically build and install +(scientific) software, potentially including additional **extensions**, using a particular compiler **toolchain**, +as specified in **easyconfig files** which each define a set of **easyconfig parameters**. + +EasyBuild ensures that the specified **(build) dependencies** are in place, +and automatically generates a set of **(environment) modules** that facilitate access to the installed software. + +An **easystack file** can be used to specify a collection of software to install with EasyBuild. + +--- + +*[[next: Installation]](installation.md)* diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md index 5e6b8b53..6b4c8613 100644 --- a/docs/2022-CSC_and_LO/index.md +++ b/docs/2022-CSC_and_LO/index.md @@ -10,9 +10,10 @@ See that site for [credits to the contributors of those tutorials](https://easyb ## Contents -- [Part I: **Introduction to EasyBuild**](1_00_part1_intro.md) +- [Part I: **Introduction to EasyBuild on Cray systems**](1_00_part1_intro.md) - [What is EasyBuild?](1_01_what_is_easybuild.md) - - [Terminology](terminology.md) + - [The HPE Cray Programming Environment and Lmod](1_02_CPE_and_Lmod.md) + - [Terminology](1_03_terminology.md) - [Installation](installation.md) *(hands-on)* - [Configuration](configuration.md) *(hands-on)* - [Basic usage](basic_usage.md) *(hands-on)* From b2a55cd3e9d98c16970fd492f80c4357f3072f12 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 10 Mar 2022 14:37:51 +0100 Subject: [PATCH 004/159] Additions to the Lmod section and some restructuring, and better information in the README for the repository. --- .../2022-CSC_and_LO/1_01_what_is_easybuild.md | 2 +- docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md | 71 ------ docs/2022-CSC_and_LO/1_02_Lmod.md | 211 ++++++++++++++++++ docs/2022-CSC_and_LO/1_03_CPE.md | 30 +++ ..._03_terminology.md => 1_04_terminology.md} | 2 +- docs/2022-CSC_and_LO/index.md | 5 +- 6 files changed, 246 insertions(+), 75 deletions(-) delete mode 100644 docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md create mode 100644 docs/2022-CSC_and_LO/1_02_Lmod.md create mode 100644 docs/2022-CSC_and_LO/1_03_CPE.md rename docs/2022-CSC_and_LO/{1_03_terminology.md => 1_04_terminology.md} (99%) diff --git a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md index 5ccdb1db..979c6e90 100644 --- a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md @@ -318,4 +318,4 @@ More information, including the dates and times of the next calls, is available --- -*[[next: The HPE Cray Programming Environment and Lmod]](1_02_CPE_and_Lmod.md)* +*[[next: The Lmod module system]](1_02_Lmod.md)* diff --git a/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md b/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md deleted file mode 100644 index 01812057..00000000 --- a/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md +++ /dev/null @@ -1,71 +0,0 @@ -# The HPE Cray Programming Environment and Lmod - -*[[back: What is EasyBuild?]](1_01_what_is_easybuild.md)* - ---- - -On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated -as Cray PE). The environment provides several tools, including compilers, communication libraries, -optmized math libraries and various other libraries, analyzers and debuggers. - -The Cray PE is made available through *environment modules* tha allow to select particular versions of -tools and to configure the environment in a flexible way. - ---- - -## Modules - -*Module* is a massively overloaded term in (scientific) software and IT in general -(kernel modules, Python modules, and so on). -In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. - -[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept -on HPC systems: it is a way to specify changes that should be made to one or more -[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a -[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file -is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or -[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, -and specifies which environment variables should be updated, and how (append, -prepend, (re)define, undefine, etc.) upon loading the environment module. -Unloading the environment module will restore the shell environment to its previous state. - -Environment module files are processed via a **modules tool**, of which there -are several conceptually similar yet slightly different implementations. -The oldest module tool still in use today is Environment Modules 3.2, implemented in C and -supporting module files written in Tcl. After a gap in developement, Xavier Delaruelle of CEA -developed [Environment Modules 4 and 5](https://sourceforge.net/projects/modules/) which is -fully implemented on Tcl. An alternative module tool is [Lmod](https://lmod.readthedocs.io), -developed by Robert McLay at TACC and implemented in LUA. This tool supports natively LUA -module files but also offers a high degree of compatibility with Tcl-based module files -developed for Environment Modules fia a translation layer and some API translation. - -The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no -packages or official support for Environment Modules 4 or 5. On LUMI, Lmod was selected as -the module tool. At the user level, Lmod and the various versions of Emvironment Modules -have very simmilar commands for managing the environment, but with different options. -The commands for searching for modules are very different though so if you are not familiar -with Lmod and its commands for users, it is worthwile to read the -[LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). - ---- - -## Lmod features - - ---- - -## Cray PE components - - ---- - -## Configuring the Cray PE through modules - ---- - -## Further reading - - ---- - -*[[next: Terminology]](1_03_terminology.md)* diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md new file mode 100644 index 00000000..91cb7571 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -0,0 +1,211 @@ +# The Lmod module system + +*[[back: What is EasyBuild?]](1_01_what_is_easybuild.md)* + +--- + +On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated +as Cray PE). The environment provides several tools, including compilers, communication libraries, +optmized math libraries and various other libraries, analyzers and debuggers. + +The Cray PE is made available through *environment modules* tha allow to select particular versions of +tools and to configure the environment in a flexible way. + +--- + +## Modules + +*Module* is a massively overloaded term in (scientific) software and IT in general +(kernel modules, Python modules, and so on). +In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. + +[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept +on HPC systems: it is a way to specify changes that should be made to one or more +[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a +[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file +is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or +[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, +and specifies which environment variables should be updated, and how (append, +prepend, (re)define, undefine, etc.) upon loading the environment module. +Unloading the environment module will restore the shell environment to its previous state. + +Environment module files are processed via a **modules tool**, of which there +are several conceptually similar yet slightly different implementations. +The oldest module tool still in use today is Environment Modules 3.2, implemented in C and +supporting module files written in Tcl. After a gap in developement, Xavier Delaruelle of CEA +developed [Environment Modules 4 and 5](https://sourceforge.net/projects/modules/) which is +fully implemented on Tcl. An alternative module tool is [Lmod](https://lmod.readthedocs.io), +developed by Robert McLay at TACC and implemented in LUA. This tool supports natively LUA +module files but also offers a high degree of compatibility with Tcl-based module files +developed for Environment Modules fia a translation layer and some API translation. + +The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no +packages or official support for Environment Modules 4 or 5. On LUMI, Lmod was selected as +the module tool. At the user level, Lmod and the various versions of Emvironment Modules +have very simmilar commands for managing the environment, but with different options. +The commands for searching for modules are very different though so if you are not familiar +with Lmod and its commands for users, it is worthwile to read the +[LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). + +--- + +## Lmod hierarchy + +### User view + +Lmod supports a module hierarchy. In a hierarchy, there is a distinction between the *installed +modules* and the *available modules*. Available modules are those that can be loaded directly +without first loading any other module, while the installed modules is the complete set of +modules that one could load one way or another. A typical use case +is a hierarchy to deal with different compilers on a system and different MPI implementations. +After all, it is a common practice to only link libraries and application code compiled with the +same compiler to avoid compatibility problems between compilers (and to be able to use advanced +features such as link time optimization). This is even more important for MPI, as Open MPI and +MPCIH-derived MPI implementations have incompatible Application Binary Interfaces. This would lead +to a hierarchy with 3 levels: + +1. The ``Core`` level containing the modules for the compilers themselves, e.g., one or more versions + of the GNU compiler suite and one or more versions of LLVM-based compilers. + + Loading a compiler module would then make the next level available: + +2. The ``Compiler`` level, containing modules for libraries and packages that only rely on the compilers + but do not use MPI, as well as the MPI modules, e.g., a version of Open MPI and a version of MPICH. + + Loading one of the MPI modules would then make the next level available: + +3. The ``MPI`` level, containing libraries and applications that depend on the compiler used and the MPI + implementation. + +Now assume that we have two compilers in the hierarchy, Compiler_A and Compiler_B. Their modules would reside +at the ``Core`` level. Both copilers provide the same MPI implementation, MPI_C. So there would be two modules +for ``MPI_C`` in two different subdirectories at the ``Compiler`` level. And further assume that we have an +application, Appl_E, compiled with both Compiler_A and Compiler_B and using MPI_C. For that application there would +also be two module files at the ``MPI`` level, one in a subdirectory corresponding ao Compiler_A and MPI_C and one +in a subdirectory corresponding to Compiler_B and MPI_C. To be able to load the module for Appl_E, a user should +first load Compiler_A, then load MPI_C and only then is it possible to load the module for Appl_E: + +```bash +module load Compiler_A MPI_C Appl_E +``` + +What is interesting is what happens if the user now loads Compiler_B: + +```bash +module load Compiler_B +``` + +In a properly designed and implemented hierarchy, Lmod will unload Compiler_A which will also trigger the unloading/deactivation +of MPI_C and Appl_E. It will then load the module for Compiler_B and proceed with looking if it can find another module for +MPI_C. That will then be loaded which now makes a different module for Appl_E available, which Lmod will proceed to load. If it +cannot find an exact match for the version, Lmod will even try to locate a different version. Hence the situation after loading +Compiler_B is that now modules are loaded for Compiler_B, MPI_C for Compiler_B and Appl_E for Compiler_A with MPI_C. +All this requires very little effort from the module file programmer and very little logic in the module files. E.g., rather +then implementing a single module file for Appl_E that would require logic to see which compiler and MPI implementation is loaded +and depending on those adapt the path to the binaries, several very simple modules need to be written with very little +logic, and one could add an Appl_E module for a different compiler or MPI implementation without touching any of the already +existing module files for that application. + + +### Building blocks + +Some mechanisms in Lmod make implementing a hierarchy fairly easy (though there are a lot of hidden pitfalls) + +- The *MODULEPATH* environment variable determines which modules are available. MODULEPATH is different from any other + path-style variable in Lmod in that any change will immediately trigger a re-evaluation of which modules are available + and trigger deactivating modules that are no longer available when a directory is removed from the MODULEPATH or + looking for alternatives for deactivated modules when a directory is added to the MODULEPATH. + +- The *"one name rule"*: Lmod cannot have two modules loaded with the same name (but a different version). By default, when loading + a module with the name of an already loaded module, Lmod will automatically swap the old one with the new one, i.e., unload the + already loaded module and load the new one. + +- The *family* concept: It is possible to declare a module to be part of a family using a command in the module file. No two modules + of the same family can be loaded at the same time, and Lmod will again by default auto-swap the already loaded one with the one + being loaded. The procedure is different though as Lmod now first has to read the new module file to discover the family, and this + may lead to more side effects. But that discussion is outside the scope of this tutorial. + + The family concept was for a long time a unique feature of Lmod, but it has been added now also to Environment Modules version 5.1. + + +### Implementation details + +The above example could be implemented using 6 module files: One for each compiler, two for the MPI module and +two for the application module. + +``` +moduleroot +├── Core +│ ├── Compiler_A +│ │ └── version_A.lua +│ └── Compiler_B +│ └── version_B.lua +├── Compiler +│ ├── Compiler_A +│ │ └── version_A +│ │ └── MPI_C +│ │ └── version_C.lua +│ └── Compiler_B +│ └── version_B +│ └── MPI_C +│ └── version_C.lua +└── MPI + ├── Compiler_A + │ └── version_A + │ └── MPI_C + │ └── version_C + │ └── Appl_E + │ └── version_E.lua + └── Compiler_B + └── version_B + └── MPI_C + └── version_C + └── Appl_E + └── version_E.lua +``` + +Besides the module functions needed to create the environment needed to run the compiler, the module file for +Compiler_A would need only two lines to implement the hierarchy: + +```Lua +family('Compiler') +prepend_path('MODULEPATH', 'moduleroot/Compiler/Compiler_A/version_A') +``` + +There are now two different ``version_C.lua`` files. One contains the necessary calls to module functions to +initialise the environment to use the version compiled with Compiler_A/version_A while the other contains the +necessary functions to do that for Compiler_B/version_B. Again, two more lines are needed to implement the hierarchy. +E.g., for ``moduleroot/Compiler/Compiler_A/version_A/MPI_C/version_C.lua``: + +```Lua +family('MPI') +prepend_path('MODULEPATH', 'moduleroot/MPI/Compiler_A/version_A/MPI_C/version_C') +``` + +Finally two vesions of the ``version_E.lua`` file are needed, one to prepare the environment for using the +package with Compiler_A anmd MPI_C and one for using the package with Compiler_B and MPI_C. However, these +are just regular modules and no additions are needed to work for the hierarchy. + +Both EasyBuild and Spack support Lmod hierarchies and with these tools it is also fairly automatic to create +different versions of the module files for each compiler and MPI library used to build the application. When +hand-writing modules it may be more interesting to have a generic module which would work for all those cases +and that is also possible with Lmod. Lmod does have a range of *introspection functions* that a module can use +to figure out its name, version and place in the module tree. All that would be needed is that the various +instances of the module file are at the correct location in the module tree and link to the generic file which +can be outside the module tree. In fact, this feature is used on LUMI to implement the modules that load a +particular version of the hardware for a particular section of LUMI. + +--- + +## Finding modules + +TODO: module spider, module help, module whatis, module keyword and where do they get their information. + +--- + +## Further reading + + +--- + +*[[next: The Cray Programming Environment]](1_03_CPE.md)* diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md new file mode 100644 index 00000000..ff054066 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -0,0 +1,30 @@ +# The HPE Cray Programming Environment + +*[[back: The Lmod module system]](1_02_Lmod.md)* + +--- + +On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated +as Cray PE). The environment provides several tools, including compilers, communication libraries, +optmized math libraries and various other libraries, analyzers and debuggers. + +The Cray PE is made available through *environment modules* tha allow to select particular versions of +tools and to configure the environment in a flexible way. + +--- + +## Cray PE components + + +--- + +## Configuring the Cray PE through modules + +--- + +## Further reading + + +--- + +*[[next: Terminology]](1_04_terminology.md)* diff --git a/docs/2022-CSC_and_LO/1_03_terminology.md b/docs/2022-CSC_and_LO/1_04_terminology.md similarity index 99% rename from docs/2022-CSC_and_LO/1_03_terminology.md rename to docs/2022-CSC_and_LO/1_04_terminology.md index ec085b0c..3450c8f0 100644 --- a/docs/2022-CSC_and_LO/1_03_terminology.md +++ b/docs/2022-CSC_and_LO/1_04_terminology.md @@ -1,6 +1,6 @@ # Terminology -*[[back: The HPE Cray Programming Environment and Lmod]](1_02_CPE_and_Lmod.md)* +*[[back: The HPE Cray Programming Environment]](1_03_CPE.md)* --- diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md index 6b4c8613..6c99a412 100644 --- a/docs/2022-CSC_and_LO/index.md +++ b/docs/2022-CSC_and_LO/index.md @@ -12,8 +12,9 @@ See that site for [credits to the contributors of those tutorials](https://easyb - [Part I: **Introduction to EasyBuild on Cray systems**](1_00_part1_intro.md) - [What is EasyBuild?](1_01_what_is_easybuild.md) - - [The HPE Cray Programming Environment and Lmod](1_02_CPE_and_Lmod.md) - - [Terminology](1_03_terminology.md) + - [The Lmod module system](1_02_Lmod.md) + - [The HPE Cray Programming Environment and Lmod](1_03_CPE.md) + - [Terminology](1_04_terminology.md) - [Installation](installation.md) *(hands-on)* - [Configuration](configuration.md) *(hands-on)* - [Basic usage](basic_usage.md) *(hands-on)* From 1692a80267639f5f658faabe01f85ac27139dd0f Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 10 Mar 2022 14:40:29 +0100 Subject: [PATCH 005/159] Corrections to the README file. --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index f7ed0418..e84c0d54 100644 --- a/README.md +++ b/README.md @@ -22,11 +22,11 @@ which makes it very easy to preview the result of the changes you make locally. make preview - or + or mkdocs serve - Visit http://127.0.0.1:8000 to see the local live preview of the changes you make. + Visit http://127.0.0.1:8000 to see the local live preview of the changes you make. * If you prefer building a static preview you can use ``make`` or ``mkdocs build``, which should result in a ``site/`` subdirectory that contains the rendered documentation. From 82dbfaed8a9aaab80a9eae8f26d67f610e03a9bc Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 10 Mar 2022 14:42:32 +0100 Subject: [PATCH 006/159] Corrections to the overview for part I. --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md index e381f45d..bdfddfc5 100644 --- a/docs/2022-CSC_and_LO/1_00_part1_intro.md +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -2,10 +2,10 @@ *[[back to start page]](index.md)* -*(Tue March 4th 2021, 9am-12 CET)* - - [What is EasyBuild?](1_01_what_is_easybuild.md) -- [Terminology](1_02_terminology.md) +- [The Lmod module system](1_02_Lmod.md) +- [The HPE Cray Programming Environment and Lmod](1_03_CPE.md) +- [Terminology](1_04_terminology.md) - [Installation](installation.md) *(hands-on)* - [Configuration](configuration.md) *(hands-on)* From 29d5ba53b484a2199afa21a77b38eac08c7ead09 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 11 Mar 2022 14:16:34 +0100 Subject: [PATCH 007/159] Further work on the Lmod section. --- docs/2022-CSC_and_LO/1_02_Lmod.md | 314 ++++++++++++++++++++++++++++-- 1 file changed, 299 insertions(+), 15 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md index 91cb7571..04ccff8e 100644 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -4,15 +4,6 @@ --- -On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated -as Cray PE). The environment provides several tools, including compilers, communication libraries, -optmized math libraries and various other libraries, analyzers and debuggers. - -The Cray PE is made available through *environment modules* tha allow to select particular versions of -tools and to configure the environment in a flexible way. - ---- - ## Modules *Module* is a massively overloaded term in (scientific) software and IT in general @@ -40,12 +31,16 @@ module files but also offers a high degree of compatibility with Tcl-based modul developed for Environment Modules fia a translation layer and some API translation. The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no -packages or official support for Environment Modules 4 or 5. On LUMI, Lmod was selected as -the module tool. At the user level, Lmod and the various versions of Emvironment Modules -have very simmilar commands for managing the environment, but with different options. -The commands for searching for modules are very different though so if you are not familiar -with Lmod and its commands for users, it is worthwile to read the -[LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). +packages or official support for Environment Modules 4 or 5. At the user level, +Environment Modules 3.2 and Lmod have many commands in common, but with different options. +Lmod also has some powerful features that are lacking in Environment Modules 3.2. + +!!! Note "The Cray PE on LUMI" + On LUMI, Lmod was selected as the module tool. One area where there are significant + differences between Environment Modules 3.2 (and also the newer versions) and Lmod is + in the commands for discovering modules on the system. If you are not familiar with Lmod + and its commands for users, it is worthwile to read the + [LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). --- @@ -199,8 +194,297 @@ particular version of the hardware for a particular section of LUMI. ## Finding modules +In a hierarchical setup, not all modules are available at login. This implies that a user cannot use +``module avail`` to discover which software is available on the system. To this end Lmod has powerful +search commands. It is important to understand how these commands work to ensure that the proper information +is included in the module files to improve discoverability of software. + +!!! Note "Documentation in the LUMI documentation" + Extensive information on search commands with examples of how to use them on LUMI can be found + in the [LUMI documentation]([https://)](https://docs.lumi-supercomputer.eu/), in + [the computing section, "Module environment page", "Finding modules" section](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/#finding-modules). + TODO: module spider, module help, module whatis, module keyword and where do they get their information. +### module spider command + +The available modules at any point in time are often only a subset of all installed modules on a +system. However, Lmod provides the ``module spider`` command to search for a module with a given name +among all installed modules and to tell you how this module can be loaded (i.e., which other modules +need to be loaded to make the module available). + +The ``module spider`` command has three levels, producing different outputs: + + 1. ``module spider`` without further arguments will produce a list of all + installed software and show some basic information about those packages. + Some packages may have an ``(E)`` behind their name and will appear in blue + (in the default colour scheme) which means that they are part of a different + package. These are called *extensions* of packages or modules. + This is explained a little further in this page. + + Note that ``module spider`` will also search in packages that are hidden from + being displayed. These packages can be loaded and used. However administrators + may have decided to hide them + either because they are not useful to regular users or because they think that + they will rarely or never be directly loaded by a user and want to avoid + overloading the module display. + + 2. ``module spider `` will search for the specific package. This + can be the name of a module, but it will also search some other information + that can be included in the modules. The search is also case-insensitive. + E.g., on LUMI + ```bash + module spider GNUplot + ``` + will show something along the lines of + ``` + ------------------------------------------------------------------ + gnuplot: + ------------------------------------------------------------------ + Description: + Gnuplot is a portable command-line driven graphing utility + + Versions: + gnuplot/5.4.2-cpeCray-21.08 + gnuplot/5.4.2-cpeGNU-21.08 + ``` + so even though the capitalisation of the name was wrong, it can tell us that + there are two versions of gnuplot. The ``cpeGNU-21.08`` and ``cpeCray-21.08`` + tell that the difference is the compiler that was used to install gnuplot, + being the GNU compiler (PrgEnv-gnu) and the Cray compiler (PrgEnv-cray) + respectively. + + In some cases, if there is no ambiguity, `module spider` will actually + already produce help about the package, which is the next level. + + 3. `module spider /` will show more help information + about the package, including information on which other modules need to be + loaded to be able to load the package. E.g., + ```bash + module spider git/2.35.1 + ``` + will return something along the lines of + ``` + ------------------------------------------------------------------- + git: git/2.35.1 + ------------------------------------------------------------------- + Description: + Git is a free and open source distributed version control + system + + You will need to load all module(s) on any one of the lines below + before the "git/2.35.1" module is available to load. + + CrayEnv + LUMI/21.12 partition/C + LUMI/21.12 partition/D + LUMI/21.12 partition/G + LUMI/21.12 partition/L + + Help: + ``` + (abbreviated output). Note that it also tells you which other modules need + to be loaded. You need to choose the line which is appropriate for you and + load all modules on that line, not the whole list of in this case 9 + modules. + +!!! failure "Known issue" + The Cray PE uses Lmod in an unconventional manner with the hierarchy not + build fully in the way Lmod expects. As a consequence Lmod is not always + able to generate the correct list of modules that need to be loaded to make + a package available, and the list of ways to make a module available may + also be incomplete. + + The problem is somewhat aggrevated on LUMI because the Cray PE hierarchy sits + next to the hierarchy of the software stack as the Cray PE is installed + separately and hence cannot be integrated in the way the Lmod developer had + in mind. + + +#### Module extensions + +Certain packages, e.g., Python, Perl or R, get a lot of their functionality through +other packages that are installed together with them and extend the functionity, +e.g., NumPy and SciPy for Python. Installing all those packages as separate modules +to make it easy to see if they are installed or not on a system would lead to an +overload of modules on the system. + +Similary, admins of a software stack may chose to bundle several libraries or tools +that are often used together in a single module (and single installation directory), +e.g., to reduce module clutter but also to reduce the length of the search paths for +binaries, libraries or manual pages to speed up loading of applications. + +Lmod offers a way to make those individual packages installed in a module discoverable +by declaring them as *extensions* of the module. The ``module spider`` command will +search for those too. + + 1. ``module spider`` without further arguments: The output may contain lines similar + to + ``` + ----------------------------------------------------------------------- + The following is a list of the modules and extensions currently available: + ----------------------------------------------------------------------- + Autoconf: Autoconf/2.71 (E) + + CMake: CMake/3.21.2 (E), CMake/3.22.2 (E) + ``` + which tells that ``Autoconf`` and ``CMake`` are not available as modules themselves + but as extensions of another module, and it also tells the versions that are available, + though that list may not be complete (and is not always complete for modules either + as it is limited to one line of output). + +2. ``module spider `` will search for extensions also. E.g., + ``` + module spider CMake + ``` + on LUMI will return something along the lines of + ``` + ----------------------------------------------------------------------- + CMake: + ----------------------------------------------------------------------- + Versions: + CMake/3.21.2 (E) + CMake/3.22.2 (E) + ``` + (output abbreviated). + This tells that there is no ``CMake`` module on the system but that two versions + of ``CMake`` are provided in another module. + + 3. `module spider /` will show more information on the + extension, including which module provides the extension and which other modules + have to be loaded to make that module available. E.g., on LUMI, + ``` + module spider CMake/3.22.2 + ``` + will output something along the lines of + ``` + ----------------------------------------------------------------------- + CMake: CMake/3.22.2 (E) + ----------------------------------------------------------------------- + This extension is provided by the following modules. To access the + extension you must load one of the following modules. Note that any + module names in parentheses show the module location in the software + hierarchy. + + buildtools/21.12 (LUMI/21.12 partition/L) + buildtools/21.12 (LUMI/21.12 partition/G) + buildtools/21.12 (LUMI/21.12 partition/D) + buildtools/21.12 (LUMI/21.12 partition/C) + buildtools/21.12 (CrayEnv) + ``` + (output abbreviated and slightly reformatted for readability). This tells that + ``CMake/3.22.2`` is provided by the ``bvuildtools/21.12`` module and that there + are 5 different ways to make that package available. + +??? Bug "Restrictions with older Lmod versions" + At the time of development of this tutorial, Cray is still using the pretty old + 8.3.1 version of Lmod. Even though extensions were supported since Lmod version 8.2.5, + Lmod 8.3.1 has several problems: + + - It is not possible to hide extensions in the output of ``module avail``, a feature + that only became available in version 8.5. This may be annoying to many users as + the extension list of packages such as Python, R and Perl can be very long (the + default EasyBuild installation of R contains on the order of 600 packages). + + For that reason on LUMI extensions are only used for some modules. + + - ``module avail`` also shows extensions for modules that are not available which + makes no sense. This bug was only corrected in Lmod 8.6.13 and 8.6.14. + + +### module keyword + +Another search command that is sometimes useful is `module keyword`. It really +just searches for the given word in the short descriptions that are included in +most module files and in the name of the module. The output is not always +complete since not all modules may have a complete enough short description. + +Consider we are looking for a library or package that supports MP3 audio +encoding. +```bash +module keyword mp3 +``` +will return something along the lines of +``` +---------------------------------------------------------------- + +The following modules match your search criteria: "mp3" +---------------------------------------------------------------- + + LAME: LAME/3.100-cpeCray-21.08, LAME/3.100-cpeGNU-21.08 + LAME is a high quality MPEG Audio Layer III (mp3) encoder +``` +though the output will depend on the version of Lmod. This may not be the most +useful example on a supercomputer, but the library is in fact needed to be able +to install some other packages even though the sound function is not immediately +useful. + +??? bug "Know issue: Irrelevant output" + At the moment of the development of this tutorial, this command actually + returns a lot more output, referring to completely irrelevant extensions. + This is a bug in the HPE-Cray-provided version of Lmod (8.3.1 at the time + of development of this tutorial) that was only solved in more recent versions. + + +### module avail + +The `module avail` command is used to show only available modules, i.e., modules +that can be loaded directly without first loading other modules. It can be used +in two ways: + + 1. Without a further argument it will show an often lengthy list of all + available modules. Some modules will be marked with `(D)` which means that + they are the default module that would be loaded should you load the module + using only its name. + + 2. With the name of a module (or a part of the name) it will show all modules + that match that (part of) a name. E.g., + ```bash + module avail gnuplot + ``` + will show something along the lines of + ``` + ------ EasyBuild managed software for software stack LUMI/21.08 on LUMI-L ------ + gnuplot/5.4.2-cpeCray-21.08 gnuplot/5.4.2-cpeGNU-21.08 (D) + + Where: + D: Default Module + (output abbreviated). + ``` + but + ```bash + module avail gnu + ``` + will show you an often lengthy list that contains all packages with gnu + (case insensitive) in their name or version. + + +### Getting help + +One way to get help on a particular module has already been discussed on this +page: ``module spider /`` will produce help about the package as +soon as it can unambiguously determine the package. It is the only command that +can produce help for all installed packages. The next two commands can only +produce help about available packages. + +A second command is ``module whatis`` with the name or name and version of a +module. It will show the brief description of the module that is included in +most modules on the system. If the full version of the module is not given, it +will display the information for the default version of that module. + +The third command is ``module help``. Without any further argument it will display +some brief help about the module command. However, when used as +``module help `` or ``module help /`` it will produce help for either the +default version of the package (if the version is not specified) or the +indicated version. + + +### Implementation details + +**TODO** Tell about the extensions, whatis and help functions in module file and maybe a +little bit about how module spider works. + + --- ## Further reading From 2eae491dafec302ffced90b4bb62c7b343e39f67 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 14 Mar 2022 14:38:34 +0100 Subject: [PATCH 008/159] Finished the Lmod section. --- docs/2022-CSC_and_LO/1_02_Lmod.md | 74 +++++++++++++++++++++++++++++-- 1 file changed, 71 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md index 04ccff8e..2d1d5121 100644 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -41,6 +41,7 @@ Lmod also has some powerful features that are lacking in Environment Modules 3.2 in the commands for discovering modules on the system. If you are not familiar with Lmod and its commands for users, it is worthwile to read the [LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). + Some of those commands are also discussed on this page. --- @@ -204,7 +205,6 @@ is included in the module files to improve discoverability of software. in the [LUMI documentation]([https://)](https://docs.lumi-supercomputer.eu/), in [the computing section, "Module environment page", "Finding modules" section](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/#finding-modules). -TODO: module spider, module help, module whatis, module keyword and where do they get their information. ### module spider command @@ -481,14 +481,82 @@ indicated version. ### Implementation details -**TODO** Tell about the extensions, whatis and help functions in module file and maybe a -little bit about how module spider works. +Lmod works by executing the module file. However, the actions of all Lmod-defined +functions will depend upon the mode in which Lmod is executing the module function, +and the module file can also detect in which mode it is executing. +Modes include "load", "unload"but also "spider". E.g., when te mode is "load", the +``setenv`` function will set an environment variable to the indicated value while in +"unload" mode that environment variable will be unset, and in "spider" mode the +environment variable is left untouched. The working of ``prepend_path``, a function +that modifies PATH-style variables, depends a bit on how Lmod is configured (as it is +possible to work with reference counts), but in its most basic mode, ``prepend_path`` +will add a given directory to a given PATH-style environment variable (or move it to +the front of the PATH-style variable if the directory is already in there), while in +"unload" mode that specific directory will be removed from the PATH (but no error will +be generated should the directory that is used as the argument not be part of the path +in that PATH-style variable). When the mode is "spider", the function has special behaviour +if it is used to change the ``MODULEPATH``. It will then note the change and add that +directory to the list of directories that has to be searched for module files. +This makes ``module spider`` a very expensive command as it may have to traverse a lot +of directories and has to execute all module files in there. Therefor Lmod will build +a so-called spider cache which can be pre-built in the system for certain directories +and otherwise will be build in the user's home directory (in the ``.lmod.d/.cache`` +subdirectory). Our experience is that this cache tends to be rather fragile, +in particular on Cray systems (and that has been confirmed in discussions with +people with access to some other Cray systems) so from time to time Lmod fails to +note changes to the modules, at least when using commands such as ``module spider``. +The actual loading and unloading of the module is not based on cached information. + +Lmod has several functions that can be used in module files to provide the information +that Lmod needs for the search-related and help commands. + +The ``help`` function defines the long help text used by ``module help`` and by +``module spider`` as soon as there is no ambiguity anymore about which module is +being searched for. + +The ``whatis`` function is used to provide short information about a module. That +information is then used by ``module whatis`` and ``module keyword`` , but also +for brief information shown by ``module spider`` when multiple modules or versions +of modules are found by the command. A module file can contain multiple ``whatis`` +commands and the Lmod manuel suggests to use those lines as a kind of database +record. See, e.g., +[the Lmod manual page with module file examples](https://lmod.readthedocs.io/en/latest/100_modulefile_examples.html?highlight=whatis). +One such example is +```Lua +whatis("Name: valgrind") +whatis("Version: 3.7.0") +whatis("Category: tools") +whatis("URL: http://www.valgrind.org") +whatis("Description: memory usage tester") +``` +It is not all that important to include all those lines in a module file, but some of +those lines get a special treatment from Lmod. The line starting with ``Description`` +is used by ``module spider`` to provide some brief information about the module if it +is not totally resolved. This comes with a limitation though: It is not show for each +version of the module, so ideally all "GROMACS" modules should contain the same +description line and use other lines to provide further information about what +distinguished a particular version. +Likewise the ``Category:`` line is used by the ``spider_decoration`` hook that can be +used to add decoration to the spider level 1 output. +All in all the ``whatis`` function if often overlooked in Lmod-based module functionx +but it is a very useful function to include in the proper way in module files. + +A third function that provides information to the search commands is ``extensions``. +It can be used to list up the extensions supported by the module. The argument list +may seem strange as it takes only a single argument, a string of comma-separated ``extension/version`` +elements, but that is because the number of arguments to a function is limited in +Lua and that limit can actually be met easily by modules for Python, Perl or R packages. --- ## Further reading +- [Lmod documentation](https://lmod.readthedocs.io/en/latest/index.html) +- [Lmod on LUMI in the LUMI documentation](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/) +- [Documentation of Environment Modules 5](https://modules.readthedocs.io/en/latest/), + an alternative to Lmod (though not currently supported by HPE Cray) + --- From 8ca50b89053ea9723a0b4f358f83f49db7351714 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 14 Mar 2022 15:36:44 +0100 Subject: [PATCH 009/159] Continued merging of the new CSC tutorial in the structure of the repository. --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 2 +- docs/2022-CSC_and_LO/index.md | 58 ++++++++++++++++++++++-- mkdocs.yml | 8 ++++ 3 files changed, 62 insertions(+), 6 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md index bdfddfc5..7b138670 100644 --- a/docs/2022-CSC_and_LO/1_00_part1_intro.md +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -4,7 +4,7 @@ - [What is EasyBuild?](1_01_what_is_easybuild.md) - [The Lmod module system](1_02_Lmod.md) -- [The HPE Cray Programming Environment and Lmod](1_03_CPE.md) +- [The HPE Cray Programming Environment](1_03_CPE.md) - [Terminology](1_04_terminology.md) - [Installation](installation.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md index 6c99a412..49f4a4bb 100644 --- a/docs/2022-CSC_and_LO/index.md +++ b/docs/2022-CSC_and_LO/index.md @@ -1,19 +1,67 @@ -# EasyBuild tutorial for LUST - -Overview page of the introductory tutorial on [EasyBuild](https://easybuild.io) for the CSC -and Local Organizations who want to contribute to the LUMI EasyBuild setup. +# EasyBuild tutorial for CSC and the LUMI consortium This tutorial is based extensively on the more generic EasyBuild tutorials build by the EasyBuild community and maintained on [the EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). See that site for [credits to the contributors of those tutorials](https://easybuilders.github.io/easybuild-tutorial/#contributors). +## Scope + +This is an introductory tutorial to [EasyBuild](https://easybuild.io), +a command line tool for installing (scientific) software on High Performance Computing (HPC) systems. +These tutorials are specifically for EasyBuild as implemented on [LUMI](https://lumi-supercomputer.eu) +and has been developed for CSC and the Local Organisations within the LUMI consortium. Yet +much of the material is useful to a broader community of EasyBuild users on Cray systems +or even EasyBuild users in general. +For more generic EasyBuild tutorials, see the [EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). + +This tutorial aims to explain the core concepts of EasyBuild, +get you started with using it, make you familiar with some of the features it provides, +and show how it is used on LUMI to maintain the central software stacks and offer the users +an easy environment to install packages on top of the central stack and thus create their own +customised environment. + +Through hands-on exercises and demos, you will learn how EasyBuild can help you +to get scientific software installed in an efficient way. + + +## Intended audience + +This tutorial is primarily intended for people new to EasyBuild, but even if you're already familiar +with the project it could be interesting to step through it. + +Our main target audience includes: + +- Application experts in LUST and the local organizations who want to contribute to the + software stack on LUMI or support their users +- Developers who want to make their developments available to LUMI users +- Advanced users who want to customize available build recipes or develop their own recipes + + +## Prerequisites + +We expect you to be (a little bit) familiar with: + +- using a Linux command line interface +- the (absolute) basics of compiling software from source + +EasyBuild requires: + +- GNU/Linux (any distribution) +- Python 2.7 or 3.5+ +- an environment modules tool (see the ``module`` command). On LUMI we use [Lmod](https://lmod.readthedocs.io), + a modern environment modules tool implemented in Lua. + +However, the LUMI version of the tutorial is currently specifically for the Cray Programming Environment which is not +freely available, so unless you have access to a system with this environment you cannot really do local development. + + ## Contents - [Part I: **Introduction to EasyBuild on Cray systems**](1_00_part1_intro.md) - [What is EasyBuild?](1_01_what_is_easybuild.md) - [The Lmod module system](1_02_Lmod.md) - - [The HPE Cray Programming Environment and Lmod](1_03_CPE.md) + - [The HPE Cray Programming Environment](1_03_CPE.md) - [Terminology](1_04_terminology.md) - [Installation](installation.md) *(hands-on)* - [Configuration](configuration.md) *(hands-on)* diff --git a/mkdocs.yml b/mkdocs.yml index 1e3ef2c9..bcdd0889 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -33,6 +33,14 @@ nav: - Contributing to EasyBuild: 2022-isc22/contributing.md - Comparison with other tools: 2022-isc22/comparison_other_tools.md - Archive: + - "CSC (spring '22)": + - (overview): 2022-CSC_and_LO/index.md + - Introduction to EasyBuild: + - (overview): 2022-CSC_and_LO/1_00_part1_intro.md + - What is EasyBuild?: 2022-CSC_and_LO/1_01_what_is_easybuild.md + - The Lmod module system: 2022-CSC_and_LO/1_02_Lmod.md) + - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_03_CPE.md + - Te0rminology: 2022-CSC_and_LO/1_04_terminology.md - "ISC'21 (25 June 2021)": - (overview): 2021-isc21/index.md - Practical info: 2021-isc21/practical_info.md From 71966b53dd603a9641664e25343c1790fea72642 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 14 Mar 2022 18:27:32 +0100 Subject: [PATCH 010/159] Tutorial page about the Cray PE, and correction of a typo. --- docs/2022-CSC_and_LO/1_03_CPE.md | 251 +++++++++++++++++++++++++++++++ mkdocs.yml | 2 +- 2 files changed, 252 insertions(+), 1 deletion(-) diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md index ff054066..8de332a4 100644 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -15,16 +15,267 @@ tools and to configure the environment in a flexible way. ## Cray PE components +- **Cray Compiling Environment (CCE)**: + + CCE consists of Cray compilers performing code analysis during compilation to generate highly optimized code. + Supported languages include Fortran, C and C++, and UPC (Unified Parallel C). + The Cray C/C++ compiler is basically the Clang/LLVM compiler with a back-end configured by HPE Cray that also + adds some Cray-specific options while the Cray Fortran compiler uses an HPE-Cray specific front-end + with a back-end based on LLVM. The Fortran compiler supports most of Fortran 2018 and tends to be + considerably stricter than the GNU or Intel Fortran compilers. + + The CCE compilers will also support OpenMP offload to NVIDIA and AMD GPUs but that is still work in progress + at the time of the development of this tutorial. The Fortran compiler (but not the C/C++ compiler) also supports + OpenACC for offloading to GPU. + + The classic pre-LLVM Cray C/C++ compiler is not available on Cray EX systems. + +- **Cray Scientific and Mathematical Libraries (CSML)**: + + A set of high performance libraries providing portability for scientific applications, sparse and dense linear + algebra (BLAS, LAPACK, ScaLAPACK) and fast Fourier transforms (FFTW). + + There is a special (more limited) version of the library with GPU-optimized routines. + + CSML also includes the HDF5 and netCDF libraries, in versions with and without MPI. + +- **Cray Message Passing Toolkit**: + + A collection of software libraries used to perform data transfers between nodes running in parallel applications. + It includes the Message Passing Interface (MPI) and OpenSHMEM parallel programming models. + + The HPE Cray MPI implementation is based on the MPICH3 sources from ANL. However, some algorithms are tweaked + specifically for HPE Cray hardware and the Cray interconnects. + +- **Cray Environment Setup and Compiling Support**: + + Libraries supporting code compilation and setting up the development environment, including compiler drivers. + +- **Cray Performance and Analysis Tools**: + + Tools to analyze the performance and behavior of programs that are run on Cray systems, and a Performance API (PAPI). + +- **Cray Debugging Support Tools**: + + Debugging tools, including `gdb4hpc` and `Valgrind4hpc`. + +- **Third-party compilers**: + + The Cray PE also supports several third-party compilers. Not all compilers are supported on all systems: + + - The GNU C/C++ and Fortran compilers will be present on most systems with the Cray PE. + + - The AMD AOCC C/C++ and Fortran compilers are supported on systems with AMD CPUs. + + - The AMD ROCm compilers (sometimes known as AOMP) are supported on systems with AMD GPU technology. + + - The Intel C/C++ and Fortran compilers (also the oneAPI versions) are supported on systems with Intel + processors. HPE Cray does not include them in licenses for AMD CPUs. + + - The NVIDIA HPC Toolkit compilers are supported on systems with NVIDIA GPUs. + --- ## Configuring the Cray PE through modules +Compilers in the Cray Programming Environment are almost always used through the Cray compiler wrappers. In fact, +until recently, some compilers could not be used without the wrappers as they failed to even detect their own +include files and some components. The behaviour of the wrapper will depend on the other modules that are +loaded in the environment. E.g., there is no separate wrapper for MPI as the MPI header files and libraries are +automatically activated through the regular wrappers as soon as a Cray MPI module is loaded. + +Multiple releases of the HPE Cray PE can be installed simultaneously on the system and users can mix-and-match +components. However, it is also possible to load only components for a specific release of the PE. +Cray PE versions have version numbers of the form ``yy.mm``, e.g., ``22.02`` for the version released in +February of 2022. However, each of the components have their own version number and it is not easy to see +which version of a component came with which version(s) of the Cray PE. + +Below we only discuss those modules that are important when building software with EasyBuild. Debuggers, +profilers, etc., are not included in the list. + + +### The PrgEnv-* modules + +The usual way to initialise the Cray PE is by loading one of the ``PrgEnv-*`` modules. This module will then +load the appropriate compiler, compiler wrappers and other libraries. Some of the components that will be +loaded are configured through the ``/etc/cray-pe.d/cray-pe-configuration.sh`` file, so the list of modules +may depend on the actual system that you are using. + +The Cray PE supports the following ``PrgEnv-*`` modules. On LUMI, only the first three are currently available: + +| Module | Compiler module | What? | +|:---------------|-----------------|:--------------------------------------------| +| PrgEnv-cray | cce | The Cray Compiling Environment compilers | +| PrgEnv-gnu | gcc | The GNU compilers | +| PrgEnv-aocc | aocc | AMD compilers for CPU-only systems | +| PrgEnv-amd | rocm | AMD ROCm compilers for GPU systems | +| PrgEnv-intel | intel | The Intel compilers | +| PrgEnv-nvidia | nvidia | NVIDIA HPC toolkit compilers (formerly PGI) | + +All ``PrgEnv-*`` modules belong to the same module family. Hence Lmod will automatically unload any +already loaded ``PrgEnv-*`` module when you load a different one. + + +### Selecting the version through the cpe module + +Loading one of the ``cpe/yy.mm`` modules (e.g., ``cpe/22.02``) has the following effects: + +- It sets the default versions of each of the Cray PE modules to the version that comes with that + particular release of the Cray PE. E.g., + + ``` bash + module load cpe/22.02 + module load cce + ``` + + would load that version of the ``cce`` compiler that comes with the 22.02 release of the Cray PE. + +- It will reload all already loaded Cray PE modules and switch them over to the version corresponding + to that particular release of the Cray PE. + +???+ Bug "Limitations and bugs" + Due to the way Lmod works and implementation bugs in the ``cpe`` modules, loading the ``cpe`` module + does not always have the desired effect. + + - The Cray PE sets the default version of each module by adding a file to the list of files in the + ``LMOD_MODULERCFILE`` environment variable. This is because Lmod does not re-evaluate the visibility + of modules and the internal list of default version immediately when the value of ``LMOD_MODULERCFILE`` + is changed but only the next time the ``module`` command is executed. Hence + ``` bash + module load cpe/22.02 ; module load cce + ``` + and + ``` + module load cpe/22.02 cce + ``` + do not have the same effect. In the first version, the version of ``cce`` loaded is the version that + corresponds to the 22.02 release of the Cray PE. In the second case however the default version of the + ``cce`` module is determined by whatever list of default modules was used when calling the ``module`` + command so may or may not be the one of the 22.02 release. + + - Loading the ``cpe`` module after loading the other Cray PE modules also does not always have the + desired effect in many versions of the Cray PE. This is because of a bug in the ``cpe `` module that + reloads the modules in the wrong order which may trigger the reload of a module with whatever version + was the default when the ``module`` command was called rather than the version the the ``cpe`` module + intends to (re-)load. + + +### The compiler wrapper module craype + +The ``craype`` module loads the compiler wrappers. There is only one set of compiler wrappers for all compilers. +Which compiler will be called, which libraries will be included, but also processor and GPU target options will +be used, is all determined by other modules. Hence it is in principle possible to use a single Makefile for +a project and still reconfigure the build by loading certain modules. + + +### Target modules + +The targets for CPU and GPU optimization, the network library for MPI bt also some other compiler options, +can be set through target modules: + +- ``craype-x86-*`` (and similar options can be expected on ARM-based systems) set the target for CPU + optimisations. For LUMI, the ``crape-x86-rome``, ``craype-x86-milan`` and ``craype-x86-trento`` modules + are relevant. + + This can also be used to cross-compile to a different target architecture unless the compiler target + gets overwritten by a compiler flag added to the command line through the Makefile or other means, + something that unfortunately happens more and more often in faulty software installation procedures. + +- ``creype-accel-*`` sets the target for OpenMP offload (and likely other technologies in the future). E.g., + loading ``craype-accel-amd-gfx90a`` tells the compilers to target AMD MI200 family GPUs, while loading + ``craype-accel-host`` tells the compiler to use the CPU instead (according to the documentation, the latter + is for ``PrgEnv-cray`` only). + +- ``craype-network-*`` selects the communication library to be used by Cray MPICH. On Slingshot 11 EX systems, + only ``craype-network-ofi`` is supported, but Slingshot 10 EX systems also offer support for UCX through the + ``craype-network-ucx`` module. + +- The ``craype-hugepages*`` modules enable Cray Huge Pages support. To fully enable this support they have to + be used at link-time and at run-time. At link time, support is compiled into the binary wile at run-time they + are used to set the actual size of the huge pages. + + The ``craype-hugepages*`` modules are not supported by all compilers. E.g., the AOCC compiler does not support + huge pages at the moment, and loading the module at link time will cause an error message from the linker. + + +### The compiler modules + +The compiler modules have already been discussed with the ``PrgEnv-*`` modules above. The different regular +compiler modules also all belong to the same family so no two different compilers can be loaded simultaneously +and Lmod will automatically unload the other compiler when a new one is loaded. + + +### The MPI modules + +To load the Cray MPI libraries, both one of the ``craype-network-*`` modules and a compiler module has to be +loaded as the MPI libraries are both network- and compiler specific. + +For some unknown reason, the MPI module for the libfabric (``craype-network-ofi``) transport is called ``cray-mpich`` +while the library for the UCX transport (``craype-network-ucx``) is called ``craype-network-ucx``. As a result, +the MPI module fails to reload automatically when switching between both transports, but it does reload automatically +when switching compilers. + +Loading an MPI module will also automatically configure the regular compiler wrappers to compile with support for +that MPI module. However, the libfabric and UCX versions of the MPI library have compatible interfaces, so it is +always possible to swap between those versions at runtime. + + +### The Cray Scientific libraries + +The Cray Scientific Libraries are loaded through the ``cray-libsci`` module (or ``cray-libsci_acc`` for the GPU +versions). Loading this module makes the BLAS, LAPACK, and ScaLAPACK libraries available, and also the Cray IRT +(Iterative Refinement Toolkit), a Cray-specific library. It will also configure the compiler wrappers to link +with these libraries, so no additional include or link options are needed. + + +### The ``fftw`` module + +As FFTW is a third party library, it gets its own module (``cray-fftw``) and is not included in the ``cray-libsci`` module. + + +--- + +## Some unexpected behaviour of the Cray PE + +On Cray EX systems, dynamic linking is the preferred way of linking applications. Whith that comes some +unexpected behaviour of the Cray modules. EasyBuild users expect that at run time the versions of the +libraries that are used, are the ones from the modules that are loaded. This is not always the case +for the runtime libraries of the Cray PE. By default the Cray PE will use a default set of libraries +that is determined by the system default version of the Cray PE (which is set by the sysadmins, not +determined by the ``cpe`` module). This can only be avoided by either using\ +rpath-linking (which is also special in the Cray PE as the wrappers activate rpath linking if the +environment variable ``CRAY_ADD_RPATH`` is defined and set to ``yes``) or by manually adjusting +the library path after loading the modules: +``` bash +export LD_LIBRARY_PATH=${CRAY_LD_LIBRARY_PATH}:$LD_LIBRARY_PATH +``` +The latter cannot be easily automated in modulefiles. Any technique that can be used (without actually +reworking the Cray PE modules) has nasty side effects in some scenarios. + +The net result of this feature is that some reproducibility of the results is lost. Programs will +react differently if the system default version of the Cray PE is changed as that will change the +set of default libraries that will be used at runtime unless rpath-linking is used or users +redfine ``LD_LIBRARY_PATH``. + + --- ## Further reading +- [LUMI documentation: "Developing" section](https://docs.lumi-supercomputer.eu/development/) +- The Cray PE is mostly documented through man pages. There used to be some documentation on the + Cray web site also but the documentation system got reworked after the merger with HPE. + The documentation is now in the [HPE Support Centre](https://support.hpe.com/) where it + is very difficult to find the right version of the documents. +- The [PE-Cray](https://github.com/PE-Cray) GitHub project also provides some additional + documentation, including + - [some whitepapers](https://github.com/PE-Cray/whitepapers), + - [documentation for Cray OpenSMHEMX](https://github.com/PE-Cray/cray-openshmemx) and + - [information about Cray DSMML](https://github.com/PE-Cray/cray-dsmml) + though much of that information ia also not updated regularly anymore. + --- *[[next: Terminology]](1_04_terminology.md)* diff --git a/mkdocs.yml b/mkdocs.yml index bcdd0889..f397c9e4 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -40,7 +40,7 @@ nav: - What is EasyBuild?: 2022-CSC_and_LO/1_01_what_is_easybuild.md - The Lmod module system: 2022-CSC_and_LO/1_02_Lmod.md) - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_03_CPE.md - - Te0rminology: 2022-CSC_and_LO/1_04_terminology.md + - Terminology: 2022-CSC_and_LO/1_04_terminology.md - "ISC'21 (25 June 2021)": - (overview): 2021-isc21/index.md - Practical info: 2021-isc21/practical_info.md From 91e7c8614f13f3ed421ec79602917647d793b230 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 16 Mar 2022 15:23:45 +0100 Subject: [PATCH 011/159] Corrected a typo in mkdocs.yml --- mkdocs.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index f397c9e4..6ee7889e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -38,7 +38,7 @@ nav: - Introduction to EasyBuild: - (overview): 2022-CSC_and_LO/1_00_part1_intro.md - What is EasyBuild?: 2022-CSC_and_LO/1_01_what_is_easybuild.md - - The Lmod module system: 2022-CSC_and_LO/1_02_Lmod.md) + - The Lmod module system: 2022-CSC_and_LO/1_02_Lmod.md - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_03_CPE.md - Terminology: 2022-CSC_and_LO/1_04_terminology.md - "ISC'21 (25 June 2021)": From 766a0f20556c65e6d7004571b8bae1acf766fa00 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 16 Mar 2022 15:24:08 +0100 Subject: [PATCH 012/159] Adding an example of implementing a hierarchy to the Lmod section of the tutorial. --- docs/2022-CSC_and_LO/1_02_Lmod.md | 191 ++++++++++++++++++++++++++++++ 1 file changed, 191 insertions(+) diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md index 2d1d5121..0caa8a47 100644 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -548,6 +548,197 @@ elements, but that is because the number of arguments to a function is limited i Lua and that limit can actually be met easily by modules for Python, Perl or R packages. +--- + +## Some warnings about writing modulefiles + +***This section is very technical and only useful if you want to manually implement +modules that depend on each other one way or another.*** + +Lmod cannot guarantee that the order of unloading the modules will be the inverse of +the order in which they were loaded. Moreover, unloading a module is not done by reverting +stored actions done when loading the module, but by executing the modulefile again +in a mode that reverts certain actions. This can lead to subtle problems when modulefiles +communicate with each other through environment variables or by detecting which other +modules are loaded. These problems are usually solved by using a proper hierarchy +and basing actions of modulefiles on their position in the hierarchy. + +One case where passing information between modules through environment variables will +go wrong is when that environment variable is subsequently used to compute a directory +name that should be added to a PATH-like variable. Assume we have two versions of +a ``MyPython`` module, e.g., ``MyPython/2.7.18`` and ``MyPython/3.6.10``. That module then +sets an environment variable ``PYTHON_API_VERSION`` to either ``2.7`` or ``3.6``. +Next we have a module ``MyPythonPackage`` that makes a number of Python packages available +for both Python modules. However, as some Python packages have to be installed separately +for each Python version, it does so by adding a directory to the environment variable +``PYTHON_PATH`` that contains the version which it gets by using the Lua function +``os.getenv`` to request the value of ``PYTHON_API_VERSION``. + +One problem becomes clear in the following scenario: +``` bash +module load MyPython/2.7.18 +module load MyPythonPackage/1.0 +module load MyPython/3.6.10 +``` +The ``module load MyPythonPackage`` will find the environment variable ``PYTHON_PACKAGE_API`` +with the value ``2.7`` as set by ``module load MyPython/2.7.18`` and hence add the directory +for the packages for version 2.7 to ``PYTHON_PATH``. The ``module load MyPython/3.6.10`` +command will trigger two operations because of the *"one name rule"*: First it will +automatically unload ``MyPython/2.7.18`` (which will unset ``PYTHON_API_VERSIUON``) and +next it will load ``MyPython/3.6.10`` which will set ``PYTHON_API_VERSION`` to ``3.6``. +However, ``MyPythonPackage`` is not reloaded so the ``PYTHON_PATH`` variable will now point +to the wrong directory. One would be tempted to think that the easy fix for the user would +be to reload ``MyPythonPackage/1.0``: +``` bash +module load MyPythonPackage/1.0 +``` +Because of the *"one name rule"* this will again trigger an unload followed by a load +of the module. The problem is in the unload. One would expect that first unloading +``MyPythonPackage`` would remove the 2.7 directory from the ``PYTHON_PATH`` but it +will not. Lmod does not remeber that last time it loaded ``MyPythonPackage`` it added +the 2.7 directory to ``PythonPath``. Instead it will execute the commands in the +modulefile and reverse certain commands. Since ``PYTHON_API_VERSION`` has now the value +``3.6``, it will try to remove the directory for version ``3.6`` which is not in the +``PYTHON_PATH``. The subsequent load will then add the 3.6 directory to ``PYTHON_PATH`` +so the environment variable now contains both directories. + +In this simple case, a ``module purge`` after the first two ``module load`` commands would +still work as Lmod is able to figure out the right order to unload modules, but in more +complicated examples this may also go wrong. However, a ``module purge`` command after +the load of ``MyPython/3.6.10`` would also fail to clean up the environment as it would +still fail to remove the 2.7 directory from ``PYTHONPATH``. + +??? Note "Running the example" + To test this example for yourself, create a directory and add that directory to + the ``MODULEPATH`` using ``module use``. In that directory, create the following + subdirectories and files: + 1. ``MyPython/2.7.18.lua`` with content: + ``` lua + LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) + setenv( 'PYTHON_API_VERSION', '2.7' ) + ``` + 2. ``MyPython/3.6.10.lua`` with content: + ``` lua + LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) + setenv( 'PYTHON_API_VERSION', '3.6' ) + ``` + 3. ``MyPythonPackage/1.0.lua`` with content: + ``` lua + LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) + LmodMessage( 'PYTHON_API_VERSION = ' .. ( os.getenv( 'PYTHON_API_VERSION' ) or '') ) + prepend_path( 'PYTHON_PATH', 'someroot/python' .. + ( os.getenv( 'PYTHON_API_VERSION' ) or 'TTT' ) .. '/packages' ) + LmodMessage( 'PYTHON_PATH = ' .. ( os.getenv( 'PYTHON_PATH' ) or '') ) + ``` +??? Note "Solution with a hierarchy" + The better way in Lmod to implement the above scenario would be in a module hierarchy. + + Just to show the power of Lmod introspection functions combined with a proper hierarchy + we present a solution using only one version of the code for ``MyPython`` and one version + of the code for ``MyPythonPackages``. + + It is best to start from a clean directory. In that directory, create: + + 1. The files ``level1/MyPython/2.7.18.lua`` and ``level1/MyPython/3.6.10.lua``, + both with the same contents: + ``` lua + LmodMessage( '\nIn ' .. myModuleFullName() .. ' in mode ' .. mode() ) + + local api_version = myModuleVersion():match( '(%d+%.%d+)%..*' ) + + -- Set the variable PYTHON_API_VERSION but not for internal use in the modules. + setenv( 'PYTHON_API_VERSION', api_version ) + + local module_root = myFileName():match( '(.*)/level1/' .. myModuleFullName() ) + prepend_path( 'MODULEPATH', pathJoin( module_root, 'level2/PythonAPI', api_version ) ) + LmodMessage( 'MODULEPATH is now\n ' .. + os.getenv( 'MODULEPATH' ):gsub( ':', '\n ' ) ) + ``` + + 2. The files ``level2/PythonAPI/2.7/MyPythonPackage/1.0.lua`` and + ``level2/PythonAPI/3.6/MyPythonPackage/1.0.lua``, both with the contents: + ``` lua + LmodMessage( '\nIn ' .. myFileName() .. ' in mode ' .. mode() ) + + local python_api_version = myFileName():match( '.*/level2/PythonAPI/([^/]+)/.*' ) + LmodMessage( 'Detected Python API version from hierarchy: ' .. python_api_version ) + LmodMessage( 'Detected Python API version from environment: ' .. + ( os.getenv( 'PYTHON_API_VERSION' ) or '' ) ) + + prepend_path( 'PYTHON_PATH', 'someroot/python' .. python_api_version .. '/packages' ) + + LmodMessage( 'PYTHON_PATH = ' .. (os.getenv( 'PYTHON_PATH' ) or '') ) + ``` + + Now add the ``level1`` subdirectory to ``MODULEPATH``, e.g., if you're in the directory + containing the ``level1`` and ``level2`` subdirectories: + ``` bash + module use $PWD/level1 + ``` + and then try the following commands: + ``` bash + module avail + module load MyPython/2.7.18 + module avail + module load MyPythonPackage/1.0 + module load MyPython/3.6.10 + ``` + and pay attention to the output. + + Initially ``module avail`` will show none of the ``MyPythonPackage`` modules. These are + installed modules but not available modules. ``module load MyPython/2.7.18`` will set the + environment variable ``PYTHON_API_VERSION`` to ``2.7`` and also add a directory to the front + of the ``MODULEPATH`` with the directory name ending on ``level2/PythonAPI/2.7``. Now + ``module avail`` will show the ``MyPythonPackage/1.0`` module. + + The ``MyPythonPackage`` shows two ways to get the version of the Python API to use for + determining the right directory to add to ``PYTHON_PATH``. The fragile way is to enquire + the value of the environment variable ``PYTHON_API_VERSION`` set by loading ``MyPython/2.7.18``. + The more robust way is to use the Lmod introspection function ``myFileName()`` which returns + the full path and file name of the module file that is executing, and extracting the version + from the path with a pattern matching function. In this particular situation both computed + values are the same so both would have worked to correctly add + ``somedir/python2.7/packages`` to the front of ``PYTHON_PATH``. + + The next command, ``module load MyPython/3.6.10`` triggers a chain of events. + + First, Lmod notices that there is already a module loaded with the same name, so it will + unload ``MyPython/2.7.18``. This will unset the environment variable ``PYTHON_API_VERSION`` + (the inverse operation of ``setenv``) and will remove the ``.../level2/PythonAPI/2.7`` + subdirectory from the ``MODULEPATH`` (the inverse action of ``prepend_path``). + + Now due to + the change of the ``MODULEPATH`` the ``MyPythonPackage/1.0`` module which was loaded from + ``.../level2/PythonAPI/2.7`` is no longer available so Lmod will continue with unloading + that module. The interesting bit now is that ``PYTHON_API_VERSION`` is unset. So had we + computed the name of the directory to add to ``PYTHON_PATH`` using the value of that + environment variable, the module would have failed to compute the correct directory name + to remove so ``prepend_path`` would have left the ``PYTHON_PATH`` environment variable + untouched. However, by computing that value from the directory of the modulefile, we get + the right value and can correctly remove ``somedir/python2.7/packages`` from ``PYTHON_PATH``. + Lmod will also remember that the module was only unloaded due to a change in the + ``MODULEPATH`` and not because a user explicitly unloaded the module. I.e., it considers + the module as deactivated but not as unloaded. + + Lmod proceeds with loading the ``MyPython/3.6.10`` module. This will now set + ``PTHON_API_VERSION`` to ``3.6`` and add a directory with name ending on + ``level2/PythonAPI/3.6`` to ``MODULEPATH``. + + Things are not done yet though. As the ``MODULEPATH`` has changed, Lmod looks at its list + of deactivated modules and notices that a different version of ``MyPythonPackage/1.0`` is + now available. Hence it will now automatically load that module from the + ``.../level2/PythonAPI/3.6`` subdirectory so that that module now correctly detects + that ``somedir/python3.6/package`` should be added to ``PYTHON_PATH``. + + Hence at the end of the cycle we have again a correctly configured environment with no + trace of the ``2.7`` version that was loaded initially and with no action required from + the user to ensure that ``MyPythonPackage`` is unloaded and reloaded to ensure the + correct configuration. + + This idea is used on LUMI to implement the various versions of the software stack with + for each software stack also optimised binaries for each of the node types. + + --- ## Further reading From 70889a5c7f94f98ea6c46707cf3f204790d192ff Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 10:08:23 +0100 Subject: [PATCH 013/159] More information about hierarchy in the Cray PE, and definition of toolchain in Terminology. --- docs/2022-CSC_and_LO/1_03_CPE.md | 3 +++ docs/2022-CSC_and_LO/1_04_terminology.md | 21 ++++++++++++++++++--- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md index 8de332a4..42eb197e 100644 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -233,6 +233,9 @@ with these libraries, so no additional include or link options are needed. As FFTW is a third party library, it gets its own module (``cray-fftw``) and is not included in the ``cray-libsci`` module. +The ``cray-fftw`` module can only be loaded if one of the processor target modules (the ``craype-x86-*`` modules) +is loaded first. + --- diff --git a/docs/2022-CSC_and_LO/1_04_terminology.md b/docs/2022-CSC_and_LO/1_04_terminology.md index 3450c8f0..19f1e01f 100644 --- a/docs/2022-CSC_and_LO/1_04_terminology.md +++ b/docs/2022-CSC_and_LO/1_04_terminology.md @@ -28,22 +28,37 @@ can also be included. Additional toolchain components are usually special-purpose libraries: -- an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/)); +- an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/) + or vendor-specific MPI implementations like Intel MPI in a toolchain based on the Intel compilers); - libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), - [LAPACK](http://performance.netlib.org/lapack/)); + [LAPACK](http://performance.netlib.org/lapack/), [ScaLAPACK](http://www.netlib.org/scalapack/)); These libraries are included because they tend to provide a common API, so mixing them in a single executable would cause name conflicts when linking. - a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); +These libraries are included because they are some of the most popular libraries in scientific software, +but also because they provide an API that is available through different implementations. Mixing +implementations in a single build is a recipe for problems as the various implementations might +define the same symbols. + A toolchain that includes all of these libraries is referred to as a **full toolchain**, while a **subtoolchain** is a toolchain that is missing one or more of these libraries. A **compiler-only toolchain** only consists of compilers (no additional libraries). +The toolchain concept also maps nicely on the Programming Environment concept in the Cray +Programming Environment. Each ``PrgEnv-*`` module in fact provides a full toolchain in a +typical Cray PE installation, except for the FFTW library. Note that systems could configure +the contents of a ``PrgEnv-*`` module differently as the MPI library and scientific library are +optional. In the Cray PE, the MPI library is provided by the ``cray-mpic`` module, +the BLAS, LAPACK and ScaLAPACK libraries by the ``cray-libsci`` module (for CPU-only nodes) +and the FFTW library through the ``cray-fftw`` module. + Many EasBuild toolchains are organised in a hierarchy. Each toolchain can use components compiled -with itself or with one of the subtoolchains. +with itself or with one of the subtoolchains. The Cray toolchains however are not currently +organised in a hierarchy. ### System toolchain From e101d7baf65bdd4b27a3824b80d762916106393b Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 10:14:56 +0100 Subject: [PATCH 014/159] Proposed change to mkdocs.yml to support mermaid.js graphs. --- mkdocs.yml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index 6ee7889e..c5103eaa 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -127,7 +127,11 @@ markdown_extensions: # notes, warnings, hints, ... - admonition # code blocks with syntax highlighting, graphs - - pymdownx.superfences + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format # clickable details - pymdownx.details # tabbed contents From df081ae9122a3f29095cb038c7509c647589309d Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 10:33:37 +0100 Subject: [PATCH 015/159] Add a mermaid.js diagram with the toolchain hierarchy. --- docs/2022-CSC_and_LO/1_04_terminology.md | 42 +++++++++++++++++------- 1 file changed, 30 insertions(+), 12 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_04_terminology.md b/docs/2022-CSC_and_LO/1_04_terminology.md index 19f1e01f..ebb99963 100644 --- a/docs/2022-CSC_and_LO/1_04_terminology.md +++ b/docs/2022-CSC_and_LO/1_04_terminology.md @@ -12,7 +12,7 @@ It is important to be familiar with these terms, so we'll briefly cover them one --- -## *Toolchains* +## Toolchains A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. @@ -82,6 +82,8 @@ because they are widely adopted by the EasyBuild community. The `foss` toolchain consists of all open source components (hence the name: "FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), [ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). +In recent versions (since 2021a), [FlexiBLAS](https://www.mpi-magdeburg.mpg.de/projects/flexiblas) +is used as the BLAS library, with OpenBLAS and LAPACK as the backend. The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. @@ -89,11 +91,27 @@ controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.c Roughly every 6 months, a new version of these common toolchains is agreed upon in the EasyBuild community, after extensive testing. +The common toolchains are also organised in a toolchain hierarchy: + +```mermaid +graph TD; +A[SYSTEM] --> B[GCCcore]; +B --> C[GCC]; +B --> D[iccifort]; +C --> E[gompi: Adds OpenMPI]; +D --> F[iimpi: Adds Intel MPI]; +D --> G[imkl: Adds Intel MKL]; +E --> H[foss: Adds OpenBLAS, LAPACK, ScaLAPACK, FFTW]; +F --> I[intel]; +G --> I; +``` + More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). + --- -## *Modules* +## Modules *Module* is a massively overloaded term in (scientific) software and IT in general (kernel modules, Python modules, and so on). @@ -121,7 +139,7 @@ session such that the corresponding software installation can be used. --- -## *EasyBuild framework* +## EasyBuild framework The EasyBuild *framework* consists of a set of Python modules organised in packages (``easybuild.framework``, ``easybuild.toolchains``, ``easybuild.tools``, etc.) that collectively form **the core of EasyBuild**, @@ -137,7 +155,7 @@ the necessary functionality to facilitate this. --- -## *Easyblocks* +## Easyblocks An *easyblock* is **a Python module that implements a specific software installation procedure**, and can be viewed as a plugin to the EasyBuild framework. Easyblocks can be either *generic* or *software-specific*. @@ -158,7 +176,7 @@ in the [``easybuild-easyblocks`` repository on GitHub](https://github.com/easybu --- -## *Easyconfig parameters* +## Easyconfig parameters An **easyconfig parameter** specifies a particular aspect of a software installation that should be performed by EasyBuild. @@ -188,7 +206,7 @@ only supported by certain easyblocks, and only make sense for particular (types --- -## *Easyconfig files* +## Easyconfig files *Easyconfig files* (or *easyconfigs* for short), are **simple text files written in Python syntax that specify what EasyBuild should install**. @@ -211,7 +229,7 @@ the EasyBuild community maintains a large (and growing) collection of easyconfig --- -## *Easystack files* +## Easystack files [**Easystack files**](https://docs.easybuild.io/en/latest/Easystack-files.html) are a new concept in EasyBuild, providing a way to define a *software stack* @@ -227,7 +245,7 @@ which means it is subject to change in future EasyBuild releases, and may be pro --- -## *Extensions* +## Extensions *Extensions* is the collective term we use for **additional software packages that can be installed on top of another software package**. Common examples are *Python packages*, *R libraries*, and *Perl modules*. @@ -244,7 +262,7 @@ Extensions can be installed in different ways: --- -## *Dependencies* +## Dependencies A *dependency* is a common term in the context of software. It refers to **a software package that is either strictly required by other software, or that can be leveraged to @@ -266,7 +284,7 @@ but we will discriminate build-only dependencies. --- -## *Toolchains* +## Toolchains A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. @@ -319,7 +337,7 @@ More information on these toolchains is available [in the EasyBuild documentatio --- -## *Modules* +## Modules *Module* is a massively overloaded term in (scientific) software and IT in general (kernel modules, Python modules, and so on). @@ -347,7 +365,7 @@ session such that the corresponding software installation can be used. --- -## *Bringing it all together* +## Bringing it all together The EasyBuild **framework** leverages **easyblocks** to automatically build and install (scientific) software, potentially including additional **extensions**, using a particular compiler **toolchain**, From 0fd57c5308e361611205bf330777bf39efe05bd4 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 15:02:25 +0100 Subject: [PATCH 016/159] Finished terminology session, extended the installation section with LUMI-specific material. --- docs/2022-CSC_and_LO/1_04_terminology.md | 141 ++---- docs/2022-CSC_and_LO/1_05_installation.md | 543 ++++++++++++++++++++++ docs/2022-CSC_and_LO/index.md | 4 +- mkdocs.yml | 1 + 4 files changed, 573 insertions(+), 116 deletions(-) create mode 100644 docs/2022-CSC_and_LO/1_05_installation.md diff --git a/docs/2022-CSC_and_LO/1_04_terminology.md b/docs/2022-CSC_and_LO/1_04_terminology.md index ebb99963..9c643579 100644 --- a/docs/2022-CSC_and_LO/1_04_terminology.md +++ b/docs/2022-CSC_and_LO/1_04_terminology.md @@ -109,38 +109,13 @@ G --> I; More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). ---- - -## Modules - -*Module* is a massively overloaded term in (scientific) software and IT in general -(kernel modules, Python modules, and so on). -In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. - -[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept -on HPC systems: it is a way to specify changes that should be made to one or more -[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a -[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file -is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or -[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, -and specifies which environment variables should be updated, and how (append, -prepend, (re)define, undefine, etc.) upon loading the environment module. -Unloading the environment module will restore the shell environment to its previous state. - -Environment module files are processed via a **modules tool**, of which there -are several conceptually similar yet slightly different implementations. -The Tcl-based [Environment Modules](https://sourceforge.net/projects/modules/) implementation, and -[Lmod](https://lmod.readthedocs.io), a more recent Lua-based implementation (which also supports module -files written in Tcl syntax), are the most commonly used ones. - -**Environment module files are automatically generated for each software installation** by EasyBuild, -and *loading* a module results in changes being made to the environment of the current shell -session such that the corresponding software installation can be used. - --- ## EasyBuild framework +EasyBuild is written in Python and organised in three layers, represented by the three blocks in the logo: +the EasyBuild framework, easyblocks and Easyconfig files. + The EasyBuild *framework* consists of a set of Python modules organised in packages (``easybuild.framework``, ``easybuild.toolchains``, ``easybuild.tools``, etc.) that collectively form **the core of EasyBuild**, and is developed in the [``easybuild-framework`` repository on GitHub](https://github.com/easybuilders/easybuild-framework). @@ -165,12 +140,20 @@ multiple different software packages. Commonly used examples include the ``Confi which implements the ubiquitous ``configure``-``make``-``make install`` procedure, and the ``PythonPackage`` easyblock that can be used to install a Python package. +*In Spack this functionality is provided in the core of the package.* + A **software-specific easyblock** implements an installation procedure that is specific to a particular software packages. Infamous examples include the easyblocks we have for ``GCC``, ``OpenFOAM``, ``TensorFlow``, ``WRF``, ... The installation procedure performed by an easyblock can be controlled by defining **easyconfig parameters** in an easyconfig file. +*In Spack, this functionality is provided by the ``package.py`` files for the various packages. However, EasyBuild +does not need a software-specific easyblock for every package as for many packages the combination of a +generic easyblock with the much simpler description in the easyconfig file is sufficient. +The software-specific easyblocks can be a bit troublesome on Cray PE systems. Some easyblocks contain code that +is specific for certain compilers or toolchains and do not support all compilers or all toolchains.* + A collection of (generic and software-specific) easyblocks is developed by the EasyBuild community in the [``easybuild-easyblocks`` repository on GitHub](https://github.com/easybuilders/easybuild-easyblocks). @@ -227,6 +210,17 @@ In the [``easybuild-easyconfigs`` repository on GitHub](https://github.com/easyb the EasyBuild community maintains a large (and growing) collection of easyconfig files, for a wide range of (scientific) software. +*EasyBuild does not have an equivalent of the concretiser of Spack. EasyBuild always starts from a complete, +precise configuration of a package to install, determined by the easyconfig parameters and possibly a +custom easyblock. Both approaches have their pros and cons. The concretiser in Spack combined with the +configuration of a software package via command line parameters makes it easy to quickly generate a +number of different configurations of a package to test performance or to check which combinations work +well. But one also has to be careful to not cause an explosion of modules and software installations, and +take care of using the appropriate features to ensure that one can generate the same configurations again. +The precise configurations of EayBuild can sometimes slow down things as they may need a lot of file editing, +but when it comes to building a reproducible software stack and sharing precise building instructions, this +approach may be preferred.* + --- ## Easystack files @@ -273,95 +267,14 @@ There are three main types of dependencies for computer software: * a **build dependency** is only required when building/installing a software package; once the software package is installed, it is no longer needed to *use* that software (examples: `CMake`, `pkg-config`); -* a **run-time dependency** (often referred to simply as *dependency*) is a software package that is - required to *use* (or *run*) another software package (example: `Python`); * a **link-time dependency** is somewhere in between a build and runtime dependency: it is only needed when *linking* a software package; it can become either a build or runtime dependency, depending on exactly how the software is installed (example: `OpenBLAS`); +* a **run-time dependency** (often referred to simply as *dependency*) is a software package that is + required to *use* (or *run*) another software package (example: `Python`); -The distinction between link-time and run-time dependencies is mostly irrelevant for this tutorial, -but we will discriminate build-only dependencies. - ---- - -## Toolchains - -A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, -which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. - -We refer to the different parts of a toolchain as **toolchain components**. - -The *compiler component* typically consists of [C](https://en.wikipedia.org/wiki/C_(programming_language)), -[C++](https://en.wikipedia.org/wiki/C%2B%2B), and [Fortran](https://en.wikipedia.org/wiki/Fortran) -compilers in the context of HPC, but additional compilers (for example, -a [CUDA](https://developer.nvidia.com/cuda-zone) compiler for -[GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units) software) -can also be included. - -Additional toolchain components are usually special-purpose libraries: - -* an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/)); -* libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), - [LAPACK](http://performance.netlib.org/lapack/)); -* a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); - -A toolchain that includes all of these libraries is referred to as a **full toolchain**, while -a **subtoolchain** is a toolchain that is missing one or more of these libraries. -A **compiler-only toolchain** only consists of compilers (no additional libraries). - -### System toolchain - -The **`system` toolchain** is a special case which corresponds to using the compilers and libraries -*provided by the operating system*, rather than using toolchain components that were installed using EasyBuild. - -It used sparingly, mostly to install software where no actual compilation is done or -to build a set of toolchain compilers and its dependencies, since the versions of the system compilers -and libraries are beyond the control of EasyBuild, which could affect the reproducibility of the installation. - -### Common toolchains - -The `foss` and `intel` toolchains are also known as the *common toolchains*, -because they are widely adopted by the EasyBuild community. - -The `foss` toolchain consists of all open source components (hence the name: -"FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), -[ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). - -The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version -controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. - -Roughly every 6 months, a new version of these common toolchains is agreed upon -in the EasyBuild community, after extensive testing. - -More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). - ---- - -## Modules - -*Module* is a massively overloaded term in (scientific) software and IT in general -(kernel modules, Python modules, and so on). -In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. - -[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept -on HPC systems: it is a way to specify changes that should be made to one or more -[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a -[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file -is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or -[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, -and specifies which environment variables should be updated, and how (append, -prepend, (re)define, undefine, etc.) upon loading the environment module. -Unloading the environment module will restore the shell environment to its previous state. - -Environment module files are processed via a **modules tool**, of which there -are several conceptually similar yet slightly different implementations. -The Tcl-based [Environment Modules](https://sourceforge.net/projects/modules/) implementation, and -[Lmod](https://lmod.readthedocs.io), a more recent Lua-based implementation (which also supports module -files written in Tcl syntax), are the most commonly used ones. - -**Environment module files are automatically generated for each software installation** by EasyBuild, -and *loading* a module results in changes being made to the environment of the current shell -session such that the corresponding software installation can be used. +EasyBuild currently doesn't treat link-time dependencies differently from run-time dependencies +and hence has no specific easyconfig parameter for them. --- @@ -378,4 +291,4 @@ An **easystack file** can be used to specify a collection of software to install --- -*[[next: Installation]](installation.md)* +*[[next: Installation]](1_05_installation.md)* diff --git a/docs/2022-CSC_and_LO/1_05_installation.md b/docs/2022-CSC_and_LO/1_05_installation.md new file mode 100644 index 00000000..163262ea --- /dev/null +++ b/docs/2022-CSC_and_LO/1_05_installation.md @@ -0,0 +1,543 @@ +# Installing EasyBuild + +*[[back: Terminology]](1_04_terminology.md)* + +--- + +Before deep diving into the wonderful world of EasyBuild and getting your hands dirty with +the hands on exercises coming up in this tutorial, you will need to install EasyBuild. + +In this section we outline a couple of different ways of doing this, and also the things you should pay attention to. +By the end, you will have a functional EasyBuild installation that you can use +for the remainder of this tutorial. + +!!! Remark "EasyBuild on LUMI" + + EasyBuild is already installed for each of the LUMI software stacks. The LUMI software + stacks are based on the Cray PE (using the Cray-specific toolchains). + Moreover, in case you want to install a full shadow stack rather than build on top + of one of the LUMI software stacks, scripts are provided to initialise a new version + of the LUMI software stack based on a specific version of the Cray PE, and those + scripts also take care of the installation of EasyBuild. + + Hence this section of the tutorial is only useful for people who want more background + knowledge or who want to experiment with other EasyBuild toolchains. + + +!!! summary + + * *Requirements*: Linux, Python 2.7 or 3.5+, environment modules tool (Lmod is recommended). + * *Installation methods*: + * ``pip install easybuild`` + * installing EasyBuild with EasyBuild + * development setup + * Verify EasyBuild installation using `eb --version` and `eb --help`. + + +--- + +## Requirements + +### Linux + +The main target platform for EasyBuild is *Linux*, since it is heavily focused on installing software +on HPC system where Linux is dominant operating system (to the point where 100% of the current Top500 list of supercomputers are running Linux). + +EasyBuild is also compatible with *macOS*, but the included easyconfig files are heavily focused +on Linux so most software installations supported by EasyBuild won't work out-of-the-box on macOS. +You can still use the EasyBuild command line interface on macOS for other tasks though, +like development or contributing back to the project. + +### Python + +EasyBuild is implemented in Python, and is **compatible with both Python 2.7 and Python 3.5+** +(that is, Python 3.5 or a newer version of Python 3). + +To check which Python version you have, use: + +```shell +python -V +``` + +*No additional Python packages are required by EasyBuild*, the ones that come with the standard +Python distribution are sufficient. Some additional Python packages *can* be leveraged for specific features. +More on that later. + +### Environment modules tool + +**An *environment modules tool* is required for using EasyBuild.** + +We strongly recommend using [Lmod](https://lmod.readthedocs.io), a Lua-based modern environment +modules implementation and the most commonly used modules tool in the EasyBuild community. +Other implementations, like the original Tcl-based one, are also supported. + +To check if you have a modules tool installed, use: + +``` +module --version +``` + +If this produces output that starts with something like "`Modules based on Lua: Version 8.3.1`" you have Lmod installed, +which is the default modules tool used by EasyBuild, and you are all set for installing and using EasyBuild. +Any sufficiently recent Lmod version (8.x or even 7.x) should be fine. + +If you see output that starts with a line like "`VERSION=3.2.10`" or "`Modules Release 4.5.0`", +you have the original Tcl-based environment modules tool installed, either in its original +C-based imnplementation (``VERSION=3.2.10``) or the new Tcl-based implementation developed at CEA +(``Modules Release 4.5.0``) +and *EasyBuild will need to be configured to use it after installation*. + +If the `module` function is not defined either you do not have a modules tool installed +or your environment is not properly set up to use it. In this case, +please refer to the EasyBuild documentation here for more information. + +HPE Cray supports both the old-style Environments Modules (based on version 3.2.10) and Lmod. On LUMI, +we use Lmod as the default module system for users, and EasyBuild is configured to use that one. + +--- + +## EasyBuild as a Python package + +EasyBuild consists of a number of interdependent Python packages, +and is available via both GitHub at https://github.com/easybuilders, +as well as via the standard Python Package Index (PyPI) at +https://pypi.org/project/easybuild. + +As you may be aware the Python packaging ecosystem is bit convoluted, +which is reflected in the many different ways in which you can install a Python package. +In addition, EasyBuild is packaged in 3 components (framework, easyblocks, easyconfigs) +which slightly complicates the installation. + +
+ +Nevertheless, you don't need to be a rocket scientist to install EasyBuild (and even if you are, +that's OK too), so don't worry. + +You can install EasyBuild just like you can install any other Python software that is released +via the standard *Python Package Index* (PyPI), through one of the standard Python installation tools +(like `pip`, `virtualenv`, `pipenv`, ...). +And since EasyBuild is a software installation tool in its own right, we actually have a couple +of additional tricks up our sleeve! + +### Python 2 or Python 3? + +For EasyBuild it does not matter much whether you install it on top of Python 2 or Python 3. +Since version 4.5, Python 3 does offer some optional features (requiring additional packages) +that are not available with Python 2. +Since Python 2 is end-of-life, +we strongly recommend using Python 3 if you have the choice. + +By default EasyBuild will use the `python` command to run, +but you can control this if needed via ``$EB_PYTHON``. + + +## Installing EasyBuild + +We present three methods for installing EasyBuild. +It is up to you which one you prefer, both result a fully functional EasyBuild installation. + +Time to get your hands dirty! + +* Method 1: Using ``pip`` +* Method 2: installing EasyBuild with EasyBuild +* Method 3: Development setup + +--- + +### Method 1: Using `pip` + +Since EasyBuild is released as a [Python package on PyPI](https://pypi.org/project/easybuild) +you can install it using `pip`, the most commonly used tool for installing Python packages. + +You may need to take additional steps after the installation, depending on the exact installation command. + +!!! note + There are various other ways of installing Python packages, which we won't cover here. + If you are familiar with other tools like `virtualenv` or `pipenv`, feel free to use those + instead to install EasyBuild. + +#### Running `pip install` + +Installing EasyBuild with `pip` is as simple as running the following command: + +```shell +pip install easybuild +``` + +However, you may need to slightly change this command depending on the context and your personal preferences: + +* To install EasyBuild *system-wide*, you can use `sudo` (if you have admin privileges): + ```shell + sudo pip install easybuild + ``` + +* To install EasyBuild *in your personal home directory*, you can use the `--user` option: + ```shell + pip install --user easybuild + ``` + This will result in an EasyBuild installation in `$HOME/.local/`. + +* To install EasyBuild in a *specific directory* you can use the `--prefix` option: + ```shell + pip install --prefix _PREFIX_ easybuild + ``` + In this command, you should replace '`_PREFIX_`' with the location where you want to have EasyBuild installed + (for example, `$HOME/tools` or `/tmp/$USER`). + +#### `pip` vs `pip3` + +On systems where both Python 2 and Python 3 are installed you may also have different `pip` commands +available. Or maybe `pip` is not available at all, and only "versioned" `pip` commands like `pip3` are +available. + +If you (only) have `pip3` available, you can replace `pip` with `pip3` in any of the `pip install` commands +above. + +If you want to ensure that you are using the ``pip`` installation that corresponds to the Python 3 installation +that you intend to use, you can use ``python3 -m pip`` rather than ``pip3``. + +#### Updating your environment + +If you used the `--user` or `--prefix` option in the `pip install` command, +or if you installed EasyBuild with a `pip` version that does not correspond +to your default Python installation, +you will need to update your environment to make EasyBuild ready for use. +This is not required if you did a system-wide installation in a standard location with the default Python version. + +!!! note + Keep in mind that you will have to make these environment changes again if you start a new shell session. + To avoid this, you can update one of the shell startup scripts in your home directory (`.bashrc` for example). + +#### Updating ``$PATH`` + +Update the `$PATH` environment variable to make sure the `eb` command is available: +```shell + +export PATH=_PREFIX_/bin:$PATH +``` +**Replace '`_PREFIX_`' in this command** with the directory path where EasyBuild was installed into +(use `$HOME/.local` if you used `pip install --user`). + +This is not required if you installing EasyBuild in a standard system location. + +You can check with the `which eb` command to determine whether or not you need to update the ``$PATH`` environment variable. + +#### Updating ``$PYTHONPATH`` + +If you installed EasyBuild to a non-standard location using `pip install --prefix`, +you also need to update the Python search path environment variable +[`$PYTHONPATH`](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH) to instruct Python where +it can find the EasyBuild Python packages. + +This is not required if you used the `--user` option, since Python will automatically consider +`$HOME/.local` when searching for installed Python packages, or if you installed EasyBuild in a standard +system-wide location. + +Update `$PYTHONPATH` by running a command like: + +```shell + +export PYTHONPATH=_PREFIX_/lib/pythonX.Y/site-packages:$PYTHONPATH +``` + +Here, you need to replace the `X` and `Y` with the major and minor version of your Python installation, +which you can determine by running `python -V`. +For example, if you are using Python 3.6, make sure you are using `/python3.6/` in the command to update `$PYTHONPATH`. + +And of course, you again need to **replace '`_PREFIX_`'** with the installation prefix where EasyBuild was installed +into. + +For example: + +```shell +# update $PYTHONPATH if EasyBuild was installed in $HOME/tools with Python 3.6 +export PYTHONPATH=$HOME/tools/lib/python3.6/site-packages:$PYTHONPATH +``` + +#### Setting ``$EB_PYTHON`` + +If you want to control which Python version is used to run EasyBuild, +you can specify the name or the full path to the `python` command that should be used by the `eb` command +via the `$EB_PYTHON` environment variable. + +This may be required when you installing EasyBuild with a version of `pip` that does not correspond +with the default Python version. + +For example, to ensure that `eb` uses `python3.6`: + +```shell +export EB_PYTHON=python3.6 +``` + +#### Setting ``$EB_VERBOSE`` + +To determine which `python` commands are being considered by the `eb` command, +you can define the `$EB_VERBOSE` environment variable. For example: + +```shell +$ EB_VERBOSE=1 eb --version +>> Considering 'python3.6'... +>> 'python3' version: 3.6.8, which matches Python 3 version requirement (>= 3.5) +>> Selected Python command: python3 (/usr/bin/python3.6) +>> python3.6 -m easybuild.main --version +This is EasyBuild 4.3.3 (framework: 4.3.3, easyblocks: 4.3.3) on host example +``` + +!!! Tip "Debugging startup problems" + + Using `ÈB_VERBOSE` is useful if EasyBuild fails to start up and complains it + cannot find a suitable Python executable. Rather often the error message is + caused by a failure elsewhere in EasyBuild. + + +--- + +### Method 2: Installing EasyBuild with EasyBuild + +!!! note + This section covers an alternative method for installing EasyBuild. + + If you already have EasyBuild installed through ``pip``, + you can skip ahead to the next section. + +If you prefer having EasyBuild available through an environment module file, +you can consider installing EasyBuild with EasyBuild. This can be done in 3 steps: + +* Step 1: Installing EasyBuild with ``pip`` into a temporary location (only needed if EasyBuild is not installed yet) +* Step 2: Using EasyBuild to install EasyBuild as a module +* Step 3: Loading the EasyBuild module + +!!! note + A [bootstrap script](https://docs.easybuild.io/en/latest/Installation.html#bootstrapping-easybuild) + is available that automates this procedure, + but is known to be problematic in some contexts, and is not being actively + maintained anymore. + + As a result, we do not recommend using the bootstrap script anymore. + + +#### Step 1: Installing EasyBuild into a temporary location + +If you don't have EasyBuild installed yet, you need to install it in a temporary location first. +The recommended way of doing this is using ``pip``. + +For example, to install EasyBuild into a subdirectory `/tmp/$USER` using the default Python 3 version: + +```shell +# pick installation prefix, and install EasyBuild into it +export EB_TMPDIR=/tmp/$USER/eb_tmp +python3 -m pip install --ignore-installed --prefix $EB_TMPDIR easybuild +``` + +```shell +# update environment to use this temporary EasyBuild installation +export PATH=$EB_TMPDIR/bin:$PATH +export PYTHONPATH=$(/bin/ls -rtd -1 $EB_TMPDIR/lib*/python*/site-packages | tail -1):$PYTHONPATH +export EB_PYTHON=python3 +``` + +#### Step 2: Using EasyBuild to install EasyBuild + +Once you have a working (recent) temporary EasyBuild installation, you can use it to +install EasyBuild as a module. Usually this is done in the location where you would +like to install other software too. + +You can use the ``eb --install-latest-eb-release`` command for this, +combined with the ``--prefix`` option to control which directories are used by EasyBuild for the installation. + +For example, to install the latest version of EasyBuild as a module into ``$HOME/easybuild``: + +```shell +eb --install-latest-eb-release --prefix $HOME/easybuild +``` + +!!! note + You may see a harmless deprecation warning popping up when performing this installation, just ignore it. + +#### Step 3: Loading the EasyBuild module + +Once step 2 is completed, you should be able to load the module that was generated alongside +the EasyBuild installation. You will need to do this every time you start a new shell session. + +First, make the module available by running the following command (which will update the module search path +environment variable `$MODULEPATH`): + +```shell + +module use _PREFIX_/modules/all +``` + +**Replace '`_PREFIX_`'** with the path to the directory that you used when running step 2 +(for example, ``$HOME/easybuild``). + +Then, load the `EasyBuild` module to update your environment and make EasyBuild available for use: + +```shell + +module load EasyBuild +``` + +Note that in this case, we don't need to make any changes to our environment for EasyBuild to work correctly. +The environment module file that was generated by EasyBuild specifies all changes that need to be made. + +### Method 3: Development setup + +If you are planning to make changes to EasyBuild, or if you prefer using the latest *bleeding edge* +version of EasyBuild that is being developed, you can consider *cloning* the 3 main EasyBuild repositories +from GitHub, and updating your environment to run EasyBuild from there. + +This can be done as follows (into ``$HOME/easybuild``): + +```shell +mkdir -p $HOME/easybuild +cd $HOME/easybuild +``` + +```shell +# clone EasyBuild repositories from GitHub +git clone https://github.com/easybuilders/easybuild-framework.git +git clone https://github.com/easybuilders/easybuild-easyblocks.git +git clone https://github.com/easybuilders/easybuild-easyconfigs.git +``` +```shell +# update environment for running EasyBuild from there +export PATH=$HOME/easybuild/easybuild-framework:$PATH +export PYTHONPATH=$HOME/easybuild/easybuild-framework:$HOME/easybuild/easybuild-easyblocks:$HOME/easybuild/easybuild-easyconfigs:$PYTHONPATH + +# control which Python command is used to run EasyBuild +export EB_PYTHON=python3 +``` + + +### Approach on LUMI + +*Documentation on the inner workings of the LUMI software stack can be found in the +[LUMI-SoftwareStack GitHub, docs subdirectory](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/docs).* + +To keep the different versions of the LUMI software stack as independent from one another as possible, +EasyBuild is bootstrapped for each software stack by the ``prepare_LUMI_stack.sh`` script. We use the +system Python 3 for running EasyBuild. That Python version currently doesn't have ``pip`` installed. +EasyBuild is first installed in a temporary work directory from files downloaded from PyPi. These +are each untarred and then installed into their location by running +``python3 setup.py install --prefix=...`` in their unpacked directory. +We do so only for the framework and easyblocks files as the easconfig files are not used to +install EasyBuild. Instead we create our own EasyConfig file for EasyBuild which contains +some additional packages that enable extra features in EasyBuild and also provide more information +to Lmod. Next the configuration module for EasyBuild (see the next section, +["Configuring EasyBuild"](1_06_configuring.md), for more information) and use the temporary +installation of EasyBuild with our own EasyConfig file to do a proper installation of EasyBuild +with module in the final location. + + +TODO: follow development setup but with downloading rather than cloning, then use that one with the +EasyBuild configuration module to install EasyBuild with itself. + + + +## Verifying the installation + +Regardless of how EasyBuild was installed, you can now run a couple of basic commands to verify the installation: + +#### Checking the version + +To check which EasyBuild version you have installed, run: + +```shell + +eb --version +``` + +The output should match with the latest EasyBuild version. + + +#### Consulting the help output + +You can consult the help output of the `eb` command, which produces a long list of available options +along with a short informative message. + +```shell +eb --help +``` + +#### Showing the default EasyBuild configuration + +To inspect the current EasyBuild configuration, you can use this command: + +```shell +eb --show-config +``` + +This should tell you that EasyBuild (ab)uses `$HOME/.local/easybuild` as a default location. +More on configuring EasyBuild in the [next part of the tutorial](configuration.md). + +#### System information + +You ask EasyBuild to collect and print some information about the +system you are using it on (OS, CPU, Python, etc.) using this command: + +```shell +eb --show-system-info +``` + +## Updating EasyBuild + +Before we wrap up here, a brief word about updating EasyBuild. + +Once you have EasyBuild installed, the easiest way to update to a newer version is by instructing EasyBuild +to install the latest available version as a module: + +``` + +eb --install-latest-eb-release +``` + +This will result in a *new* EasyBuild installation, which is entirely separate from the EasyBuild installation +you are currently using (so it is *not* an in-place update). +The location where this new EasyBuild version will be installed is determined by the active +EasyBuild configuration. + +If you have installed EasyBuild through ``pip``, and you prefer updating that installation, +you can use ``pip install --upgrade easybuild`` (perhaps with additional options like ``--user`` or ``--prefix``). + +!!! Note "EasyBuild upgrade policy on LUMI" + + Even minor version or patch level updates of EasyBuild may break things (and this has happned + in the past on other systems managed previously by LUMI User Support Team members). Hence a very + conservative upgrade policy is used on LUMI. + + In general we fix the version of EasyBuild within a particular version of the LUMI software stack + and only transition to the latest version when starting a new software stack. The reason is that we + want to be able to rebuild an existing software stack with as little unexpected problems as possible. + A move to a newer version of EasyBuild is acceptable should we run into major problems with an existing + version that cannot be solved in an easyconfig file or LUMI-specific custom easyblock, but in those + cases we will first rebuild the whole software stack on a test setup to ensure that no problems are + introduced by upgrading EasyBuild. + + +--- + +## Exercise (optional) + +*This exercise is not needed to proceed the course when working on LUMI as we will +use the installation that is already present.* + +Install EasyBuild in your home directory. + +Make sure that the EasyBuild installation uses the `python3` command to run, +rather than the standard `python` command. + +Choose your own adventure, or try all these installation methods! + +* install EasyBuild with `pip` (or another very similar command...) using either the `--user` or `--prefix` option; +* perform a (manual) "bootstrap" installation into `$HOME/easybuild`, as outlined in installation method 2; +* set up a development installation; + +Check that the installation works by running the verification commands outlined +[above](#verifying-the-installation). + +--- + +**Make sure you have a working EasyBuild installation before proceeding +with the rest of the tutorial!** + +*[[next: Configuring EasyBuild]](1_06_configuration.md)* diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md index 49f4a4bb..29fa0359 100644 --- a/docs/2022-CSC_and_LO/index.md +++ b/docs/2022-CSC_and_LO/index.md @@ -63,8 +63,8 @@ freely available, so unless you have access to a system with this environment yo - [The Lmod module system](1_02_Lmod.md) - [The HPE Cray Programming Environment](1_03_CPE.md) - [Terminology](1_04_terminology.md) - - [Installation](installation.md) *(hands-on)* - - [Configuration](configuration.md) *(hands-on)* + - [Installation](1_05_installation.md) + - [Configuration](configuration.md) - [Basic usage](basic_usage.md) *(hands-on)* diff --git a/mkdocs.yml b/mkdocs.yml index c5103eaa..a4831aa1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -41,6 +41,7 @@ nav: - The Lmod module system: 2022-CSC_and_LO/1_02_Lmod.md - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_03_CPE.md - Terminology: 2022-CSC_and_LO/1_04_terminology.md + - Installation: 2022-CSC_and_LO/1_05_installation.md - "ISC'21 (25 June 2021)": - (overview): 2021-isc21/index.md - Practical info: 2021-isc21/practical_info.md From 2aee856de3f80d7177ec137dd974cf826f8bd81f Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 17:31:43 +0100 Subject: [PATCH 017/159] Tutorial on EasyBuild configuration added, and some restructuring. --- .../1_04_LUMI_software_stack.md | 17 + ..._04_terminology.md => 1_05_terminology.md} | 0 ...5_installation.md => 1_06_installation.md} | 0 docs/2022-CSC_and_LO/1_07_configuration.md | 610 +++++++++++ docs/2022-CSC_and_LO/1_08_basic_usage.md | 976 ++++++++++++++++++ 5 files changed, 1603 insertions(+) create mode 100644 docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md rename docs/2022-CSC_and_LO/{1_04_terminology.md => 1_05_terminology.md} (100%) rename docs/2022-CSC_and_LO/{1_05_installation.md => 1_06_installation.md} (100%) create mode 100644 docs/2022-CSC_and_LO/1_07_configuration.md create mode 100644 docs/2022-CSC_and_LO/1_08_basic_usage.md diff --git a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md new file mode 100644 index 00000000..0e578a59 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md @@ -0,0 +1,17 @@ +# LUMI software stacks + +*[[back: The Cray Programming Environment]](1_03_CPE.md)* + +--- + + + + + + + + +--- + +*[[next: Terminology]](1_05_terminology.md)* + diff --git a/docs/2022-CSC_and_LO/1_04_terminology.md b/docs/2022-CSC_and_LO/1_05_terminology.md similarity index 100% rename from docs/2022-CSC_and_LO/1_04_terminology.md rename to docs/2022-CSC_and_LO/1_05_terminology.md diff --git a/docs/2022-CSC_and_LO/1_05_installation.md b/docs/2022-CSC_and_LO/1_06_installation.md similarity index 100% rename from docs/2022-CSC_and_LO/1_05_installation.md rename to docs/2022-CSC_and_LO/1_06_installation.md diff --git a/docs/2022-CSC_and_LO/1_07_configuration.md b/docs/2022-CSC_and_LO/1_07_configuration.md new file mode 100644 index 00000000..17e60afe --- /dev/null +++ b/docs/2022-CSC_and_LO/1_07_configuration.md @@ -0,0 +1,610 @@ +# Configuring EasyBuild + +*[[back: Installation]](1_06_installation.md)* + +--- + +After installing EasyBuild, you should **configure** it. + +EasyBuild should work fine out-of-the-box if you use Lmod as your modules tool. +If you are not using Lmod, please see here for more information. + +Nevertheless, we strongly recommend you to inspect the default configuration, +and to configure EasyBuild according to your preferences and the system on which you will use it. + +!!! Note "EasyBuild configuration on LUMI" + + On LUMI serveral configurations of EasyBuild are already available. + + The ``EasyBuild-user`` module is the most important one. It will configure EasyBuild + to install software in either a default location in the user's home directory + (``$HOME/EasyBuild``) or the location pointed to by the environment variable + ``EBU_USER_PREFIX``. The latter can be used to install software in the project directory + so that it is available to all users of the project. It enables the creation of a custom + software environment build on top of the LUMI software stack. Software in this environment is + build in exactly the same way as it would if it were installed in the central repository, but + one can now also easily create custom configurations without confusing other projects with + different or non-standard configurations of software. + + There are two more admin-only configuration modules for EasyBuild. The main one is + ``EasyBuild-production`` which is used for software installations in the central software stack. + The other one is ``EasyBuild-infrastructure`` which is used to install infrastructure modules + that need to be installed in a pure Lmod hierarchy, e.g., the modules that load the toolchains. + + +--- + +## Available configuration settings + +One of the central policies in the EasyBuild project is to avoid hardcoded settings in the codebase. +While this significantly increases the ability to configure EasyBuild to your liking, it also results +in a large amount of available configuration settings. + +The full list of configuration settings can be consulted via `eb --help`, +which shows the corresponding command line option accompanied by a short description. +At the time of writing, *over 240 different configuration settings* are supported by EasyBuild. + +For the sake of this tutorial we will focus on a specific subset of configuration settings, +and cover only the most prominent and important ones. + +We will refer to EasyBuild configuration settings using the names as they appears in the output of `eb --help`, +and omit the leading dashes (`--`) for the sake of clarity. + +Keep in mind that *every* configuration setting can be defined in 3 different ways, +see below for more details. + +--- + +### Overall prefix + +*(default: `HOME/.local/easybuild`)* + +The `prefix` configuration setting specifies the overall prefix that EasyBuild should use, +which **determines the default value for various other configuration settings**: + +* `installpath`: `` +* `buildpath`: `/build` +* `sourcepath`: `/sources` +* `repositorypath` (easyconfigs archive): `/ebfiles_repo` +* `containerpath`: `/containers` + +Here, `` represents the value of the `prefix` configuration setting. + +If one of the configuration settings affected by `prefix` is defined specifically, +the `prefix` value becomes irrelevant for that specific configuration setting. + +--- + +### Install path + +*(default: ``)* + +The **location for both the software installation directories and generated module files** +can be controlled via the `installpath` configuration setting. +Software installation directories will be placed in `/software`, while +`/modules/all` will be used for generated module files. + +The `installpath` location is usually set to a directory on a *shared filesystem* when installing +software for an HPC cluster. Of course, software can also be installed on a local filesystem, +which is particularly useful to test and evaluate software installations. + +Separate configuration settings are available for both software and modules locations, +as well as for controlling the name of the `software` and `modules/all` subdirectories. + +*The EasyBuild community recommends to only change the `installpath` configuration setting to control the location +of software installations and accompanying module files, +such that the software and modules directories are located in the same parent directory, +and the default `software` and `modules/all` names for the subdirectories are used.* + +!!! Note "Path for software and for modules on LUMI" + + On LUMI we do not follow that recommendation. There is a three for the software installations + themselves with subdirectories based on the version of the software stack and LUMI hardware + partition, and a separate tree for the modules organised in a similar way. + + This makes it slightly easier to organise the module tree with user-friendly labeling, but above + all also makes the synchronisation process of the 4 instances of the software directory more robust + as it is now easy to synchonise all modules in the last step, which is a much quicker process than + syncrhonising the software installations. + + We also use short paths for software installations (to avoid overrunning the maximum length of a + shebang line in scripts) while we use longer, more descriptive names for subdirectories in the + module tree. + + +--- + +### Build path + +*(default: `/build`)* + +For each installation it performs, EasyBuild creates a **separate build directory** where software will be compiled +before installing it. This directory is cleaned up automatically when the installation is successfully completed. +To control the location where these build directories are created, you can use the `buildpath` configuration setting. + +Keep in mind that build directories may grow out to several GBs in size during an installation, +and that the commands that run in there can be fairly I/O-intensive since they may involve +manipulating lots of small files. In addition, a build directory that corresponds to a failing installation +is *not* cleaned up automatically, but it will be cleaned up and recycled when the same installation is re-attempted. +Running out of disk space in the location where build directories are created will result in failing +installations. + +It is strongly recommend to use the path to a directory on a *local filesystem* for the value of the +`buildpath` configuration setting, since using a shared filesystem like Lustre or GPFS is known to cause +problems when building certain software packages. Using an in-memory location (like `/dev/shm/$USER`) can +significantly speed up the build process, but may also lead to problems (due to space limitations, +or specific mount options like `noexec`). + +!!! Note "buildpath on LUMI" + + The configuration modules on LUMI will use a RAM disk for the build path. On the login nodes, + ``$XDG_RUNTIME_DIR`` is used as that space is automatically cleared when the last session of a user + ends. However, on the compute nodes a job- or user-specific subdirectory of ``/dev/shm`` is currently used + as ``$XDG_RUNTIME_DIR`` does not exist. + + +--- + +### Source path + +*(default: `/sources`)* + +For most supported software, EasyBuild can **automatically download the source files** required for the installation. +Before trying to download a source file, EasyBuild will first check if it is already present in the source path. + +The locations considered by EasyBuild when checking for available source files, as well as the location to +store downloaded source files, can be controlled via the ``sourcepath`` configuration setting. + +The `sourcepath` value is a colon (`:`) separated list of directory paths. +Each of these paths will be considered in turn when checking for available source files, +until one of them provides the desired source file. Searching for source files is done +based on filename, and a couple of subdirectories are considered. +For example, for a software package named '`Example'`, EasyBuild will consider locations +like `/e/Example/`, `/Example/`, and so on. + +The first path listed in `sourcepath` is the location where EasyBuild will store downloaded source files, +organised by software name through subdirectories, so EasyBuild expects to have write permissions to this path. +For the other paths listed in `sourcepath` only read permissions are required. + +*Make sure you have write permissions to the first path listed in `sourcepath`, so EasyBuild is able +to store downloaded files there. Feel free to list additional paths if you already have a cache of downloaded +files available somewhere.* + +*Storing the downloaded files not only reduces the amount of downloads while developing new easyconfig files +but also greatly helps when recompiling a software stack, as it is not that uncommon that download sites change +of files become unavailable.* + + +--- + +### Easyconfigs archive + +*(default: `/ebfiles_repo`)* + +EasyBuild keeps track of the easyconfig files that were used for installations in the easyconfigs +archive, the location of which is specified by the `repositorypath` configuration setting. + +By default the specified path is assumed to be a regular directory, but using a Git repository +as easyconfigs archive is also supported (for more details, see +[the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Configuration.html#easyconfigs-repository-repository-repositorypath)). + + +--- + +### Modules tool & module syntax + +*(default: `Lmod` as modules tool, and `Lua` as module syntax)* + +By default, EasyBuild assumes you are using **[Lmod](https://lmod.readthedocs.io) as modules tool**. +In addition, it will generate module files in [Lua](https://www.lua.org/) syntax, as supported by Lmod +(next to `Tcl` syntax). + +To diverge from this, you can define the `modules-tool` configuration setting to indicate you +are using a different modules tool; see the output of `eb --avail-modules-tools` for a list of supported +modules tools. +Note that for anything other than Lmod, you *must* make +sure that the actual modules tool binary command is available through `$PATH` (more information +on this [in the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Configuration.html#modules-tool-modules-tool)). + +If you prefer that EasyBuild generates module files in [Tcl](https://tcl.tk/) syntax, you can instruct +it to do so via the `module-syntax` configuration setting. Note that this is *required* if you are using +a modules tool other than Lmod, since only Lmod supports module files in `Lua` syntax. + +*We recommend using EasyBuild with the defaults settings: Lmod as modules tool, and Lua as module syntax.* + +--- + +### Robot search path + +*(default: `robot-paths` specifies the location to the easyconfig files included with EasyBuild installation; +dependency resolution is* ***not*** *enabled)* + +When EasyBuild needs to locate one or more easyconfig files, it will do so via the **robot search path**. +This applies to both easyconfig files that were specified using only their filename as an argument to the +`eb` command, as well as to easyconfigs required to resolve dependencies (more on that later). + +To control the robot search path, two configuration settings are available: `robot` and `robot-paths`. +Both accept a colon-separated list of locations to consider when looking for easyconfig files, +with `robot` having a higher priority than `robot-paths` (i.e., the directories specified with `robot` +appear before those specified by `robot-paths` in the robot search path). + +The key difference between these two configuration settings is that defining `robot` also enables dependency resolution, +and hence makes EasyBuild install missing dependencies, alongside +specifying a list of paths to consider when searching for easyconfig files. +On the other hand, defining `robot-paths` does not have the side effect of enabling dependency resolution. + +In addition, you can use the `--robot` command line option without specifying any paths to it, +to only enable dependency resolution. ***This option is needed on LUMI if you want automatic +dependency resolution as it is disabled on LUMI to prevent accidental mistakes when installing +software.*** + +!!! note "Easyconfig files included with EasyBuild and the robot search path" + Keep in mind that when either of the `robot` or `robot-paths` configuration settings are + defined, the default value corresponding to the location of easyconfigs included with the + EasyBuild is *no longer considered*. + + There are ways around this however, which are outside of the scope of this tutorial. + + For more information, + [see the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Using_the_EasyBuild_command_line.html#controlling-robot-search-path). + + *On LUMI the included easyconfig files cannot be used without changes as they are for the + common toolchains which are currently not supported in the LUMI software stacks.* + + +!!! Note "Easyconfigs archive on LUMI and the robot search path" + + On LUMI we also include the easyconfigs archive at the front of the search path for easyconfig + files. This has the advantage that EasyBuild when checking dependencies will always find the + configuration that is actually installed on the system, even if the easyconfig file in one of + the regular repositories has changed. It also has the disadvantage that one may accidentally + be re-installing with the already installed configuration while one really wants to re-install + the module in a different configuration. + + + +--- + +### Module naming scheme + +*(default: `EasyBuildMNS`)* + +EasyBuild will use a specific naming scheme for the module files it generates for each of the software installations. +This ensures consistency, regardless of who uses EasyBuild to perform the installation. + +Different types of module naming schemes are supported (flat, hierarchical, ...) and you can provide an implementation of +your own custom module naming scheme if desired. A number of different naming schemes are included +with EasyBuild, which you can consult via `eb --avail-module-naming-schemes`. + +The default `EasyBuildMNS` module naming scheme roughly corresponds to the filename of easyconfig files, +and consists of the software name followed by a combination of the software version, toolchain and +an optional label (which corresponds to the value of the `versionsuffix` easyconfig parameter): +`/<-toolchain>`. Just like with [names of easyconfig +files](terminology.md#easyconfig-files), the `<-toolchain>` part is omitted when the `system` toolchain is used, +and the `` value is empty by default. + +!!! Note "Module naming scheme on LUMI" + + On LUMI we use a custom variant of the standard EasyBuild flat naming scheme. The EasyBuild default + naming scheme creates an ``all`` subdirectory in the module directory containing all modules, but also + a directory per category, with the categories determined from the ``moduleclass`` easyconfig parameter. + However, that choice is often rather arbitrary as modules can belong to different categories depending + on who you ask. E.g., GROMACS is used in physics, chemistry and life sciences while EasyBuild can only + assign a single category. + + Hence we do not generate the categories and drop the ``all`` level in the module subdirectory. + + +--- + +## Configuration levels + +Configuring EasyBuild can be done in different ways: + +* through one or more configuration files; +* via `$EASYBUILD_*` environment variables; +* using `eb` command line options; + +Each of the methods corresponds to a *configuration level*. + +*Every* configuration setting can be defined via one of these +mechanisms, without exception! + +--- + +### Configuration level hierarchy + +There is a strict **hierarchy between the different configuration levels** supported by EasyBuild. + +Settings defined via a *configuration file* only override default values. + +A configuration setting that is defined via the corresponding *`$EASYBUILD_*` environment variable* +takes precedence over the value specified in a configuration file (if any). + +Finally, values specified through `eb` command line options **always** win, +regardless of whether the corresponding +configuration setting was already defined some other way, be it via a configuration file or +an environment variable. + +For example, let us consider a fictional configuration setting named `magic`: + +* If a value for `magic` is specified in an EasyBuild configuration file, + then this value will only be used if the corresponding environment variable + (`$EASYBUILD_MAGIC`) is *not* defined **and** if the `--magic` command line option is *not* used; +* If the `$EASYBUILD_MAGIC` environment *is* defined however, then its value + will be used for the `this-is-magic` configuration setting; +* *Unless* the `--magic` command line option is used, + since in that case the value provided as an argument there will be used instead. + +--- + +### Configuration files + +*Configuration files* are the most basic way of configuring EasyBuild. +Two types of are supported by EasyBuild: *user-level* and *system-level* +configuration files. The output of `eb --show-default-configfiles` tells you which locations +are considered, and whether any configuration files were found. + +EasyBuild configuration files are written in the standard [INI format](https://en.wikipedia.org/wiki/INI_file), +and the configuration settings are grouped into different *sections*. + +To create an EasyBuild configuration file, the output of `eb --confighelp` is very useful: +it produces the list of all supported configuration settings which are readily grouped in sections +and with every entry commented out, along with accompanying descriptive comments mentioning the default values, +and in the correct syntax. + +*Configuration files are the recommended mechanism to define configuration settings +that should always be in place, regardless of the software you are installing.* + +#### EasyBuild configuration files vs easyconfig files + +EasyBuild configuration files are sometimes confused with easyconfig files, +due to the similar name. However, these are two entirely different concepts! + +EasyBuild configuration files (usually `*.cfg`) are a way of *configuring the general behaviour of EasyBuild* +across different software installations. They define *configuration settings*, +such as the location where software should be installed, or the syntax that should +be used when generating module files. + +An [easyconfig file](terminology.md#easyconfig-files) (`*.eb`) on the other hand *specifies the details for one particular software installation*. +It does this by defining a set of *easyconfig parameters*, which tell EasyBuild the name and version +of the software to install, which [toolchain](terminology.md#toolchains) and [easyblock](terminology.md#easyblocks) to use, etc. + +For each software installation performed by EasyBuild, there is a corresponding easyconfig file. +There typically are only a handful of configuration files used however, for example a system-level +configuration file, perhaps combined with a user-level one. Or there may be no configuration files involved +at all, since EasyBuild can also be configured through other mechanisms: environment variables and command line +options. + +--- + +### `$EASYBUILD_*` environment variables + +A particularly easy way to configure EasyBuild is through *environment variables*. + +At startup, EasyBuild will pick up any environment variable of which the name starts with '`EASYBUILD_`'. +For each of these, it will determine the corresponding configuration setting (or exit with an error if +none was found). + +Mapping the name of a configuration setting to the name of the corresponding environment variable is straightforward: +use capital letters, replace dashes (`-`) with underscores (`_`), and prefix with `EASYBUILD_`. + +For example: the `module-syntax` configuration setting can be specified by defining +the `$EASYBUILD_MODULE_SYNTAX` environment variable: + +```shell +export EASYBUILD_MODULE_SYNTAX=Tcl +``` + +*Configuring via environment variables is especially practical for controlling the EasyBuild configuration +in a more dynamic way. For example, you can implement a simple shell script that defines `$EASYBUILD_*` +environment variables based on the current context (user, hostname, other environment variables), and +configure EasyBuild through [sourcing](https://bash.cyberciti.biz/guide/Source_command) it.* + +!!! note + Keep in mind that environment variables are only defined for the shell session you are currently working in. + If you want to configure EasyBuild through environment variables in a more persistent way, + you can leverage one of the [shell startup scripts](https://bash.cyberciti.biz/guide/Startup_scripts) (for example `$HOME/.bash_profile` or `$HOME/.bashrc`). + +--- + +### `eb` command line options + +Finally, you can also configure EasyBuild by specifying one or options to the `eb` command. + +As mentioned earlier, the values for configuration settings defined this way override the value that +is specified through any other means. So if you want to be sure that a particular configuration setting +is defined the way you want it to be, you can use the corresponding command line option. + +There are various configuration settings for which it only makes sense to use the command line option. +An example of this is letting the `eb` command print the EasyBuild version (via `eb --version`). +Although you could configure EasyBuild to always print its version and then exit whenever the `eb` command is +run, that would not be very useful... + +*Command line options are typically used to define configuration settings that are only relevant to +that particular EasyBuild session. One example is doing a test installation into a temporary directory:* + +```shell +eb --installpath /tmp/$USER example.eb +``` + +## Inspecting the current configuration (`--show-config`) + +Given the large amount of available configuration settings in EasyBuild and the different configuration levels, +you can easily lose track of exactly how EasyBuild is configured. + +Through the `--show-config` command line option you can +easily inspect the currently active EasyBuild configuration. + +The output of `--show-config` includes a sorted list of all configuration settings that are defined to a +*non-default* value, +along with a couple of important ones that are always shown (like `buildpath`, `installpath`, `sourcepath`, and so on). +In addition, it also indicates at which configuration level each setting was defined, +so you can trace down *where* it was defined if needed. + +This is the output produced by `eb --show-config` for the default EasyBuild configuration, +where EasyBuild was installed via `pip install --user` (which results in the value shown for the +`robot-paths` configuration setting): + +```shell +# +# Current EasyBuild configuration +# (C: command line argument, D: default value, E: environment variable, F: configuration file) +# +buildpath (D) = /home/example/.local/easybuild/build +containerpath (D) = /home/example/.local/easybuild/containers +installpath (D) = /home/example/.local/easybuild +repositorypath (D) = /home/example/.local/easybuild/ebfiles_repo +robot-paths (D) = /home/example/.local/easybuild/easyconfigs +sourcepath (D) = /home/example/.local/easybuild/sources +``` + +As shown here, all configuration settings shown follow the default `prefix` value (`$HOME/.local/easybuild`), +and none of the values diverge from the default value, since all entries are marked with `(D)` for "default value"). + +**Example** + +Now let us do some basic configuring and inspect the resulting output of `--show-config`. + +First, create a user-level EasyBuild configuration file to define the `prefix` configuration setting: + +```shell +mkdir -p $HOME/.config/easybuild +echo '[config]' > $HOME/.config/easybuild/config.cfg +echo 'prefix=/apps' >> $HOME/.config/easybuild/config.cfg +``` + +In addition, define the `buildpath` configuration setting using the corresponding +environment variable: + +```shell +export EASYBUILD_BUILDPATH=/tmp/$USER +``` + +Then run `--show-config` while you specify that the `installpath` configuration +setting should be defined as `/tmp/$USER`: + +```shell +$ eb --installpath=/tmp/$USER --show-config +# +# Current EasyBuild configuration +# (C: command line argument, D: default value, E: environment variable, F: configuration file) +# +buildpath (E) = /tmp/easybuild +containerpath (F) = /apps/containers +installpath (C) = /tmp/easybuild +packagepath (F) = /apps/packages +prefix (F) = /apps +repositorypath (F) = /apps/ebfiles_repo +robot-paths (D) = /home/example/.local/easybuild/easyconfigs +sourcepath (F) = /apps/sources +``` + +The output indicates that the `installpath` setting was specified through a command line option (indicated +with `(C)`), that the `buildpath` setting was defined via an environment +variable (indicated with `(E)`), that the `robot-paths` setting still has the default value (indicated with `(D)`), and that all other configuration +settings were specified via a configuration file, some of which indirectly through the `prefix` value (indicated with +`(F)`). + +--- + +## Exercises (optional) + +*These exercises are not very relevant for LUMI as LUMI already offers a complete configuration +also for user installations of software. However, if you are a very advanced user, you may still +want to make changes to that configuration, and all three options (configuration files, +environment variables and command line parameters) are available to users in the LUMI setup.* + +***Exercise C.1* - Configure EasyBuild** + +Configure EasyBuild to use the `easybuild` subdirectory in your home directory for everything, except for: + +* The location of the build directories: use `/tmp/$USER` for this; +* The locations that should be considered when searching for source files: + include both `$HOME/easybuild/sources` and `/easybuild/sources`, but make + sure that source files that are downloaded by EasyBuild are stored in + `$HOME/easybuild/sources`. + +Leave other configuration settings set to their default value. + +??? success "(click to show solution)" + + This is pretty straightforward. + + Here we just define the corresponding environment variables: + + ```shell + export EASYBUILD_PREFIX=$HOME/easybuild + export EASYBUILD_BUILDPATH=/tmp/$USER + export EASYBUILD_SOURCEPATH=$HOME/easybuild/sources:/easybuild/sources + ``` + + The location where EasyBuild should download source files to + must be listed first in the `sourcepath` configuration setting. + + The output of `--show-config` should look like this: + + ```shell + buildpath (E) = /tmp/example + containerpath (E) = /home/example/easybuild/containers + installpath (E) = /home/example/easybuild + packagepath (E) = /home/example/easybuild/packages + prefix (E) = /home/example/easybuild + repositorypath (E) = /home/example/easybuild/ebfiles_repo + robot-paths (D) = /home/example/easybuild/easyconfigs + sourcepath (E) = /home/example/easybuild/sources:/easybuild/sources + ``` + +--- + +***Exercise C.2* - Install a trivial software package with EasyBuild** + +Try running the following command: + +```shell +eb bzip2-1.0.6.eb +``` + +Where do you expect to find the installation? + +??? success "(click to show solution)" + + The software was installed in `$HOME/easybuild`, + since that's how we configured EasyBuild in *Exercise 3.1*: + + ```shell + $ ls $HOME/easybuild + ebfiles_repo modules software sources + ``` + + The actual installation is in `$HOME/easybuild/software`, + while the module file was generated in `$HOME/easybuild/modules/all`: + + ```shell + $ ls $HOME/easybuild/software + bzip2 + $ ls $HOME/easybuild/software/bzip2 + 1.0.6 + $ ls $HOME/easybuild/software/bzip2/1.0.6 + bin easybuild include lib man + ``` + + ```shell + $ ls $HOME/easybuild/modules/all + bzip2 + $ ls $HOME/easybuild/modules/all/bzip2 + 1.0.6.lua + ``` + + The source file for bzip2 1.0.6 was downloaded to `$HOME/easybuild/sources`: + + ```shell + $ ls $HOME/easybuild/sources/b/bzip2 + bzip2-1.0.6.tar.gz + ``` + + We will discuss this in more detail in the next part of the tutorial. + +--- + +*[[next: Basic usage]](1_08_basic_usage.md)* diff --git a/docs/2022-CSC_and_LO/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_08_basic_usage.md new file mode 100644 index 00000000..5566d376 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_08_basic_usage.md @@ -0,0 +1,976 @@ +# Basic usage + +*[[back: Configuring EasyBuild]](1_07_configuration.md)* + +--- + +Now that we have installed and configured EasyBuild, we can start using it for what it is intended for: +getting scientific software installed without breaking a sweat, or having to resist the urge to +shout out four-letter words. + +We will look at the high-level workflow first, and then cover each aspect in more detail. + +A couple of exercises are available at the end to help to make you more familiar with the +EasyBuild command line interface, so pay attention! + +--- + +## Overall workflow + +Installing software with EasyBuild is as easy as specifying to the **`eb` command** what we +want to install, and then sitting back to enjoy a coffee or tea (or whatever beverage you prefer). + +This is typically done by **specifying the name of one or more easyconfig files**, often combined +with the `--robot` option to let EasyBuild also install missing dependencies. + +It is recommended to first assess the current situation before letting EasyBuild install the software, +and to check which **dependencies** are already installed and which are still missing. In addition, +you may want to inspect the specifics of the **installation procedure** that will be performed by EasyBuild, +and ensure that the active EasyBuild configuration is what it should be. + +--- + +## Specifying easyconfigs + +Letting EasyBuild know what should be installed can be done by specifying one or more easyconfig files, +which is also the most common way. Alternative methods like using the `--software-name` option won't be +covered in this tutorial, since they are not commonly used. + +Arguments passed to the `eb` command, being anything that is *not* an option (which starts with `-` or `--`) or +is a value for a preceding configuration option, are assumed to refer to easyconfig files (with some exceptions). +These could be: + +* the *(absolute or relative) path* to an easyconfig file; +* the *name* of an easyconfig file; +* the path to a *directory* containing easyconfig files; + +Specified paths to files must of course point to existing files; if not, EasyBuild will print an appropriate error message: + +```shell +$ eb /tmp/does_not_exist.eb +ERROR: Can't find path /tmp/does_not_exist.eb +``` + +When only the *name* of an easyconfig file is specified, EasyBuild will automatically try and locate it. +First, it will consider the *current directory*. If no file with the specified name is found there, +EasyBuild will search for the easyconfig file in the [robot search path](../configuration/#robot-search-path). + +If the path to an existing *directory* is provided, EasyBuild will walk through the entire directory +(including all subdirectories), retain all files of which the name ends with `.eb`, and (try to) use these +as easyconfig files. + + +#### Example command + +Suppose we have the current situation in our home directory: + +* two (easyconfig) files named `example1.eb` and `example2.eb`; +* a subdirectory named `some_deps`, which has two easyconfig files `dep1.eb` and `dep2.eb` + alongside a text file named `list.txt`; +* a subdirectory named `more_deps` located *in* the `some_deps` subdirectory, + which contains another easyconfig file `dep3.eb`; + +Or, visually represented: + +```shell +example1.eb +example2.eb +some_deps/ +|-- dep1.eb +|-- dep2.eb +|-- list.txt +|-- more_deps/ + |-- dep3.eb +``` + +In this context, we run the following EasyBuild command from our home directory: + +```shell +eb example1.eb bzip2-1.0.6.eb $HOME/example2.eb some_deps +``` + +EasyBuild will interpret each of these arguments as follows: + +* `example1.eb` is the name of a file in the current directory, so it can be used directly; +* `bzip2-1.0.6.eb` is the name of an easyconfig file to locate via the robot search path + (since it does not exist in the current directory); +* `$HOME/example2.eb` specifies the full path to an existing file, which can be used directly; +* `some_deps` is the relative path to an existing directory, so EasyBuild will scan it and find three + easyconfig files: `some_deps/dep1.eb`, `some_deps/dep2.eb` and `some_deps/more_deps/dep3.eb`, + The `list.txt` file will be ignored since its name does not end with `.eb`. + +--- + +### Easyconfig filenames + +Note that the `eb` command does not care how easyconfig files are named, at least to some extent: the `.eb` +file extension *does* matter when easyconfig files are being picked up in subdirectories. + +File names for easyconfigs being mostly irrelevant is only correct with respect to the arguments passed to +the `eb` command however. As we will learn soon, the name of easyconfig files *does* matter (a lot) +when EasyBuild needs to locate easyconfigs that can be used to resolve a specified dependency +(see [here](#enabling-dependency-resolution)). + +This explains why easyconfig files usually adher to a very specific naming scheme, +corresponding to `--.eb`, where: + +* `` represents the software name; +* `` represents the software version; +* `` represents the toolchain used in the easyconfig file, which consists of the toolchain name + and version separated with a dash (`-`), and which is omitted (including the preceding `-`) when the + [`system` toolchain](../introduction#system-toolchain) is used; +* `` represents the value of the `versionsuffix` easyconfig parameter, + which is sometimes used to distinguish multiple variants of particular software installations + (and is empty by default); + +--- + +### Searching for easyconfigs + +You will frequently need to determine the exact name of an easyconfig file you want to install, +or just check which easyconfigs are available for a given software package. +This can be done by searching for easyconfigs using **`eb --search`** or **`eb -S`**. + +By default all directories listed in the [robot search path](../configuration#robot-search-path) will be +searched. If you want to search in additional directories without changing the robot search path, +you can use the `search-paths` configuration setting, or you can change the robot search path via either +the `robot` or `robot-paths` configuration options. + +Both the `--search` and `-S` options trigger the same search operation, but yield different output: +`eb --search` will print the full path to each easyconfig file that matches the specified search pattern, +while `eb -S` produces a more concise output. + +For example, let's check which easyconfig files are available for OpenFOAM 8 with a `foss` toolchain: + +```shell +$ eb --search openfoam-8-foss + * /home/example/.local/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-8-foss-2020a.eb + * /home/example/.local/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-8-foss-2020b.eb +``` + +The output is a bit more condensed when using `eb -S`: + +```shell +$ eb -S openfoam-8-foss +CFGS1=/home/example/.local/easybuild/easyconfigs/o/OpenFOAM + * $CFGS1/OpenFOAM-8-foss-2020a.eb + * $CFGS1/OpenFOAM-8-foss-2020b.eb +``` + +Note that the search is performed *case-insensitive*. + +The search pattern can include wildcards like `.*` and/or character groups like `[0-9]`, +or other special characters like `^` or `$` to mark the start/end of the filename, +but you need to be careful that `bash` does not expand these before the `eb` command is started, +so it is recommended to wrap the search pattern in single quotes (`'...'`) when using wildcards. + +For example, to check which easyconfigs are available to install TensorFlow 2.4.1 with the `2020b` version of a toolchain: + +```shell +$ eb -S '^tensorflow-2.4.1.*2020b' +CFGS1=/home/example/.local/easybuild/easyconfigs/t/TensorFlow + * $CFGS1/TensorFlow-2.4.1-foss-2020b.eb + * $CFGS1/TensorFlow-2.4.1-fosscuda-2020b.eb +``` + +--- + +#### Search index + +When searching for easyconfig files, you may see a message like this pop up: + +``` +== found valid index for , so using it... +``` + +This indicates that a **search index** was used for this particular directory which significantly speeds +up the search procedure, especially when the easyconfig files are located on a shared filesystem where metadata +operations involving lots of (small) files can be slow. + +For the easyconfig files included with an EasyBuild release, a search index is readily provided. +For other directories, you can create a search index using `eb --create-index `. +[See the EasyBuild documentation for more information](https://easybuild.readthedocs.io/en/latest/Easyconfigs_index.html). + +--- + +## Inspecting easyconfigs + +Once you have determined the name of the easyconfig file that corresponds to the software you want to install, +you may want to take a closer look at its contents before employing it. + +Since easyconfig files are simple text files (in Python syntax), you could use the ubiquitous `cat` command +or your favorite text editor (`vim`, what else). To avoid that you need to locate the easyconfig file first +and copy-paste the full path to it, you can use **`eb --show-ec`**. + +For example, let's inspect the contents of the `bzip2-1.0.6.eb` easyconfig file: + +```shell +$ eb --show-ec bzip2-1.0.6.eb +== temporary log file in case of crash /tmp/eb-jnpzclhl/easybuild-e37cbrj1.log +== Contents of /home/example/.local/easybuild/easyconfigs/b/bzip2/bzip2-1.0.6.eb: +name = 'bzip2' +version = '1.0.6' + +homepage = 'https://sourceware.org/bzip2' +description = """bzip2 is a freely available, patent free, high-quality data compressor. It typically +compresses files to within 10% to 15% of the best available techniques (the PPM family of statistical +compressors), whilst being around twice as fast at compression and six times faster at decompression.""" + +toolchain = SYSTEM +toolchainopts = {'pic': True} + +source_urls = ['https://sourceware.org/pub/bzip2/'] +sources = [SOURCE_TAR_GZ] +patches = ['bzip2-%(version)s-pkgconfig.patch'] +checksums = [ + 'a2848f34fcd5d6cf47def00461fcb528a0484d8edef8208d6d2e2909dc61d9cd', # bzip2-1.0.6.tar.gz + '5a823e820b332eca3684416894f58edc125ac3dace9f46e62f98e45362aa8a6d', # bzip2-1.0.6-pkgconfig.patch +] + +buildopts = "CC=gcc CFLAGS='-Wall -Winline -O3 -fPIC -g $(BIGFILES)'" + +# building of shared libraries doesn't work on OS X (where 'gcc' is actually Clang...) +with_shared_libs = OS_TYPE == 'Linux' + +moduleclass = 'tools' + +== Temporary log file(s) /tmp/eb-jnpzclhl/easybuild-e37cbrj1.log* have been removed. +== Temporary directory /tmp/eb-jnpzclhl has been removed. +``` + +We'll get back to what all of this means later... + +--- + +## Checking dependencies + +!!! note + In some of the examples below, we assume that some software is already installed with EasyBuild. + + If you are following hands-on in a prepared environment, make sure these installations are visible + in the output of "`module avail`". + + +Before kicking off an installation, it is good practice to check which of the required dependencies +are already installed, and which ones are still missing. + +This can be helpful to ensure that your EasyBuild configuration is set up correctly, +and to prevent from accidentally installing an entirely new software stack from scratch +in an unintended location (like `$HOME/.local/easybuild`, the +[default installation path](../configuration/#install-path)). + +--- + +### Dry run + +To get a complete overview of all required dependencies, and see which ones are already installed +and which ones aren't yet, you can use **`eb --dry-run`**. + +Since `--dry-run` produces rather verbose output including the full path to each easyconfig file, +there is a more concise equivalent available as well: `eb --dry-run-short`, which is equivalent with **`eb -D`**. + +For example, to check which of the required dependencies for `SAMtools-1.11-GCC-10.2.0.eb` are already installed: + +```shell +$ eb SAMtools-1.11-GCC-10.2.0.eb -D +== temporary log file in case of crash /tmp/eb-x4qofiph/easybuild-ehhi9fb1.log +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +Dry run: printing build status of easyconfigs and dependencies +CFGS=/home/example/.local/easybuild/easyconfigs + ... + * [x] $CFGS/b/bzip2/bzip2-1.0.8-GCCcore-10.2.0.eb (module: bzip2/1.0.8-GCCcore-10.2.0) + * [x] $CFGS/x/XZ/XZ-5.2.5-GCCcore-10.2.0.eb (module: XZ/5.2.5-GCCcore-10.2.0) + * [x] $CFGS/c/cURL/cURL-7.72.0-GCCcore-10.2.0.eb (module: cURL/7.72.0-GCCcore-10.2.0) + * [x] $CFGS/g/GCC/GCC-10.2.0.eb (module: GCC/10.2.0) + * [x] $CFGS/n/ncurses/ncurses-6.2-GCCcore-10.2.0.eb (module: ncurses/6.2-GCCcore-10.2.0) + * [ ] $CFGS/s/SAMtools/SAMtools-1.11-GCC-10.2.0.eb (module: SAMtools/1.11-GCC-10.2.0) +``` + +(We've trimmed the output a bit here, for the sake of brevity.) + +This output tells us that all dependencies required by `SAMtools-1.11-GCC-10.2.0.eb` are already installed, +since they are all marked with `[x]`, whereas the easyconfig for `SAMtools` itself is not installed yet, +as indicated by lack of an `x` in `[ ]`. + +--- + +### Missing dependencies + +If you are only interested in which dependencies are still *missing*, +you can consult the output of **`eb --missing`**, or the equivalent **`eb -M`**. + +For example, let's see which dependencies are missing in order to get version 3.1.0 of the h5py +Python package installed using the `2020b` version of the `foss` toolchain: + +```shell +$ eb h5py-3.1.0-foss-2020b.eb -M + +2 out of 61 required modules missing: + +* pkgconfig/1.5.1-GCCcore-10.2.0-python (pkgconfig-1.5.1-GCCcore-10.2.0-python.eb) +* h5py/3.1.0-foss-2020b (h5py-3.1.0-foss-2020b.eb) +``` + +That should be pretty self-explanatory: out of the 63 required dependencies (which includes the `foss` toolchain +and everything needed to install it), only 2 dependencies are missing. Great! + +--- + +## Inspecting install procedures + +Finally, before actually installing something you may want to assess *how* exactly EasyBuild is going +to install the software. + +Perhaps you don't trust EasyBuild yet (you will eventually though, hopefully), +or maybe you just want to double check that you have made the right choice before going through with +the actual installation. + +Using **`eb --extended-dry-run`**, or just **`eb -x`** for short, +you can get a **detailed overview of the installation procedure that would be performed by EasyBuild**, +**in a matter of seconds**. + +By means of example, let's inspect some parts of the installation procedure for `Boost-1.74.0-GCC-10.2.0.eb`: + +```shell +$ eb Boost-1.74.0-GCC-10.2.0.eb -x +... + +preparing... [DRY RUN] + +[prepare_step method] +Defining build environment, based on toolchain (options) and specified dependencies... + +Loading toolchain module... + +module load GCC/10.2.0 + +Loading modules for dependencies... + +module load bzip2/1.0.8-GCCcore-10.2.0 +module load zlib/1.2.11-GCCcore-10.2.0 +module load XZ/5.2.5-GCCcore-10.2.0 + +... + +Defining build environment... + + ... + export CXX='mpicxx' + export CXXFLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno -fPIC' + ... + +configuring... [DRY RUN] + +[configure_step method] + running command "./bootstrap.sh --with-toolset=gcc --prefix=/tmp/example/Boost/1.74.0/GCC-10.2.0/obj --without-libraries=python,mpi" + (in /tmp/example/build/Boost/1.74.0/GCC-10.2.0/Boost-1.74.0) + +... + +[sanity_check_step method] +Sanity check paths - file ['files'] + * lib/libboost_system.so + * lib/libboost_thread-mt-x64.so +Sanity check paths - (non-empty) directory ['dirs'] + * include/boost +Sanity check commands + (none) + +... +``` + +We've obviously trimmed the generated output a bit, but it should be sufficient. + +An overview of the installation procedure is shown, following the installation steps as they would be +performed by EasyBuild. The output above shows: + +* how the build environment will be set up during the `prepare` step, by loading the module for both the + toolchains and the dependencies, and defining a set of environment variables like `$CXX`, `$CXXFLAGS`, etc. +* which command will be executed during the configuration step, and in which directory; +* the list of files and directories that will be checked during the sanity check step; + +If you were concerned about EasyBuild being too much of a black box, that is hopefully resolved now. + +!!! note + It is important to highlight here that the reported installation procedure *may* not be 100% correct, + since the [easyblock](../introduction/#easyblocks) can change its mind based on the output of shell commands + that were executed, or based on the contents of a file that was generated during the installation. + Since all "actions" that would be performed during the installation are actually skipped when using `eb -x`, + the reported installation procedure could be partially incorrect. + + In addition, the easyblock may trip over the fact that the installation procedure is not actually being + executed, which sometimes leads to an unexpected error. These situations are handled gracefully by `eb -x` + though, which will report any errors that occurred and then continue anyway with inspecting the remainder of the + installation procedure. Although this obviously limits the value of the generated output, + it doesn't make it completely useless. + +--- + +## Installing software + +You should now be able to make an informed decision on which easyconfig file you should use to +make EasyBuild install the software you require. + +As mentioned before, installing an easyconfig is as simple as passing it to the `eb` command. + +So, let's try to install SAMtools version 1.11: + +```shell +$ eb SAMtools-1.11-GCC-10.2.0.eb +== temporary log file in case of crash /tmp/eb-zh7_fyre/easybuild-4q_lo57b.log +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +== processing EasyBuild easyconfig /home/example/.local/easybuild/easyconfigs/s/SAMtools/SAMtools-1.11-GCC-10.2.0.eb +== building and installing SAMtools/1.11-GCC-10.2.0... +== fetching files... +== creating build dir, resetting environment... +== unpacking... +== patching... +== preparing... +== configuring... +== building... +== testing... +== installing... +== taking care of extensions... +== restore after iterating... +== postprocessing... +== sanity checking... +== cleaning up... +== creating module... +== permissions... +== packaging... +== COMPLETED: Installation ended successfully (took 17 sec) +== Results of the build can be found in the log file(s) /home/example/easybuild/software/SAMtools/1.11-GCC-10.2.0/easybuild/easybuild-SAMtools-1.11-20210309.105601.log +== Build succeeded for 1 out of 1 +== Temporary log file(s) /tmp/eb-zh7_fyre/easybuild-4q_lo57b.log* have been removed. +== Temporary directory /tmp/eb-zh7_fyre has been removed. +``` + +That was... easy. Is that really all there is to it? Well, almost... + +### Enabling dependency resolution + +The SAMtools installation worked like a charm, but remember that all required dependencies were already +available (see [above](#dry-run)). + +If we try this with the `BCFtools-1.11-GCC-10.2.0.eb`, for which the required `GSL` and `HTSlib` dependencies are not available yet, it's less successful: + +```shell +$ eb BCFtools-1.11-GCC-10.2.0.eb -M + +3 out of 23 required modules missing: + +* GSL/2.6-GCC-10.2.0 (GSL-2.6-GCC-10.2.0.eb) +* HTSlib/1.11-GCC-10.2.0 (HTSlib-1.11-GCC-10.2.0.eb) +* BCFtools/1.11-GCC-10.2.0 (BCFtools-1.11-GCC-10.2.0.eb) +``` + +```shell +$ eb BCFtools-1.11-GCC-10.2.0.eb +... +== preparing... +== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/build/BCFtools/1.11/GCC-10.2.0): build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0 (took 2 sec) +== Results of the build can be found in the log file(s) /tmp/eb-3v1dfvnk/easybuild-BCFtools-1.11-20210308.195024.FlxkH.log +ERROR: Build of /home/example/.local/easybuild/easyconfigs/b/BCFtools/BCFtools-1.11-GCC-10.2.0.eb failed (err: 'build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0') +``` + +Oh my, what's this all about? + +If we filter the output a bit and focus on the actual error, the problem is clear: + +``` +Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0 +``` + +The required dependencies `HTSlib/1.11-GCC-10.2.0` and `GSL/2.6-GCC-10.2.0` are not installed yet, +and EasyBuild does not automatically install missing dependencies unless it is told to do so. + +It helpfully suggests to use the `--robot` command line option, so let's try that: + +```shell +$ eb BCFtools-1.11-GCC-10.2.0.eb --robot +... +== resolving dependencies ... +... +== building and installing HTSlib/1.11-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 13 sec) +... +== building and installing GSL/2.6-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 1 min 10 sec) +... +== building and installing BCFtools/1.11-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 8 sec) +... +== Build succeeded for 3 out of 3 +``` + +With dependency resolution enabled the `HTSlib` and `GSL` modules get installed first, +before EasyBuild proceeds with installing `BCFtools`. Great! + +--- + +### Trace output + +As you may have noticed if you tried the previous example hands-on, +some installations take a while. +An installation can be spending quite a bit of time during the build step, but what is actually going on there? + +To provide some more feedback as the installation progresses, you can enable the "`trace`" configuration setting. +Let's do this by defining the `$EASYBUILD_TRACE` environment variable, just to avoid having to type `--trace` +over and over again. + +We will redo the installation of `BCFtools-1.11-GCC-10.2.0.eb` by passing the `--rebuild` +option to the `eb` command (try yourself what happens if you don't use the `--rebuild` option!): + +```shell +$ export EASYBUILD_TRACE=1 +$ eb BCFtools-1.11-GCC-10.2.0.eb --rebuild +... +== configuring... + >> running command: + [started at: 2021-03-08 19:54:53] + [working dir: /tmp/example/build/BCFtools/1.11/GCC-10.2.0/bcftools-1.11] + [output logged in /tmp/eb-9u_ac0nv/easybuild-run_cmd-17m_he2x.log] + ./configure --prefix=/home/example/easybuild/software/BCFtools/1.11-GCC-10.2.0 --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --with-htslib=$EBROOTHTSLIB --enable-libgsl +== building... + >> running command: + [started at: 2021-03-08 19:54:54] + [working dir: /tmp/example/BCFtools/1.11/GCC-10.2.0/bcftools-1.11] + [output logged in /tmp/example/eb-9u_ac0nv/easybuild-run_cmd-bhkgjxi7.log] + make -j 8 + >> command completed: exit 0, ran in 00h00m03s +``` + +That's a bit more comforting to stare at... + +During the *configure* step, the `./configure` command is run with option to +enable support for leveraging `HTSlib` and `GSL`. + +During the *build* step, the software is actually being compiled +by running the `make` command. EasyBuild automatically uses the available cores on the system (in this case 8). + +We even get a pointer to a log file that contains the output of the command being run, +so we can use `tail -f` to see in detail how it progresses. + +Once the `make` command completes, we get a message that the command completed with a exit code 0 +(implying success), and that it took 3 seconds to run. That's good to know. + +Later during the installation, we now also see this output during the sanity check step: + +``` +== sanity checking... + >> file 'bin/bcftools' found: OK + >> file 'bin/plot-vcfstats' found: OK + >> file 'bin/vcfutils.pl' found: OK + >> (non-empty) directory 'libexec/bcftools' found: OK +``` + +Thanks to enabling trace mode, EasyBuild tells us which files & directories it is checking for +in the installation, before declaring it a success. Nice! + +The extra output you get when trace mode is enabled is concise and hence not overwhelming, +while it gives a better insight into what is going on during the installation. +It may also help to spot unexpected actions being taken during the installation early on, +so you can interrupt the installation before it completes, if deemed necessary. + +--- + +## Using installed software + +So far, we have already installed 4 different software packages (SAMtools, HTSlib, GSL, and BCFtools); +we even installed BCFtools twice! + +A lot was going on underneath the covers: locating and unpacking +the source tarballs, setting up the build environment, configuring the build, compiling, +creating and populating the installation directory, performing a quick sanity check on the installation, +cleaning things up, and finally generated the environment module file corresponding to the installation. + +That's great, but how do we now actually *use* these installations? + +This is where the generated module files come into play: they form the access portal to the software +installations, and we'll use the ubiquitous `module` command to digest them. + +First, we need to make sure that the modules tool is aware of where the module files for +our installations are located. If you're unsure where EasyBuild is installing stuff at this point, +check the output of `eb --show-config`; the value of the `installpath` configuration setting is what we are interested in now: + +```shell +$ eb --show-config +... +installpath (E) = /home/example/easybuild +... +repositorypath (E) = /home/example/easybuild/ebfiles_repo +... +sourcepath (E) = /home/example/easybuild/sources +... +``` + +So, what's in this directory? + +```shell +$ ls -l $HOME/easybuild +total 16 +drwxrwxr-x 5 example example 4096 Jun 10 20:11 ebfiles_repo +drwxrwxr-x 5 example example 4096 Jun 10 20:10 modules +drwxrwxr-x 6 example example 4096 Jun 10 20:10 software +drwxrwxr-x 6 example example 4096 Jun 10 20:10 sources +``` + +The `ebfiles_repo` and `sources` directories correspond to the `repositorypath` and `sourcepath` configuration +settings, respectively. The `modules` and `software` directories are what we need now. + +The `modules` subdirectory consists of multiple subdirectories: + +```shell +$ ls $HOME/easybuild/modules +all bio devel numlib tools +``` + +Directories like `bio` and `numlib` correspond to different software categories, +and contain symbolic links to the module files in the `all` directory, +which contains all actual module files for software installed in this EasyBuild installation path. +We'll ignore these separate category directories for now. + +Let's inform the modules tool about the existence of these module files using `"module use"`: + +```shell +module use $HOME/easybuild/modules/all +``` + +This command does little more that updating the `$MODULEPATH` environment variable, +which contains a list of paths that the modules tool should consider when looking for module files. + +Now the modules tool should be aware of our brand new installations: + +```shell +$ module avail + +---------------------- /home/example/easybuild/modules/all ----------------------- + BCFtools/1.11-GCC-10.2.0 GSL/2.6-GCC-10.2.0 SAMtools/1.11-GCC-10.2.0 + EasyBuild/4.3.3 HTSlib/1.11-GCC-10.2.0 bzip2/1.0.6 + +---------------------------- /easybuild/modules/all ----------------------------- + ... +``` + +This output shows both the modules for our own installations as well as the "central" installations in `/easybuild` (which we omitted above for brevity). + +Now we can load these modules and start using these software installations. + +Let's test this for BCFtools. In our current environment, the `bcftools` command is not available yet: + +```shell +$ module list +No modules loaded + +$ bcftools +-bash: bcftools: command not found +``` + +Loading the module for BCFtools changes that: + +```shell +$ module load BCFtools/1.11-GCC-10.2.0 + +$ module list +Currently Loaded Modules: + 1) GCCcore/10.2.0 6) XZ/5.2.5-GCCcore-10.2.0 + 2) zlib/1.2.11-GCCcore-10.2.0 7) cURL/7.72.0-GCCcore-10.2.0 + 3) binutils/2.35-GCCcore-10.2.0 8) HTSlib/1.11-GCC-10.2.0 + 4) GCC/10.2.0 9) GSL/2.6-GCC-10.2.0 + 5) bzip2/1.0.8-GCCcore-10.2.0 10) BCFtools/1.11-GCC-10.2.0 + +$ bcftools --version +bcftools 1.11 +Using htslib 1.11 +... +``` + +Note that the modules for the required dependencies, including the compiler toolchain (which provides runtime libraries +like `libstdc++.so`), are loaded automatically. The "`module load`" command changes the active environment, +by updating environment variables like `$PATH` for example, to make the software available for use. + +##### Resetting your environment + +To restore your environment to a pristine state in which no modules are loaded, you can either +unload the loaded modules one by one using "`module unload`", or you can unload all of them at once using +"`module purge`". + +**If you are using an EasyBuild installation provided by a module, +don't forget to load the `EasyBuild` module again after running "`module purge`".** + +--- + +## Stacking software + +Maybe you have overlooked how the software we are playing around with was not only installed across multiple +different installation directories per software, we are also "stacking" our own installations (in `$HOME/easybuild`) +on top of installations that are provided in a totally different location (`/easybuild`). + +**EasyBuild doesn't care *where* software is installed: as long as the module file that provides access to it +is available, it is happy to pick it up and use it when required.** + +This implies that end users of an HPC system can easily install their +own small software stack on top of what is provided centrally by the HPC support team, +for example. They can even +"replace" a central software installation for their purposes if they need to, since the modules tool will +load the first module file that matches the request being made (there are some caveats with this, but we +won't go into those here). + +--- + +## Exercises + +***Guidelines*** + +Do yourself a favor: don't peek at the solution until you have made an attempt to solve the exercise yourself! + +Please do not spoil solutions to others before they have been discussed by the tutorial organisers. + +The exercises are based on the easyconfig files included with EasyBuild 4.3.3. + +--- + +***Exercise U.0**** - Making installed software available* + +Before working on the exercises for this part of the tutorial, +make sure that the software that is already installed in the prepared environment is available. + +We will assume that you have a small software stack installed using the `2020b` version of the `foss` toolchain. + +**Tip:** execute a "`module use`" command, and verify with "`module avail`" that a bunch of software modules +are available for loading. + +??? success "(click to show solution)" + + Use the following command to make the modules for the software available that is pre-installed + in the prepared environment: + ```shell + module use /easybuild/modules/all + ``` + + If software is installed in a different location than `/easybuild/` in your environment, + you should adjust the command accordingly. + + +--- + +***Exercise U.1**** - Searching easyconfigs* + +See if EasyBuild provides any easyconfig files for installing GROMACS version 2020/5. + +??? success "(click to show solution)" + To check for available easyconfig files, we can use `eb --search` or `eb -S`: + ```shell + $ eb -S gromacs-2020.5 + == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + CFGS1=/home/example/.local/easybuild/easyconfigs/g/GROMACS + * $CFGS1/GROMACS-2020.5-fosscuda-2020a-Python-3.8.2.eb + * $CFGS1/GROMACS-2020.5_fix_threads_gpu_Gmxapitests.patch + ``` + This actually shows one easyconfig file but also a patch file. We can also search specifically + for GROMACS 2020.5 in the `foss` and `fosscuda` toolchains using + ```shell + $ eb -S gromacs-2020.5-foss + == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + CFGS1=/home/example/.local/easybuild/easyconfigs/g/GROMACS + * $CFGS1/GROMACS-2020.5-fosscuda-2020a-Python-3.8.2.eb + ``` + and now we find a single easyconfig file. + +--- + +***Exercise U.2**** - Checking dependencies* + +Check which dependencies are missing to install QuantumESPRESSO version 6.6 with the `2020b` version of the `foss` toolchain. + +??? success "(click to show solution)" + First, we need to determine the name of the easyconfig file for QuantumESPRESSO version 6.6: + ```shell + $ eb -S 'QuantumESPRESSO-6.6.*foss-2020b' + == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + CFGS1=/home/example/.local/easybuild/easyconfigs/q/QuantumESPRESSO + * $CFGS1/QuantumESPRESSO-6.6-foss-2020b.eb + ``` + To determine which dependencies are missing to install this QuantumESPRESSO easyconfig file, we can use `--missing`: + ```shell + $ eb QuantumESPRESSO-6.6-foss-2020b.eb --missing + + 3 out of 58 required modules missing: + + * libxc/4.3.4-GCC-10.2.0 (libxc-4.3.4-GCC-10.2.0.eb) + * ELPA/2020.11.001-foss-2020b (ELPA-2020.11.001-foss-2020b.eb) + * QuantumESPRESSO/6.6-foss-2020b (QuantumESPRESSO-6.6-foss-2020b.eb) + ``` + (some nonessential output removed). + +--- + +***Exercise U.3**** - Performing a dry run* + +Figure out which command EasyBuild would use to compile +the software provided by the `Bowtie2-2.4.2-GCC-9.3.0.eb` easyconfig file, +without actually installing `Bowtie2`. + +Also, which binaries will EasyBuild check for to sanity check the installation? + +??? success "(click to show solution)" + To inspect the installation procedure, we can use `eb -x Bowtie2-2.4.2-GCC-9.3.0.eb`. + + The output for the build step shows the actual compilation command that would be performed (`make ...`): + + ```shell + [build_step method] + >> running command: + [started at: 2021-03-08 20:15:08] + [working dir: /local_scratch/hkenneth/eb-1wodfohg/__ROOT__/local_scratch/hkenneth/Bowtie2/2.4.2/GCC-9.3.0/Bowtie2-2.4.2] + [output logged in /local_scratch/hkenneth/eb-1wodfohg/easybuild-run_cmd-haojzisn.log] + make -j 48 CC="gcc" CPP="g++" CXX="g++" RELEASE_FLAGS="-O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -std=gnu++98" + (in /local_scratch/hkenneth/Bowtie2/2.4.2/GCC-9.3.0/Bowtie2-2.4.2) + ``` + + If the output you get is less detailed, you may not have set `export EASYBUILD_TRACE=1`. + + The output for the sanity check step shows which binaries are expected to be installed: + ``` + [sanity_check_step method] + Sanity check paths - file ['files'] + * bin/bowtie2 + * bin/bowtie2-align-l + * bin/bowtie2-align-s + * bin/bowtie2-build + * bin/bowtie2-build-l + * bin/bowtie2-build-s + * bin/bowtie2-inspect + * bin/bowtie2-inspect-l + * bin/bowtie2-inspect-s + ``` + +--- + +***Exercise U.4**** - Installing software* + +Install version 3.1.0 of the `h5py` Python package and all missing dependencies, +using the `foss/2020b` toolchain, into `/tmp/$USER/easybuild`, +while leveraging the already installed software available from `/easybuild`. + +Enable trace output so you can see which parts of the installation take a while. + +??? success "(click to show solution)" + First, determine the easyconfig file we can use for this: + ```shell + $ eb -S 'h5py-3.1.0.*foss-2020b' + CFGS1=/home/example/.local/easybuild/easyconfigs/h/h5py + * $CFGS1/h5py-3.1.0-foss-2020b.eb + ``` + + Make sure the pre-install software in `/easybuild/` is available: + ```shell + module use /easybuild/modules/all + ``` + + Check which dependencies are missing to install this `h5py` easyconfig: + ```shell + $ eb h5py-3.1.0-foss-2020b.eb --missing + + 2 out of 63 required modules missing: + + * pkgconfig/1.5.1-GCCcore-10.2.0-python (pkgconfig-1.5.1-GCCcore-10.2.0-python.eb) + * h5py/3.1.0-foss-2020b (h5py-3.1.0-foss-2020b.eb) + ``` + + Install `h5py` by specifying the easyconfig file and enabling dependency resolution via `--robot`, + while indicating that we want to install the software into `/tmp/$USER/easybuild` using the `--installpath` + option. Also make sure that trace mode is enabled by defining the `$EASYBUILD_TRACE` environment variable. + ```shell + $ export EASYBUILD_TRACE=1 + $ eb h5py-3.1.0-foss-2020b.eb --robot --installpath /tmp/$USER/easybuild + ... + == building and installing pkgconfig/1.5.1-GCCcore-10.2.0-python... + ... + == building and installing h5py/3.1.0-foss-2020b... + ... + == building... + >> running command: + [started at: 2020-06-10 21:47:32] + [working dir: /tmp/example/h5py/3.1.0/foss-2020b/h5py-3.1.0] + [output logged in /tmp/eb-rjjkbqe1/easybuild-run_cmd-d_dkc4iz.log] + python setup.py configure --mpi --hdf5=$EBROOTHDF5 && /easybuild/software/Python/3.8.6-GCCcore-10.2.0/bin/python setup.py build + >> command completed: exit 0, ran in 00h01m27s + ... + == COMPLETED: Installation ended successfully (took 2 min 46 sec) + ... + == Build succeeded for 2 out of 2 + ``` + + The trace output shows that most time is spent in the build command, + which runs both `python setup.py configure` and `python setup.py build`. + +--- + +***Exercise U.5**** - Using installed software* + +Using the `h5py` installation from the previous exercise to create an empty HDF5 file, +using the following Python statements: + +```python +import h5py +f = h5py.File("empty.hdf5", "w") +f.close() +``` + +Check the resulting file using the `h5stat` command. + +??? success "(click to show solution)" + First, we need to make the modules tool aware of the module files that were installed into `/tmp/$USER/easybuild`: + ```shell + module use /tmp/$USER/easybuild/modules/all + ``` + + Then we can check the `h5py` module is available, and load it: + ```shell + $ module avail h5py + ------------ /tmp/example/easybuild/modules/all ------------ + h5py/3.1.0-foss-2020b + ``` + + ```shell + module load h5py/3.1.0-foss-2020b + ``` + + The Python code snippet can be run directly on the command line using "`python -c '...'`", since it's tiny: + ```shell + python -c 'import h5py; f = h5py.File("empty.hdf5", "w"); f.close()' + ``` + Of course you can also copy the Python code snippet in a file named `test_h5py.py`, + and then run it with `python test_h5py.py`. + + Checking with the `h5stat` command shows that the resulting `empty.hdf5` is indeed a valid HDF5 file: + ```shell + $ ls -l empty.hdf5 + -rw-rw-r-- 1 example example 800 Jun 10 21:54 empty.hdf5 + + $ h5stat empty.hdf5 + Filename: empty.hdf5 + File information + # of unique groups: 1 + # of unique datasets: 0 + ... + ``` + +--- + +If you've made it through the hands-on exercises, congratulations! + +If not, don't worry too much about it. We covered a lot of ground here, +and it's a lot to take in at once, take your time... + +Feel free to ask question in the `#tutorial` channel in the [EasyBuild +Slack](https://docs.easybuild.io/en/latest/#getting-help), +we're happy to help! + +--- + +*[[next: Part 2: Using EasyBuild]](2_00_part2_using.md)* From 53d564d8931f3526fb98b6d00aa9150d1cb0bb7f Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 17:33:23 +0100 Subject: [PATCH 018/159] Restructuring continued. --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 10 +++++----- docs/2022-CSC_and_LO/1_03_CPE.md | 2 +- docs/2022-CSC_and_LO/1_05_terminology.md | 4 ++-- docs/2022-CSC_and_LO/1_06_installation.md | 10 ++++++---- docs/2022-CSC_and_LO/index.md | 9 +++++---- mkdocs.yml | 7 +++++-- 6 files changed, 24 insertions(+), 18 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md index 7b138670..0674be92 100644 --- a/docs/2022-CSC_and_LO/1_00_part1_intro.md +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -5,8 +5,8 @@ - [What is EasyBuild?](1_01_what_is_easybuild.md) - [The Lmod module system](1_02_Lmod.md) - [The HPE Cray Programming Environment](1_03_CPE.md) -- [Terminology](1_04_terminology.md) - -- [Installation](installation.md) *(hands-on)* -- [Configuration](configuration.md) *(hands-on)* -- [Basic usage](basic_usage.md) *(hands-on)* +- [LUMI software stacks](1_04_LUMI_software_stack.md) +- [Terminology](1_05_terminology.md) +- [Installation](1_06_installation.md) +- [Configuration](1_07_configuration.md) +- [Basic usage](1_08_basic_usage.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md index 42eb197e..077fbc36 100644 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -281,4 +281,4 @@ redfine ``LD_LIBRARY_PATH``. --- -*[[next: Terminology]](1_04_terminology.md)* +*[[next: LUMI software stacks]](1_04_LUMI_software_stack.md)* diff --git a/docs/2022-CSC_and_LO/1_05_terminology.md b/docs/2022-CSC_and_LO/1_05_terminology.md index 9c643579..8e05012f 100644 --- a/docs/2022-CSC_and_LO/1_05_terminology.md +++ b/docs/2022-CSC_and_LO/1_05_terminology.md @@ -1,6 +1,6 @@ # Terminology -*[[back: The HPE Cray Programming Environment]](1_03_CPE.md)* +*[[back: LUMI software stacks]](1_04_LUMI_software_stack.md)* --- @@ -291,4 +291,4 @@ An **easystack file** can be used to specify a collection of software to install --- -*[[next: Installation]](1_05_installation.md)* +*[[next: Installation]](1_06_installation.md)* diff --git a/docs/2022-CSC_and_LO/1_06_installation.md b/docs/2022-CSC_and_LO/1_06_installation.md index 163262ea..6e68b500 100644 --- a/docs/2022-CSC_and_LO/1_06_installation.md +++ b/docs/2022-CSC_and_LO/1_06_installation.md @@ -1,6 +1,6 @@ # Installing EasyBuild -*[[back: Terminology]](1_04_terminology.md)* +*[[back: Terminology]](1_05_terminology.md)* --- @@ -537,7 +537,9 @@ Check that the installation works by running the verification commands outlined --- -**Make sure you have a working EasyBuild installation before proceeding -with the rest of the tutorial!** +**You'll need a working custom EasyBuild installation for some of the +optional generic EasyBuild exercises, but in general you can use the +installation provided by LUMI.** -*[[next: Configuring EasyBuild]](1_06_configuration.md)* + +*[[next: Configuring EasyBuild]](1_07_configuration.md)* diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md index 29fa0359..9287b960 100644 --- a/docs/2022-CSC_and_LO/index.md +++ b/docs/2022-CSC_and_LO/index.md @@ -62,10 +62,11 @@ freely available, so unless you have access to a system with this environment yo - [What is EasyBuild?](1_01_what_is_easybuild.md) - [The Lmod module system](1_02_Lmod.md) - [The HPE Cray Programming Environment](1_03_CPE.md) - - [Terminology](1_04_terminology.md) - - [Installation](1_05_installation.md) - - [Configuration](configuration.md) - - [Basic usage](basic_usage.md) *(hands-on)* + - [LUMI software stacks](1_04_LUMI_software_stack.md) + - [Terminology](1_05_terminology.md) + - [Installation](1_06_installation.md) + - [Configuration](1_07_configuration.md) + - [Basic usage](1_08_basic_usage.md) *(hands-on)* diff --git a/mkdocs.yml b/mkdocs.yml index a4831aa1..bb2dc75d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -40,8 +40,11 @@ nav: - What is EasyBuild?: 2022-CSC_and_LO/1_01_what_is_easybuild.md - The Lmod module system: 2022-CSC_and_LO/1_02_Lmod.md - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_03_CPE.md - - Terminology: 2022-CSC_and_LO/1_04_terminology.md - - Installation: 2022-CSC_and_LO/1_05_installation.md + - LUMI software stacks: 2022-CSC_and_LO/1_04_LUMI_software_stack.md + - Terminology: 2022-CSC_and_LO/1_05_terminology.md + - Installation: 2022-CSC_and_LO/1_06_installation.md + - Configuration: 2022-CSC_and_LO/1_07_configuration.md + - Basic usage: 2022-CSC_and_LO/1_08_basic_usage.md - "ISC'21 (25 June 2021)": - (overview): 2021-isc21/index.md - Practical info: 2021-isc21/practical_info.md From fea45d27687bef7f05c5584f935e408a8a8a9564 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 30 Mar 2022 12:07:24 +0200 Subject: [PATCH 019/159] Finished reworking the basic usage section, except for the exercises. --- docs/2022-CSC_and_LO/1_08_basic_usage.md | 644 ++++++++++++++++------- 1 file changed, 442 insertions(+), 202 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_08_basic_usage.md index 5566d376..f5031e48 100644 --- a/docs/2022-CSC_and_LO/1_08_basic_usage.md +++ b/docs/2022-CSC_and_LO/1_08_basic_usage.md @@ -21,7 +21,7 @@ Installing software with EasyBuild is as easy as specifying to the **`eb` comman want to install, and then sitting back to enjoy a coffee or tea (or whatever beverage you prefer). This is typically done by **specifying the name of one or more easyconfig files**, often combined -with the `--robot` option to let EasyBuild also install missing dependencies. +with the `--robot` (or `-r`) option to let EasyBuild also install missing dependencies. It is recommended to first assess the current situation before letting EasyBuild install the software, and to check which **dependencies** are already installed and which are still missing. In addition, @@ -53,7 +53,7 @@ ERROR: Can't find path /tmp/does_not_exist.eb When only the *name* of an easyconfig file is specified, EasyBuild will automatically try and locate it. First, it will consider the *current directory*. If no file with the specified name is found there, -EasyBuild will search for the easyconfig file in the [robot search path](../configuration/#robot-search-path). +EasyBuild will search for the easyconfig file in the [robot search path](../1_07_configuration/#robot-search-path). If the path to an existing *directory* is provided, EasyBuild will walk through the entire directory (including all subdirectories), retain all files of which the name ends with `.eb`, and (try to) use these @@ -75,12 +75,12 @@ Or, visually represented: ```shell example1.eb example2.eb -some_deps/ -|-- dep1.eb -|-- dep2.eb -|-- list.txt -|-- more_deps/ - |-- dep3.eb +some_deps +├── deb2.eb +├── dep1.eb +├── list.txt +└── more_deps + └── dep3.eb ``` In this context, we run the following EasyBuild command from our home directory: @@ -118,7 +118,7 @@ corresponding to `--.eb`, where: * `` represents the software version; * `` represents the toolchain used in the easyconfig file, which consists of the toolchain name and version separated with a dash (`-`), and which is omitted (including the preceding `-`) when the - [`system` toolchain](../introduction#system-toolchain) is used; + [`system` toolchain](../1_05_terminology#system-toolchain) is used; * `` represents the value of the `versionsuffix` easyconfig parameter, which is sometimes used to distinguish multiple variants of particular software installations (and is empty by default); @@ -131,7 +131,7 @@ You will frequently need to determine the exact name of an easyconfig file you w or just check which easyconfigs are available for a given software package. This can be done by searching for easyconfigs using **`eb --search`** or **`eb -S`**. -By default all directories listed in the [robot search path](../configuration#robot-search-path) will be +By default all directories listed in the [robot search path](../1_07_configuration#robot-search-path) will be searched. If you want to search in additional directories without changing the robot search path, you can use the `search-paths` configuration setting, or you can change the robot search path via either the `robot` or `robot-paths` configuration options. @@ -202,41 +202,91 @@ Since easyconfig files are simple text files (in Python syntax), you could use t or your favorite text editor (`vim`, what else). To avoid that you need to locate the easyconfig file first and copy-paste the full path to it, you can use **`eb --show-ec`**. +!!! Hint + To follow the examples below on LUMI, load ``LUMI/21.12`` and ``EasyBuild-user`` + (though results may differ or the examples not work anymore as the software installation + on LUMI evolves). + For example, let's inspect the contents of the `bzip2-1.0.6.eb` easyconfig file: ```shell -$ eb --show-ec bzip2-1.0.6.eb -== temporary log file in case of crash /tmp/eb-jnpzclhl/easybuild-e37cbrj1.log -== Contents of /home/example/.local/easybuild/easyconfigs/b/bzip2/bzip2-1.0.6.eb: -name = 'bzip2' -version = '1.0.6' - -homepage = 'https://sourceware.org/bzip2' -description = """bzip2 is a freely available, patent free, high-quality data compressor. It typically -compresses files to within 10% to 15% of the best available techniques (the PPM family of statistical -compressors), whilst being around twice as fast at compression and six times faster at decompression.""" - -toolchain = SYSTEM +$ eb --show-ec bzip2-1.0.8-cpeCray-21.12.eb +== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-53o823qb/easybuild-xn6nmt61.log +== Contents of /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeCray-21.12.eb: +# Contributed by Kurt Lust, LUMI project & UAntwerpen + +local_bzip2_version = '1.0.8' # http://www.bzip.org/downloads.html + +name = 'bzip2' +version = local_bzip2_version + +homepage = 'https://www.sourceware.org/bzip2/' + +whatis = [ + 'Description: bzip2 is a freely available, patent free, high-quality data compressor.', + 'The module contains both executables and libraries.' + 'Keywords: BZ2', +] + +description = """ +bzip2 is a freely available, patent free, high-quality data compressor. It +typically compresses files to within 10% to 15% of the best available techniques +(the PPM family of statistical compressors), whilst being around twice as fast +at compression and six times faster at decompression. It is based on the +Burrows-Wheeler block-sorting text compression algorithm and Huffman coding. +""" + +usage = """ +Check the man pages for the available commands or the web-based documentation for the +library functions. +""" + +docurls = [ + 'Web-based documentation: http://www.bzip.org/docs.html', + 'Man pages available for bzcmp, bzdiff, bzegrep, bzfgrep, bzgrep, bzip2, bunzip2, bzless and bzmore', +] + +toolchain = {'name': 'cpeCray', 'version': '21.12'} toolchainopts = {'pic': True} -source_urls = ['https://sourceware.org/pub/bzip2/'] -sources = [SOURCE_TAR_GZ] -patches = ['bzip2-%(version)s-pkgconfig.patch'] +source_urls = ['https://sourceware.org/pub/%(name)s/'] +sources = [SOURCE_TAR_GZ] +patches = ['bzip2-%(version)s-pkgconfig-manpath.patch'] checksums = [ - 'a2848f34fcd5d6cf47def00461fcb528a0484d8edef8208d6d2e2909dc61d9cd', # bzip2-1.0.6.tar.gz - '5a823e820b332eca3684416894f58edc125ac3dace9f46e62f98e45362aa8a6d', # bzip2-1.0.6-pkgconfig.patch + 'ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269', # bzip2-1.0.8.tar.gz + 'de11269dc6e4917023af4cee9ff83b204953ad9cde561dbc9d6fc70d9b9578e3', # bzip2-1.0.8-pkgconfig-manpath.patch ] -buildopts = "CC=gcc CFLAGS='-Wall -Winline -O3 -fPIC -g $(BIGFILES)'" +builddependencies = [ # Create a reproducible build environment. + ('buildtools', '%(toolchain_version)s', '', True), +] -# building of shared libraries doesn't work on OS X (where 'gcc' is actually Clang...) -with_shared_libs = OS_TYPE == 'Linux' +local_bzip2_major_minor = '.'.join(local_bzip2_version.split('.')[:2]) + +sanity_check_paths = { + 'files': [ 'lib/pkgconfig/bzip2.pc', 'lib/libbz2.a', 'lib/libbz2.%s' % SHLIB_EXT, + 'lib/libbz2.%s.%s' % (SHLIB_EXT, local_bzip2_major_minor), + 'lib/libbz2.%s.%s' % (SHLIB_EXT, local_bzip2_version), + 'include/bzlib.h' ] + + [ 'bin/b%s' % x for x in ['unzip2', 'zcat', 'zdiff', 'zgrep', 'zip2', 'zip2recover', 'zmore'] ] + + [ 'share/man/man1/bz%s.1' % x for x in ['cmp', 'diff', 'egrep', 'fgrep', 'grep', 'ip2', 'less', 'more'] ], + 'dirs': [] +} + +sanity_check_commands = [ + 'bzip2 --help', + 'pkg-config --libs bzip2', +] moduleclass = 'tools' -== Temporary log file(s) /tmp/eb-jnpzclhl/easybuild-e37cbrj1.log* have been removed. -== Temporary directory /tmp/eb-jnpzclhl has been removed. +== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-53o823qb/easybuild-xn6nmt61.log* have been removed. +== Temporary directory /run/user/10012026/easybuild/tmp/eb-53o823qb has been removed. ``` +The output may actually be longer for an easyconfig file that is already installed on the system +as a new easyconfig file is generated in the repository with some information about the installation +added to it and as on LUMI these are at the front of the robot search path to ensure that the system +finds the right easyconfig file matching with a module on the system. We'll get back to what all of this means later... @@ -272,25 +322,30 @@ there is a more concise equivalent available as well: `eb --dry-run-short`, whic For example, to check which of the required dependencies for `SAMtools-1.11-GCC-10.2.0.eb` are already installed: ```shell -$ eb SAMtools-1.11-GCC-10.2.0.eb -D -== temporary log file in case of crash /tmp/eb-x4qofiph/easybuild-ehhi9fb1.log -== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +$ eb SAMtools-1.14-cpeGNU-21.12.eb -D +== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-oo0lj9lq/easybuild-2cyomy8v.log Dry run: printing build status of easyconfigs and dependencies -CFGS=/home/example/.local/easybuild/easyconfigs - ... - * [x] $CFGS/b/bzip2/bzip2-1.0.8-GCCcore-10.2.0.eb (module: bzip2/1.0.8-GCCcore-10.2.0) - * [x] $CFGS/x/XZ/XZ-5.2.5-GCCcore-10.2.0.eb (module: XZ/5.2.5-GCCcore-10.2.0) - * [x] $CFGS/c/cURL/cURL-7.72.0-GCCcore-10.2.0.eb (module: cURL/7.72.0-GCCcore-10.2.0) - * [x] $CFGS/g/GCC/GCC-10.2.0.eb (module: GCC/10.2.0) - * [x] $CFGS/n/ncurses/ncurses-6.2-GCCcore-10.2.0.eb (module: ncurses/6.2-GCCcore-10.2.0) - * [ ] $CFGS/s/SAMtools/SAMtools-1.11-GCC-10.2.0.eb (module: SAMtools/1.11-GCC-10.2.0) +CFGS=/appl/lumi + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeGNU/cpeGNU-21.12.eb (module: cpeGNU/21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ncurses/ncurses-6.2-cpeGNU-21.12.eb (module: ncurses/6.2-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/zlib/zlib-1.2.11-cpeGNU-21.12.eb (module: zlib/1.2.11-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeGNU-21.12.eb (module: bzip2/1.0.8-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12-minimal.eb (module: gettext/0.21-cpeGNU-21.12-minimal) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/XZ/XZ-5.2.5-cpeGNU-21.12.eb (module: XZ/5.2.5-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/Brotli/Brotli-1.0.9-cpeGNU-21.12.eb (module: Brotli/1.0.9-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cURL/cURL-7.78.0-cpeGNU-21.12.eb (module: cURL/7.78.0-cpeGNU-21.12) + * [ ] $CFGS/LUMI-EasyBuild-contrib/easybuild/easyconfigs/h/HTSlib/HTSlib-1.14-cpeGNU-21.12.eb (module: HTSlib/1.14-cpeGNU-21.12) + * [ ] $CFGS/LUMI-EasyBuild-contrib/easybuild/easyconfigs/s/SAMtools/SAMtools-1.14-cpeGNU-21.12.eb (module: SAMtools/1.14-cpeGNU-21.12) +== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-oo0lj9lq/easybuild-2cyomy8v.log* have been removed. +== Temporary directory /run/user/10012026/easybuild/tmp/eb-oo0lj9lq has been removed. ``` -(We've trimmed the output a bit here, for the sake of brevity.) +This output tells you that most of the dependencies required by ``SAMtools-1.14-cpeGNU-21.12.eb`` are +already installed, since they are marked with ``[x]``. However, the easyconfig files for +``HTSLib-1.14-cpeGNU-21.12.eb`` and SAMtools itself are not installed yet, denoted by the +lack of an ``x`` in ``[ ]``. -This output tells us that all dependencies required by `SAMtools-1.11-GCC-10.2.0.eb` are already installed, -since they are all marked with `[x]`, whereas the easyconfig for `SAMtools` itself is not installed yet, -as indicated by lack of an `x` in `[ ]`. --- @@ -299,20 +354,19 @@ as indicated by lack of an `x` in `[ ]`. If you are only interested in which dependencies are still *missing*, you can consult the output of **`eb --missing`**, or the equivalent **`eb -M`**. -For example, let's see which dependencies are missing in order to get version 3.1.0 of the h5py -Python package installed using the `2020b` version of the `foss` toolchain: +For example, for the SAMtools easyconfig file used in the previous example we get (with +some lines removed from the output): ```shell -$ eb h5py-3.1.0-foss-2020b.eb -M - -2 out of 61 required modules missing: +$ eb SAMtools-1.14-cpeGNU-21.12.eb -M +2 out of 11 required modules missing: -* pkgconfig/1.5.1-GCCcore-10.2.0-python (pkgconfig-1.5.1-GCCcore-10.2.0-python.eb) -* h5py/3.1.0-foss-2020b (h5py-3.1.0-foss-2020b.eb) +* HTSlib/1.14-cpeGNU-21.12 (HTSlib-1.14-cpeGNU-21.12.eb) +* SAMtools/1.14-cpeGNU-21.12 (SAMtools-1.14-cpeGNU-21.12.eb) ``` -That should be pretty self-explanatory: out of the 63 required dependencies (which includes the `foss` toolchain -and everything needed to install it), only 2 dependencies are missing. Great! +That should be pretty self-explanatory: out of the 113 required dependencies (which includes the `cpeGNU` toolchain +and everything needed to install it), only 2 dependencies (including SAMtools itself) are missing. Great! --- @@ -329,52 +383,78 @@ Using **`eb --extended-dry-run`**, or just **`eb -x`** for short, you can get a **detailed overview of the installation procedure that would be performed by EasyBuild**, **in a matter of seconds**. -By means of example, let's inspect some parts of the installation procedure for `Boost-1.74.0-GCC-10.2.0.eb`: +By means of example, let's inspect some parts of the installation procedure for ``HTSLib-1.14-cpeGNU-21.12.eb``: ```shell -$ eb Boost-1.74.0-GCC-10.2.0.eb -x +$ eb HTSlib-1.14-cpeGNU-21.12.eb -x ... -preparing... [DRY RUN] +*** DRY RUN using 'ConfigureMake' easyblock (easybuild.easyblocks.generic.configuremake @ /appl/lumi/SW/LUMI-21.12/common/EB/EasyBuild/4.5.3/lib/python3.6/site-packages/easybuild/easyblocks/generic/configuremake.py) *** + +== building and installing HTSlib/1.14-cpeGNU-21.12... +fetching files... [DRY RUN] + +... [prepare_step method] Defining build environment, based on toolchain (options) and specified dependencies... Loading toolchain module... -module load GCC/10.2.0 +module load cpeGNU/21.12 Loading modules for dependencies... -module load bzip2/1.0.8-GCCcore-10.2.0 -module load zlib/1.2.11-GCCcore-10.2.0 -module load XZ/5.2.5-GCCcore-10.2.0 +module load buildtools/21.12 +module load zlib/1.2.11-cpeGNU-21.12 +module load bzip2/1.0.8-cpeGNU-21.12 +module load XZ/5.2.5-cpeGNU-21.12 +module load cURL/7.78.0-cpeGNU-21.12 ... Defining build environment... - ... - export CXX='mpicxx' - export CXXFLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno -fPIC' - ... +... + + export CC='cc' + export CFLAGS='-O2 -ftree-vectorize -fno-math-errno' + +... configuring... [DRY RUN] [configure_step method] - running command "./bootstrap.sh --with-toolset=gcc --prefix=/tmp/example/Boost/1.74.0/GCC-10.2.0/obj --without-libraries=python,mpi" - (in /tmp/example/build/Boost/1.74.0/GCC-10.2.0/Boost-1.74.0) + running command "./configure --prefix=/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/HTSlib/1.14-cpeGNU-21.12" + (in /run/user/10012026/easybuild/build/HTSlib/1.14/cpeGNU-21.12/HTSlib-1.14) + +building... [DRY RUN] + +[build_step method] + running command "make -j 256" + (in /run/user/10012026/easybuild/build/HTSlib/1.14/cpeGNU-21.12/HTSlib-1.14) + +testing... [DRY RUN] + +[test_step method] + +installing... [DRY RUN] ... +sanity checking... [DRY RUN] + [sanity_check_step method] Sanity check paths - file ['files'] - * lib/libboost_system.so - * lib/libboost_thread-mt-x64.so + * bin/bgzip + * bin/tabix + * lib/libhts.so Sanity check paths - (non-empty) directory ['dirs'] - * include/boost + * include Sanity check commands - (none) + * bgzip --version + * htsfile --version + * tabix --version ... ``` @@ -385,7 +465,7 @@ An overview of the installation procedure is shown, following the installation s performed by EasyBuild. The output above shows: * how the build environment will be set up during the `prepare` step, by loading the module for both the - toolchains and the dependencies, and defining a set of environment variables like `$CXX`, `$CXXFLAGS`, etc. + toolchains and the dependencies, and defining a set of environment variables like `$CC`, `$CFLAGS`, etc. * which command will be executed during the configuration step, and in which directory; * the list of files and directories that will be checked during the sanity check step; @@ -393,7 +473,7 @@ If you were concerned about EasyBuild being too much of a black box, that is hop !!! note It is important to highlight here that the reported installation procedure *may* not be 100% correct, - since the [easyblock](../introduction/#easyblocks) can change its mind based on the output of shell commands + since the [easyblock](1_05_terminology.md#easyblocks) can change its mind based on the output of shell commands that were executed, or based on the contents of a file that was generated during the installation. Since all "actions" that would be performed during the installation are actually skipped when using `eb -x`, the reported installation procedure could be partially incorrect. @@ -413,64 +493,105 @@ make EasyBuild install the software you require. As mentioned before, installing an easyconfig is as simple as passing it to the `eb` command. -So, let's try to install SAMtools version 1.11: +So, let's try to install libdap version 3.20.9 with the cpeGNU/21.12 toolchain. + +Let's first check if it has any dependencies that still need to be installed: ```shell -$ eb SAMtools-1.11-GCC-10.2.0.eb -== temporary log file in case of crash /tmp/eb-zh7_fyre/easybuild-4q_lo57b.log -== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... -== processing EasyBuild easyconfig /home/example/.local/easybuild/easyconfigs/s/SAMtools/SAMtools-1.11-GCC-10.2.0.eb -== building and installing SAMtools/1.11-GCC-10.2.0... +$ eb libdap-3.20.9-cpeGNU-21.12.eb -D +== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-wm_bk3j6/easybuild-puyu_559.log +Dry run: printing build status of easyconfigs and dependencies + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeGNU/cpeGNU-21.12.eb (module: cpeGNU/21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/zlib/zlib-1.2.11-cpeGNU-21.12.eb (module: zlib/1.2.11-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libtirpc/libtirpc-1.3.2-cpeGNU-21.12.eb (module: libtirpc/1.3.2-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/Brotli/Brotli-1.0.9-cpeGNU-21.12.eb (module: Brotli/1.0.9-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ICU/ICU-69.1-cpeGNU-21.12.eb (module: ICU/69.1-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cURL/cURL-7.78.0-cpeGNU-21.12.eb (module: cURL/7.78.0-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeGNU-21.12.eb (module: bzip2/1.0.8-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ncurses/ncurses-6.2-cpeGNU-21.12.eb (module: ncurses/6.2-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libreadline/libreadline-8.1-cpeGNU-21.12.eb (module: libreadline/8.1-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12-minimal.eb (module: gettext/0.21-cpeGNU-21.12-minimal) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/PCRE2/PCRE2-10.37-cpeGNU-21.12.eb (module: PCRE2/10.37-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/XZ/XZ-5.2.5-cpeGNU-21.12.eb (module: XZ/5.2.5-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libxml2/libxml2-2.9.12-cpeGNU-21.12.eb (module: libxml2/2.9.12-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libxslt/libxslt-1.1.34-cpeGNU-21.12.eb (module: libxslt/1.1.34-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12.eb (module: gettext/0.21-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/file/file-5.41-cpeGNU-21.12.eb (module: file/5.41-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/util-linux/util-linux-2.37.1-cpeGNU-21.12.eb (module: util-linux/2.37.1-cpeGNU-21.12) + * [ ] /pfs/lustrep3/users/kurtlust/LUMI/LUMI-EasyBuild-contrib/easybuild/easyconfigs/l/libdap/libdap-3.20.9-cpeGNU-21.12.eb (module: libdap/3.20.9-cpeGNU-21.12) +== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-wm_bk3j6/easybuild-puyu_559.log* have been removed. +== Temporary directory /run/user/10012026/easybuild/tmp/eb-wm_bk3j6 has been removed. +``` + +and now instal the library: + +```shell +$ eb libdap-3.20.9-cpeGNU-21.12.eb +== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-kfphjoi8/easybuild-kcs00ai5.log +== processing EasyBuild easyconfig +/pfs/lustrep3/users/kurtlust/LUMI/LUMI-EasyBuild-contrib/easybuild/easyconfigs/l/libdap/libdap-3.20.9-cpeGNU-21.12.eb +== building and installing libdap/3.20.9-cpeGNU-21.12... == fetching files... +== ... (took 2 secs) == creating build dir, resetting environment... == unpacking... == patching... == preparing... +== ... (took 6 secs) == configuring... +== ... (took 1 min 6 secs) == building... +== ... (took 53 secs) == testing... == installing... +== ... (took 5 secs) == taking care of extensions... == restore after iterating... == postprocessing... == sanity checking... +== ... (took 3 secs) == cleaning up... == creating module... +== ... (took 2 secs) == permissions... == packaging... -== COMPLETED: Installation ended successfully (took 17 sec) -== Results of the build can be found in the log file(s) /home/example/easybuild/software/SAMtools/1.11-GCC-10.2.0/easybuild/easybuild-SAMtools-1.11-20210309.105601.log +== COMPLETED: Installation ended successfully (took 2 mins 20 secs) +== Results of the build can be found in the log file(s) +/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/libdap/3.20.9-cpeGNU-21.12/easybuild/easybuild-libdap-3.20.9-20220329.154535.log == Build succeeded for 1 out of 1 -== Temporary log file(s) /tmp/eb-zh7_fyre/easybuild-4q_lo57b.log* have been removed. -== Temporary directory /tmp/eb-zh7_fyre has been removed. +== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-kfphjoi8/easybuild-kcs00ai5.log* have been removed. +== Temporary directory /run/user/10012026/easybuild/tmp/eb-kfphjoi8 has been removed. ``` That was... easy. Is that really all there is to it? Well, almost... ### Enabling dependency resolution -The SAMtools installation worked like a charm, but remember that all required dependencies were already +The libdap installation worked like a charm, but remember that all required dependencies were already available (see [above](#dry-run)). -If we try this with the `BCFtools-1.11-GCC-10.2.0.eb`, for which the required `GSL` and `HTSlib` dependencies are not available yet, it's less successful: +If we try this with the `SAMtools-1.14-cpeGNU-21.12.eb`, for which the required `HTSlib` dependencies is not available yet, it's less successful: ```shell -$ eb BCFtools-1.11-GCC-10.2.0.eb -M +$ eb SAMtools-1.14-cpeGNU-21.12.eb -M -3 out of 23 required modules missing: +2 out of 11 required modules missing: -* GSL/2.6-GCC-10.2.0 (GSL-2.6-GCC-10.2.0.eb) -* HTSlib/1.11-GCC-10.2.0 (HTSlib-1.11-GCC-10.2.0.eb) -* BCFtools/1.11-GCC-10.2.0 (BCFtools-1.11-GCC-10.2.0.eb) +* HTSlib/1.14-cpeGNU-21.12 (HTSlib-1.14-cpeGNU-21.12.eb) +* SAMtools/1.14-cpeGNU-21.12 (SAMtools-1.14-cpeGNU-21.12.eb) ``` ```shell -$ eb BCFtools-1.11-GCC-10.2.0.eb +$ eb SAMtools-1.14-cpeGNU-21.12.eb ... == preparing... -== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/build/BCFtools/1.11/GCC-10.2.0): build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0 (took 2 sec) -== Results of the build can be found in the log file(s) /tmp/eb-3v1dfvnk/easybuild-BCFtools-1.11-20210308.195024.FlxkH.log -ERROR: Build of /home/example/.local/easybuild/easyconfigs/b/BCFtools/BCFtools-1.11-GCC-10.2.0.eb failed (err: 'build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0') +== FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12): build failed +(first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12 (took 3 secs) +== Results of the build can be found in the log file(s) +/run/user/10012026/easybuild/tmp/eb-rgj1v43y/easybuild-SAMtools-1.14-20220329.155911.ZtDcX.log + +ERROR: Build of /appl/lumi/LUMI-EasyBuild-contrib/easybuild/easyconfigs/s/SAMtools/SAMtools-1.14-cpeGNU-21.12.eb failed (err: 'build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12') ``` Oh my, what's this all about? @@ -478,36 +599,33 @@ Oh my, what's this all about? If we filter the output a bit and focus on the actual error, the problem is clear: ``` -Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0 +Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12') ``` -The required dependencies `HTSlib/1.11-GCC-10.2.0` and `GSL/2.6-GCC-10.2.0` are not installed yet, -and EasyBuild does not automatically install missing dependencies unless it is told to do so. +The required dependency `HTSlib/1.14-cpeGNU-21.12` is not installed yet, +and EasyBuild does not automatically install missing dependencies unless it is told to do so +(which we didn't do in the configuration for LUMI). -It helpfully suggests to use the `--robot` command line option, so let's try that: +It helpfully suggests to use the `--robot` (or '-r') command line option, so let's try that: ```shell -$ eb BCFtools-1.11-GCC-10.2.0.eb --robot +$ eb SAMtools-1.14-cpeGNU-21.12.eb --robot ... == resolving dependencies ... ... -== building and installing HTSlib/1.11-GCC-10.2.0... +== building and installing HTSlib/1.14-cpeGNU-21.12... ... == COMPLETED: Installation ended successfully (took 13 sec) ... -== building and installing GSL/2.6-GCC-10.2.0... -... -== COMPLETED: Installation ended successfully (took 1 min 10 sec) -... -== building and installing BCFtools/1.11-GCC-10.2.0... +== building and installing SAMtools/1.14-cpeGNU-21.12... ... == COMPLETED: Installation ended successfully (took 8 sec) ... -== Build succeeded for 3 out of 3 +== Build succeeded for 2 out of 2 ``` -With dependency resolution enabled the `HTSlib` and `GSL` modules get installed first, -before EasyBuild proceeds with installing `BCFtools`. Great! +With dependency resolution enabled the `HTSlib` module gets installed first, +before EasyBuild proceeds with installing `SAMtools`. Great! --- @@ -521,35 +639,50 @@ To provide some more feedback as the installation progresses, you can enable the Let's do this by defining the `$EASYBUILD_TRACE` environment variable, just to avoid having to type `--trace` over and over again. -We will redo the installation of `BCFtools-1.11-GCC-10.2.0.eb` by passing the `--rebuild` +We will redo the installation of `SAMtools-1.14-cpeGNU-21.12.eb` by passing the `--rebuild` option to the `eb` command (try yourself what happens if you don't use the `--rebuild` option!): ```shell $ export EASYBUILD_TRACE=1 -$ eb BCFtools-1.11-GCC-10.2.0.eb --rebuild +$ eb SAMtools-1.14-cpeGNU-21.12.eb --rebuild ... == configuring... >> running command: - [started at: 2021-03-08 19:54:53] - [working dir: /tmp/example/build/BCFtools/1.11/GCC-10.2.0/bcftools-1.11] - [output logged in /tmp/eb-9u_ac0nv/easybuild-run_cmd-17m_he2x.log] - ./configure --prefix=/home/example/easybuild/software/BCFtools/1.11-GCC-10.2.0 --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --with-htslib=$EBROOTHTSLIB --enable-libgsl -== building... + [started at: 2022-03-29 18:46:31] + [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] + [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-g7vd83qv.log] + /users/kurtlust/LUMI-user-appl/sources/generic/eb_v4.5.3/ConfigureMake/config.guess + >> command completed: exit 0, ran in < 1s >> running command: - [started at: 2021-03-08 19:54:54] - [working dir: /tmp/example/BCFtools/1.11/GCC-10.2.0/bcftools-1.11] - [output logged in /tmp/example/eb-9u_ac0nv/easybuild-run_cmd-bhkgjxi7.log] - make -j 8 + [started at: 2022-03-29 18:46:31] + [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] + [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-k0etfv8i.log] + ./configure --prefix=/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/SAMtools/1.14-cpeGNU-21.12 --build=x86_64-pc-linux-gnu +--host=x86_64-pc-linux-gnu --with-htslib=$EBROOTHTSLIB >> command completed: exit 0, ran in 00h00m03s +== ... (took 3 secs) +== building... + >> running command: + [started at: 2022-03-29 18:46:34] + [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] + [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-svcps0yj.log] + make -j 256 CC="cc" CXX="CC" CFLAGS="-O2 -ftree-vectorize -fno-math-errno -fPIC" CXXFLAGS="-O2 -ftree-vectorize -fno-math-errno +-fPIC" + >> command completed: exit 0, ran in 00h00m06s +== ... (took 6 secs) ``` That's a bit more comforting to stare at... -During the *configure* step, the `./configure` command is run with option to -enable support for leveraging `HTSlib` and `GSL`. +SAMtools uses a custom easyblock that is derived from the generic `ConfigureMake` easyblock. +During the *configure* step, the `./configure` command is run with `--build` and `--host` +options added by the generic ConfigureMake easyblock (and the other command, `config.guess` plays +a role in determining the value of those flags). The `--with-htslib=$EBROOTHTSLIB` flag is added +via our easyconfig file to tell SAMtools to use an already available version of HTSlib rather +than the built-in one. During the *build* step, the software is actually being compiled -by running the `make` command. EasyBuild automatically uses the available cores on the system (in this case 8). +by running the `make` command. EasyBuild automatically uses the available cores on the system (in this case 256). We even get a pointer to a log file that contains the output of the command being run, so we can use `tail -f` to see in detail how it progresses. @@ -561,14 +694,19 @@ Later during the installation, we now also see this output during the sanity che ``` == sanity checking... - >> file 'bin/bcftools' found: OK - >> file 'bin/plot-vcfstats' found: OK - >> file 'bin/vcfutils.pl' found: OK - >> (non-empty) directory 'libexec/bcftools' found: OK + >> file 'bin/blast2sam.pl' found: OK + >> file 'bin/bowtie2sam.pl' found: OK + >> file 'bin/export2sam.pl' found: OK + >> file 'bin/interpolate_sam.pl' found: OK +... + >> loading modules: SAMtools/1.14-cpeGNU-21.12... + >> running command 'samtools version' ... + >> result for command 'samtools version': OK ``` -Thanks to enabling trace mode, EasyBuild tells us which files & directories it is checking for -in the installation, before declaring it a success. Nice! +Thanks to enabling trace mode, EasyBuild tells us which files (& directories, but there are non in this case) +it is checking for +in the installation, and which command it is trying to run before declaring it a success. Nice! The extra output you get when trace mode is enabled is concise and hence not overwhelming, while it gives a better insight into what is going on during the installation. @@ -579,13 +717,13 @@ so you can interrupt the installation before it completes, if deemed necessary. ## Using installed software -So far, we have already installed 4 different software packages (SAMtools, HTSlib, GSL, and BCFtools); -we even installed BCFtools twice! +So far, we have already installed 4 different software packages (SAMtools, HTSlib, libdap); +we even installed SAMtools twice! A lot was going on underneath the covers: locating and unpacking the source tarballs, setting up the build environment, configuring the build, compiling, creating and populating the installation directory, performing a quick sanity check on the installation, -cleaning things up, and finally generated the environment module file corresponding to the installation. +cleaning things up, and finally generating the environment module file corresponding to the installation. That's great, but how do we now actually *use* these installations? @@ -593,98 +731,165 @@ This is where the generated module files come into play: they form the access po installations, and we'll use the ubiquitous `module` command to digest them. First, we need to make sure that the modules tool is aware of where the module files for -our installations are located. If you're unsure where EasyBuild is installing stuff at this point, -check the output of `eb --show-config`; the value of the `installpath` configuration setting is what we are interested in now: +our installations are located. On LUMI, when using the EasyBuild-user module to configure EasyBuild, +everything is taken care of for you and the LUMI modules will also automatically add the +suitable module directories for user-installed software to the search path for modules. +By default, EasyBuild-config will install in `$HOME/EasyBuild`, but it is possible to +build the installation in a different directory by pointing to it with the environment +variable `EBU_USER_PREFIX`. Of course this variable needs to be set before loading the `LUMI` +module. (Note that one reason why we don't load a software stack by default is that in +the current setup of LUMI this module would be loaded before the user gets the chance to set +that environment variable in `.bash_profile` or `.bashrc`.) + +When loading the `EasyBuild-user` module, the module command will show you were EasyBuild +will install the software and put the modules, and also put its repository of +processed easyconfig file. ```shell -$ eb --show-config -... -installpath (E) = /home/example/easybuild -... -repositorypath (E) = /home/example/easybuild/ebfiles_repo -... -sourcepath (E) = /home/example/easybuild/sources -... +ml EasyBuild-user + +EasyBuild configured to install software from the LUMI/21.12 software stack for the LUMI/L +partition in the user tree at /users/kurtlust/LUMI-user-appl. + * Software installation directory: /users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L + * Modules installation directory: /users/kurtlust/LUMI-user-appl/modules/LUMI/21.12/partition/L + * Repository: /users/kurtlust/LUMI-user-appl/ebrepo_files/LUMI-21.12/LUMI-L + * Work directory for builds and logs: /run/user/10012026/easybuild + Clear work directory with clear-eb ``` -So, what's in this directory? - -```shell -$ ls -l $HOME/easybuild -total 16 -drwxrwxr-x 5 example example 4096 Jun 10 20:11 ebfiles_repo -drwxrwxr-x 5 example example 4096 Jun 10 20:10 modules -drwxrwxr-x 6 example example 4096 Jun 10 20:10 software -drwxrwxr-x 6 example example 4096 Jun 10 20:10 sources -``` +EasyBuild will copy each easyconfig file it installs to +the repository and add some lines to it with information about the installation. It +also has some options that may edit the source easyconfig, e.g., when asking EasyBuild +to try to build with another toolchain. -The `ebfiles_repo` and `sources` directories correspond to the `repositorypath` and `sourcepath` configuration -settings, respectively. The `modules` and `software` directories are what we need now. +You can always check where EasyBuild is installing stuff by checking the output of +`eb --show-config`. -The `modules` subdirectory consists of multiple subdirectories: +If you're unsure where EasyBuild is installing stuff at this point, +check the output of `eb --show-config`; the value of the `installpath` configuration setting is what we are interested in now: ```shell -$ ls $HOME/easybuild/modules -all bio devel numlib tools +$ eb --show-config +... +buildpath (E) = /run/user/XXXXXXXX/easybuild/build +... +installpath-modules (E) = /users/XXXXXXXX/LUMI-user/modules/LUMI/21.12/partition/L +installpath-software (E) = /users/XXXXXXXX/LUMI-user/SW/LUMI-21.12/L... +... +repositorypath (E) = /users/XXXXXXXX/LUMI-user/ebrepo_files/LUMI-21.12/LUMI-L +... +sourcepath (E) = /users/XXXXXXXX/LUMI-user/sources:/appl/lumi/sources/easybuild +... ``` -Directories like `bio` and `numlib` correspond to different software categories, -and contain symbolic links to the module files in the `all` directory, -which contains all actual module files for software installed in this EasyBuild installation path. -We'll ignore these separate category directories for now. +This is sligthly different from the default EasyBuild setup, where the modules, software, +repository and sources would be installed in respectively the subdirectories `modules`, +`software`, `ebfiles_repo` and `sources` of the directory pointed to by the `installpath` +line. -Let's inform the modules tool about the existence of these module files using `"module use"`: +The modules directory is also a simplified one from the standard EasyBuild one as that also +provides a module categorisation besides a directory containing all modules. As this categorisation +is largely arbitrary and hard to use in the module system, we decided simply not to use it in +our installation and use a custom naming scheme. -```shell -module use $HOME/easybuild/modules/all -``` - -This command does little more that updating the `$MODULEPATH` environment variable, -which contains a list of paths that the modules tool should consider when looking for module files. +However, if you would be using EasyBuild on another system with its default configuration, the +above setup would be used. For more information, we refer to the generic EasyBuild tutorials on the +[EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). Now the modules tool should be aware of our brand new installations: ```shell $ module avail +... +-- EasyBuild managed user software for software stack LUMI/21.12 on LUMI-L --- + HTSlib/1.14-cpeGNU-21.12 libdap/3.20.9-cpeGNU-21.12 + SAMtools/1.14-cpeGNU-21.12 ----------------------- /home/example/easybuild/modules/all ----------------------- - BCFtools/1.11-GCC-10.2.0 GSL/2.6-GCC-10.2.0 SAMtools/1.11-GCC-10.2.0 - EasyBuild/4.3.3 HTSlib/1.11-GCC-10.2.0 bzip2/1.0.6 - ----------------------------- /easybuild/modules/all ----------------------------- - ... +----- EasyBuild managed software for software stack LUMI/21.12 on LUMI-L ----- +... ``` -This output shows both the modules for our own installations as well as the "central" installations in `/easybuild` (which we omitted above for brevity). +This output shows both the modules for our own installations as well as the "central" installations +(which we omitted for brevity). Now we can load these modules and start using these software installations. -Let's test this for BCFtools. In our current environment, the `bcftools` command is not available yet: +Let's test this for SAMtools. In our current environment, the `samtools` command is not available yet: ```shell $ module list -No modules loaded -$ bcftools --bash: bcftools: command not found +Currently Loaded Modules: + 1) perftools-base/21.12.0 + 2) cce/13.0.0 + 3) craype/2.7.13 + 4) cray-dsmml/0.2.2 + 5) cray-mpich/8.1.12 + 6) cray-libsci/21.08.1.2 + 7) PrgEnv-cray/8.2.0 + 8) ModuleLabel/label (S) + 9) init-lumi/0.1 (S) + 10) craype-x86-rome + 11) craype-accel-host + 12) libfabric/1.11.0.4.106 + 13) craype-network-ofi + 14) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta + 15) partition/L (S) + 16) LUMI/21.12 (S) + + Where: + S: Module is Sticky, requires --force to unload or purge +$ samtools +-bash: samtools: command not found ``` -Loading the module for BCFtools changes that: +Loading the module for SAMtools changes that: ```shell -$ module load BCFtools/1.11-GCC-10.2.0 +$ module load SAMtools/1.14-cpeGNU-21.12 + +Lmod is automatically replacing "cce/13.0.0" with "gcc/11.2.0". +Lmod is automatically replacing "PrgEnv-cray/8.2.0" with "cpeGNU/21.12". + +Due to MODULEPATH changes, the following have been reloaded: + 1) cray-mpich/8.1.12 $ module list + Currently Loaded Modules: - 1) GCCcore/10.2.0 6) XZ/5.2.5-GCCcore-10.2.0 - 2) zlib/1.2.11-GCCcore-10.2.0 7) cURL/7.72.0-GCCcore-10.2.0 - 3) binutils/2.35-GCCcore-10.2.0 8) HTSlib/1.11-GCC-10.2.0 - 4) GCC/10.2.0 9) GSL/2.6-GCC-10.2.0 - 5) bzip2/1.0.8-GCCcore-10.2.0 10) BCFtools/1.11-GCC-10.2.0 - -$ bcftools --version -bcftools 1.11 -Using htslib 1.11 + 1) perftools-base/21.12.0 + 2) ModuleLabel/label (S) + 3) init-lumi/0.1 (S) + 4) craype-x86-rome + 5) craype-accel-host + 6) libfabric/1.11.0.4.106 + 7) craype-network-ofi + 8) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta + 9) partition/L (S) + 10) LUMI/21.12 (S) + 11) gcc/11.2.0 + 12) craype/2.7.13 + 13) cray-mpich/8.1.12 + 14) cray-libsci/21.08.1.2 + 15) cray-dsmml/0.2.2 + 16) cpeGNU/21.12 + 17) ncurses/6.2-cpeGNU-21.12 + 18) zlib/1.2.11-cpeGNU-21.12 + 19) bzip2/1.0.8-cpeGNU-21.12 + 20) gettext/0.21-cpeGNU-21.12-minimal + 21) XZ/5.2.5-cpeGNU-21.12 + 22) Brotli/1.0.9-cpeGNU-21.12 + 23) cURL/7.78.0-cpeGNU-21.12 + 24) HTSlib/1.14-cpeGNU-21.12 + 25) SAMtools/1.14-cpeGNU-21.12 + + Where: + S: Module is Sticky, requires --force to unload or purge + +$ samtools --version +samtools 1.14 +Using htslib 1.14 +Copyright (C) 2021 Genome Research Ltd.... ... ``` @@ -696,24 +901,61 @@ by updating environment variables like `$PATH` for example, to make the software To restore your environment to a pristine state in which no modules are loaded, you can either unload the loaded modules one by one using "`module unload`", or you can unload all of them at once using -"`module purge`". +"`module purge`". On LUMI, `module purge` will unload all application modules but will not undo the selection +of the software stack. It will reset the software stack though to use those modules that fit best +with the hardware of the current node (i.e., you may find a different `partition` module). -**If you are using an EasyBuild installation provided by a module, -don't forget to load the `EasyBuild` module again after running "`module purge`".** +```shell +$ module purge +The following modules were not unloaded: + (Use "module --force purge" to unload all): + + 1) LUMI/21.12 6) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta + 2) craype-x86-rome 7) partition/L + 3) craype-accel-host 8) init-lumi/0.1 + 4) libfabric/1.11.0.4.106 9) ModuleLabel/label + 5) craype-network-ofi +$ module list + +Currently Loaded Modules: + 1) LUMI/21.12 (S) + 2) craype-x86-rome + 3) craype-accel-host + 4) libfabric/1.11.0.4.106 + 5) craype-network-ofi + 6) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta + 7) partition/L (S) + 8) init-lumi/0.1 (S) + 9) ModuleLabel/label (S) + + Where: + S: Module is Sticky, requires --force to unload or purge +``` + +Runnin `module --force purge` instead will remove all modules, including the `init-lumi` +module which does part of the initialisation. You will not be able to use the software +stacks completely as before without first loading `init-lumi` in its most recent (or default) +version again! + +```shell +$ module --force purge +$ module list +No modules loaded +``` --- ## Stacking software Maybe you have overlooked how the software we are playing around with was not only installed across multiple -different installation directories per software, we are also "stacking" our own installations (in `$HOME/easybuild`) -on top of installations that are provided in a totally different location (`/easybuild`). +different installation directories per software, we are also "stacking" our own installations (in `$HOME/EasyBuild` +or `$EBU_USER_PREFIX`) on top of installations that are provided in a totally different location (`/appl/lumi`). **EasyBuild doesn't care *where* software is installed: as long as the module file that provides access to it is available, it is happy to pick it up and use it when required.** -This implies that end users of an HPC system can easily install their -own small software stack on top of what is provided centrally by the HPC support team, +This implies that end users of LUMI can easily install their +own small software stack on top of what is provided centrally by the LUMI User Support, for example. They can even "replace" a central software installation for their purposes if they need to, since the modules tool will load the first module file that matches the request being made (there are some caveats with this, but we @@ -723,6 +965,8 @@ won't go into those here). ## Exercises +TODO: Rework for LUMI and provide the necessary EasyConfig files. + ***Guidelines*** Do yourself a favor: don't peek at the solution until you have made an attempt to solve the exercise yourself! @@ -967,10 +1211,6 @@ If you've made it through the hands-on exercises, congratulations! If not, don't worry too much about it. We covered a lot of ground here, and it's a lot to take in at once, take your time... -Feel free to ask question in the `#tutorial` channel in the [EasyBuild -Slack](https://docs.easybuild.io/en/latest/#getting-help), -we're happy to help! - --- *[[next: Part 2: Using EasyBuild]](2_00_part2_using.md)* From c844b969ac44457f17d1e45d54e54f9968d2ed40 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 30 Mar 2022 12:27:23 +0200 Subject: [PATCH 020/159] Additions to the LUMI software stack page of the tutorial. --- .../1_04_LUMI_software_stack.md | 86 ++++++++++++++++++- 1 file changed, 85 insertions(+), 1 deletion(-) diff --git a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md index 0e578a59..3bfae70c 100644 --- a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md +++ b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md @@ -1,9 +1,93 @@ -# LUMI software stacks +# LUMI software stacks (technical) *[[back: The Cray Programming Environment]](1_03_CPE.md)* --- +The user-facing documentation on how to use the LUMI software stacks is +avialable in [the LUMI documentation](https://docs.lumi-supercomputer.eu/computing/softwarestacks/). +On this page we focus more on the technical implementation behind it. + +--- + +# An overview of LUMI + +LUMI has different node types providing compute resources: + +- LUMI has 16 login nodes, though many of those are reserved for special purposes and not + available to all users. TODO + + +--- + +## CrayEnv and LUMI modules + +On LUMI, two types of software stacks are currently offered: + + - ``CrayEnv`` (module name) offers the Cray PE and enables one to use + it completely in the way intended by HPE-Cray. The environment also offers a + limited selection of additional tools, often in updated versions compared to + what SUSE Linux, the basis of the Cray Linux environment, offers. Those tools + are installed and managed via EasyBuild. However, EasyBuild is not available + in that partition. + + It also rectifies a problem caused by the fact that there is only one + configuration file for the Cray PE on LUMI, so that starting a login shell + will not produce an optimal set of target modules for all node types. + The ``CrayEnv`` module recognizes on which node type it is running and + (re-)loading it will trigger a reload of the recommended set of target + modules for that node. + + - ``LUMI`` is an extensible software stack that is mostly managed through + [EasyBuild][easybuild]. Each version of the LUMI software stack is based on + the version of the Cray Programming Environment with the same version + number. + + A deliberate choice was made to only offer a limited number of software + packages in the globally installed stack as the setup of redundancy on LUMI + makes it difficult to update the stack in a way that is guaranteed to not + affect running jobs and as a large central stack is also hard to manage. + However, the EasyBuild setup is such that users can easily install + additional software in their home or project directory using EasyBuild build + recipes that we provide or they develop, and that software will fully + integrate in the central stack (even the corresponding modules will be made + available automatically). + + Each ``LUMI`` module will also automatically activate a set of application + modules tuned to the architecture on which the module load is executed. To + that purpose, the ``LUMI`` module will automatically load the ``partition`` + module that is the best fit for the node. After loading a version of the + ``LUMI`` module, users can always load a different version of the ``partition`` + module. + +Note that the ``partition`` modules are only used by the ``LUMI`` module. In the +``CrayEnv`` environment, users should overwrite the configuration by loading their +set of target modules after loading the ``CrayEnv`` module. + + +--- + +## The ``partition`` module + +The ``LUMI`` module currently supports four partition modules, but that number may +be reduced in the future: + +| Partition | CPU target | Accelerator | +|:----------------|-----------------------|:----------------------------| +| ``partition/L`` | ``craype-x86-rome`` | ``craype-accel-host`` | +| ``partition/C`` | ``craype-x86-milan`` | ``craype-accel-host`` | +| ``partition/G`` | ``craype-x86-trento`` | ``craype-accel-amd-gfx90a`` | +| ``partition/D`` | ``craype-x86-rome`` | ``craype-accel-nvidia80`` | + +All ``partition`` modules also load `craype-network-ofi``. + +``pattition/D`` may be dropped in the future as it seems we have no working CUDA setup +and can only use the GPU nodes in the LUMI-D partition for visualisation and not with CUDA. + +Furthermore if it would turn out that there is no advantage in optimizing for Milan +specifically, or that there are no problems at all in running Milan binaries on Rome +generation CPUs, ``partition/L`` and ``partition/C`` might also be united in a single +partition. From cbdb67ea939779bfc83258d7f966047fc82791ce Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 30 Mar 2022 12:28:17 +0200 Subject: [PATCH 021/159] Integrating parts of the old tutorial into the new one, texts still need to be rewritten. --- docs/2022-CSC_and_LO/2_00_part2_using.md | 7 + docs/2022-CSC_and_LO/2_01_troubleshooting.md | 488 +++++++++ .../2_02_creating_easyconfig_files.md | 998 ++++++++++++++++++ .../2_03_implementing_easyblocks.md | 665 ++++++++++++ docs/2022-CSC_and_LO/3_00_part3_advanced.md | 9 + .../2022-CSC_and_LO/3_01_easybuild_library.md | 293 +++++ docs/2022-CSC_and_LO/3_02_hooks.md | 190 ++++ docs/2022-CSC_and_LO/3_03_slurm_jobs.md | 304 ++++++ .../3_04_module_naming_schemes.md | 617 +++++++++++ .../3_05_github_integration.md | 442 ++++++++ docs/2022-CSC_and_LO/index.md | 4 + mkdocs.yml | 12 + 12 files changed, 4029 insertions(+) create mode 100644 docs/2022-CSC_and_LO/2_00_part2_using.md create mode 100644 docs/2022-CSC_and_LO/2_01_troubleshooting.md create mode 100644 docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md create mode 100644 docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md create mode 100644 docs/2022-CSC_and_LO/3_00_part3_advanced.md create mode 100644 docs/2022-CSC_and_LO/3_01_easybuild_library.md create mode 100644 docs/2022-CSC_and_LO/3_02_hooks.md create mode 100644 docs/2022-CSC_and_LO/3_03_slurm_jobs.md create mode 100644 docs/2022-CSC_and_LO/3_04_module_naming_schemes.md create mode 100644 docs/2022-CSC_and_LO/3_05_github_integration.md diff --git a/docs/2022-CSC_and_LO/2_00_part2_using.md b/docs/2022-CSC_and_LO/2_00_part2_using.md new file mode 100644 index 00000000..e033499c --- /dev/null +++ b/docs/2022-CSC_and_LO/2_00_part2_using.md @@ -0,0 +1,7 @@ +# Part II: Using EasyBuild + +*[[back to start page]](index.md)* + +* [Troubleshooting](2_01_troubleshooting.md) *(hands-on)* +* [Creating easyconfig files](2_02_creating_easyconfig_files.md) *(hands-on)* +* [Implementing easyblocks](2_03_implementing_easyblocks.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/2_01_troubleshooting.md b/docs/2022-CSC_and_LO/2_01_troubleshooting.md new file mode 100644 index 00000000..86a2f1d3 --- /dev/null +++ b/docs/2022-CSC_and_LO/2_01_troubleshooting.md @@ -0,0 +1,488 @@ +# Troubleshooting + +*[[back: Overview]](index.md)* + +--- + +When installing scientific software you are bound to run into problems +that make the installation fail sooner or later, even when using EasyBuild. + +In this part we take a look at how you can **troubleshoot a failing installation**, +and focus on a couple of EasyBuild aspects that can be helpful in that context, +like the **error messages produced by EasyBuild**, the detailed **log file** that is +available for the installation and the **build directory** where most software is +being compiled before it actually gets installed. + +At the end of this part an exercise is available in which you get +to fix a failing installation, so pay attention! + +## EasyBuild error messages + +When EasyBuild detects that something went wrong, it will usually produce a +short (and hopefully helpful) error message. + +Things that could go wrong during an installation include: + +* missing source or patch files; +* a checksum error on a downloaded source or patch file; +* required dependencies that are not specified in the easyconfig file; +* failing shell commands; +* running out of available memory or disk space; +* a segmentation fault caused by a flipped bit triggered by a cosmic ray ([really, it happens!](https://blogs.oracle.com/linux/attack-of-the-cosmic-rays-v2)); + +Unfortunately this is not an exhaustive list, there are plenty of other +potential problems that could result in a failing installation... + +For each of the shell commands that EasyBuild executes during an +installation, it will check the exit status. +If the exit status is zero, EasyBuild will usually assume that the shell command +ran correctly, and it will continue with the rest of the installation procedure. +If the exit status is anything but zero, EasyBuild will assume that a problem +has occurred, and the installation will be interrupted. + +### Example + +Here is an example of an EasyBuild error message (slightly reformatted for clarity): + +``` +$ eb example.eb +... +== building... +== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): +build failed (first 300 chars): cmd "make" exited with exit code 2 and output: +/usr/bin/g++ -O2 -ftree-vectorize -march=native -fno-math-errno -std=c++14 -c -o core.o core.cpp +g++: error: unrecognized command line option '-std=c++14' (took 1 sec) +== Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log +ERROR: Build of /home/easybuild/subread.eb failed (err: ...) +``` + +Let's break this down a bit: during the `build` step of the installation +procedure EasyBuild was running `make` as a shell command, which +failed (exit code 2, so not zero). +The `make` command tripped over the compilation of `core.cpp` that failed because +`-std=c++14` is not a known option to the `g++` command. + +OK fine, but now what? Can you spot something suspicious here? +Wait a minute... Why is `make` using `/usr/bin/g++` for the compilation?! +That's not where our toolchain compiler is installed, +that's somewhere under `/easybuild/software`. + +Let's see what `/usr/bin/g++` is: + +```shell +$ /usr/bin/g++ --version +g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) +``` + +Oh my that's pretty ancient (GCC 4.8.5 was released in June 2015). +That could definitely explain why it doesn't know about the C++14 standard yet... + +Your next step in this case should probably be figuring +out why `/usr/bin/g++` is being used rather than just `g++`, which would +result in using the right compiler version because EasyBuild sets up the build +environment carefully. + +This is a fictitious example of course, but hopefully it gives you a feeling +of how errors that occur during the installation are handled. + +## EasyBuild log files + +Finding the cause of a problem that made the installation fail is, unfortunately, not always that straightforward... + +EasyBuild includes the first 300 characters of the output produced by a failing +shell command in the error message, which is a simple way to try include +actionable information in the error message while not flooding your terminal +with the full output of the command that failed. + +In some cases there won't be any useful information in there however, +since the actual error message(s) could only appear way later, perhaps even after +the command was already running for several minutes. + +In that case, you will have the dive into the log file that is created by EasyBuild for +every installation, which is located in the unique temporary directory for the EasyBuild session. + +See for example this output line from our earlier example error message: + +``` +== Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log +``` + +You can open this file with your favorite text editor or a tool like `less` +to take a look at the information collected in the log file, which includes +things like: + +* informative messages produced by both the EasyBuild framework and the easyblock + describing how the installation is progressing; +* how the build environment was set up: which modules were loaded, which environment variables were set; +* the exact shell commands that were executed, and in which directory they were run; +* the full output produced by these commands, and their exit code; + +Note that the installation log is also copied into each software installation +directory for successful installation, into the `easybuild` subdirectory. +For example: + +``` +/easybuild/software/HDF5/1.10.7-gompi-2020b/easybuild/easybuild-HDF5-1.10.7-20210308.214453.log +``` + +### Last log + +The `eb` command supports a handy little option that prints the location +to the most recently updated build log. You can leverage this to quickly +open the build log of the last failed EasyBuild session in an editor: + +``` +vim $(eb --last-log) +``` + +### Navigating log files + +Usually you want to go to the end of the log file and then work your way up, +either by scrolling or by searching for specific patterns. Here are a couple +of suggestions of patterns you can use to locate errors: + +* `ERROR` +* `Error 1` +* `error: ` +* `failure` +* `not found` +* `No such file or directory` +* `bazel` +* `Segmentation fault` + +Using "`error`" as a search pattern is not very useful: you will hit a lot of log lines +that are not actually errors at all (like the compilation of an `error.c` file), +and you'll miss others that do include errors but mention `ERROR` or `Error` +rather than `error`. + +When using `less` to view a log file, you can navigate it by: + +* hitting '`$`' followed by '`G`' to go to the end of the log file; +* using your arrow keys to scroll up/down; +* typing '`?`' followed by some text and Enter to search backwards for a particular +pattern ('`/`' to search forwards, '`n`' for next match); +* hitting '`q`' to exit; + +It can also be helpful to zoom in on a specific step of the installation procedure, +which you can do by looking for step markers like these: + +``` +== 2020-06-13 01:34:48,816 example INFO configuring... +== 2020-06-13 01:34:48,817 example INFO Starting configure step +... +== 2020-06-13 01:34:48,823 main.EB_HPL INFO Running method configure_step part of step configure +``` + +If you want to look at the start of the output produced by a particular command, +you can look for the log message that looks like this (this is from the installation +log for `HDF5`): + +``` +== 2020-06-09 13:11:19,968 run.py:222 INFO running cmd: make install +== 2020-06-09 13:11:25,186 run.py:538 INFO cmd " make install " exited with exit code 0 and output: +Making install in src +make[1]: Entering directory `/tmp/example/HDF5/1.10.7/gompi-2020b/hdf5-1.10.7/src' +... +``` + +It can be useful to look for the *first* error that occurred in the output of a command, since subsequent errors are +often fallout from earlier errors. You can do this by first navigating +to the start of the output for a command using "`INFO running cmd`" as a search pattern, and then looking for patterns +like "`error:`" from there. + +## Inspecting the build directory + +When an installation fails the corresponding build directory is *not* cleaned up +automatically, that is only done for successful installations. +This allows you to dive in and check for clues in the files that are stored there. + +The location of the build directory is mentioned in the EasyBuild error message: + +``` +== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): ... +``` + +For software using a classic `configure` script, you may have to locate +and inspect the `config.log` file in the build directory to determine the underlying cause of an error. For software using CMake as a configuration tool you often have to check in +`CMakeOutput.log` or `CMakeError.log` for clues, which are sneakily hidden by CMake in a `CMakeFiles` subdirectory of the build directory. + +As a side note here: as EasyBuild does not clean out old and failed builds you will need to eventually manually remove these build directories from the `buildpath` directory. + +## Exercise + +Let's work our way through a less smooth software installation, +using the easyconfig file that is provided below. + +Don't worry if most of this is still unclear to you, we'll get +to writing easyconfig files from scratch [later in this tutorial](../creating_easyconfig_files). + +```python +easyblock = 'MakeCp' + +name = 'Subread' +version = '2.0.1' + +homepage = 'http://subread.sourceforge.net' +description = "High performance read alignment, quantification and mutation discovery" + +toolchain = {'name': 'GCC', 'version': '8.5.0'} + +# download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz +sources = ['subread-%(version)s-source.tar.gz'] +checksums = ['d808eb5b1823c572cb45a97c95a3c5acb3d8e29aa47ec74e3ca1eb345787c17b'] + +start_dir = 'src' + +# -fcommon is required to compile Subread 2.0.1 with GCC 10, +# which uses -fno-common by default (see https://www.gnu.org/software/gcc/gcc-10/porting_to.html) +buildopts = '-f Makefile.Linux CFLAGS="-fast -fcommon"' + +files_to_copy = ['bin'] + +sanity_check_paths = { + 'files': ['bin/featureCounts', 'bin/subread-align'], + 'dirs': ['bin/utilities'], +} + +sanity_check_commands = ["featureCounts --version"] + +moduleclass = 'bio' +``` + +Do you spot any potential problems yet with this easyconfig file? + +***Preparation*** + +Start by copying the text above in a file named `subread.eb`, +so you can gradually fix the problem you'll encounter. + +Also make sure that the pre-installed software stack is available, +and that the EasyBuild module is loaded (unless you installed EasyBuild +yourself): + +``` +module use /easybuild/modules/all +module load EasyBuild +``` + +For this exercise, make sure EasyBuild is configured to +use `$HOME/easybuild` as `prefix`, and to use `/tmp/$USER` as `buildpath`: + +```shell +export EASYBUILD_PREFIX=$HOME/easybuild +export EASYBUILD_BUILDPATH=/tmp/$USER +``` + +Check your configuration via `eb --show-config`. + +Strictly speaking the configuration doesn't matter much for the sake of this +exercise, but it may help with the step-wise approach we'll take and +grasping the solutions. + +Remember though: *no peeking* before you tried to solve each step yourself! + +--- + +***Exercise T.1**** - Sources* + +Try to install the `subread.eb` easyconfig file, see what happens. + +Can you fix the problem you run into, perhaps without even changing +the easyconfig file? + +??? success "(click to show solution)" + + The installation fails because the source file `subread-2.0.1-source.tar.gz` + is not found: + ``` + $ eb subread.eb + ... + == FAILED: Installation ended unsuccessfully (build directory: /tmp/example/Subread/2.0.1/GCC-8.5.0): build failed (first 300 chars): + Couldn't find file subread-2.0.1-source.tar.gz anywhere, and downloading it didn't work either... + Paths attempted (in order): ... + ``` + + In this case, the problem is that the easyconfig file does not specify + where the sources can be downloaded from. Not automatically at least, + but there is a helpful comment included: + ```python + # download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz + sources = ['subread-%(version)s-source.tar.gz'] + ``` + + We can download the source tarball ourselves, + and move it to the location where EasyBuild expects to find it + (in the `sourcepath` directory): + ``` + curl -OL https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz + mv subread-2.0.1-source.tar.gz $HOME/easybuild/sources/s/Subread/ + ``` + + If downloading is problematic for some reason, the source tarball is also available + in `/easybuild/sources/s/Subread`. + + Or, we can change the easyconfig file to specify the location where + the easyconfig file can be downloaded from: + ```python + source_urls = ['https://download.sourceforge.net/subread/'] + sources = ['subread-%(version)s-source.tar.gz'] + ``` + Note that the `source_urls` value is a *list* of candidate download URLs, + *without* the filename of the source file itself. + + This way, EasyBuild will download the source file when running `eb subread.eb`. + + The source tarball is fairly large (23MB), so don't be alarmed if the download takes a little while. + + ```shell + $ ls -lh $HOME/easybuild/sources/s/Subread + total 23M + -rw-rw-r-- 1 easybuild easybuild 23M Jun 13 17:42 subread-2.0.1-source.tar.gz + ``` + +--- + +***Exercise T.2**** - Toolchain* + +After fixing the problem with missing source file, try the installation again. + +What's wrong now? How can you fix it quickly? + +Take into account that we just want to get this software package installed, +we don't care too much about details like the version of the dependencies or +the *toolchain* here... + + +??? success "(click to show solution)" + + The installation fails because the easyconfig specifies that GCC 8.5.0 + should be used as toolchain: + ``` + $ eb subread.eb + ... + == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-8.5.0): build failed (first 300 chars): + No module found for toolchain: GCC/8.5.0 (took 1 sec) + ``` + + We don't have this GCC version installed, but we do have GCC 10.2.0: + + ```shell + $ module avail GCC/ + ----------------- /easybuild/modules/all ------------------ + GCC/10.2.0 + ``` + + So let's try using that instead. + + Edit the easyconfig file so it contains this: + + ```python + toolchain = {'name': 'GCC', 'version': '10.2.0'} + ``` +--- + +***Exercise T.3**** - Build step* + +With the first two problems fixed, now we can actually try to build the software. + +Can you fix the next problem you run into? + +??? success "(click to show solution)" + + The compilation fails, but the error message we see is incomplete due to + EasyBuild truncating the command output (only the 300 first characters of the output are shown): + ``` + == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-10.2.0): build failed (first 300 chars): + cmd " make -j 1 -f Makefile.Linux CFLAGS="-fast -fcommon"" exited with exit code 2 and output: + gcc -mtune=core2 -O3 -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\""2.0.1"\" -D_FILE_OFFSET_BITS=64 -fmessage-length=0 -ggdb -fast -fcommon -c -o core.o core.c + gcc: error: unrecognized command line opti (took 1 sec) + ``` + + If you open the log file and scroll to the end, + the error is pretty obvious: + ``` + gcc: error: unrecognized command line option -fast; did you mean -Ofast? + make: *** [core.o] Error 1 + ``` + + The easyconfig file hard specifies the `-fast` compiler flag via the `CFLAGS` argument to the build command: + ```python + # -fcommon is required to compile Subread 2.0.1 with GCC 10, + # which uses -fno-common by default (see https://www.gnu.org/software/gcc/gcc-10/porting_to.html) + buildopts = '-f Makefile.Linux CFLAGS="-fast -fcommon"' + ``` + + EasyBuild sets up the build environment, so there should be no need + to hard specify compiler flags (certainly not incorrect ones). + The comment above the `buildopts` definition makes it clear that the `-fcommon` + flag *is* required though, because GCC 10 became a bit stricter by + using `-fno-common` by default. Note that we are using `-fcommon` + as an escape mechanism here: it would be better to fix the source code + and create a patch file instead. + + An easy way to fix this problem is to replace the `-fast` with `-Ofast`, + as the compiler error suggests. + + In this case it is advised to change the `CFLAGS` argument that is added + to be build command to replace the `-fast` with `$CFLAGS`, + which is defined in the build environment by EasyBuild. + + ```python + buildopts = '-f Makefile.Linux CFLAGS="$CFLAGS -fcommon"' + ``` + + Note that we need to be careful with quotes here: we use inner double quotes + to ensure that `$CFLAGS` will be expanded to its value when the build command is run. + +--- + +***Exercise T.4**** - Sanity check* + +After fixing the compilation issue, you're really close to getting the installation working, we promise! + +Don't give up now, try one last time and fix the last problem that occurs... + +??? success "(click to show solution)" + + Now the installation itself works but the sanity check fails, + and hence the module file does not get generated: + ``` + $ eb subread.eb + ... + == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-10.2.0): build failed (first 300 chars): + Sanity check failed: sanity check command featureCounts --version exited with code 255 + (output: featureCounts: unrecognized option '--version' + ... + ``` + + If you look at the full output in the log file you can see + that the correct option to check the version of the `featureCounts` command + is "`-v`" rather than "`--version`", so we need to fix this in the easyconfig file. + + Make the following change in the easyconfig file: + ```python + sanity_check_commands = ["featureCounts -v"] + ``` + + After doing so, **you don't have to redo the installation + from scratch**, you can use the `--module-only` option to only run the + sanity check and generate the module file again: + ``` + eb subread.eb --module-only + ``` + +--- + +In the end, you should be able to install Subread 2.0.1 with the GCC 10.2.0 toolchain by fixing the problems with the `subread.eb` easyconfig file. + +Check your work by manually loading the module and checking the version +via the `featureCounts` command, which should look like this: + +```shell +$ featureCounts -v +featureCounts v2.0.1 +``` + +--- + +*[[next: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md new file mode 100644 index 00000000..d46e003a --- /dev/null +++ b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md @@ -0,0 +1,998 @@ +# Creating easyconfig files + +*[[back: Troubleshooting]](2_02_troubleshooting.md)* + +--- + +We already know how to install easyconfig files that are included with EasyBuild, +but what about installing software for which no easyconfig is available yet? + +To do this we will need to **create additional easyconfig files**, +since every software installation performed by EasyBuild is done based on an easyconfig file. + +In this part of the tutorial we will look at the guts of easyconfig files and even create some ourselves! + +## Easyconfigs vs easyblocks + +Before we dive into writing [easyconfig files](../introduction/#easyconfig-files), +let us take a brief look at how they relate to [easyblocks](../introduction/#easyblocks). + +As we discussed [earlier](../introduction/#terminology), an easyconfig file (`*.eb`) is required +for each installation +performed by EasyBuild which specifies the details of the installation (which software +version, toolchain, etc.), while the installation procedure is implemented +in an easyblock (a Python module). + +When can we leverage a *generic easyblock*, perhaps via a "fat" easyconfig file that includes +a lot of carefully defined easyconfig parameters, and when should we use a minimal easyconfig file +together with a custom *software-specific* easyblock? + +This is not an easy question to answer in a general sense, since it depends on several factors: +the complexity of the software you want to get installed, how much flexibility you want, +how "intelligent" the installation procedure should be with respect to the compiler toolchain and dependencies +that are used for the installation, etc. + +In a nutshell, custom software-specific easyblocks are "do once and forget": they are central solution to peculiarities in the installation procedure of a particular software package. + +Reasons to consider implementing a software-specific easyblock rather than using a generic easyblock include: + +* 'critical' values for easyconfig parameters required to make installation succeed; +* toolchain-specific aspects of the build and installation procedure (e.g., configure options); +* interactive commands that need to be run; +* custom (configure) options for dependencies; +* having to create or adjust specific (configuration) files; +* 'hackish' usage of a generic easyblock; +* complex or very non-standard installation procedure; + +Implementing easyblocks is out of scope for this basic tutorial, for more information please consult +the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). + +## Writing easyconfig files + +Writing an easyconfig file boils down to defining a set of easyconfig parameters in a text file, +which we give a '`.eb`' extension by convention. The name of an easyconfig file doesn't matter +when using it directly to install software, but it does matter when EasyBuild needs to find it +to resolve a dependency for example (as we [discussed earlier](../basic_usage/#easyconfig-filenames)). + +The syntax for easyconfig files is *Python syntax*: you are basically defining a bunch of Python variables +that correspond to easyconfig parameters. + +The order in which the easyconfig parameters are defined doesn't matter, but we generally try to strick to a particular +order which roughly corresponds to the order in which the easyconfig parameters are used during the installation. +That is mostly helpful for humans staring at easyconfig files or comparing them. + +### Mandatory parameters + +A limited number of easyconfig parameters are *mandatory*, they must be defined in every easyconfig file: + +* `name`: the name of the software to install; +* `version`: the version of the software to install; +* `homepage`: a URL to the website of the software; +* `description`: a short description of the software; +* `toolchain`: the compiler toolchain to use for the installation; + +**`name`, `version`** + +It should be no surprise that specifying the name and version of the software you want to install is mandatory. +This information may influence the value of several other easyconfig parameters (like the name of the source file), and is also used to the determine the name of the module file to install. + +```python +name = 'example' +version = '1.0' +``` + +**`homepage`, `description`** + +The homepage and description are included in the generated module file for the installation. +That way the "`module show`" command provides some useful high-level information about the installation. + +```python +homepage = 'https://example.org' +description = "This is just an example." +``` + +Usually it does not matter whether you use single or double quotes to specify string values, +but you will often see that single quotes are used for values that don't have spaces (words) +and double quotes for values that do have spaces (sentences). There is no technical reason for +this, it just feels more natural to some people. There are cases where it is important to use +the right type of quotes however, we will get back to that later (keep it in mind for the exercises!). + +For multi-line descriptions, you will need to use "triple quoting" (which is standard Python syntax): + +```python +description = """This is an example + of a multi-line description. + It is spread across multiple lines.""" +``` + +**`toolchain`** + +EasyBuild also requires that the [compiler toolchain](../introduction/#toolchains) is specified, via the `toolchain` +easyconfig parameter. + +This can either be the [`system` toolchain](../introduction/#system-toolchain), for which a constant named `SYSTEM` is available: + +```python +toolchain = SYSTEM +``` + +Usually we specify a 'proper' toolchain like the compiler-only toolchain GCC 10.2.0 which we used before, +or the full toolchain `foss` 2020b. The name and version of the toolchain can be specified using a small Python dictionary, +for example: + +```python +toolchain = {'name': 'GCC', 'version': '10.2.0'} +``` + +### Commonly used parameters + +You will often need to specify additional easyconfig parameters to get something useful done. +We will cover the most commonly used ones here, but keep in mind that these are *not* mandatory. + +A full overview of all known easyconfig parameters can be obtained via "`eb --avail-easyconfig-params`" +or just "`eb -a`" for short, or can be consulted in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_parameters.html). + +#### Sources, patches, and checksums + +In most easyconfig files you will see that a list of source files is specified via the `sources` +easyconfig parameter, usually combined +with one or more URLs where these sources can be downloaded specified via `source_urls`. +There also may be patch files listed (specified via `patches`), +and checksums for both the source files and patches (specified via `checksums`). + +The `sources` easyconfig parameter is commonly defined but it is *not* mandatory, +because some easyconfig files only specify bundles of software packages and hence only +serve to generate a module file. + +Here is an example of how these easyconfig parameters can be specified: + +```python +source_urls = [ + 'https://example.org/download/', + 'https://example.org/download/archive/', +] +sources = ['example-1.0-src.tar.gz'] +patches = ['example-fix.patch'] +checksums = [ + '9febae18533d035ac688d977cb2ca050e6ca8379311d7a14490ad1ef948d45fa', + '864395d648ad9a5b75d1a745c8ef82b78421d571584037560a22a581ed7a261c', +] +``` + +Each of these require a *list* of values, so even if there is only a single source file or download URL +you must use square brackets as shown in the example. The default value for each of these is an empty list (`[]`). + +Some things worth pointing out here: + +* The download URLs specified via `source_urls` do *not* include the name of the file, that is added + automatically by EasyBuild when it tries to download the file (only if it's not available already.) +* If multiple download URLs are specified, they are each tried once in order until the download of the source file was + successful. This can be useful to include backup locations where source files can be downloaded from. +* Names of source files and patches should not include hardcoded software versions, they usually use a + template value like `%(version)s` instead: + ```python + sources = ['example-%(version)s-src.tar.gz'] + ``` + EasyBuild will use the value of the `version` easyconfig parameter to determine the actual name of the source + file. This way the software version is only specified in one place and the easyconfig file is easier to + update to other software versions. A list of template values can be consulted via the EasyBuild command + line via the `--avail-easyconfig-templates` option, or in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_templates.html). +* Source files can also be specified in ways other than just using a filename, see the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-param-sources-alt) for more information. +* Specified checksums are usually SHA256 checksum values, but [other types are also supported](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html?highlight=checksums#checksums). + + +#### Easyblock + +The easyblock that should be used for the installation can be specified via the `easyblock` easyconfig parameter. + +This is not mandatory however, because by default EasyBuild will determine the easyblock to use based on the +name of the software. If '`example`' is specified as software name, EasyBuild will try to locate a +software-specific easyblock named `EB_example` (in a Python module named `example.py`). Software-specific +easyblocks follow the convention that the class name starts with `'EB_`', followed by the software name +(where some characters are replaced, like '`-`' with '`_minus_`'). + +**Generic easyblocks** + +Usually the `easyblock` value is the name of a *generic* easyblock, if it is specified. The name of +a generic easyblock does *not* start with '`EB_`', so you can easily distinguish it from a software-specific +easyblock. + +Here are a couple of commonly used generic easyblocks: + +* `ConfigureMake`: implements the standard `./configure`, `make`, `make install` installation procedure; +* `CMakeMake`: same as `ConfigureMake`, but with `./configure` replaced with `cmake` for the configuration step; +* `PythonPackage`: implements the installation procedure for a single Python package, by default using + "`python setup.py install`" but other methods like using "`pip install`" are also supported; +* `Bundle`: a simple generic easyblock to bundle a set of software packages together in a single installation directory; +* `PythonBundle`: a customized version of the `Bundle` generic easyblock to install a bundle of Python packages + in a single installation directory; + +A full overview of the available generic easyblocks is available in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/generic_easyblocks.html). You can also consult the output of +`eb --list-easyblocks`, which gives an overview of *all* known easyblocks, and how they relate to each other. + +**Custom easyconfig parameters** + +Most generic easyblocks provide additional easyconfig parameters to steer their behaviour. +You can consult these via "`eb -a --easyblock`" or just "`eb -a -e`", which results in an +additional "`EASYBLOCK-SPECIFIC`" section to be added. See the (partial) output of this command for example: + +```shell +$ eb -a -e ConfigureMake +Available easyconfig parameters (* indicates specific to the ConfigureMake easyblock): +... +EASYBLOCK-SPECIFIC +------------------ +build_cmd* Build command to use [default: "make"] +build_type* Value to provide to --build option of configure script, e.g., x86_64-pc-linux-gnu (determined by config.guess shipped with EasyBuild if None, False implies to leave it up to the configure script) [default: None] +configure_cmd* Configure command to use [default: "./configure"] +configure_cmd_prefix* Prefix to be glued before ./configure [default: ""] +host_type* Value to provide to --host option of configure script, e.g., x86_64-pc-linux-gnu (determined by config.guess shipped with EasyBuild if None, False implies to leave it up to the configure script) [default: None] +install_cmd* Build command to use [default: "make install"] +prefix_opt* Prefix command line option for configure script ('--prefix=' if None) [default: None] +tar_config_opts* Override tar settings as determined by configure. [default: False] +``` + +#### Dependencies + +You will often need to list one or more [dependencies](../introduction/#dependencies) that are required +to install or run the software. +We distinguish between two main different types of dependencies: runtime dependencies and build dependencies. + +*Runtime dependencies* are required for using the installed software, and may also have to be available +during the installation. These dependencies can be specified via the `dependencies` easyconfig parameter. +EasyBuild will load the modules for these dependencies when setting up the build environment, +and will include load statements for them in the generated module file. + +*Build dependencies* are only required during the installation of the software, not for using the +software once it is installed. The modules for these dependencies will be loaded in the build environment +set up by EasyBuild during the installation, but they will *not* be loaded by the generated module file. +You can specify build dependencies via the `builddependencies` easyconfig parameter. +One typical example of a build dependency is `CMake`, which is only needed for configuring +the build. + +Here is a simple example of specifying dependencies: + +```python +builddependencies = [('CMake', '3.18.4')] + +dependencies = [ + ('Python', '3.8.2'), + ('HDF5', '1.10.6'), + ('SciPy-bundle', '2020.03', '-Python-%(pyver)s'), +] +``` + +Both `builddependencies` and `dependencies` require a list of tuples, +each of which specifying one dependency. +The name and version of a dependency is specified with a 2-tuple (a tuple with two string values). + +In some cases additional information may have to be provided, as is shown in the example above for the `SciPy-bundle` +dependency where a 3rd value is specified corresponding to the `versionsuffix` value of this dependency. +If this is not specified, it is assumed to be the empty string (`''`). + +Note how we use the '`%(pyver)s'` template value in the `SciPy-bundle` dependency +specification, to avoid hardcoding the Python version in different places. + +See also the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#dependencies) for additional options on specifying dependencies. + + +#### Version suffix + +In some cases you may want to build a particular software package in different configurations, or include a label in the module name to highlight a particular aspect +of the installation. + +The `versionsuffix` easyconfig parameter can be used for this purpose. The name of this parameter implies that this label will be added after the +software version (and toolchain label) in the standard module naming scheme. + +If you are configuring the software to build with a particular non-default value, +you can indicate this as follows for example: + +```python +versionsuffix = '-example-label' +``` + +This mechanism is used frequently to indicate that a software installation depends +on a particular version of Python, where the `%(pyver)s` template comes in useful again: + +```python +versionsuffix = '-Python-%(pyver)s' +... +dependencies = [('Python', '2.7.18')] +``` + +Even though Python 2 is officially dead and +buried some scientific software still requires it, and mixing modules where +some use Python 2 and other use Python 3 doesn't work well. +The `versionsuffix` label is helpful to inform the user that a particular Python version is required by the installation. + +#### Customizing configure, build, test, and install commands + +When using a generic easyblock like `ConfigureMake` or `CMakeMake`, you will often +find yourself having to specify options to the configure, build, test, or install commands, or to inject additional commands right before them. + +For this the following standard easyconfig parameters are available: + +* `preconfigopts`: string value that is glued *before* the configure command; +* `configopts`: string value that is added *after* the configure command, which can be used to specify configuration options; + +Equivalent easyconfig parameters are available for the `build`, `test` and `install` steps: `prebuildopts`, `buildopts`, `pretestopts`, `testopts`, `preinstallopts`, and `installopts`. + +Here is a fictitious example of how they can be used: + +```python +easyblock = 'ConfigureMake' +... +dependencies = [('HDF5', '1.10.6')] +... +configopts = '--enable-hdf5-support' + +prebuildopts = 'export HDF5_PREFIX="$EBROOTHDF5" && ' + +installopts = "PREFIX='%(installdir)s'" +``` + +Here we are: + +* Adding the `--enable-hdf5-support` configure option, to convince the `ConfigureMake` easyblock to run the following + command during the configure step: + ```shell + ./configure --prefix ... --enable-hdf5-support + ``` + (where the '`...`' represents the path to installation directory where the software should be installed). + +* Specifying that an additional command has to be run before running `make` in the build step. We use '`&&`' to glue the + command to the `make` command, so `make` will only be run if the command we specified ran correctly. So, the build step will run something like: + ```shell + export HDF5_PREFIX="$EBROOTHDF5" && make -j 4 + ``` + The '`4`' value passed to the `-j` option shown here, which specifies how many commands `make` can run in parallel, is automatically determined by EasyBuild based on the number of available cores (taking into account `ulimit` settings, and cpuset and cgroup restrictions). + +* Passing the location where the software should be installed via the `PREFIX` argument to the `make install` command during the installation step. This results in the + following command being run: + ```shell + make install PREFIX=... + ``` + (where the '`...`' again represents the path to installation directory). + Even though the + installation directory is already specified in the configure command, it is + apparently blatantly ignored by the software we are installing here, and we are expected to specify it + this way instead. How rude! + +The `$EBROOTHDF5` environment variable that we are using in `prebuildopts` corresponds to the path of +the installation directory of the HDF5 dependency. EasyBuild includes a statement +to define an `$EBROOT*` environment variable +like this in every environment module file it generates (see the output of "`module show HDF5`"). + +#### Sanity check + +One seemingly trivial yet important aspect of the installation procedure that EasyBuild performs +is the sanity check step. + +By default EasyBuild does a simple sanity check that verifies whether there is a non-empty `bin` subdirectory +in the installation, next to a non-empty `lib` or `lib64` directory (either is sufficient). + +It is recommended to customize the sanity check and check for something more specific, like a particular +binary or directory, or making sure that a trivial command (like `example -V` or `example --help`) +runs correctly. + +To specify a custom set of files and/or directories to check, +you can use the `sanity_check_paths` easyconfig parameter. The expected value is Python dictionary +with two keys: `files` and `dirs`. For example: + +```python +sanity_check_paths = { + 'files': ['bin/example'], + 'dirs': ['examples/one', 'examples/two'], +} +``` + +In addition, you can specify one or more commands that should be working without a problem (that is, have a zero exit status) via the `sanity_check_commands` easyconfig parameter. +These commands will be run just like a user would: after loading the module that was generated for this installation. +Here is an example: + +```python +sanity_check_commands = [ + "example --version", + "example --help", +] +``` + + +#### Module class + +Finally, you will usually see the `moduleclass` easyconfig parameter to be defined as well, for example: + +```python +moduleclass = 'lib' +``` + +This is done to categorize software, and it is used to group the generated module files into smaller sets ([remember what we saw when installing software earlier](../basic_usage/#using-installed-software)). + +## Generating tweaked easyconfigs + +Sometimes you may want to install software that differs only slightly from an +existing easyconfig file, like a newer software version or using a different +compiler toolchain. Do we need to create an easyconfig file for this too? + +We do, but EasyBuild does provide some help so you don't need to *manually* +create the easyconfig file. You can use one of the `--try-*` options provided +by the `eb` command to make EasyBuild *generate* a new easyconfig file based on +an existing one. + +For example, to try installing a different software version you can use the `--try-software-version` option: + +```shell +eb example-1.2.3.eb --try-software-version 1.2.4 +``` + +Or, to try using a different compiler toolchain you can use `--try-toolchain`: + +```shell +eb example-1.2.3-foss-2020b.eb --try-toolchain intel,2020b +``` + +It is important to keep in mind the *"try"* aspect here: while easyconfigs that +are generated by EasyBuild via a `--try-*` option often do work fine, there is +no strong guarantee they will. Newer software versions may come with changes to +the installation procedure, additional dependencies that are required, etc. +Using a different compiler toolchain may be as simple as just switching one for +another, but it may require additional changes to be made to configure options, for example. + +## Copying easyconfigs + +One additional handy command line option we want to highlight is `--copy-ec`, which can be used to +copy easyconfig files to a specific location. That may sound trivial, but +keep in mind that you can specify easyconfigs to the `eb` command using only +the filename, and letting the robot search mechanism locate them. + +So to copy an easyconfig file, we would have to use `eb --search` first to +get the full location to it, copy-paste that, and then use the `cp` command. + +It is a lot easier with `--copy-ec`: + +```shell +$ eb --copy-ec SAMtools-1.11-GCC-10.2.0.eb SAMtools.eb +... +SAMtools-1.10-GCC-10.2.0.eb copied to SAMtools.eb +``` + +If you omit the target location, the easyconfig file will simply be copied +to the current working directory, retaining the original filename. + +You can copy multiple easyconfig files at once, as long as the target location +is an existing directory. + +## Example + +By means of example, we are going to puzzle together an easyconfig file to install the +example software package `eb-tutorial`. + +The sources for `eb-tutorial` version 1.0.1 are available at: + +``` +https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.0.1.tar.gz +``` + +You can consult the unpacked sources at [https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1). + +### Preparation + +Make sure EasyBuild is properly configured before you start: + +```shell +export EASYBUILD_PREFIX=$HOME/easybuild +export EASYBUILD_BUILDPATH=/tmp/$USER +``` + +and that the installed software in `/easybuild` is available: + +```shell +module use /easybuild/modules/all +``` + +### Mandatory parameters + +Let's start by getting the mandatory easyconfig parameters defined in the easyconfig file: + +```python +name = 'eb-tutorial' +version = '1.0.1' + +homepage = 'https://easybuilders.github.io/easybuild-tutorial' +description = "EasyBuild tutorial example" +``` + +If we try using this (very) minimal easyconfig file, EasyBuild will inform us that we failed to specify one of the mandatory easyconfig parameters: `toolchain`: + +``` +$ eb example.eb +== Temporary log file in case of crash /tmp/eb-90j723rl/easybuild-q21plqvx.log +== found valid index for /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs, so using it... +ERROR: Failed to process easyconfig /home/example/example.eb: mandatory parameters not provided in pyheader: toolchain +``` + +We will use `GCC/10.2.0` as toolchain, since we know it is already installed in `/easybuild` in the prepared environment, so we also define the `toolchain` easyconfig parameter: + +```python +toolchain = {'name': 'GCC', 'version': '10.2.0'} +``` + +In addition, we'll also specify the `moduleclass`. +This is not required, but it is usually set to a sensible value: + +```python +moduleclass = 'tools' +``` + +The default value is '`base`', at least '`tools`' has *some* meaning. + +### Easyblock + +Let us see what happens if we take our current easyconfig file for a spin: + +```shell +$ eb example.eb +== temporary log file in case of crash /tmp/eb-8_vxjfn7/easybuild-k3aaoan2.log +ERROR: Failed to process easyconfig /home/example/example.eb: +No software-specific easyblock 'EB_eb_minus_tutorial' found for eb-tutorial +``` + +That didn't get us very far... + +The error shows that there is no software-specific easyblock available for installing the software with the name '`eb-tutorial`'. +Does that mean we have to implement an easyblock? + +In this simple case it doesn't, since we can leverage one of the available *generic easyblocks*. +But, which one? + +Build instructions are usually included in a `README` file, or in the documentation. +In this case, there's indeed a minimal [`README` +file](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1/README) available, +which tells us that we should use the `cmake` command to configure the installation, followed by `make` and `make install`. + +[We briefly discussed](#easyblock) a generic easyblock that does exactly this: `CMakeMake`. + +```python +easyblock = 'CMakeMake' +``` + +The "`easyblock =`" line is usually at the top of the easyconfig file, but strictly speaking +the order of the parameter definitions doesn't matter (unless one is defined in terms of another one). + +### CMake build dependency + +Does using the `CMakeMake` generic easyblock help at all? + +``` +$ eb example.eb +== temporary log file in case of crash /tmp/eb-yutbor1p/easybuild-4jc9v1u9.log +== found valid index for /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs, so using it... +== processing EasyBuild easyconfig /home/example/example.eb +== building and installing eb-tutorial/1.0.1-GCC-10.2.0... +== fetching files... +== creating build dir, resetting environment... +== unpacking... +== patching... +== preparing... +== configuring... +== FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/ebtutorial/1.0.1/GCC-10.2.0): +build failed (first 300 chars): cmd " cmake -DCMAKE_INSTALL_PREFIX=/home/example/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER='gcc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno' -DCMAKE_CXX_COMPILER='g++' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=native (took 0 sec) +``` + +It did help: EasyBuild made an attempt to configure the build using the `cmake` command, but that failed almost +instantly. We need to dive into the log file to see the actual reason. By starting at the end of the log file and +scrolling up, you should be able to locate the following error message: + +``` +/bin/bash: cmake: command not found +``` + +Ah, that explains it, `cmake` isn't even installed on this system. Or is it? + +```shell +$ module avail CMake + +--------------------------- /easybuild/modules/all ---------------------------- + CMake/3.18.4-GCCcore-10.2.0 +``` + +Since a module is available for `CMake` that is compatible with the toolchain we are using (GCC 10.2.0), +we can use it as a dependency for the installation. +It is only needed for building the software, not for running it, so it's only a *build* dependency: + +```python +builddependencies = [('CMake', '3.18.4')] +``` + +There is usually no need to specify toolchain for (build) dependencies, EasyBuild will automatically consider +[subtoolchains](../terminology/#toolchains) compatible with the specified toolchain to locate module for the dependencies. + +You can verify this via `eb -D` (equivalent with `eb --dry-run`): + +``` +$ eb example.eb -D + ... + * [x] /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs/g/GCC/GCC-10.2.0.eb (module: GCC/10.2.0) + * [x] /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs/c/CMake/CMake-3.18.4-GCCcore-10.2.0.eb (module: CMake/3.18.4-GCCcore-10.2.0) + * [ ] /home/example/example.eb (module: eb-tutorial/1.0.1-GCC-10.2.0) +``` + +### Sources + +If you try again after adding `CMake` as a build dependency, you will see the installation fail again in the +configuration step. Inspecting the log file reveals this: + +``` +CMake Error: The source directory "/tmp/example/ebtutorial/1.0.1/GCC-10.2.0" does not appear to contain CMakeLists.txt. +``` + +Wait, but there *is* a `CMakeLists.txt`, we can see it in the [unpacked sources](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1)! + +Let's inspect the build directory: + +``` +$ ls /tmp/$USER/ebtutorial/1.0.1/GCC-10.2.0 +easybuild_obj +$ ls /tmp/$USER/ebtutorial/1.0.1/GCC-10.2.0/easybuild_obj +$ +``` + +There's nothing there at all! And that's not strange because we didn't actually +specify any sources in our easyconfig file... + +The `sources` easyconfig parameter is commonly defined but it is *not* mandatory, +because some easyconfig files only specify bundles of software packages and hence only +serve to generate a module file. + +So we need to specify one or more source files that should be used, +via the `sources` easyconfig parameter which specifies a *list* of +names of source files: + +```python +sources = ['eb-tutorial-1.0.1.tar.gz'] +``` + +We can avoid hardcoding the version number here by using a *template value*: + +```python +sources = ['eb-tutorial-%(version)s.tar.gz'] +``` + +And since this is a standard way of naming software files, there's +even a constant available that we can use: + +```python +sources = [SOURCE_TAR_GZ] +``` + +That way, we only have the software version specified *once* in the easyconfig file, +via the `version` easyconfig parameter. That will come in useful later (see [Exercise 7.2](#exercises))... + +If now we try installing the easyconfig file again, EasyBuild complains +that it can't find the specified source file anywhere: + +``` +Couldn't find file eb-tutorial-1.0.1.tar.gz anywhere, and downloading it didn't work either... +``` + +To let EasyBuild automatically download the source file if it is not available yet, +we have to specify *where* it can be downloaded. This is done via `source_urls`: + +```python +source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] +sources = [SOURCE_TAR_GZ] +``` + +### Required configure option + +With `sources` and `source_urls` defined, we can try again. Yet again we see the configure step fail. +Is this a ruse to make you hate CMake with a passion? Maybe... + +Here's what we find in the log file: + +``` +CMake Error at CMakeLists.txt:7 (message): + EBTUTORIAL_MSG is not set! +``` + +Apparently the `eb-tutorial` software has a required configure option. It's almost as if that +was done on purpose, how silly! + +Options to the configure command can be specified by the `configopts` easyconfig parameter. +To define the value of a CMake option, we need to use `-DNAME_OF_OPTION`, so: + +```python +configopts = "-DEBTUTORIAL_MSG='Hello from the EasyBuild tutorial!' " +``` + +We need to be a little bit careful with quotes here. If we use outer double quotes, +we have to use single quotes to specify the actual value for the `EBTUTORIAL_MSG` configure option. +That works fine here, but that's not always the case! +In some cases we will have to use inner doubles quotes, for example to get environment variables +expanded when the configure command is run (see [Exercise 7.1](#exercises)). + +### Sanity check + +Hopefully that brings us closer to getting the installation to work... + +``` +$ eb example.eb +.... +== sanity checking... +== FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/ebtutorial/1.0.1/GCC-10.2.0): build failed (first 300 chars): +Sanity check failed: no (non-empty) directory found at 'lib' or 'lib64' in /home/easybuild/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 (took 2 sec) +``` + +It got all the way to the sanity check step, that's great! + +The sanity check failed because no '`lib`' or `'lib64'` directory was found. +Indeed: + +``` +$ ls $HOME/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 +bin +$ ls $HOME/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0/bin +eb-tutorial +``` + +There is only a binary named `eb-tutorial` in the `bin` subdirectory. +So we need to customize the standard sanity check: + +```python +sanity_check_paths = { + 'files': ['bin/eb-tutorial'], + 'dirs': [], +} +``` + +Since we want to obtain a *working* installation, we might as well try to run this `eb-tutorial` command as well: + +```python +sanity_check_commands = ['eb-tutorial'] +``` + +Let us now retry, but use `--module-only` rather than redoing the whole installation. +`--module-only` still sanity checks the installation, so if it creates +a module, we know it will work as expected. +By enabling trace mode via `--trace` we can get some more information too: + +```shell +$ eb example.eb --module-only --trace +... +== sanity checking... + >> file 'bin/eb-tutorial' found: OK + >> running command 'eb-tutorial' ... + >> result for command 'eb-tutorial': OK +... +== COMPLETED: Installation ended successfully (took 4 sec) +``` + +Yes, great success! + +To convince yourself that the installation works as intended, try to load the `eb-tutorial` module and +run the `eb-tutorial` command yourself: + +``` +$ module use $HOME/easybuild/modules/all +$ module load eb-tutorial +$ eb-tutorial +Hello from the EasyBuild tutorial! +``` + +### Complete easyconfig + +Here is the complete easyconfig we puzzled together for this example: + +```python +easyblock = 'CMakeMake' + +name = 'eb-tutorial' +version = '1.0.1' + +homepage = 'https://easybuilders.github.io/easybuild-tutorial' +description = "EasyBuild tutorial example" + +toolchain = {'name': 'GCC', 'version': '10.2.0'} + +source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] +sources = [SOURCE_TAR_GZ] +checksums = ['d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9'] + +builddependencies = [('CMake', '3.18.4')] + +configopts = "-DEBTUTORIAL_MSG='Hello from the EasyBuild tutorial!' " + +sanity_check_paths = { + 'files': ['bin/eb-tutorial'], + 'dirs': [], +} + +sanity_check_commands = ['eb-tutorial'] + +moduleclass = 'tools' +``` + +One additional easyconfig parameter we defined here is `checksums`, +which specifies SHA256 checksums for source (and patch) files, +so EasyBuild can verify them before performing an installation. + +You can let EasyBuild determine *and* inject these SHA256 checksums +automatically via `eb --inject-checksums`: + +``` +$ eb example.eb --inject-checksums +... +== injecting sha256 checksums for sources & patches in example.eb... +== * eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 +``` + +--- + +## Exercises + +--- + +***Exercise E.1**** - Making `eb-tutorial` a bit more personal* + +Change the easyconfig file for `eb-tutorial` to make the message printed by the `eb-tutorial` command +a bit more personal: include the username of the account that was used to install the software in it +(using the `$USER` environment variable). + +??? success "(click to show solution)" + For this we need to change the value that is passed to the `EBTUTORIAL_MSG` configure option: + ```python + configopts = '-DEBTUTORIAL_MSG="Hello from the EasyBuild tutorial! I was installed by $USER." ' + ``` + Here we have to use inner double quotes, to ensure that the `$USER` environment variable is expanded + by the shell when running the `cmake` configure command. + + When you run the `eb-tutorial` command yourself, you should get output like this (not a message that + includes a literal '`$USER`' string): + + ```shell + Hello from the EasyBuild tutorial! I was installed by example. + ``` + + To re-install the `example.eb` easyconfig, you will need to use `eb --rebuild` or `eb --force`. + +--- + +***Exercise E.2**** - Installing eb-tutorial version 1.1.0* + +Install version 1.1.0 of the `eb-tutorial` example software, +which is a trivial version bump compared to version 1.0.1. + +The sources are available via: + +``` +https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.1.0.tar.gz +``` + +You can leverage the `eb-tutorial` easyconfig file we have composed in the example above, +**but you should not make any manual changes to it!** + +??? success "(click to show solution)" + You can use the `--try-software-version` option for this: + ```shell + $ eb example.eb --try-software-version 1.1.0 + ... + == building and installing eb-tutorial/1.1.0-GCC-10.2.0... + ... + == COMPLETED: Installation ended successfully (took 4 sec) + ``` + To test: + ``` + $ module use $HOME/easybuild/modules/all + $ module load eb-tutorial/1.1.0-GCC-10.2.0 + $ eb-tutorial + I have a message for you: + Hello from the EasyBuild tutorial! + ``` + (`eb-tutorial` version 1.0.1 doesn't print "`I have a message for you:`") + +--- + +***Exercise E.3**** - Installing py-eb-tutorial 1.0.0* + +Try composing an easyconfig file for the `py-eb-tutorial` example software, which is a tiny Python package. +The source tarball can be downloaded from this link: [py-eb-tutorial-1.0.0.tar.gz](https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/py-eb-tutorial-1.0.0.tar.gz). + +A couple of tips: + +* There is a [generic easyblock](#easyblock) available for installing Python packages, which will come in useful here. + +* By default EasyBuild performs an `import` check when install Python packages, using a Python module name that is derived from the software name by default, which will be incorrect in this case. You can specify the correct name to use in the import check by specifying it via the `options` +easyconfig parameter in your easyconfig file: + ```python + options = {'modulename': 'example'} + ``` + (you will need to change '`example`' here, of course) + +* Leverage the software that is already pre-installed in `/easybuild` in the prepared environment. + Remember that some already installed modules may be a *bundle* of a couple of other software packages. + +Please also take this into account: + +* Unfortunately this software doesn't come with documentation. That is done to make it an example that + is representative for software that you may run into in the wild (it's *not* because + we were lazy when preparing the exercises, really!). + You can inspect the sources of this software [here](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/py-eb-tutorial-1.0.0). Definitely take a look at the `setup.py` file, it includes some clues + about the requirements to get this software installed. + +* Make sure the installation actually *works*, by checking that the `py-eb-tutorial` command runs correctly. + Maybe you will need to make sure other required software is available as well, for it to work correctly... + + +??? success "(click to show solution)" + + Here is a complete working easyconfig file for `py-eb-tutorial`: + ```python + + easyblock = 'PythonPackage' + + name = 'py-eb-tutorial' + version = '1.0.0' + versionsuffix = '-Python-%(pyver)s' + + homepage = 'https://easybuilders.github.io/easybuild-tutorial' + description = "EasyBuild tutorial Python example" + + source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] + sources = [SOURCE_TAR_GZ] + checksums = ['fcf73a9efc65527a210b993e8889d41ebf05977eef1f6a65ebac3188152cd496'] + + toolchain = {'name': 'foss', 'version': '2020b'} + + dependencies = [ + ('Python', '3.8.6'), + ('SciPy-bundle', '2020.11'), + ('eb-tutorial', '1.0.1'), + ] + + use_pip = True + + options = {'modulename': 'eb_tutorial'} + + sanity_check_paths = { + 'files': ['bin/py-eb-tutorial'], + 'dirs': ['lib/python%(pyshortver)s/site-packages'], + } + + sanity_check_commands = ["py-eb-tutorial"] + + moduleclass = 'tools' + ``` + + Some remarks: + + * We used the `PythonPackage` generic easyblock. There is also a `PythonBundle` easyblock for installing + bundles of Python packages, which is used for `SciPy-bundle` for example. But we don't need that here, + since we are only dealing with a single Python package. + + * The `versionsuffix` is not strictly needed, but it's common to tag Python packages with the Python version + for which they were installed. + * The SHA256 checksum for the source tarball was added automatically via `eb py-eb-tutorial.eb --inject-checksums`. + + * `py-eb-tutorial` only wants to be installed with `pip install`, so we had to set `use_pip = True`. + You can consult the custom easyconfig parameters supported by the `PythonPackage` easyblock via + "`eb -a -e PythonPackage`", see the `EASYBLOCK-SPECIFIC` part of the output. + Even when the default installation mechanism used by `PythonPackage` + (which consists of running `python setup.py install`) works fine, + it is recommended to instruct EasyBuild to use `pip install` instead. + + * By default EasyBuild will try to import `py_eb_tutorial`, while the actual name of the Python package + provided by `py-eb-tutorial` is just `eb_tutorial`. We fixed this by specifying the correct Python module name to + use via `options`. + + * Strictly speaking we don't need to specify a custom `sanity_check_paths`, since the default used + by Python package is already pretty decent (it will check for a non-empty `lib/python3.8/site-packages` + directory in the installation). We also want to make sure the `py-eb-tutorial` command is available in + the `bin` subdirectory however. Hardcoding to `python3.8` can be avoided using the `%(pyshortver)s` + template value. + + * A good way to check whether the `py-eb-tutorial` command works correctly is by running it as a sanity check + command. If the `eb-tutorial` command is not available the `py-eb-tutorial` command will fail, + since it basically just runs the `eb-tutorial` command. So we need to include `eb-tutorial` as a (runtime) + dependency in the `py-eb-tutorial` easyconfig file. + +*[[next: Implementing easyblocks]](2_03_implementing_easyblocks.md)* diff --git a/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md new file mode 100644 index 00000000..5355548e --- /dev/null +++ b/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md @@ -0,0 +1,665 @@ +# Implementing easyblocks + +*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* + +--- + +## The basics + +An *easyblock* is a Python module that implements a software build and installation procedure. + +This concept is essentially implemented as a Python script that plugs into the EasyBuild framework. + +EasyBuild will leverage easyblocks as needed, depending on which software packages it needs to install. Which easyblock is required is determined by the ``easyblock`` easyconfig parameter, if it is present, or by the software name. + + +## Generic vs software-specific easyblocks + +Easyblocks can either be *generic* or *software-specific*. + +*Generic* easyblocks implement a "standard" software build and installation procedure that is used by multiple different +software packages. +A commonly used example is the +[``ConfigureMake``](https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/generic/configuremake.py) +generic easyblock, which implements the standard ``configure`` - ``make`` - ``make install`` installation procedure used +by most GNU software packages. + +*Software-specific* easyblocks implement the build and installation procedure for a particular software package. +Typically this involves highly customised steps, for example specifying dedicated configuration options, creating +or adjusting specific files, executing non-standard shell commands, etc. Usually a custom implementation of the +sanity check is also included. + +Using a generic easyblock requires specifying the ``easyblock`` parameter in the easyconfig file. +If it is not specified, EasyBuild will try and find the software-specific easyblock derived from the software name. + +The distinction between generic and software-specific easyblocks can be made based on the naming scheme that is used +for an easyblock (see below). + + +## Naming + +Easyblocks need to follow a strict naming scheme, to ensure that EasyBuild can pick them up as needed. This involves two aspects: + +* the name of the Python class; +* the name and location of the Python module file. + +### Python class name + +The name of the Python class is determined by the *software name* for software-specific easyblocks. +It consists of a prefix '``EB_``', followed by the (encoded) software name. + +Because of limitations in Python on characters allowed in names of Python classes, +only alphanumeric characters and underscores (``_``) are allowed. Any other characters are replaced following an encoding scheme: + +* spaces are replaced by underscores (``_``); +* dashes ``-`` are replaced by ``_minus_``; +* underscores are replaced by ``_underscore_``; + +The ``encode_class_name`` function provided in ``easybuild.tools.filetools`` returns the expected class name +for a given software name; for example: + +```shell +$ python3 -c "from easybuild.tools.filetools import encode_class_name; print(encode_class_name('netCDF-Fortran'))" +EB_netCDF_minus_Fortran +``` + +**Python class name for *generic* easyblocks** + +For *generic* easyblocks, the class name does *not* include an ``EB_`` prefix (since there is no need for an escaping +mechanism) and hence the name is fully free to choose, taking into account the restriction to alphanumeric characters +and underscores. + +For code style reasons, the class name should start with a capital letter and use CamelCasing. + +Examples include ``Bundle``, ``ConfigureMake``, ``CMakePythonPackage``. + +### Python module name and location + +The *filename* of the Python module is directly related to the name of Python class it provides. + +It should: + +* *not* include the ``EB_`` prefix of the class name for software-specific easyblocks; +* consists only of lower-case alphanumeric characters (``[a-z0-9]``) and underscores (``_``); + * dashes (``-``) are replaced by underscores (``_``); + * any other non-alphanumeric characters (incl. spaces) are simply dropped; + +Examples include: + +* ``gcc.py`` (for *GCC*) +* ``netcdf_fortran.py`` (for *netCDF-Fortran*) +* ``gamess_us.py`` (for *GAMESS (US)*) + +The ``get_module_path`` function provided by the EasyBuild framework in the +``easybuild.framework.easyconfig.easyconfig`` module returns the (full) +module location for a particular software name or easyblock class name. For example: + +```python +>>> from easybuild.framework.easyconfig.easyconfig import get_module_path +>>> get_module_path('netCDF-Fortran') +'easybuild.easyblocks.netcdf_fortran' +>>> get_module_path('EB_netCDF_minus_Fortran') +'easybuild.easyblocks.netcdf_fortran' +``` + +The location of the Python module is determined by whether the easyblock is generic or software-specific. +Generic easyblocks are located in the ``easybuid.easyblocks.generic`` namespace, while software-specific easyblocks +live in the ``easybuild.easyblocks`` namespace directly. + +To keep things organised, the actual Python module files +for software-specific easyblocks are kept in 'letter' subdirectories, +rather than in one large '``easyblocks``' directory +(see +[https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/](https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/)), +but this namespace is collapsed transparently by EasyBuild (you don't need to import from letter subpackages). + +To let EasyBuild pick up one or more new or customized easyblocks, you can use the [``--include-easyblocks``](https://docs.easybuild.io/en/latest/Including_additional_Python_modules.html#including-additional-easyblocks-include-easyblocks) +configuration option. As long as both the filename of the Python module and the name of the Python class +are correct, EasyBuild will use these easyblocks when needed. + +## Structure of an easyblock + +The example below shows the overal structure of an easyblock: + +```python +from easybuild.framework.easyblock import EasyBlock +from easybuild.tools.run import run_cmd + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def configure_step(self): + """Custom implementation of configure step for Example""" + + # run configure.sh to configure the build + run_cmd("./configure.sh --install-prefix=%s" % self.installdir) +``` + + +Each easyblock includes an implementation of a ``class`` that (directly or indirectly) derives from the abstract +``EasyBlock`` class. + +Typically some useful functions provided by the EasyBuild framework are imported at the top of the Python module. + +In the class definition, one or more '``*_step``' methods (and perhaps a couple of others) are redefined, +to implement the corresponding step in the build and installation procedure. + +Each easyblock *must* implement the ``configure``, ``build`` and ``install`` steps, since these are not implemented +in the abstract ``EasyBlock`` class. This could be done explicitly by redefining the corresponding ``*_step`` methods, +or implicitly by deriving from existing (generic) easyblocks. + +The full list of methods that can be redefined in an easyblock can be consulted in +the [API documentation](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyblock.html). + +## Deriving from existing easyblocks + +When implementing an easyblock, it is common to derive from an existing (usually generic) easyblock, +and to leverage the functionality provided by it. This approach is typically used when only a specific part +of the build and installation procedure needs to be customised. + +In the (fictitious) example below, we derive from the generic ``ConfigureMake`` easyblock to redefine the ``configure`` +step. In this case, we are *extending* the ``configure`` step as implemented by ``ConfigureMake`` rather than +redefining it entirely, since we call out to the original ``configure_step`` method at the end. + +```python +from easybuild.easyblocks.generic.configuremake import ConfigureMake +from easybuild.tools.filetools import copy_file + + +class EB_Example(ConfigureMake): + """Custom easyblock for Example""" + + def configure_step(self): + """Custom implementation of configure step for Example""" + + # use example make.cfg for x86-64 + copy_file('make.cfg.x86', 'make.cfg') + + # call out to original configure_step implementation of ConfigureMake easyblock + super(EB_Example, self).configure_step() +``` + +## Easyconfig parameters + +All of the easyconfig parameters that are defined in an easyconfig file +are available via the ``EasyConfig`` instance that can be accessed through ``self.cfg`` in an easyblock. +For instance, if the easyconfig file specifies + +```python +name = 'example' +version = '2.5.3' +versionsuffix = '-Python-3.7.4' +``` + +then these three parameters are accessible within an easyblock via ``self.cfg['name']``, ``self.cfg['version']`` +and ``self.cfg['versionsuffix']``. + +A few of the most commonly used parameters can be referenced directly: + +* **``self.name``** is equivalent with ``self.cfg['name']``; +* **``self.version``** is equivalent with ``self.cfg['version']``; +* **``self.toolchain``** is equivalent with ``self.cfg['toolchain']``; + + +### Updating parameters + +You will often find that you need to *update* some easyconfig parameters in an easyblock, +for example ``configopts`` which specifies options for the configure command. + +Because of implementation details (related to +how template values like ``%(version)s`` are handled), you need to be a bit careful here... + +To completely redefine the value of an easyconfig parameter, you can use simple assignment. For example: + +```python +self.cfg['example'] = "A new value for the example easyconfig parameter." +``` + +If want to *add* to the existing value however, you *must* use the ``self.cfg.update`` method. For example: + +```python +self.cfg.update('some_list', 'example') +``` + +This will ***not*** work (because ``self.cfg['some_list']`` does not return a reference to the original value, +but to a *temporary copy* thereof): + +```python +# anti-pattern, this does NOT work as expected! +self.cfg['some_list'].append('example') +``` + +### Custom parameters + +Additional custom easyconfig parameters can be defined in an easyblock to allowing steering its behaviour. +This is done via the ``extra_options`` *static* method. Custom parameters can be specified to be mandatory. + +The example below shows how this can be implemented: + +```python +from easybuild.easyblocks.generic.configuremake import ConfigureMake +from easybuild.framework.easyconfig import CUSTOM, MANDATORY + + +class EB_Example(ConfigureMake): + """Custom easyblock for Example""" + + @staticmethod + def extra_options(): + """Custom easyconfig parameters for Example""" + extra_vars = { + 'required_example_param': [None, "Example required custom parameter", MANDATORY], + 'optional_example_param': [None, "Example optional custom parameter", CUSTOM], + } + return ConfigureMake.extra_options(extra_vars) +``` + +The first element in the list of a defined custom parameter corresponds to the default value for that parameter +(both ``None`` in the example above). The second element provides some informative help text, and the last element +indicates whether the parameter is mandatory (``MANDATORY``) or just an optional custom parameter (``CUSTOM``). + +## Easyblock constructor + +In the ``class`` constructor of the easyblock, i.e. the ``__init__`` method, one or more class variables +can be initialised. These can be used for sharing information between different ``*_step`` methods in the easyblock. + +For example: + +```python +from easybuild.framework.easyblock import EasyBlock + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def __init__(self, *args, **kwargs): + """Constructor for Example easyblock, initialises class variables.""" + + # call out to original constructor first, so 'self' (i.e. the class instance) is initialised + super(EB_Example, self).__init__(*args, **kwargs) + + # initialise class variables + self.example_value = None + self.example_list = [] +``` + +## File operations + +File operations is a common use case for implementing easyblocks, hence the EasyBuild framework provides a +number of useful functions related to this, including: + +* ``read_file()``: read file at a specified location and returns its contents; + +* ``write_file(, )`` at a specified location with provided contents; + to append to an existing file, use ``append=True`` as an extra argument; + +* ``copy_file(, )`` to copy an existing file; + +* ``apply_regex_substitutions(, )`` to patch an existing file; + +All of these functions are provided by the [``easybuild.tools.filetools``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html) module. + +## Executing shell commands + +For executing shell commands two functions are provided by the +[``easybuild.tools.run``](https://docs.easybuild.io/en/latest/api/easybuild.tools.run.html) module: + +* ``run_cmd()`` to run a non-interactive shell command; + +* ``run_cmd_qa(, )`` to run an interactive shell command; + +Both of these accept a number of optional arguments: + +* ``simple=True`` to just return ``True`` or ``False`` to indicate a successful execution, + rather than the default return value, i.e., a tuple that provides the command output and the exit code (in that + order); + +* ``path=`` to run the command in a specific subdirectory; + +The ``run_cmd_qa`` function takes two additional specific arguments: + +* ``no_qa=`` to specify a list of patterns to recognize non-questions; + +* ``std_qa=`` to specify regular expression patterns for common questions, and the matching answer; + +## Manipulating environment variables + +To (re)define environment variables, the ``setvar`` function provided by the +[``easybuild.tools.environment``](https://docs.easybuild.io/en/latest/api/easybuild.tools.environment.html) +module should be used. + +This makes sure that the changes being made to the specified environment variable are kept track of, +and that they are handled correctly under ``--extended-dry-run``. + +## Logging and errors + +It is good practice to include meaningful log messages in the ``*_step`` methods being customised in the easyblock, +to enrich the EasyBuild log with useful information for later debugging or diagnostics. + +For logging, the provided ``self.log`` logger class should be used. +You can use the ``self.log.info`` method to log an informative message. +Similar methods are available for logging debug messages (``self.log.debug``), which are +only emitted when ``eb`` is run with debugging mode enabled (``--debug`` or ``-d``), +and for logging warning messages (``self.log.warning``). + +If something goes wrong, you can raise an ``EasyBuildError`` instance to report the error. + +For example: + +```python +from easybuild.framework.easyblock import EasyBlock +from easybuild.tools.build_log import EasyBuildError +from easybuild.tools.run import run_cmd + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def configure_step(self): + """Custom implementation of configure step for Example""" + + cmd = "./configure --prefix %s" % self.installdir) + out, ec = run_cmd(cmd) + + success = 'SUCCESS' + if success in out: + self.log.info("Configuration command '%s' completed with success." % cmd) + else: + raise EasyBuildError("Pattern '%s' was not found in output of '%s'." % (success, cmd)) +``` + +## Custom sanity check + +For software-specific easyblocks, a custom sanity check is usually included to verify that the installation was +successful or not. + +This is done by redefining the ``sanity_check_step`` method in the easyblock. For example: + +```python +from easybuild.framework.easyblock import EasyBlock + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def sanity_check_step(self): + """Custom sanity check for Example.""" + + custom_paths = { + 'files': ['bin/example'], + 'dirs': ['lib/examples/'], + } + custom_commands = ['example --version'] + + # call out to parent to do the actual sanity checking, pass through custom paths and commands + super(EB_Example, self).sanity_check_step(custom_paths=custom_paths, custom_commands=custom_commands) +``` + +You can both specify file paths and subdirectories to check for, which are specified relative to the installation directory, +as well as simple commands that should execute successfully after completing the installation and loading the generated module file. + +It is up to you how extensive you make the sanity check, but it is recommended to make the check as complete +as possible to catch any potential build or installation problems that may occur, while ensuring that it can +run relatively quickly (in seconds, or at most a couple of minutes). + +## Version-specific parts + +In some cases version-specific actions or checks need to be included in an easyblock. +For this, it is recommended to use ``LooseVersion`` rather than directly comparing version numbers using string values. + +For example: + +```python +from distutils.version import LooseVersion +from easybuild.framework.easyblock import EasyBlock + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def sanity_check_step(self): + """Custom sanity check for Example.""" + + custom_paths = { + 'files': [], + 'dirs': [], + } + + # in older versions, the binary used to be named 'EXAMPLE' rather than 'example' + if LooseVersion(self.version) < LooseVersion('1.0'): + custom_paths['files'].append('bin/EXAMPLE') + else: + custom_paths['files'].append('bin/example') + + super(EB_Example, self).sanity_check_step(custom_paths=custom_paths) +``` + +## Compatibility with ``--extended-dry-run`` and ``--module-only`` + +Some special care must be taken to ensure that an easyblock is fully compatible with ``--extended-dry-run`` / ``-x`` +(see [Inspecting install procedures](../basic_usage/#inspecting-install-procedures)) and ``--module-only``. + +For compatibility with ``--extended-dry-run``, you need to take into account that specified operations +like manipulating files or running shell commands will not actually be executed. You can check +whether an easyblock is being run in dry run mode via ``self.dry_run``. + +For example: + +```python +from easybuild.framework.easyblock import EasyBlock +from easybuild.tools.build_log import EasyBuildError +from easybuild.tools.run import run_cmd + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def configure_step(self): + """Custom implementation of configure step for Example""" + + cmd = "./configure --prefix %s" % self.installdir) + out, ec = run_cmd(cmd) + + success = 'SUCCESS' + if success in out: + self.log.info("Configuration command '%s' completed with success." % cmd) + + # take into account that in dry run mode we won't get any output at all + elif self.dry_run: + self.log.info("Ignoring missing '%s' pattern since we're running in dry run mode." % success) + + else: + raise EasyBuildError("Pattern '%s' was not found in output of '%s'." % (success, cmd)) +``` + +For ``--module-only``, you should make sure that both the ``make_module_step``, including the ``make_module_*`` submethods, +and the ``sanity_check_step`` methods do not make any assumptions about the presence of certain environment variables, or that class variables have been defined already. + +This is required because under ``--module-only`` the large majority of the ``*_step`` functions are +simply *skipped* entirely. So, if the ``configure_step`` method is responsible for defining class variables that are +picked up in ``sanity_check_step``, the latter may run into unexpected initial values like ``None``. +A possible workaround is to define a separate custom method to define the class variables, and to call out to this +method from ``configure_step`` and ``sanity_check_step`` (for the latter, conditionally, i.e., only if the class +variables still have the initial values). + +For example: + +```python +from easybuild.framework.easyblock import EasyBlock + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def __init__(self, *args, **kwargs): + """Easyblock constructor.""" + super(EB_Example, self).__init__(*args, **kwargs) + + self.command = None + + def set_command(self): + """Initialize 'command' class variable.""" + # $CC environment variable set by 'prepare' step determines exact command + self.command = self.name + '-' + os.getenv('CC') + + def configure_step(self): + """Custom configure step for Example.""" + + self.set_command() + self.cfg.update('configopts', "COMMAND=%s" % self.command) + + super(EB_Example, self).configure_step() + + def sanity_check_step(self): + """Custom implementation of configure step for Example""" + + if self.command is None: + self.set_command() + + super(EB_Example, self).sanity_check_step(custom_commands=[self.command]) +``` + +## Exercise + +### Exercise I.1 + +Try implementing a new custom easyblock for ``eb-tutorial``, which derives directly +from the base ``EasyBlock`` class. + +Your easyblock should: + +* define a custom mandatory easyconfig parameter named ``message``; +* run `cmake` to configure the installation, which includes at least: + * specifying the correct installation prefix (using the `-DCMAKE_INSTALL_PREFIX=...` option); + * passing down the value of ``message`` easyconfig parameter via `-DEBTUTORIAL_MSG=...` +* run `make` to build `eb-tutorial`; +* run `make install` to install the generated binary; +* perform a custom sanity check to ensure the installation is correct; +* pick up on commonly used easyconfig parameters like `configopts` and `preinstallopts` where appropriate; + +??? success "(click to show solution)" + + Here's a complete custom easyblock for ``eb-tutorial`` that derives from the base ``EasyBlock`` class, + which should be included in a file named ``eb_tutorial.py``. + + We need to implement the ``configure_step``, ``build_step``, and ``install_step`` methods in + order to have a fully functional easyblock. + + The configure, build, and install steps take into account the corresponding easyconfig + parameters that allow customizing these commands from an easyconfig file. + + ```python + from easybuild.framework.easyblock import EasyBlock + from easybuild.framework.easyconfig import MANDATORY + from easybuild.tools.run import run_cmd + + + class EB_eb_minus_tutorial(EasyBlock): + """Custom easyblock for eb-tutorial.""" + + @staticmethod + def extra_options(): + extra = EasyBlock.extra_options() + extra.update({ + 'message': [None, "Message that eb-tutorial command should print", MANDATORY], + }) + return extra + + def configure_step(self): + """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" + + cmd = ' '.join([ + self.cfg['preconfigopts'], + "cmake", + "-DCMAKE_INSTALL_PREFIX='%s'" % self.installdir, + "-DEBTUTORIAL_MSG='%s'" % self.cfg['message'], + self.cfg['configopts'], + ]) + run_cmd(cmd) + + def build_step(self): + """Build step for eb-tutorial""" + + cmd = ' '.join([ + self.cfg['prebuildopts'], + "make", + self.cfg['buildopts'], + ]) + run_cmd(cmd) + + def install_step(self): + """Install step for eb-tutorial""" + + cmd = ' '.join([ + self.cfg['preinstallopts'], + "make install", + self.cfg['installopts'], + ]) + run_cmd(cmd) + + def sanity_check_step(self): + custom_paths = { + 'files': ['bin/eb-tutorial'], + 'dirs': [], + } + custom_commands = ['eb-tutorial'] + return super(EB_eb_minus_tutorial, self).sanity_check_step(custom_paths=custom_paths, + custom_commands=custom_commands) + ``` + +### Exercise I.2 + +Try implementing another new custom easyblock for ``eb-tutorial``, +which derives from the generic ``CMakeMake`` easyblock. + +Your easyblock should only: + +* define a custom mandatory easyconfig parameter named ``message``; +* pass down the value of ``message`` easyconfig parameter via `-DEBTUTORIAL_MSG=...` +* perform a custom sanity check to ensure the installation is correct; + +??? success "(click to show solution)" + + When deriving from the ``CMakeMake`` generic easyblock, there is a lot less to worry about. + + We only need to customize the ``configure_step`` method to ensure that the ``-DEBTUTORIAL_MSG`` configuration + option is specified; the ``CMakeMake`` easyblock already takes care of specifying the location of + the installation directory (and a bunch of other configuration options, like compiler commands and flags, etc.). + + Implementing the ``build_step`` and ``install_step`` methods is no longer needed, + the standard procedure that is run by the ``CMakeMake`` generic easyblock is fine, + and even goes beyond what we did in the previous exercise (like building in parallel with ``make -j``). + + ```python + from easybuild.easyblocks.generic.cmakemake import CMakeMake + from easybuild.framework.easyconfig import MANDATORY + from easybuild.tools.run import run_cmd + + class EB_eb_minus_tutorial(CMakeMake): + """Custom easyblock for eb-tutorial.""" + + @staticmethod + def extra_options(): + extra = CMakeMake.extra_options() + extra.update({ + 'message': [None, "Message that eb-tutorial command should print", MANDATORY], + }) + return extra + + def configure_step(self): + """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" + self.cfg.update('configopts', "-DEBTUTORIAL_MSG='%s'" % self.cfg['message']) + + super(EB_eb_minus_tutorial, self).configure_step() + + def sanity_check_step(self): + custom_paths = { + 'files': ['bin/eb-tutorial'], + 'dirs': [], + } + custom_commands = ['eb-tutorial'] + return super(EB_eb_minus_tutorial, self).sanity_check_step(custom_paths=custom_paths, + custom_commands=custom_commands) + ``` + +*[[next: PArt 3: OVerview]](3_00_part3_advanced.md)* diff --git a/docs/2022-CSC_and_LO/3_00_part3_advanced.md b/docs/2022-CSC_and_LO/3_00_part3_advanced.md new file mode 100644 index 00000000..708c57f8 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_00_part3_advanced.md @@ -0,0 +1,9 @@ +# Part III: Advanced topics + +*[[back to start page]](index.md)* + +* [Using EasyBuild as a library](3_01_easybuild_library.md) *(hands-on)* +* [Using hooks to customise EasyBuild](3_02_hooks.md) *(hands-on)* +* [Submitting installations as Slurm jobs](3_03_slurm_jobs.md) *(hands-on)* +* [Module naming schemes (incl. hierarchical)](3_04_module_naming_schemes.md) *(hands-on)* +* [GitHub integration to facilitate contributing to EasyBuild](3_05_github_integration.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/3_01_easybuild_library.md b/docs/2022-CSC_and_LO/3_01_easybuild_library.md new file mode 100644 index 00000000..c037f49f --- /dev/null +++ b/docs/2022-CSC_and_LO/3_01_easybuild_library.md @@ -0,0 +1,293 @@ +# Using EasyBuild as a library + +*[[back: Configuring EasyBuild]](1_07_configuration.md)* + +--- + +You can use EasyBuild as a Python library and implement a script to automate a particular task. + +All the functionality provided by the EasyBuild framework can be leveraged, +so you only have to glue things together and add the missing pieces. + + +## Setting up the EasyBuild configuration + +Before you call any of the functions provided by the EasyBuild framework, +you should initialize EasyBuild by **setting up the configuration**. +Some of the EasyBuild framework functions assume that EasyBuild has been initialized properly, +and calling them without doing so will result in errors. + +To set up the EasyBuild configuration, you should call the ``set_up_configuration`` function +that is provided by the ``easybuild.tools.options`` module. + +This function takes a couple of optional arguments that are useful in the context of +a Python script that uses EasyBuild as a library: + +* Via ``args`` you can provide a list of command line arguments that should be passed to the option parser. + By default, the standard command line arguments are picked up via ``sys.args``, which may not be what you want. +* By passing ``silent=True`` you can specify that the ``set_up_configuration`` function should not print + anything when it is called. By default, it will print the location to the temporary EasyBuild log file. + +### Cleaning up the temporary directory + +When EasyBuild is configured, a temporary directory specific to that EasyBuild session will be created automatically. +You should clean up that directory, especially if you will be creating temporary files, or if the script will be run +often. + +Note that cleaning up the temporary directory implies removes the temporary log files, +so you probably only want to do this if no errors occurred. + +### Minimal example script + +Here is a minimal (and pretty useless) example Python script, which sets up the EasyBuild configuration +and cleans up the temporary directory: + +```python +#!/usr/bin/env python3 +from easybuild.tools.filetools import remove_dir +from easybuild.tools.options import set_up_configuration + +opts, _ = set_up_configuration(args=[], silent=True) + +remove_dir(opts.tmpdir) +``` + + +## Example use cases + +Once the EasyBuild configuration has been set up, the functions provided by the EasyBuild framework +can be called from a Python script (or directly from the Python interpreter). + +A full overview of all functions is available via the [EasyBuild API documentation](https://docs.easybuild.io/en/latest/api/easybuild.html). + +We highlight a couple commonly used functions in the sections below. + +### File operations + +The [``easybuild.tools.filetools``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html) +module provides a bunch of functions for performing file operations. + +That includes straightforward things like reading, writing, and copying files +(see [``read_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.read_file), [``write_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.write_file), [``copy_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.copy_file)), but also more specific functionality like applying a patch file ([``apply_patch``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.apply_patch)), +computing different types of checksums for a file ([``compute_checksum``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.compute_checksum)), and downloading files +([``download_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.download_file)). + +Here is a trivial example where we download a file from a specified URL to a temporary directory, +compute the SHA256 checksum, and then copy the downloaded file to the current working directory: + +```python +#!/usr/bin/env python3 +import os +import sys +import tempfile + +from easybuild.tools.filetools import download_file, compute_checksum, copy_file, remove_dir +from easybuild.tools.options import set_up_configuration + +url = sys.argv[1] + +opts, _ = set_up_configuration() + +fn = os.path.basename(url) +tmp_path = os.path.join(opts.tmpdir, fn) +download_file(fn, url, tmp_path) + +sha256 = compute_checksum(tmp_path, checksum_type='sha256') +print("SHA256 checksum for %s: %s" % (fn, sha256)) + +target_dir = os.getcwd() +copy_file(tmp_path, target_dir) +print("%s copied to %s" % (fn, target_dir)) + +# cleanup temp dir +remove_dir(opts.tmpdir) +``` + +Example output: +```shell +$ export TEST_URL='https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.0.1.tar.gz' +$ python3 download_and_copy.py $TEST_URL +== Temporary log file in case of crash /tmp/eb-or_xhaw8/easybuild-ewooa00c.log +SHA256 checksum for eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 +eb-tutorial-1.0.1.tar.gz copied to /home/example +``` +We are deliberately *not* specifying the `args` optional argument to the `set_up_configuration` call here, +so we can pass EasyBuild configuration options as command line arguments to this script. +Of course, only those configuration options that are taken into account by the EasyBuild +framework functions being called have any impact. For example: + +```shell +# wait for max. 1h for the download to complete +python3 download_and_copy.py $TEST_URL --download-timeout 3600 +``` + +### Running shell commands + +To run shell commands, we can leverage the ``run_cmd`` functions from the ``easybuild.tools.run`` module. + +Here is a simple example for running the '`make`' command via a Python script: + +```python +#!/usr/bin/env python3 +import sys + +from easybuild.tools.filetools import remove_dir, which +from easybuild.tools.run import run_cmd +from easybuild.tools.options import set_up_configuration + +opts, _ = set_up_configuration(args=[]) + +cmd = 'make' +cmd_path = which(cmd) +if cmd_path: + print(">>> '%s' command found at %s" % (cmd, cmd_path)) +else: + sys.stderr.write("ERROR: '%s' command not found!\n" % cmd) + sys.exit(1) + +cmd = ' '.join(["make"] + sys.argv[1:]) +out, ec = run_cmd(cmd) + +print("\n>>> Output of '%s' (exit code %s):\n\n%s" % (cmd, ec, out)) + +remove_dir(opts.tmpdir) +``` + +Note that in this case it is important to use `args=[]` when calling `set_up_configuration`, +to avoid that the additional script arguments which are options for the '`make`' command +are picked up by the EasyBuild option parser. + +Example usage: + +```shell +$ python3 run_make.py --version +== Temporary log file in case of crash /tmp/eb-zfvbceg6/easybuild-7tynetaj.log + +>> Output of 'make --version' (exit code 0): + +GNU Make 3.81 +``` + +### Interacting with the modules tool + +You can interact with the environment modules tool in a Python script using the +[``easybuild.tools.modules``](https://docs.easybuild.io/en/latest/api/easybuild.tools.modules.html) +Python module that is a part of the EasyBuild framework. +The modules tool that will be used is determined by the active EasyBuild configuration. + +Here is an example script that checks which modules are available and currently loaded, +loads the default module file for ``bzip2``, and inspects the resulting changes to the environment. + +```python +#!/usr/bin/env python3 +import os +import sys + +from easybuild.tools.filetools import remove_dir +from easybuild.tools.modules import get_software_root_env_var_name, modules_tool +from easybuild.tools.options import set_up_configuration + +opts, _ = set_up_configuration() + +# obtain ModulesTool instance for preferred modules tool (determined by active EasyBuild configuration) +mod_tool = modules_tool() +print("Active modules tool: %s version %s" % (mod_tool.NAME, mod_tool.version)) + +avail_modules = mod_tool.available() +print("Found %d available modules in total" % len(avail_modules)) + +avail_eb_modules = mod_tool.available('EasyBuild') +print("Found %d available modules for EasyBuild: %s" % (len(avail_eb_modules), ', '.join(avail_eb_modules))) + +loaded_modules = mod_tool.loaded_modules() +print("%d modules are currently loaded: %s" % (len(loaded_modules), ', '.join(loaded_modules))) + +# load default module for bzip2, check changes to environment +name = 'bzip2' +env_var_name = get_software_root_env_var_name(name) +if any(m.startswith(name + '/') for m in avail_modules): + + print("Current $%s value: %s" % (env_var_name, os.getenv(env_var_name, '(no set)'))) + print("Loading (default) '%s' module..." % name) + + mod_tool.load([name]) + print("Loaded modules: %s" % ', '.join(mod_tool.loaded_modules())) + + # inspect $_LMFILES_ environment variable to determine path to loaded bzip2 module file + for mod_file_path in os.getenv('_LMFILES_').split(':'): + if name in mod_file_path: + print("Path to loaded %s module: %s" % (name, mod_file_path)) + break + + # $EBROOTBZIP2 should be set now (if the bzip2 module was installed with EasyBuild) + print("Current $%s value: %s" % (env_var_name, os.getenv(env_var_name, '(no set)'))) +else: + sys.stderr.write("No modules available for %s\n" % name) + sys.exit(1) + +remove_dir(opts.tmpdir) +``` + +### Parsing easyconfig files + +Here is another small example Python script, which uses the EasyBuild framework functionality +to locate and parse an easyconfig file, and inspect the value of specific easyconfig parameters. + +We define a small helper function named ``parse_easyconfig``, because the EasyBuild framework API +is a bit awkward to use for this simple use case. + +```python +#!/usr/bin/env python3 +import sys + +from easybuild.framework.easyconfig.tools import det_easyconfig_paths, parse_easyconfigs +from easybuild.tools.options import set_up_configuration + + +def parse_easyconfig(ec_fn): + """ + Helper function: find and parse easyconfig with specified filename, + and return parsed easyconfig file (an EasyConfig instance). + """ + # determine path to easyconfig file + ec_path = det_easyconfig_paths([ec_fn])[0] + + # parse easyconfig file; + # the 'parse_easyconfigs' function expects a list of tuples, + # where the second item indicates whether or not the easyconfig file was automatically generated or not + ec_dicts, _ = parse_easyconfigs([(ec_path, False)]) + + # only retain first parsed easyconfig, ignore any others (which are unlikely anyway) + return ec_path, ec_dicts[0]['ec'] + + +# check whether required arguments are provided +if len(sys.argv) < 3: + sys.stderr.write("ERROR: Usage: %s ") + sys.exit(1) + +ec_fn = sys.argv[1] +keys = sys.argv[2:] + +set_up_configuration(args=[], silent=True) + +ec_path, ec = parse_easyconfig(ec_fn) + +print("Inspecting %s ..." % ec_path) +for key in keys: + print("%s: %s" % (key, ec[key])) +``` + +Example usage: + +```shell +$ ./inspect_easyconfig.py Subread-2.0.0-GCC-8.3.0.eb name version sources sanity_check_paths +name: Subread +version: 2.0.0 +sources: ['subread-2.0.0-source.tar.gz'] +sanity_check_paths: {'files': ['bin/exactSNP', 'bin/featureCounts', 'bin/subindel', 'bin/subjunc', 'bin/sublong', 'bin/subread-align', 'bin/subread-buildindex'], 'dirs': ['bin/utilities']} +``` + +--- + +*[[next: Using hooks to customise EasyBuild]](3_02_hooks.md)* diff --git a/docs/2022-CSC_and_LO/3_02_hooks.md b/docs/2022-CSC_and_LO/3_02_hooks.md new file mode 100644 index 00000000..a2663ff0 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_02_hooks.md @@ -0,0 +1,190 @@ +# Using hooks to customise EasyBuild + +*[[back: Using EasyBuild as a library]](3_01_easybuild_library.md)* + +--- + +Sometimes you may have the need to customise the behaviour of EasyBuild, +while you want to avoid copying easyconfig files in order to make small changes +(like adding an additional configuration option), having to maintain a set +of modified easyblocks, or using a patched version of the EasyBuild framework. + +EasyBuild provides support for implementing so-called *hooks*, +which are small functions that are triggered automatically at specific times. + +Through these hook functions you can take additional actions, or even tweak +internal data structures, to change the software installation procedure that EasyBuild +would normally execute. + +This is a very powerful feature, but it is important to aware of some details in order +to make good use of it. + +Support for hooks is also covered in detail in the EasyBuild documentation, see +[here](https://docs.easybuild.io/en/latest/Hooks.html). + + +## Implementing and using hooks + +Using hooks is done by implementing functions with specific names in a Python module, +and configuring EasyBuild to use them by specifying the path to the file that contains them +via the ``hooks`` configuration option. + +For example, if the file named ``my_eb_hooks.py`` in your home directory contains the implementation +of your hooks, you can configure EasyBuild to use them by defining the ``$EASYBUILD_HOOKS`` environment +variable: + +```shell +export EASYBUILD_HOOKS=$HOME/my_eb_hooks.py +``` + +## Available hooks + +EasyBuild supports various hooks, the full list can be consulted via ``eb --avail-hooks``. + +There are 3 different types of hooks: + +- start/end hooks +- parse hook +- pre/post-step hooks + +These are discussed in more details in the sections below. + +### Start/end hooks + +The first type of hooks are the ``start`` and ``end`` hooks, which are respectively triggered +at the very start of the EasyBuild session (right after setting up the EasyBuild configuration), +and at the very end of the session (right before stopping the logger and doing final cleanup). + +These hooks are only called *once* for every EasyBuild session, no matter how many installations +are performed during that session. + +The ``end`` hook is *not* triggered if an error occurred during one of the installations, +only on successful completion of the EasyBuild session. + +These hooks can be implementing by defining a function named ``start_hook`` and ``end_hook`` +in the Python module file that is provided via the ``hooks`` configuration option. +No arguments are provided when calling these hooks. + +### Parse hook + +The ``parse`` hook is triggered right after an easyconfig file is being parsed, +*before* EasyBuild sets up its internal data structures based on the parsed easyconfig. + +If you want to dynamically change one or more easyconfig parameters without changing the corresponding +easyconfig files, using this hook may be appropriate. + +Note that parsing an easyconfig file can happen for a variety of reasons, +not only when the easyconfig file will actually be installed. EasyBuild will also +parse easyconfig files to check whether they resolve required dependencies, +to check whether the corresponding module file is already installed, etc. + +This hook can be implemented via a function named ``parse_hook``, and exactly one +argument is provided when it is called: the [``EasyConfig``](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyconfig.easyconfig.html#easybuild.framework.easyconfig.easyconfig.EasyConfig) +instance that represents the parsed easyconfig file. + +### Pre/post-step hooks + +The third type of hooks are *pre/post-step* hooks, which are triggered right before or +after a particular installation step is executed. + +EasyBuild performs each software installation by stepping through over a dozen different methods, +and for each of these steps there a pre- and post-hook is triggered, which results in over 30 +additional hooks. + +To use any of these hooks, you need to implement a function that follow a strict naming scheme: +``__hook``. For example, the hook that is triggered right before the ``configure`` +step is run is a function named ``pre_configure_hook``. + +Every time these hooks are called, a single argument is provided: an [``EasyBlock``](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyblock.html#easybuild.framework.easyblock.EasyBlock) +instance that represents the easyblock that is being used to perform the installation. +The parsed easyconfig file can be accessed via the ``cfg`` class variable of the ``EasyBlock`` instance. + +These hooks are useful for influencing the installation procedure at a particular stage. + +## Caveats + +There are a couple of important caveats to take into account when implementing hooks. + +### Breaking EasyBuild with hooks + +Since hooks allow you to inject custom code into EasyBuild at runtime, +it is also easy to break EasyBuild by using hooks... + +Make sure to carefully test your hook implementations, and constrain the actions +you take a much as possible, for example by adding conditions to control for which +software names you will actually modify the installation procedure, etc. + +Any errors that are triggered or raised while a hook function is running +will interrupt the EasyBuild session. + +So don't forget: with great power comes great responsibility! + +### Template values + +Depending on the type of hook, you may observe "raw" values of easyconfig parameters where +template values have not been resolved yet, or values in which template values have been resolved already. + +In the ``parse`` hook, you will always see unresolved template values. + +In the pre/post-step hooks you will see resolved template values, +unless you explicitly disable templating. + +To obtain easyconfig parameter values with unresolved template values in step hooks, +you can use the ``disable_templating`` [context manager](https://docs.python.org/3/reference/compound_stmts.html#with). +For example: + +```python +from easybuild.framework.easyconfig.easyconfig import disable_templating +from easybuild.tools.build_log import print_warning + +def pre_source_hook(eb): + """Print warning when software version was found in 'raw' name of source file.""" + with disable_templating(eb.cfg): + for src in eb.cfg['sources']: + if eb.version in src: + msg = "Software version '%s' found in name of source file (%s), " % (eb.version, src) + msg += "please use %(version)s template value instead!" + print_warning(msg) +``` + + +### Manipulating easyconfig parameters + +If you want update a particular easyconfig parameter without overwriting the existing value, +a bit of care has to be taken: you should use the ``update`` method of the ``EasyConfig`` instance +for this, unless you disable template resolution. This is particularly important when +updating easyconfig parameters that have *mutable* value (like a ``list`` or ``dict``). + +Here's a correct example of a pre-install hook: + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # always copy the README directory too when installing pigz + eb.cfg.update('files_to_copy', 'README') +``` + +This seemingly equivalent implementation will ***not*** work (the value of the `files_to_copy` +easyconfig parameter will *not* be updated): + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # incorrect way of adding 'README' to 'files_to_copy' (DON'T USE THIS!) + eb.cfg['files_to_copy'].append('README') +``` + +To use this coding style successfully, you have to disable the templating mechanism +when updating the easyconfig parameter: + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # this works, but it is better to use the 'update' method instead... + with disable_templating(eb.cfg): + eb.cfg['files_to_copy'].append('README') +``` + +--- + +*[[next: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* diff --git a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md new file mode 100644 index 00000000..0ffa12a9 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md @@ -0,0 +1,304 @@ +# Submitting installations as Slurm jobs + +*[[back: Using hooks to customise EasyBuild]](3_02_hooks.md)* + +--- + +EasyBuild can submit jobs to different backends including Slurm to install software, +to *distribute* the often time-consuming installation of a set of software applications and +the dependencies they require to a cluster. + +This is done via the ``--job`` command line option. + +It is important to be aware of some details before you start using this, which we'll cover here. + +## Configuration + +The EasyBuild configuration that is active at the time that ``eb --job`` is used +will be *passed down* into the submitted job automatically, via command line options to the ``eb`` +command that is run in the job script. + +This includes not only command line options used next to ``--job``, but also configuration settings +that are specified via an [EasyBuild configuration file](configuration.md#configuration-files) or through +[``$EASYBUILD_*`` environment variables](configuration.md#easybuild_-environment-variables). + +This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables +that are in place in the job environment are most likely *irrelevant*, since configuration settings +they specify they will most likely be overruled by the corresponding command line options. + + +## Using ``eb --job`` + +### Job backend + +The default job backend in EasyBuild v4.x is [``GC3Pie``](https://gc3pie.readthedocs.io). +To let EasyBuild submit jobs to Slurm instead, you should set the ``job-backend`` configuration setting +to ``Slurm``, for example by setting the corresponding environment variable: + +```shell +export EASYBUILD_JOB_BACKEND='Slurm' +``` + +### Job resources + +To submit an installation as a job, simply use ``eb --job``: + +```shell +eb example.eb --job +``` + +By default, EasyBuild will submit single-core jobs requesting for 24 hours of walltime. +You can tweak the requested resources via the ``job-cores`` and ``job-max-walltime`` configuration options. +For example: + +```shell +# submit job to install example, using 5 cores and 2 hours of max. walltime +eb example.eb --job --job-cores 5 --job-max-walltime 2 +``` + +Note that not all ``job-*`` configuration settings apply to all job backends, +see the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Submitting_jobs.html) for more details. + +### Controlling Slurm submission options + +When using Slurm as a job backend, EasyBuild will automatically generate job scripts which +use the ``eb`` command to perform a single installation. These scripts will be submitted +using the ``sbatch`` command. + +EasyBuild currently doesn't provide away to customize the Slurm submission options, +for example to submit to a particular partition, or to use a particular account, +build you can set the corresponding ``$SBATCH_*`` environment variables prior to running ``eb --job``. + +For example, to specify a particular account that should be used for the jobs submitted by EasyBuild +(equivalent with using the ``-A`` or ``--account`` command line option for ``sbatch``): + +```shell +export SBATCH_ACCOUNT='example_project' +``` + +Or to submit to a particular Slurm partition (equivalent with the ``-p`` or ``--partition`` option for ``sbatch``): + +```shell +export SBATCH_PARTITION='example_partition' +``` + +For more information about supported ``$SBATCH_*`` environment variables, +see the [Slurm documentation](https://slurm.schedmd.com/sbatch.html#lbAJ). + +## Combining ``--job`` and ``--robot`` + +If one or more dependencies are still missing for the software you want to install, +you can combine ``--job`` and ``--robot`` to get EasyBuild to submit a *separate* job +for each of the installations. These jobs will *not* ``--robot``, they will each only +perform a single installation. + +Dependencies between jobs will be "registered" at submission time, so Slurm will put jobs +on hold until the jobs that install the required (build) dependencies have completed successfully, +and cancel jobs if the job to install a dependency failed for some reason. + +## Attention points + +There are a couple of important things to keep an eye on when submitting installations as jobs... + +### Differences on cluster workernodes + +Sometimes the resources available on the login nodes and cluster workernodes are slightly different, +and you may need to take this into account in your EasyBuild configuration. + +For example, plenty of disk space may be available in the `/tmp` temporary filesystem on a login node, +while the workernodes require you to use a different location for temporary files and directories. +As a result, you may need to slightly change your EasyBuild configuration when submitting installations +as jobs, to avoid that they fail almost instantly due to a lack of disk space. + +Keep in mind that the active EasyBuild configuration is passed down into the submitted jobs, +so any configuration that is present on the workernodes may not have any effect. + +For example, if you commonly use `/tmp/$USER` for build directories on a login node, +you may need to tweak that when submitting jobs to use a different location: + +```shell +# EasByuild is configured to use /tmp/$USER on the login node +login01 $ eb --show-config | grep buildpath +buildpath (E) = /tmp/example + +# use /localdisk/$USER for build directories when submitting installations as jobs +login01 $ eb --job --buildpath /localdisk/$USER example.eb --robot +``` + +### Temporary log files and build directories + +The temporary log file that EasyBuild creates is most likely going to end up on the local disk +of the workernode on which the job was started (by default in `$TMPDIR` or `/tmp`). +If an installation fails, the job will finish and temporary files will likely be cleaned up instantly, +which may leave you wondering about the actual cause of the failing installation... + +To remedy this, there are a couple of EasyBuild configuration options you can use: + +* You can use ``--tmp-logdir`` to specify a different location where EasyBuild should store temporary log files, + for example: + ```shell + $ eb --job example.eb --tmp-logdir $HOME/eb_tmplogs + ``` + +* If you prefer having the entire log file stored in the Slurm job output files, + you can use ``--logtostdout`` when submitting the jobs. This will result in extensive logging + to your terminal window when submitting the jobs, but it will also make EasyBuild + log to ``stdout`` when the installation is running in the job, and hence the log messages will be + captured in the job output files. + +The same remark applies to build directories: they should be on a local filesystem (to avoid problems +that often occur when building software on a parallel filesystem like GPFS or Lustre), +which will probably be cleaned up automatically when a job fails. Here it is less easy to provide +general advice on how to deal with this, but one thing you can consider is retrying the installation +in an interactive job, so you can inspect the build directory after the installation fails. + +### Lock files + +EasyBuild creates [locks](https://docs.easybuild.io/en/latest/Locks.html) +to prevent that the same installation is started multiple times on +different system to the same installation directory. + +If an installation fails or gets interrupted, EasyBuild cleans up those locks automatically. + +However, if a Slurm job that is using EasyBuild to install software gets cancelled (because it +ran out of walltime, tried to consume too much memory, through an ``scancel`` command, etc.), +EasyBuild will not get the chance to clean up the lock file. + +If this occurs you will need to either clean up the lock file (which is located in the `software/.locks` +subdirectory of ``installpath``) manually, or re-submit the job with ``eb --job --ignore-locks``. + +## Example + +As an example, we will let EasyBuild submit jobs to install ``AUGUSTUS`` with the ``foss/2020b`` toolchain. + +### Configuration + +Before using ``--job``, let's make sure that EasyBuild is properly configured: + +```shell +# use $HOME/easybuild for software, modules, sources, etc. +export EASYBUILD_PREFIX=$HOME/easybuild + +# use ramdisk for build directories +export EASYBUILD_BUILDPATH=/dev/shm/$USER + +# use Slurm as job backend +export EASYBUILD_JOB_BACKEND=Slurm +``` + +In addition, add the path to the centrally installed software to ``$MODULEPATH`` via ``module use``: + +```shell +module use /easybuild/modules/all +``` + +Load the EasyBuild module: + +```shell +module load EasyBuild +``` + +Let's assume that we also need to inform Slurm that jobs should be submitted into a particular account: + +```shell +export SBATCH_ACCOUNT=example_project +``` + +This will be picked up by the ``sbatch`` commands that EasyBuild will run to submit the software installation jobs. + + +### Submitting jobs to install AUGUSTUS + +Now we can let EasyBuild submit jobs for AUGUSTUS. + +Let's first check what is still missing: + +```shell +$ eb AUGUSTUS-3.4.0-foss-2020b.eb --missing +... +11 out of 61 required modules missing: + +* HTSlib/1.11-GCC-10.2.0 (HTSlib-1.11-GCC-10.2.0.eb) +* lpsolve/5.5.2.11-GCC-10.2.0 (lpsolve-5.5.2.11-GCC-10.2.0.eb) +* Boost/1.74.0-GCC-10.2.0 (Boost-1.74.0-GCC-10.2.0.eb) +* GSL/2.6-GCC-10.2.0 (GSL-2.6-GCC-10.2.0.eb) +* SAMtools/1.11-GCC-10.2.0 (SAMtools-1.11-GCC-10.2.0.eb) +* BCFtools/1.11-GCC-10.2.0 (BCFtools-1.11-GCC-10.2.0.eb) +* METIS/5.1.0-GCCcore-10.2.0 (METIS-5.1.0-GCCcore-10.2.0.eb) +* BamTools/2.5.1-GCC-10.2.0 (BamTools-2.5.1-GCC-10.2.0.eb) +* MPFR/4.1.0-GCCcore-10.2.0 (MPFR-4.1.0-GCCcore-10.2.0.eb) +* SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0.eb) +* AUGUSTUS/3.4.0-foss-2020b (AUGUSTUS-3.4.0-foss-2020b.eb) +``` + +Several dependencies are not installed yet, so we will need to use ``--robot`` to ensure that +EasyBuild also submits jobs to install these first. + +To speed up the installations a bit, we will request 10 cores for each submitted job (via ``--job-cores``). +That should be sufficient to let each installation finish in (well) under 1 hour, +so we only request 1 hour of walltime per job (via ``--job-max-walltime``). + +In order to have some meaningful job output files, we also enable trace mode (via ``--trace``). + +``` +$ eb AUGUSTUS-3.4.0-foss-2020b.eb --job --job-cores 10 --job-max-walltime 1 --robot --trace +... +== resolving dependencies ... +... +== List of submitted jobs (11): Boost-1.74.0-GCC-10.2.0 (Boost/1.74.0-GCC-10.2.0): 1000011; GSL-2.6-GCC-10.2.0 (GSL/2.6-GCC-10.2.0): 1000004; SAMtools-1.11-GCC-10.2.0 (SAMtools/1.11-GCC-10.2.0): 1000005; HTSlib-1.11-GCC-10.2.0 (HTSlib/1.11-GCC-10.2.0): 1000006; BCFtools-1.11-GCC-10.2.0 (BCFtools/1.11-GCC-10.2.0): 1000001; lpsolve-5.5.2.11-GCC-10.2.0 (lpsolve/5.5.2.11-GCC-10.2.0): 1000007; BamTools-2.5.1-GCC-10.2.0 (BamTools/2.5.1-GCC-10.2.0): 1000008; METIS-5.1.0-GCCcore-10.2.0 (METIS/5.1.0-GCCcore-10.2.0): 1000009; MPFR-4.1.0-GCCcore-10.2.0 (MPFR/4.1.0-GCCcore-10.2.0): 1000010; SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0): 1000002; AUGUSTUS-3.4.0-foss-2020b (AUGUSTUS/3.4.0-foss-2020b): 1000003 +== Submitted parallel build jobs, exiting now +``` + +### Inspecting the submitted jobs + +Once EasyBuild has submitted the jobs, we can inspect them via Slurm's ``squeue`` command: + +``` +$ squeue -u $USER -la + JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON) +1000001 small BCFtools user123 PENDING 0:00 2:00:00 1 (Dependency) +1000002 small SuiteSpa user123 PENDING 0:00 2:00:00 1 (Dependency) +1000003 small AUGUSTUS user123 PENDING 0:00 2:00:00 1 (Dependency) +1000004 small GSL-2.6- user123 RUNNING 0:21 2:00:00 1 node003 +1000005 small SAMtools user123 RUNNING 0:21 2:00:00 1 node007 +1000006 small HTSlib-1 user123 RUNNING 0:21 2:00:00 1 node007 +1000007 small lpsolve- user123 RUNNING 0:21 2:00:00 1 node011 +1000008 small BamTools user123 RUNNING 0:21 2:00:00 1 node011 +1000009 small METIS-5. user123 RUNNING 0:21 2:00:00 1 node013 +1000010 small MPFR-4.1 user123 RUNNING 0:21 2:00:00 1 node029 +1000011 small Boost-1. user123 RUNNING 0:24 2:00:00 1 node029 +``` + +Note that 3 jobs can not be started yet, because those installations require on one or more +missing dependencies. As soon as the jobs for those dependencies (successfully) complete, +these jobs will be able to start. + +### Final result + +After about 20 minutes, AUGUSTUS and all missing dependencies should be installed: + +``` +$ ls -lrt $HOME/easybuild/modules/all/*/*.lua | tail -11 +-rw-rw----. 1 example example 1634 Mar 29 10:13 /users/example/easybuild/modules/all/HTSlib/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1792 Mar 29 10:13 /users/example/easybuild/modules/all/SAMtools/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1147 Mar 29 10:13 /users/example/easybuild/modules/all/BamTools/2.5.1-GCC-10.2.0.lua +-rw-rw----. 1 example example 957 Mar 29 10:13 /users/example/easybuild/modules/all/lpsolve/5.5.2.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1549 Mar 29 10:13 /users/example/easybuild/modules/all/METIS/5.1.0-GCCcore-10.2.0.lua +-rw-rw----. 1 example example 1525 Mar 29 10:14 /users/example/easybuild/modules/all/GSL/2.6-GCC-10.2.0.lua +-rw-rw----. 1 example example 1221 Mar 29 10:15 /users/example/easybuild/modules/all/MPFR/4.1.0-GCCcore-10.2.0.lua +-rw-rw----. 1 example example 1678 Mar 29 10:15 /users/example/easybuild/modules/all/BCFtools/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1292 Mar 29 10:21 /users/example/easybuild/modules/all/Boost/1.74.0-GCC-10.2.0.lua +-rw-rw----. 1 example example 1365 Mar 29 10:28 /users/example/easybuild/modules/all/SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0.lua +-rw-rw----. 1 example example 2233 Mar 29 10:30 /users/example/easybuild/modules/all/AUGUSTUS/3.4.0-foss-2020b.lua + +$ module use $HOME/easybuild/modules/all + +$ module avail AUGUSTUS + +-------- /users/hkenneth/easybuild/modules/all -------- + AUGUSTUS/3.4.0-foss-2020b +``` + +--- + +*[[next: Module naming schemes]](3_04_module_naming_scheme.md)* diff --git a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md new file mode 100644 index 00000000..14229d97 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md @@ -0,0 +1,617 @@ +# Module naming schemes + +*[[back: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* + +--- + +Up until now we have used EasyBuild's default module naming scheme (`EasyBuildMNS`), +which produces module files with names that closely resemble to the names of the +corresponding easyconfig files. +For example, when installing `h5py-3.1.0-foss-2020b.eb` the generated module was named `h5py/3.1.0-foss-2020b`. + +EasyBuild supports several different module naming schemes: + +```shell +$ eb --avail-module-naming-schemes +List of supported module naming schemes: + CategorizedHMNS + CategorizedModuleNamingScheme + EasyBuildMNS + HierarchicalMNS + MigrateFromEBToHMNS +``` + +In this part of the tutorial we will take a closer look at `HierarchicalMNS`, +which is the standard **hierarchical** module naming scheme included with EasyBuild. + +We will also take a quick look at implementing our own custom module naming scheme. + +## Flat vs hierarchical + +The default module naming scheme `EasyBuildMNS` is an example of regular *"flat"* module naming scheme, which is characterized by: + +* all module files are directly available for loading; +* each module name uniquely identifies a particular installation; + +In contrast, a *hierarchical* module naming scheme +consists of a *hierarchy* of module files. + +The typical module hierarchy has 3 levels: + +* a **core** level, where module files for software that was installed using the + [`system` toolchain](../introduction/#system-toolchain) are kept; +* a **compiler** level, where module files for software that was installed using a +*compiler-only toolchain* are stored; +* and an **MPI** level, which houses module files for software that was installed using a toolchain that includes (at least) a compiler and MPI component; + +Here is a simple example of such a 3-level module hierarchy: + +
+ +In this example the core level only includes a single module `GCC/9.3.0`, +while the compiler level includes two modules: `OpenMPI/4.0.3` and `MPICH/3.3.2`. +In the MPI level, three modules are available: one for `FFTW`, one for `ScaLAPACK`, and one for `HDF5`. + +Initially only the modules on the top level of a module hierarchy are available for loading. +If you run "`module avail`" with the example module hierarchy, you will only see the `GCC/9.3.0` module. + +Some modules in the top level of the hierarchy act as a "gateway" to modules in the +next level below. +To make additional modules available for loading one of these gateway modules has to be loaded. In our exampe, loading the `GCC/9.3.0` module results in two additional modules coming into view from the compiler level, as indicated by the arrows: the modules for `OpenMPI` and `MPICH`. These correspond to installations of `OpenMPI` +and `MPICH` that were built using `GCC/9.3.0` as a (compiler-only) toolchain. + +Similarly, the `OpenMPI/4.0.3` module serves as a gateway to the three modules in the MPI level. Only by loading the `OpenMPI` module will these additional three modules become +available for loading. They correspond to software installations built using the `gompi/2020a` toolchain that +consists of the `GCC/9.3.0` compiler module and the `OpenMPI/4.0.3` MPI module. Software installed using +`foss/2020a` (which is a full toolchain that also includes OpenBLAS, FFTW and ScaLAPACK) would also be stored +in this level of the module hierarchy. + +The characteristics of a module hierarchy are: + +* not all module files are directly available for loading; +* some modules serve as a gateway to more modules; +* to access some software installations you will first need to load one or more gateway modules in order + to use them; + +You can probably think of other ways to organize module files in a hierarchical module tree, but here we will stick to the standard core / compiler / MPI hierarchy. + +### Pros & cons + +So why go through all this trouble of organizing modules hierarchically? + +There are a couple of advantages to this approach: + +* shorter module names; +* less overwhelming list of available modules; +* only compatible modules can be loaded together; + +However, the are some minor disadvantages too: + +* not all existing modules are directly visible; +* gateway modules may have little meaning to end users; + +#### Length of module names + +When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For our `HDF5` installation for example, +we have `HDF5/1.10.6-gompi-2020a` as module name. The `-gompi-2020a` part of the name refers to the toolchain that was +used for this installation, but it may be confusing to some people (what kind of Pokémon is a "gompi"?!). + +In the example module hierarchy shown above, the module for `HDF5` is named `HDF5/1.10.6` which is basically the bare +essentials: software name and version. That's way better, nice and clean! + +#### Amount of available modules + +The output of "`module avail`" can be quite overwhelming if lots of module files +are installed and a flat module naming scheme is used, since *all* modules are +*always* available. +EasyBuild makes it very easy to install lots of software, +so the number of installed modules can easily grow into the hundreds or even thousands... + +This often explosive growth of modules is less of an issue when using a hierarchical module naming scheme, since +initially only a small set of modules are available, and relatively limited +groups of additional modules become available as gateway modules are loaded. + +#### Loading compatible modules + +Since all modules are available at once when using a flat module naming scheme, you can easily load +modules together that are not compatible with each other. + +Imagine loading two modules that were built with a different compiler toolchain (different compiler, +different MPI library). That's likely to end in tears, unless you have the necessary technical expertise +to fully understand what is going on, *and* you are being very careful... + +In a module hierarchy this can be prevented, since modules for software that were installed with a +different compiler and/or a different MPI library are located in a different part of the module hierarchy, +and so these modules can not be loaded together easily. + + +#### Visibility of existing modules + +One downside of a module hierarchy is that not all existing modules are directly available for loading +or are even *visible* to the user, since the output of "`module avail`" only shows a subset of all modules. + +[Lmod](https://lmod.readthedocs.io) has a solution for this though: it provides a separate "`module spider`" +command to search for module files throughout the entire module hierarchy. So as long as the end users are +aware of this additional command, it should not be difficult to discover which software installations exist +and how they can be accessed. The "`module spider`" command will inform the user which of the gateway modules +need to be loaded in order to load a specific module file. + +#### Semantics of gateway modules + +An additional potential problem of a module hierarchy is that the semantics of the gateway modules may not be clear +to end users. They may wonder why they need to pick a specific compiler and MPI library, or which of the +available options is the best one. Maybe there are not even be aware what exactly a "compiler" is, or how it is +relevant to the software they need in their bioinformatics pipeline... + +This can be partially resolved by loading a default compiler and MPI module so a particular set of modules +is available right after login, which could be the ones used in the most recent toolchain, or the +recommended versions. More experienced users could then leverage the "`module spider`" command to navigate +the module hierarchy. + +## Using a custom module naming scheme + +Next to the module naming schemes that are included with EasyBuild, +you can also define your own module naming scheme (MNS), and configure EasyBuild to use it. + +### Implementation + +To use a custom module naming scheme, you need to implement a Python module file, +where you define a Python class that derives from the general [``ModuleNamingScheme``](https://docs.easybuild.io/en/latest/api/easybuild.tools.module_naming_scheme.mns.html#easybuild.tools.module_naming_scheme.mns.ModuleNamingScheme) class. + +For a flat module naming scheme, it is sufficient to implement the ``det_full_module_name`` method, +which should return a string value (the full module name). +You may also need to customize the ``is_short_modname_for`` method, which verifies whether +a given (short) module name is for the software with a particular given name (or not). + +The argument provided to ``det_full_module_name`` can be a small Python dictionary which only specifies +the value of a handful of easyconfig parameters (`name`, `version`, `toolchain`, and `versionsuffix`), +or an ``EasyConfig`` instance which represents a parsed easyconfig file and contains values for *all* +known easyconfig parameters. + +For simple module naming schemes, just have ``name``, ``version``, ``toolchain``, and ``versionsuffix`` +available is sufficient. If it is not, you can list which additional easyconfig parameters are required for +your module naming scheme via a class constant ``REQUIRED_KEYS``. + +A fairly straightforward example of a *flat* module name scheme is the default EasyBuild module naming scheme +[``EasyBuildMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/easybuild_mns.py). + +For a *hierarchical* module naming scheme, various additional methods have to be implemented. + +Some of these, like ``det_module_subdir`` and ``det_short_module_name``, determine which part of +the (full) module name determines the location of the module in the module hierarchy, and which part +is the user-facing (or "short") module name. Others, like ``det_modpath_extensions``, +determine which modules are *gateway* modules that open up an additional level of the hierarchy +(by *extending* the ``$MODULEPATH``). + +A typical example of a hierarchical module naming scheme is [``HierarchicalMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/hierarchical_mns.py) that comes with EasyBuild (which is quite complex because it includes quite a bit of special handling for particular compiler +toolchains), which implements the traditional ``Core``-``Compiler``-``MPI`` module hierarchy we discussed +above. + +### Configuring EasyBuild + +To let EasyBuild use a custom module naming scheme, you need to: + +* specify the path to the Python module file implementing it via the ``include-module-naming-schemes`` + EasyBuild configuration option; + +* indicate that you also want to *use* this custom module naming scheme via the ``module-naming-scheme`` + EasyBuild configuration option. + +For example: + +```shell +export EASYBUILD_INCLUDE_MODULE_NAMING_SCHEMES=$HOME/easybuild/example_mns.py +export EASYBUILD_MODULE_NAMING_SCHEME=ExampleMNS +``` + +#### Example custom module naming scheme + +Here is an example of a custom module naming scheme, where: + +* the ``versionsuffix`` goes directly after the ``version``; +* all dashes are replaced by underscores; +* all module names are lowercase; + +Note that we also need to customise the ``is_short_modname_for`` method, +to make sure it returns ``True`` when the EasyBuild framework checks whether +``scipy_bundle/2020.11_foss_2020b`` is a module name for ``SciPy-bundle``. + +```python +import os + +from easybuild.tools.module_naming_scheme.mns import ModuleNamingScheme + + +class ExampleMNS(ModuleNamingScheme): + + REQUIRED_KEYS = ['name', 'version', 'versionsuffix', 'toolchain'] + + def det_full_module_name(self, ec): + """ + Determine full module name: + - all lowercase + - replace all dashes with underscores + """ + parts = [ec['version']] + + # versionsuffix directly after version (but only if it's not empty) + if ec['versionsuffix']: + parts.append(ec['versionsuffix']) + + # only add toolchain name/version for non-system toolchain + tc = ec['toolchain'] + if tc['name'].lower() != 'system': + parts.extend([tc['name'], tc['version']]) + + modname = ec['name'] + '/' + '_'.join(parts) + + modname = modname.replace('-', '_').replace('__', '_') + + return modname.lower() + + def is_short_modname_for(self, short_modname, name): + """Determine whether short module name is a module for the software with specified name.""" + return short_modname.startswith(name.lower().replace('-', '_') + '/') +``` + +We can see what the module names with this module naming scheme would like like via ``eb -D``: + +``` +$ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D + ... + * [ ] $CFGS/g/GCC/GCC-10.2.0.eb (module: gcc/10.2.0) + ... + * [ ] $CFGS/p/Python/Python-2.7.18-GCCcore-10.2.0.eb (module: python/2.7.18_gcccore_10.2.0) + ... + * [ ] $CFGS/o/OpenMPI/OpenMPI-4.0.5-GCC-10.2.0.eb (module: openmpi/4.0.5_gcc_10.2.0) + ... + * [ ] $CFGS/s/SciPy-bundle/SciPy-bundle-2020.03-foss-2020a-Python-2.7.18.eb (module: scipy_bundle/2020.03_python_2.7.18_foss_2020a) +``` + +## Example module hierarchy: HDF5 + +!!! Warning + **This exercise will only work if you have write access to the software installation prefix**, + because EasyBuild will try to copy the installation log file to each installlation directory. + +Now that we know more about hierarchical module naming schemes, +let us see how EasyBuild can help us with generating a hierarchical module tree. + +In this example we will use EasyBuild to generate modules organised in a hierarchy +for some of the software that is already installed in the prepared environment. + +The good news is that the existing installations can be reused. There is absolutely no need +to reinstall the software, we are just creating a different "view" on these software installations. + +### Preparing the environment + +Before running EasyBuild to generate a hierarchical module tree, we have to be a bit careful +with preparing our environment. + +**We must absolutely avoid mixing modules from a flat and hierarchical module naming scheme!** + +Some module files will have the same name in both module trees (like `GCC/10.2.0` for example), +but their contents will be different. +Mixing modules from a flat and hierarchical module tree *will* trigger problems... + +So we have to make sure that the module files we already have in `/easybuild` are *not* visible. +The easiest way to do this is to unload all modules (using "`module purge`") +and resetting the module search path to be empty, which we can do with "`module unuse $MODULEPATH`". + +```shell +module purge +module unuse $MODULEPATH +``` + +In this part of the tutorial, we are assuming you are *not* using an EasyBuild installation provided through +a module. We have just made all modules unavailable, so we would have to first +install EasyBuild again in our hierarchical module tree before we can continue. + +**We strongly recommend using an EasyBuild installation that was [installed via "`pip install`" +or "`pip3 install`"](../installation/#method-1-using-pip-recommended) in this part of the tutorial.** + +An easy way to do this is in the prepared environment is to run: + +```shell +pip3 install --user easybuild +export PATH=$HOME/.local/bin:$PATH +export EB_PYTHON=python3 +``` + +### Configuring EasyBuild + +First of all, we need to make sure that EasyBuild is properly configured. +We can do this by defining this set of environment variables: + +```shell +export EASYBUILD_PREFIX=$HOME/easybuild +export EASYBUILD_BUILDPATH=/tmp/$USER +export EASYBUILD_INSTALLPATH_SOFTWARE=/easybuild/software +export EASYBUILD_MODULE_NAMING_SCHEME=HierarchicalMNS +export EASYBUILD_INSTALLPATH_MODULES=$HOME/hmns/modules +``` + +To make sure we didn't make any silly mistakes, we double check using `eb --show-config`: + + +```shell +$ eb --show-config +# +# Current EasyBuild configuration +# (C: command line argument, D: default value, E: environment variable, F: configuration file) +# +buildpath (E) = /tmp/example +containerpath (E) = /home/example/easybuild/containers +installpath (E) = /home/example/easybuild +installpath-modules (E) = /home/example/hmns/modules +installpath-software (E) = /easybuild/software +module-naming-scheme (E) = HierarchicalMNS +packagepath (E) = /home/example/easybuild/packages +prefix (E) = /home/example/easybuild +repositorypath (E) = /home/example/easybuild/ebfiles_repo +robot-paths (D) = /home/example/.local/easybuild/easyconfigs +sourcepath (E) = /home/example/easybuild/sources +``` + +There are a couple of things worth pointing out here: + +* We have defined the `module-naming-scheme` configuration setting to `HierarchicalMNS`, + which makes EasyBuild use the included standard hierarchical module naming scheme (the classic + core / compiler / MPI one we discussed above). +* We have specified *different* locations for the software (via `installpath-software`) + and the module files (via `installpath-modules`). This is important because we want to + reuse the software that is already installed in `/easybuild/software` while we want to + generate an entirely new module tree for it (in `$HOME/hmns/modules`). + +The other configuration settings are the same as before, and mostly irrelevant for this example. + +### Generating modules for HDF5 + +Let us now generate a hierarchical module tree for `HDF5` and all of its dependencies, +including the toolchain. That sounds complicated, and it sort of is since there are +a lot of details you have to get right for the module hierarchy to works as intended, +but EasyBuild can do all the hard work for us. + +The steps we will have to go through are: + +* Tell EasyBuild we want to "install" the `HDF5-1.10.7-gompi-2020b.eb` easyconfig file; +* Enable dependency resolution via `--robot`; +* Instruct EasyBuild to only generate the module files, not to install the software (since it is + there already in `/easybuild/software`), via the `--module-only` option. + +These steps translate to this single `eb` command: + +``` +$ eb HDF5-1.10.7-gompi-2020b.eb --robot --module-only +... +== building and installing MPI/GCC/10.2.0/OpenMPI/4.0.5/HDF5/1.10.7... +... +== sanity checking... +== cleaning up [skipped] +== creating module... +... +== COMPLETED: Installation ended successfully (took 9 sec) +... +== Build succeeded for 41 out of 41 +``` + +This should take a couple of minutes in total, for generating 41 modules. +Remember that this also includes generating module files for the toolchain and +all of its components. + +In addition, there is a bit more going on one that just generating module files, +since the sanity check step is still being run for each of the installations +when using `--module-only` to ensure the installation is actually functional. +After all, there is no point in generating a module for an obviously broken +installation... + +### Loading the HDF5 module + +After generating the hierarchical module tree for HDF5, how do we access the HDF5 installation through it? + +Here's what the module tree looks like on disk: + +``` +$ ls $HOME/hmns/modules/all +Compiler Core MPI +``` + +Those are basically the 3 levels in the module hierarchy we showed in our example earlier. + +The starting point is the top level of the module hierarchy named `Core`: + +``` +module use $HOME/hmns/modules/all/Core +``` + +Let us see what that gives us in terms of available modules: + +``` +$ module avail + +--------------------- /home/easybuild/hmns/modules/all/Core --------------------- + Bison/3.5.3 GCCcore/10.2.0 flex/2.6.4 help2man/1.47.4 + Bison/3.7.1 (D) M4/1.4.18 gettext/0.21 ncurses/6.2 + GCC/10.2.0 binutils/2.35 gompi/2020b zlib/1.2.11 +``` + + +Nice and short module names, but only a limited set of them. + +We know a module file exists for `HDF5`, but we can't see it yet (and hence +we can't load it either). + +``` +$ module avail HDF5 +No module(s) or extension(s) found! +Use "module spider" to find all possible modules and extensions. +``` + +Let us see if `module spider` is of any help, as "`module avail`" so kindly suggests: + +``` +$ module spider HDF5 +... + + You will need to load all module(s) on any one of the lines below + before the "HDF5/1.10.7" module is available to load. + + GCC/10.2.0 OpenMPI/4.0.5 +``` + +This tells us we need to load two gateway modules before we can load the module +for HDF5. + +Let us start with loading the `GCC` compiler module: + +``` +module load GCC/10.2.0 +``` + +And then check again which modules are available: + +``` +$ module avail + +-------------- /home/easybuild/hmns/modules/all/Compiler/GCC/10.2.0 -------------- + OpenMPI/4.0.5 + +------------ /home/easybuild/hmns/modules/all/Compiler/GCCcore/10.2.0 ------------ + Autoconf/2.69 UCX/1.9.0 libpciaccess/0.16 + ... + Szip/2.1.1 libfabric/1.11.0 zlib/1.2.11 (L,D) + Autoconf/2.69 XZ/5.2.5 libtool/2.4.6 + +--------------------- /home/easybuild/hmns/modules/all/Core --------------------- + Bison/3.5.3 GCCcore/10.2.0 (L) flex/2.6.4 help2man/1.47.4 + Bison/3.7.1 (D) M4/1.4.18 gettext/0.21 ncurses/6.2 + GCC/10.2.0 (L) binutils/2.35 gompi/2020b zlib/1.2.11 +``` + +Good news, we now have additional modules available! + +The compiler level of our hierarchy actually consists of two directories here: `Compiler/GCCcore/10.2.0` +and `Compiler/GCC/10.2.0`. The modules in the `GCCcore` directory are ones we can use in other compiler +toolchains that use GCC 10.2.0 as a base compiler (the details of that are out of scope here). + +The module we are interested in is `OpenMPI/4.0.5`, which is another gateway module. + +Remember that the "`module spider`" output told us that there does indeed exist a module for `HDF5`, but that +we need to load *both* the `GCC/10.2.0` and `OpenMPI/4.0.5` modules first. + +So, let us do exactly that (remember that `GCC/10.2.0` is already loaded): + +``` +module load OpenMPI/4.0.5 +``` + +If you now check the output of "`module avail`" again, you should see the `HDF5/1.10.7` module: + +``` +$ module avail + +-------- /home/easybuild/hmns/modules/all/MPI/GCC/10.2.0/OpenMPI/4.0.5 ------- + HDF5/1.10.7 + +------------ /home/easybuild/hmns/modules/all/Compiler/GCC/10.2.0 ------------ + OpenMPI/4.0.5 (L) + +... +``` + +To use HDF5, we need to load this `HDF5/1.10.7` module. We can verify that the installation works +using one of the commands provided by HDF5, `h5dump` for example: + +``` +module load HDF5/1.10.7 +``` + +``` +$ h5dump --version +h5dump: Version 1.10.7 +``` + +If you now check which modules are loaded via "`module list`", you will notice that all module names +and nice and short now, which is one of the advantages of using a hierarchical module tree: + +```shell +$ module list + +Currently Loaded Modules: + 1) GCCcore/10.2.0 5) numactl/2.0.13 9) hwloc/2.2.0 13) PMIx/3.1.5 + 2) zlib/1.2.11 6) XZ/5.2.5 10) libevent/2.1.12 14) OpenMPI/4.0.5 + 3) binutils/2.35 7) libxml2/2.9.10 11) UCX/1.9.0 15) Szip/2.1.1 + 4) GCC/10.2.0 8) libpciaccess/0.16 12) libfabric/1.11.0 16) HDF5/1.10.7 +``` + +## Exercise + +Now it is your turn! + +Try to get a feeling for how a hierarchical module tree works by: + +* installing the missing modules for the `SciPy-bundle-2020.11-foss-2020b.eb` in the module hierarchy we + generated for HDF5; +* figure out where the `SciPy-bundle` module is located in the hierarchy, and then also load it; + +You can verify your work by running this command (since `pandas` is one of the Python packages included +in the `SciPy-bundle` installation): + +```shell +python -c 'import pandas; print(pandas.__version__)' +``` + +Start from a clean slate, by first running: + +```shell +module purge +module unuse $MODULEPATH +``` + +??? success "(click to show solution)" + + * Step 0: check which modules are still missing, using `--missing` or `-M`: + ```shell + eb SciPy-bundle-2020.11-foss-2020b.eb -M + ``` + The output should tell you that 15 out of 50 required modules are still missing. + + * Install the missing modules in the module hierarchy we have generated in `$HOME/hmns/modules`: + ```shell + eb SciPy-bundle-2020.11-foss-2020b.eb --robot --module-only + ``` + Don't forget to use both `--robot` (to enable dependency resolution) and `--module-only` + (to only run the sanity check and generate module files, not install the software again). + + * Start at the top of the module hierarchy (the `Core` level), + and run module spider to check which gateway modules to load to make `SciPy-bundle` available: + ```shell + module use $HOME/hmns/modules/all/Core + module spider SciPy-bundle/2020.11 + ``` + * Load the gateway modules: + ```shell + module load GCC/10.2.0 OpenMPI/4.0.5 + ``` + * Check that the `SciPy-bundle` module is available, and load it: + ```shell + $ module avail SciPy-bundle + ----- /home/example/hmns/modules/all/MPI/GCC/10.2.0/OpenMPI/4.0.5 ------ + SciPy-bundle/2020.11 + ``` + ``` + module load SciPy-bundle/2020.11 + ``` + * Run the test command: + ``` + $ python -c 'import pandas; print(pandas.__version__)' + 1.1.4 + ``` + +!!! Warning + **This exercise will only work if you have write access to the software installation prefix**, + because EasyBuild will try to copy the installation log file to each installlation directory. + +--- + +*[[next: GitHub integration]](3_05_github_integration.md)* diff --git a/docs/2022-CSC_and_LO/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_05_github_integration.md new file mode 100644 index 00000000..59dcbb67 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_05_github_integration.md @@ -0,0 +1,442 @@ +# GitHub integration to facilitate contributing to EasyBuild + +*[[back: Module naming schemes]](3_04_module_naming_schemes.md)* + +--- + +To contribute changes to the EasyBuild code (framework or easyblocks) or easyconfigs, +you will need to be a bit familiar with Git and GitHub. Or maybe not? + +## Manual contribution procedure + +0) Create and setup a [GitHub account](https://github.com/join) (and register your SSH public key); + +1) Clone and fork the appropriate GitHub repository, for example when contributing an easyconfig file: + +```shell +git clone git@github.com:easybuilders/easybuild-easyconfigs.git +cd easybuild-easyconfigs +git remote add my_fork git@github.com:your_github_account/easybuild-easyconfigs.git +``` + +**You should change '`your_github_account`' in the last line to your own GitHub user name!** + +2) Create and check out a new branch, starting from the (up-to-date) ``develop`` branch: + +``` +git checkout develop +git pull origin develop +git checkout -b example +``` + +3) Stage the changes you want to contribute, after you make sure that your easyconfig file has the +[correct filename](../basic_usage/#easyconfig-filenames), and that it's located in the appropriate directory. + +```shell +mkdir -p easybuild/easyconfigs/e/example/ +mv example.eb easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb +git add easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb +``` + +4) Commit those changes with a sensible commit message: + +```shell +git commit -m "This is just an example" +``` + +5) Push your branch to your fork of the repository on GitHub: + +```shell +git push my_fork example +``` + +6) Open the pull request through the GitHub web interface, making sure that: + +* the target branch is correct (should be `develop`); +* an appropriate title is used; +* a short description of the changes is provided; +* the changes are indeed the ones you want to propose; +* clicking the (correct) green button; + +
Pfft
+ +That didn't exactly motivate you to contribute, did it... + +## Github integration features + +Over the years we noticed that some people were keen on contributing to EasyBuild, +but they were not very familiar with Git or GitHub. That meant they had to overcome a +relatively steep learning curve before they could contribute... + +
Gandalf vs Git
+ +In addition, the contribution workflow can be a bit daunting and time consuming, +even if you're already familiar with the procedure. You will have dozens of +branches flying around in no time, and if you get stuck in a weird corner +with `git` you may quickly end up demotivated. + +This is frustrating not only for the people who wanted to contribute but +also for the EasyBuild maintainers, and it doesn't agree with the philosophy of +a project that aims to *automate* tedious software installation procedures. + +At the end of 2015 efforts were made to tackle this issue by implementing +GitHub integration features in EasyBuild, which automate the contribution +workflow by running `git` commands and interacting with the [GitHub API](https://developer.github.com/v3/). + +We will briefly go over some of these features here, but they are also covered in detail [in the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html). + +### Requirements & configuration + +First of all, the GitHub integration features impose a couple of additional [requirements](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) +and configuration. + + +**Additional dependencies** + +Both the `GitPython` and `keyring` Python packages as well as the `keyrings.cryptfile` add-on package must be installed. +In the prepared environment, you can do this via: + +```shell +pip3 install --user GitPython keyring keyrings.cryptfile +``` + +!!! Note + You may experiences problems installing the ``cryptography`` Python packages, + which is a dependency of keyring. The underlying cause is that you need to have + the [``Rust``](https://www.rust-lang.org/) compiler installed to install the latest version + of ``cryptography`` (see [here](https://github.com/pyca/cryptography/issues/5771)). + + You can work around this issue using: + + ```shell + pip3 install --user 'cryptography<3.4' + ``` + +**SSH public key in GitHub account** + +You need to have a GitHub account that has your SSH public key registered in it +(via [https://github.com/settings/keys](https://github.com/settings/keys)). + +If you need to generate an SSH key pair, you can run the following command: + +```shell +ssh-keygen -t rsa -b 4096 +``` + +You can copy the SSH public key from the output of this command: + +```shell +cat .ssh/id_rsa.pub +``` + + +**Forked repository in GitHub** + +In addition, you must have *forked* the EasyBuild repository you want to contribute to +(for example [https://github.com/easybuilders/easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs)). + +**EasyBuild configuration, incl. GitHub token** + +You also have to configure EasyBuild a bit more, so it knows about your +GitHub user name *and* has a GitHub token available in order to perform actions +in GitHub with your credentials. + +To do this, you should define the `github-user` configuration option and +run the "`eb --install-github-token`" command: + +```shell +# replace 'ebtutorial' with your own GitHub username! +$ export EASYBUILD_GITHUB_USER=ebtutorial +$ eb --install-github-token +``` + +To create a GitHub token: + +* Visit [https://github.com/settings/tokens](https://github.com/settings/tokens). +* Click *"Personal access tokens"*. +* Click followed by *"Generate new token"*. +* Give the token a name (for example *"Token for EasyBuild"*). +* Select both the '`repo`' and '`gist`' scopes. +* Click the green *"Generate token"* button. +* Copy the generated token. +* Paste the token when asked by `--install-github-token` (and hit *Enter*). +* Enter a password to encrypt your GitHub token. + +The output should look something like this: + +```shell +$ eb --install-github-token +== temporary log file in case of crash /tmp/eb-9z0bdve9/easybuild-hfpti62w.log +Token: +Validating token... +Token seems to be valid, installing it. +Please set a password for your new keyring: +Please confirm the password: +Token 'fed..987' installed! +``` + + +**Checking status of GitHub integration** + +You can check the status of the GitHub integration using "`eb --check-github`": + +```shell +$ eb --check-github +== temporary log file in case of crash /tmp/eb-4ckdlyfy/easybuild-gp69ev2w.log + +Checking status of GitHub integration... + +Making sure we're online...OK + +* GitHub user...ebtutorial => OK +Please enter password for encrypted keyring: +* GitHub token...fed..987 (len: 40) => OK (validated) +* git command...OK ("git version 1.8.3.1; ") +* GitPython module...OK (GitPython version 3.1.3) +* push access to ebtutorial/easybuild-easyconfigs repo @ GitHub...OK +* creating gists...OK +* location to Git working dirs... not found (suboptimal) + +All checks PASSed! + +Status of GitHub integration: +* --from-pr: OK +* --new-pr: OK +* --review-pr: OK +* --update-pr: OK +* --upload-test-report: OK +``` + +If you see '`OK`' for each of the status checks, you're all set +to try out the GitHub integration features! + +!!! Note + If your SSH private key is protected with a password, you may need + to enter your password a couple of times when running "`eb --check-github`". + + You can avoid this by [using an SSH agent](https://help.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent). + +!!! Note + If you see the ``push access`` check fail with ``Failed to fetch branch 'main'``, + you will need to rename the ``master`` branch in your fork of the ``easybuild-easyconfigs`` + repository from ``master`` to ``main`` (this is required since EasyBuild v4.3.3). + + This can be done via the *pencil* icon at + [https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches](https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches) + (replace ``YOUR_GITHUB_ACCOUNT`` with the name of your GitHub account in this URL). + +### Creating pull requests + +The GitHub integration in EasyBuild allows you to **create pull requests +using the `eb` command**, without even leaving your shell environment. +How cool is that‽ + +To create a pull request to the `easybuild-easyconfigs` repository, +you can either do it in a single go by +running "`eb --new-pr`" and passing it one or more easyconfig files to add +into the pull request. + +The more detailed option is to first create a branch in your repository fork +in GitHub via "`eb --new-branch-github`" and then later open the pull request +via "`eb --new-pr-from-branch`". This method can be useful when preparing multiple +interdependent pull requests, or to check whether your changes pass the unit tests +(which are run automatically in the GitHub Actions CI environment for +all branches pushed to your fork). + +The `--new-pr` option can also be used to open pull requests to the easyblocks +and framework repositories, and it will even automatically determine the target +repository based on the contents of the files you provide. Whoa! + +You can control the target repository for your pull request using +`--pr-target-account` (default is `easybuilders`) and `--pr-target-repo`. + +If you want you can carefully double check your work before actually +opening the pull request by doing a dry run via "`eb --dry-run --new-pr`" +or "`eb -D --new-pr`". + +Finally, you can use "`eb --preview-pr`" to see how the easyconfig files +for which you plan to create a pull request differ from existing easyconfig +files. + +### Updating pull requests + +To update an existing pull request with additional changes +you can use "`eb --update-pr`" and pass the pull request ID, +alongside the paths to the updated files. + +If you have only created a branch (for example via `eb --new-branch-github`) +you can update it via `--update-branch-github` in the same way, +passing the branch name instead of a pull request ID. + +### Using a pull request + +Next to creating and updating branches and pull requests +you can also *use* easyconfig files and easyblocks from a pull request, +regardless of its status (open, merged, or closed). This is particularly +useful when testing contributions, or to install software for which +support is not yet included in the latest EasyBuild release. + +Using the `--from-pr` option you can install easyconfig files from the +pull request with specified ID. By default all easyconfig files that are +touched by the pull request will be installed, but you can specify +particular ones to use as well. It is generally advised to also use the +`--robot` option to ensure that the easyconfig files are installed in the +correct order with respect to dependencies. + +Similarly, using a new or updated easyblock from a pull request is as simple +as using the `--include-easyblocks-from-pr` option. And of course you can +combine it with `--from-pr`! + +Via `--upload-test-report` you can let EasyBuild submit a comment into the +easyconfig pull request to show that the installation worked on your system. This is +useful for others to know, in particular EasyBuild maintainers, since the comment +will include information about your system (OS, processor, etc.) and your EasyBuild configuration. + +## Demo + +That is a lot to digest, so let us make this a bit more concrete with an example: +we will open a pull request for the [`eb-tutorial` example software](../adding_support_software/#example) to *a fork* of the [`easybuild-easyconfigs` repository](https://github.com/easybuilders/easybuild-easyconfigs) using the `eb` command, +and submit a test report in it. + +!!! Note + Make sure that you have correctly configured the GitHub integration, + [see above](#requirements-configuration). + +### Creating pull request + +We first configure EasyBuild to target the `ebtutorial` GitHub account rather +than the default `easybuilders` GitHub organisation, +by defining the `pr-target-account` configuration setting: + +```shell +export EASYBUILD_PR_TARGET_ACCOUNT=ebtutorial +``` + +In the output of "`eb --show-config`" you should see a line like this: + +``` +pr-target-account (E) = ebtutorial +``` + +We only do this to avoid that lots of pull requests for the `eb-tutorial` +example software are opened in the [central easyconfigs repository](https://github.com/easybuilders/easybuild-easyconfigs). + +Opening a pull request is as simple as running "`eb --new-pr`" and passing +the easyconfig file: + +```shell +$ eb --new-pr example.eb +== temporary log file in case of crash /tmp/eb-ggr6scbq/easybuild-hnk271xj.log +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +== fetching branch 'develop' from https://github.com/ebtutorial/easybuild-easyconfigs.git... +== copying files to /tmp/eb-ggr6scbq/git-working-dirxwk1fzaw/easybuild-easyconfigs... +== pushing branch '20200622095415_new_pr_eb-tutorial100' to remote 'github_ebtutorial_qgtfU' (git@github.com:ebtutorial/easybuild-easyconfigs.git) +Enter passphrase for key '/home/example/.ssh/id_rsa': +Please enter password for encrypted keyring: + +Opening pull request +* target: ebtutorial/easybuild-easyconfigs:develop +* from: ebtutorial/easybuild-easyconfigs:20200622095415_new_pr_eb-tutorial100 +* title: "{tools}[GCC/10.2.0] eb-tutorial v1.0.1" +* labels: new +* description: +""" +(created using `eb --new-pr`) + +""" +* overview of changes: + easybuild/easyconfigs/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb | 26 ++++++++++++++++++++++++++ + 1 file changed, 26 insertions(+) + +Opened pull request: https://github.com/ebtutorial/easybuild-easyconfigs/pull/ +== Temporary log file(s) /tmp/eb-ggr6scbq/easybuild-hnk271xj.log* have been removed. +== Temporary directory /tmp/eb-ggr6scbq has been removed. +``` + +Take a moment to grasp what we did here: we ran **a single `eb` command** which +took care of the **[whole contribution procedure](#contribution-procedure)** for us, including: + +* Cloning the `easybuilders/easybuild-easyconfigs` repository and checking out the `develop` branch (in a temporary + directory); +* Picking a sensible name for a branch and creating it; +* Adding the `eb-tutorial` easyconfig file to the branch, in the correct location + (`easybuild/easyconfigs/e/eb-tutorial/`) and with the correct filename (`eb-tutorial-1.0.1-GCC-10.2.0.eb`); +* Pushing the branch to our fork (`example/easybuild-easyconfigs`); +* Actually opening the pull request, using an informative title. + +That is so... easy! + +This feature not only *significantly* lowers the bar for contributing, +it also saves quite a bit of time since you don't need to double check +various details (like targeting the `develop` branch) or spend time on +coming up with a nice looking title or funny branch name (although you +still can if you really want to). + +There are a couple of nice side effects too, like not having any local branches +to tidy up on once the pull request gets merged (since `--new-pr` created the +branch only in a temporary directory). + +If many contributions are made via `--new-pr` it also simplifies the task +of EasyBuild maintainers, since pull requests opened this way have a particular +structure to them and thus are easier to digest because they look familiar. + +### Uploading test report + +After opening the pull request, we should also upload a test report to show that the installation is working. +This is just as easy as creating the pull request. + +First make sure that the pre-installed software in the prepared environment +is available, since the required dependencies for `eb-tutorial` are already +installed there: + +```shell +module use /easybuild/modules/all +``` + +You can verify which dependencies are still missing using `--from-pr` combined with `--missing`: + +```shell +# change '1' to the ID of your own pull request (see output of --new-pr) +$ eb --from-pr 1 --missing +== temporary log file in case of crash /tmp/eb-ioi9ywm1/easybuild-e3v0xa1b.log +Please enter password for encrypted keyring: +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + +1 out of 20 required modules missing: + +* eb-tutorial/1.0.1-GCC-10.2.0 (eb-tutorial-1.0.1-GCC-10.2.0.eb) +``` + +Uploading a test report boils down to combining `--from-pr` with `--upload-test-report`: + +```shell +# change '1' to the ID of your own pull request (see output of --new-pr) +$ eb --rebuild --from-pr 1 --upload-test-report +Please enter password for encrypted keyring: +... +== processing EasyBuild easyconfig /tmp/eb-bnb1pv3n/files_pr65/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb +== building and installing eb-tutorial/1.0.1-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 2 sec) +... +Adding comment to easybuild-easyconfigs issue #65: 'Test report by @ebtutorial +**SUCCESS** +Build succeeded for 1 out of 1 (1 easyconfigs in this PR) +example - Linux centos linux 7.8.2003, x86_64, Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz (haswell), Python 3.6.8 +See https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 for a full test report.' +== Test report uploaded to https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 and mentioned in a comment in easyconfigs PR#1 +``` + +Note that we may need to use `--rebuild` here since `eb-tutorial` may already be installed. + +This results in a comment being added to the pull request: + +
Test report comment
+ +The gist linked from this comment provides more detailed information: + +
Test report gist
+ +--- + +*[[next: Overview]](index.md)* diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md index 9287b960..69ab40e8 100644 --- a/docs/2022-CSC_and_LO/index.md +++ b/docs/2022-CSC_and_LO/index.md @@ -67,6 +67,10 @@ freely available, so unless you have access to a system with this environment yo - [Installation](1_06_installation.md) - [Configuration](1_07_configuration.md) - [Basic usage](1_08_basic_usage.md) *(hands-on)* +- [Part II: **Using EasyBuild**](2_00_part2_using.md) *(Tue March 23rd 2021, 9am-12 CET)* + - [Troubleshooting](2_01_troubleshooting.md) *(hands-on)* + - [Creating easyconfig files](2_02_creating_easyconfig_files.md) *(hands-on)* + - [Implementing easyblocks](2_03_implementing_easyblocks.md) *(hands-on)* diff --git a/mkdocs.yml b/mkdocs.yml index bb2dc75d..3200ba55 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -45,6 +45,18 @@ nav: - Installation: 2022-CSC_and_LO/1_06_installation.md - Configuration: 2022-CSC_and_LO/1_07_configuration.md - Basic usage: 2022-CSC_and_LO/1_08_basic_usage.md + - Using EasyBuild: + - (overview): 2022-CSC_and_LO/2_00_part2_using.md + - Troubleshooting: 2022-CSC_and_LO/2_01_troubleshooting.md + - Creating easyconfig files: 2022-CSC_and_LO/2_02_creating_easyconfig_files.md + - Implementing easyblocks: 2022-CSC_and_LO/2_03_implementing_easyblocks.md + - Advanced topics: + - (overview): 2022-CSC_and_LO/3_00_part3_advanced.md + - Using EasyBuild as a library: 2022-CSC_and_LO/3_01_easybuild_library.md + - Using hooks to customise EasyBuild: 2022-CSC_and_LO/3_02_hooks.md + - Submitting installations as Slurm jobs: 2022-CSC_and_LO/3_03_slurm_jobs.md + - Module naming schemes (incl. hierarchical): 2022-CSC_and_LO/3_04_module_naming_schemes.md + - GitHub integration to facilitate contributing to EasyBuild: 2022-CSC_and_LO/3_05_github_integration.md - "ISC'21 (25 June 2021)": - (overview): 2021-isc21/index.md - Practical info: 2021-isc21/practical_info.md From 8d6b6f3f636df11bc4a06e0776781fdb023f2e3b Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 30 Mar 2022 18:58:28 +0200 Subject: [PATCH 022/159] Troubleshooting section reworked for LUMI. --- docs/2022-CSC_and_LO/2_01_troubleshooting.md | 159 +++++++++++++------ 1 file changed, 108 insertions(+), 51 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_01_troubleshooting.md b/docs/2022-CSC_and_LO/2_01_troubleshooting.md index 86a2f1d3..30f2f5cc 100644 --- a/docs/2022-CSC_and_LO/2_01_troubleshooting.md +++ b/docs/2022-CSC_and_LO/2_01_troubleshooting.md @@ -4,8 +4,8 @@ --- -When installing scientific software you are bound to run into problems -that make the installation fail sooner or later, even when using EasyBuild. +Whatever tool you use, when installing scientific software you'll +be running into problems rather sooner than later. In this part we take a look at how you can **troubleshoot a failing installation**, and focus on a couple of EasyBuild aspects that can be helpful in that context, @@ -50,8 +50,8 @@ $ eb example.eb == building... == FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): build failed (first 300 chars): cmd "make" exited with exit code 2 and output: -/usr/bin/g++ -O2 -ftree-vectorize -march=native -fno-math-errno -std=c++14 -c -o core.o core.cpp -g++: error: unrecognized command line option '-std=c++14' (took 1 sec) +/usr/bin/g++ -O2 -ftree-vectorize -march=znver2 -fno-math-errno -c -o core.o core.cpp +cc1plus: error: bad value (‘znver2’) for ‘-march=’ switch (took 1 sec) == Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log ERROR: Build of /home/easybuild/subread.eb failed (err: ...) ``` @@ -60,22 +60,24 @@ Let's break this down a bit: during the `build` step of the installation procedure EasyBuild was running `make` as a shell command, which failed (exit code 2, so not zero). The `make` command tripped over the compilation of `core.cpp` that failed because -`-std=c++14` is not a known option to the `g++` command. +`-march=znver2` is not a known option to the `g++` command. OK fine, but now what? Can you spot something suspicious here? Wait a minute... Why is `make` using `/usr/bin/g++` for the compilation?! That's not where our toolchain compiler is installed, -that's somewhere under `/easybuild/software`. +that's somewhere under `/opt/cray/pe/gcc`. Let's see what `/usr/bin/g++` is: -```shell +``` $ /usr/bin/g++ --version -g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) +g++ (SUSE Linux) 7.5.0 ``` -Oh my that's pretty ancient (GCC 4.8.5 was released in June 2015). -That could definitely explain why it doesn't know about the C++14 standard yet... +Oh my that's an ancient compiler (7.5 was released on November 14, 2019, +a few months after the release of the Zen2 architecture, but +the base version, 7.1, is really from May 2, 2017, long before the Zen2 architecture was around) +That could definitely explain why it doesn't know about the Zen2 architecture yet... Your next step in this case should probably be figuring out why `/usr/bin/g++` is being used rather than just `g++`, which would @@ -98,7 +100,7 @@ In some cases there won't be any useful information in there however, since the actual error message(s) could only appear way later, perhaps even after the command was already running for several minutes. -In that case, you will have the dive into the log file that is created by EasyBuild for +In that case, you will have to dive into the log file that is created by EasyBuild for every installation, which is located in the unique temporary directory for the EasyBuild session. See for example this output line from our earlier example error message: @@ -122,14 +124,14 @@ directory for successful installation, into the `easybuild` subdirectory. For example: ``` -/easybuild/software/HDF5/1.10.7-gompi-2020b/easybuild/easybuild-HDF5-1.10.7-20210308.214453.log +/appl/lumi/SW/LUMI-21.12/L/EB/ncurses/6.2-cpeGNU-21.12/easybuild/easybuild-ncurses-6.2-20220302.110244.log ``` ### Last log The `eb` command supports a handy little option that prints the location to the most recently updated build log. You can leverage this to quickly -open the build log of the last failed EasyBuild session in an editor: +open the build log of the last ***failed*** EasyBuild session in an editor: ``` vim $(eb --last-log) @@ -206,7 +208,10 @@ For software using a classic `configure` script, you may have to locate and inspect the `config.log` file in the build directory to determine the underlying cause of an error. For software using CMake as a configuration tool you often have to check in `CMakeOutput.log` or `CMakeError.log` for clues, which are sneakily hidden by CMake in a `CMakeFiles` subdirectory of the build directory. -As a side note here: as EasyBuild does not clean out old and failed builds you will need to eventually manually remove these build directories from the `buildpath` directory. +As a side note here: as EasyBuild does not clean out old and failed builds you will need to eventually manually remove these build directories from the `buildpath` directory. On the login nodes of LUMI this is currently not much of an issue as `$XDG_RUNTIME_DIR` is (ab)used for build and +temporary files and that directory is cleaned automatically. However, when building on compute nodes, where `$XDG_RUNTIME_DIR` is +not available, it is essential to manually clean that directory as the space is not automatically cleaned when your session +ends. The `EasyBuild-user` module does define the bash function `clear-eb` that can be used to clear that space. ## Exercise @@ -214,7 +219,7 @@ Let's work our way through a less smooth software installation, using the easyconfig file that is provided below. Don't worry if most of this is still unclear to you, we'll get -to writing easyconfig files from scratch [later in this tutorial](../creating_easyconfig_files). +to writing easyconfig files from scratch [later in this tutorial](2_02_creating_easyconfig_files). ```python easyblock = 'MakeCp' @@ -225,7 +230,7 @@ version = '2.0.1' homepage = 'http://subread.sourceforge.net' description = "High performance read alignment, quantification and mutation discovery" -toolchain = {'name': 'GCC', 'version': '8.5.0'} +toolchain = {'name': 'PrgEnv-gnu', 'version': '21.10'} # download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz sources = ['subread-%(version)s-source.tar.gz'] @@ -253,25 +258,23 @@ Do you spot any potential problems yet with this easyconfig file? ***Preparation*** -Start by copying the text above in a file named `subread.eb`, +Start by copying the text above in a file named `subread.eb` +(which does not follow the EasyBuild conventions but that is not a problem for this exercise), so you can gradually fix the problem you'll encounter. Also make sure that the pre-installed software stack is available, -and that the EasyBuild module is loaded (unless you installed EasyBuild +and that the EasyBuild-user module is loaded (unless you installed EasyBuild yourself): ``` -module use /easybuild/modules/all -module load EasyBuild +module load LUMI/21.12 +module load EasyBuild-user ``` -For this exercise, make sure EasyBuild is configured to -use `$HOME/easybuild` as `prefix`, and to use `/tmp/$USER` as `buildpath`: - -```shell -export EASYBUILD_PREFIX=$HOME/easybuild -export EASYBUILD_BUILDPATH=/tmp/$USER -``` +This will configure EasyBuild correctly for this exercise, though if you already have +an existing EasyBuild user installation you may want to work in a different one +by pointing `$EBU_USER_PREFIX` to the desired work directory before loading +`LUMI/21.12`. Check your configuration via `eb --show-config`. @@ -297,7 +300,7 @@ the easyconfig file? ``` $ eb subread.eb ... - == FAILED: Installation ended unsuccessfully (build directory: /tmp/example/Subread/2.0.1/GCC-8.5.0): build failed (first 300 chars): + == FAILED: Installation ended unsuccessfully (build directory: /run/user/XXXXXXXX/easybuild/build/Subread/2.0.1/cpeGNU-21.12): build failed (first 300 chars): Couldn't find file subread-2.0.1-source.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order): ... ``` @@ -315,12 +318,10 @@ the easyconfig file? (in the `sourcepath` directory): ``` curl -OL https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz - mv subread-2.0.1-source.tar.gz $HOME/easybuild/sources/s/Subread/ + mkdir -p $EBU_USER_PREFIX/sources/s/Subread + mv subread-2.0.1-source.tar.gz $EBU_USER_PREFIX/sources/s/Subread/ ``` - If downloading is problematic for some reason, the source tarball is also available - in `/easybuild/sources/s/Subread`. - Or, we can change the easyconfig file to specify the location where the easyconfig file can be downloaded from: ```python @@ -335,9 +336,9 @@ the easyconfig file? The source tarball is fairly large (23MB), so don't be alarmed if the download takes a little while. ```shell - $ ls -lh $HOME/easybuild/sources/s/Subread + $ ls -lh $EBU_USER_PREFIX/sources/s/Subread total 23M - -rw-rw-r-- 1 easybuild easybuild 23M Jun 13 17:42 subread-2.0.1-source.tar.gz + -rw-rw-r-- 1 XXXXXXXX XXXXXXXX 23M Mar 30 16:08 subread-2.0.1-source.tar.gz ``` --- @@ -350,26 +351,52 @@ What's wrong now? How can you fix it quickly? Take into account that we just want to get this software package installed, we don't care too much about details like the version of the dependencies or -the *toolchain* here... +the toolchain here... ??? success "(click to show solution)" The installation fails because the easyconfig specifies that GCC 8.5.0 should be used as toolchain: + ``` $ eb subread.eb ... - == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-8.5.0): build failed (first 300 chars): - No module found for toolchain: GCC/8.5.0 (took 1 sec) + ERROR: Failed to process easyconfig /pfs/lustrep3/users/kurtlust/easybuild-tutorial/Troubleshooting/subread.eb: Toolchain PrgEnv-gnu not found, + available toolchains: ... + ... + ``` + + `PrgEnv-gnu` is an HPE Cray PE module that may look like a toolchain - it certainly has + the same function: provide compiler, MPI and basic math libraries - but it is not + recognised as a toolchain by EasyBuild. EasyBuild prefers to manage its own modules so that it knows + well what is in it which is not the case with the `PrgEnv-*` modules from the Cray PE + as the content may differe between systems and as the versions of the compilers etc. that + are loaded differ on other modules that are loaded. Hence we created Cray-specific toolchains. + You'll actually find two series of Cray toolchains in the list of available toolchains. The + `CrayGNU`, `CrayIntel`, `CrayPGI` and `CrayCCE` are included with the EasyBuild distribution + and where developed by CSCS for their systems using Environment Modules. These were not compatible + with the initial releases of the Cray PE with Lmod modules so new ones were developed on which we + also built for the LUMI toolchains. Those are called `cpeCray`, `cpeGNU`, `cpeAOCC` and `cpeAMD` + and are maintained by LUST and available via the LUMI repositories. + + Changing the toolchain name to `cpeGNU` is not enough to solve all problems though: + + ``` + $ eb subread.eb + ... + ERROR: Build of /pfs/lustrep3/users/kurtlust/easybuild-tutorial/Troubleshooting/subread.eb failed (err: 'build failed (first 300 chars): + No module found for toolchain: cpeGNU/21.10') + ... ``` - We don't have this GCC version installed, but we do have GCC 10.2.0: + We don't have this `cpeGNU` version installed, but we do have GCC 21.12: ```shell - $ module avail GCC/ - ----------------- /easybuild/modules/all ------------------ - GCC/10.2.0 + $ module avail cpeGNU/ + ----- Infrastructure modules for the software stack LUMI/21.12 on LUMI-L ----- + cpeGNU/21.12 + ... ``` So let's try using that instead. @@ -377,7 +404,7 @@ the *toolchain* here... Edit the easyconfig file so it contains this: ```python - toolchain = {'name': 'GCC', 'version': '10.2.0'} + toolchain = {'name': 'cpeGNU', 'version': '21.12'} ``` --- @@ -392,13 +419,13 @@ Can you fix the next problem you run into? The compilation fails, but the error message we see is incomplete due to EasyBuild truncating the command output (only the 300 first characters of the output are shown): ``` - == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-10.2.0): build failed (first 300 chars): - cmd " make -j 1 -f Makefile.Linux CFLAGS="-fast -fcommon"" exited with exit code 2 and output: - gcc -mtune=core2 -O3 -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\""2.0.1"\" -D_FILE_OFFSET_BITS=64 -fmessage-length=0 -ggdb -fast -fcommon -c -o core.o core.c - gcc: error: unrecognized command line opti (took 1 sec) + == FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12): build failed + (first 300 chars): cmd " make -j 256 -f Makefile.Linux CFLAGS="-fast -fcommon"" exited with exit code 2 and output: + gcc -mtune=core2 -O3 -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\""2.0.1"\" -D_FILE_OFFSET_BITS=64 -fmessage-length=0 + -ggdb -fast -fcommon -I/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86 (took 4 secs) ``` - If you open the log file and scroll to the end, + If you open the log file (e.g., with `view $(eb --last-log)`) and scroll to the end, the error is pretty obvious: ``` gcc: error: unrecognized command line option -fast; did you mean -Ofast? @@ -416,7 +443,8 @@ Can you fix the next problem you run into? to hard specify compiler flags (certainly not incorrect ones). The comment above the `buildopts` definition makes it clear that the `-fcommon` flag *is* required though, because GCC 10 became a bit stricter by - using `-fno-common` by default. Note that we are using `-fcommon` + using `-fno-common` by default (and we're using GCC 11 in `cpeGNU/21.12`). + Note that we are using `-fcommon` as an escape mechanism here: it would be better to fix the source code and create a patch file instead. @@ -433,6 +461,32 @@ Can you fix the next problem you run into? Note that we need to be careful with quotes here: we use inner double quotes to ensure that `$CFLAGS` will be expanded to its value when the build command is run. + As you can see after the change by doing a dry-run: + + ``` + $ eb subread.eb -x + ... + + Defining build environment... + + export BLAS_INC_DIR='/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/include' + ... + export CC='cc' + export CFLAGS='-O2 -ftree-vectorize -fno-math-errno' + ... + [build_step method] + running command "make -j 256 -f Makefile.Linux CFLAGS="$CFLAGS -fcommon"" + (in /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12/Subread-2.0.1/src) + ... + ``` + + EasyBuild will launch the command + ``` + make -j 256 -f Makefile.Linux CFLAGS="$CFLAGS -fcommon" + ``` + in a shell where `CFLAGS` is defined and set to an appropriate value (determined by + defaults in EasyBuild, settings in the EasyBuild configuration and settings in the + easyconfig file that we shall discuss later). --- @@ -449,10 +503,11 @@ Don't give up now, try one last time and fix the last problem that occurs... ``` $ eb subread.eb ... - == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-10.2.0): build failed (first 300 chars): - Sanity check failed: sanity check command featureCounts --version exited with code 255 + == FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12): + build failed (first 300 chars): Sanity check failed: sanity check command featureCounts --version exited with code 255 (output: featureCounts: unrecognized option '--version' ... + ... ``` If you look at the full output in the log file you can see @@ -473,12 +528,14 @@ Don't give up now, try one last time and fix the last problem that occurs... --- -In the end, you should be able to install Subread 2.0.1 with the GCC 10.2.0 toolchain by fixing the problems with the `subread.eb` easyconfig file. +In the end, you should be able to install Subread 2.0.1 with the cpeGNU 21.12 toolchain by fixing the problems with the `subread.eb` easyconfig file. Check your work by manually loading the module and checking the version via the `featureCounts` command, which should look like this: ```shell +$ module load Subread/2.0.1-cpeGNU-21.12 +... $ featureCounts -v featureCounts v2.0.1 ``` From 51c7e3d1963aa1bb19824350d21c5c17a8c977dd Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 1 Apr 2022 18:15:00 +0200 Subject: [PATCH 023/159] Section on creating easyconfig files. --- .../2_02_creating_easyconfig_files.md | 527 +++++++++--------- 1 file changed, 278 insertions(+), 249 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md index d46e003a..99ac7d88 100644 --- a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md @@ -4,8 +4,9 @@ --- -We already know how to install easyconfig files that are included with EasyBuild, -but what about installing software for which no easyconfig is available yet? +We already know how to install easyconfig files that are provided by LUMI, +but what about installing software for which no easyconfig is available yet, +or for which we need to adapt an existing easyconfig? To do this we will need to **create additional easyconfig files**, since every software installation performed by EasyBuild is done based on an easyconfig file. @@ -14,10 +15,10 @@ In this part of the tutorial we will look at the guts of easyconfig files and ev ## Easyconfigs vs easyblocks -Before we dive into writing [easyconfig files](../introduction/#easyconfig-files), -let us take a brief look at how they relate to [easyblocks](../introduction/#easyblocks). +Before we dive into writing [easyconfig files](../1_05_terminology/#easyconfig-files), +let us take a brief look at how they relate to [easyblocks](../1_05_terminology/#easyblocks). -As we discussed [earlier](../introduction/#terminology), an easyconfig file (`*.eb`) is required +As we discussed [earlier](../1_05_terminology/#terminology), an easyconfig file (`*.eb`) is required for each installation performed by EasyBuild which specifies the details of the installation (which software version, toolchain, etc.), while the installation procedure is implemented @@ -30,34 +31,47 @@ together with a custom *software-specific* easyblock? This is not an easy question to answer in a general sense, since it depends on several factors: the complexity of the software you want to get installed, how much flexibility you want, how "intelligent" the installation procedure should be with respect to the compiler toolchain and dependencies -that are used for the installation, etc. +that are used for the installation, whether you want an easyconfig that is more user-focused (with easy-to-understand +parameters) or one that is more focused towards people who understand installation commands, etc. -In a nutshell, custom software-specific easyblocks are "do once and forget": they are central solution to peculiarities in the installation procedure of a particular software package. +In a nutshell, custom software-specific easyblocks are "do once and forget": they are central solution to peculiarities in the installation procedure of a particular software package. However, they also hide a lot from direct view, so if things go +wrong, it is often harder to debug the exact problem. And since a single easyblock has to cover +multiple software versions, multiple toolchains and multiple possible configurations of the package, +they are often harder to develop and certainly harder to test. As a result many of the easyblocks +included with EasyBuild work poorly on HPE Cray systems, e.g., because they want to add compiler flags +specific for a certain compiler and don't recognise the Cray compilers. Reasons to consider implementing a software-specific easyblock rather than using a generic easyblock include: -* 'critical' values for easyconfig parameters required to make installation succeed; -* toolchain-specific aspects of the build and installation procedure (e.g., configure options); -* interactive commands that need to be run; -* custom (configure) options for dependencies; -* having to create or adjust specific (configuration) files; -* 'hackish' usage of a generic easyblock; -* complex or very non-standard installation procedure; - -Implementing easyblocks is out of scope for this basic tutorial, for more information please consult -the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). +- 'critical' values for easyconfig parameters required to make installation succeed; +- toolchain-specific aspects of the build and installation procedure (e.g., configure options); +- interactive commands that need to be run; + *For example: The easyblock to [install maple](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/m/maple.py) + responds to a number of interactive questions.* +- custom (configure) options for dependencies; + *For example, the [easyblock for VMD](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/v/vmd.py) + will add configure options depending on the dependency list of the package.* +- having to create or adjust specific (configuration) files; +- 'hackish' usage of a generic easyblock; +- complex or very non-standard installation procedure; + *For example, the [easyblock to install the gcc compilers from source](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/g/gcc.py), + bootstrapping with the system compiler and then re-installing with itself.* + +For implementing easyblocks we refer to the +["Implementing easyblocks" section of this tutorial](2_03_implementing_easyblocks.md) +and the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). ## Writing easyconfig files Writing an easyconfig file boils down to defining a set of easyconfig parameters in a text file, which we give a '`.eb`' extension by convention. The name of an easyconfig file doesn't matter when using it directly to install software, but it does matter when EasyBuild needs to find it -to resolve a dependency for example (as we [discussed earlier](../basic_usage/#easyconfig-filenames)). +to resolve a dependency for example (as we [discussed earlier](../1_08_basic_usage/#easyconfig-filenames)). The syntax for easyconfig files is *Python syntax*: you are basically defining a bunch of Python variables that correspond to easyconfig parameters. -The order in which the easyconfig parameters are defined doesn't matter, but we generally try to strick to a particular +The order in which the easyconfig parameters are defined doesn't matter, but we generally try to stick to a particular order which roughly corresponds to the order in which the easyconfig parameters are used during the installation. That is mostly helpful for humans staring at easyconfig files or comparing them. @@ -65,13 +79,13 @@ That is mostly helpful for humans staring at easyconfig files or comparing them. A limited number of easyconfig parameters are *mandatory*, they must be defined in every easyconfig file: -* `name`: the name of the software to install; -* `version`: the version of the software to install; -* `homepage`: a URL to the website of the software; -* `description`: a short description of the software; -* `toolchain`: the compiler toolchain to use for the installation; +- `name`: the name of the software to install; +- `version`: the version of the software to install; +- `homepage`: a URL to the website of the software; +- `description`: a description of the software; +- `toolchain`: the compiler toolchain to use for the installation; -**`name`, `version`** +#### `name`, `version` It should be no surprise that specifying the name and version of the software you want to install is mandatory. This information may influence the value of several other easyconfig parameters (like the name of the source file), and is also used to the determine the name of the module file to install. @@ -81,7 +95,7 @@ name = 'example' version = '1.0' ``` -**`homepage`, `description`** +#### `homepage`, `description` The homepage and description are included in the generated module file for the installation. That way the "`module show`" command provides some useful high-level information about the installation. @@ -91,10 +105,8 @@ homepage = 'https://example.org' description = "This is just an example." ``` -Usually it does not matter whether you use single or double quotes to specify string values, -but you will often see that single quotes are used for values that don't have spaces (words) -and double quotes for values that do have spaces (sentences). There is no technical reason for -this, it just feels more natural to some people. There are cases where it is important to use +Usually it does not matter whether you use single or double quotes to specify string values. +There are cases where it is important to use the right type of quotes however, we will get back to that later (keep it in mind for the exercises!). For multi-line descriptions, you will need to use "triple quoting" (which is standard Python syntax): @@ -105,23 +117,43 @@ description = """This is an example It is spread across multiple lines.""" ``` -**`toolchain`** +If no homepage is known for the package, the convention in the EasyBuild community is to use +`(none)` as the value. + +The `description` field is used in two different places in the LUA module file generated by +EasyBuild: +- In the help block, for the first section of the help information +- If there is no `whatis` parameter in the easyconfig file, it is also used to generate + a whatis-command with the description in the module file. + +*Note:* We ask people who contribute easyconfigs to LUMI to instead use a decent description +that is useful in the help information, and to specify a short description via the `whatis` +parameter: + +```python +whatis = ['Description: Blosc is an extremely fast, multi-threaded, meta-compressor library'] +``` + +Note that in this case the word "Description:" has to be explicitly specified. -EasyBuild also requires that the [compiler toolchain](../introduction/#toolchains) is specified, via the `toolchain` + +#### `toolchain` + +EasyBuild also requires that the [compiler toolchain](../1_05_terminology/#toolchains) is specified, via the `toolchain` easyconfig parameter. -This can either be the [`system` toolchain](../introduction/#system-toolchain), for which a constant named `SYSTEM` is available: +This can either be the [`system` toolchain](../1_05_terminology/#system-toolchain), for which a constant named `SYSTEM` is available: ```python toolchain = SYSTEM ``` -Usually we specify a 'proper' toolchain like the compiler-only toolchain GCC 10.2.0 which we used before, -or the full toolchain `foss` 2020b. The name and version of the toolchain can be specified using a small Python dictionary, +Usually we specify a 'proper' toolchain like the `cpeGNU/21.12` toolchain we have used bvefore. +The name and version of the toolchain can be specified using a small Python dictionary, for example: ```python -toolchain = {'name': 'GCC', 'version': '10.2.0'} +toolchain = {'name': 'cpeGNU', 'version': '21.12'} ``` ### Commonly used parameters @@ -177,8 +209,11 @@ Some things worth pointing out here: file. This way the software version is only specified in one place and the easyconfig file is easier to update to other software versions. A list of template values can be consulted via the EasyBuild command line via the `--avail-easyconfig-templates` option, or in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_templates.html). -* Source files can also be specified in ways other than just using a filename, see the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-param-sources-alt) for more information. -* Specified checksums are usually SHA256 checksum values, but [other types are also supported](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html?highlight=checksums#checksums). +* Source files can also be specified in ways other than just using a filename, see the + [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-param-sources-alt) for more information. + It is also possible to download a given commit from a GitHub repository. +* Specified checksums are usually SHA256 checksum values, but + [other types are also supported](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html?highlight=checksums#checksums). #### Easyblock @@ -189,7 +224,9 @@ This is not mandatory however, because by default EasyBuild will determine the e name of the software. If '`example`' is specified as software name, EasyBuild will try to locate a software-specific easyblock named `EB_example` (in a Python module named `example.py`). Software-specific easyblocks follow the convention that the class name starts with `'EB_`', followed by the software name -(where some characters are replaced, like '`-`' with '`_minus_`'). +(where some characters are replaced, like '`-`' with '`_minus_`'). It is possible to use different +naming conventions for software-specific easyblocks, but then EasyBuild will not automatically detect +that there is one for the package and it will also need to be specified via the `easyblock` parameter. **Generic easyblocks** @@ -234,7 +271,7 @@ tar_config_opts* Override tar settings as determined by configure. [defau #### Dependencies -You will often need to list one or more [dependencies](../introduction/#dependencies) that are required +You will often need to list one or more [dependencies](../1_05_terminology/#dependencies) that are required to install or run the software. We distinguish between two main different types of dependencies: runtime dependencies and build dependencies. @@ -248,12 +285,17 @@ software once it is installed. The modules for these dependencies will be loaded set up by EasyBuild during the installation, but they will *not* be loaded by the generated module file. You can specify build dependencies via the `builddependencies` easyconfig parameter. One typical example of a build dependency is `CMake`, which is only needed for configuring -the build. +the build. On LUMI we define a `buildtools` module for each version of the `LUMI` software stack +which contains most popular build tools. This ensures that we use a consistent set of build tools +and that they do not change over the life time of a stack, also in case of OS upgrades, to +increase the reproducibility of the build process Here is a simple example of specifying dependencies: ```python -builddependencies = [('CMake', '3.18.4')] +builddependencies = [ + ('buildtools', '%(toolchain_version)s', '', True) +] dependencies = [ ('Python', '3.8.2'), @@ -268,20 +310,38 @@ The name and version of a dependency is specified with a 2-tuple (a tuple with t In some cases additional information may have to be provided, as is shown in the example above for the `SciPy-bundle` dependency where a 3rd value is specified corresponding to the `versionsuffix` value of this dependency. -If this is not specified, it is assumed to be the empty string (`''`). +If this is not specified, it is assumed to be the empty string (`''`). Note how we use the '`%(pyver)s'` template value in the `SciPy-bundle` dependency -specification, to avoid hardcoding the Python version in different places. +specification, to avoid hardcoding the Python version in different places. (Though this +specific parameter is less useful on LUMI as we currently try to build on top of `cray-python`.) + +The `buildtools` build dependency shows that there is a fourth parameter specifying the toolchain +used for that dependency and is needed if that toolchain is different from the one used in the example. +As it is not possible to load several Cray toolchains together (they are not in a hierachical relation) +the only useful value on LUMI is `True` which tells that `buildtools` is build with the `SYSTEM` +toolchain. Here also we use a template, `%(toolchain_version)s` which - as its name suggests - expands +to the version of the toolchain, as we version our `buildtools` modules after the version of the Cray +toolchains for which they are intended. + +See also the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#dependencies) +for additional options on specifying dependencies. That page specifies two more dependency types: -See also the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#dependencies) for additional options on specifying dependencies. +* `hiddendependencies` are currently not used on LUMI, and if we would use them in the future, it will likely + be through a way that does not require this parameter. +* `osdependencies` can be used to let EasyBuild check if certain needed OS packages are installed. + See, e.g., the [easyconfigs for the `buildtools` package](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/easybuild/easyconfigs/b/buildtools) + on LUMI. #### Version suffix -In some cases you may want to build a particular software package in different configurations, or include a label in the module name to highlight a particular aspect +In some cases you may want to build a particular software package in different configurations, +or include a label in the module name to highlight a particular aspect of the installation. -The `versionsuffix` easyconfig parameter can be used for this purpose. The name of this parameter implies that this label will be added after the +The `versionsuffix` easyconfig parameter can be used for this purpose. +The name of this parameter implies that this label will be added after the software version (and toolchain label) in the standard module naming scheme. If you are configuring the software to build with a particular non-default value, @@ -303,7 +363,8 @@ dependencies = [('Python', '2.7.18')] Even though Python 2 is officially dead and buried some scientific software still requires it, and mixing modules where some use Python 2 and other use Python 3 doesn't work well. -The `versionsuffix` label is helpful to inform the user that a particular Python version is required by the installation. +The `versionsuffix` label is helpful to inform the user that a particular Python version is +required by the installation. #### Customizing configure, build, test, and install commands @@ -330,6 +391,7 @@ prebuildopts = 'export HDF5_PREFIX="$EBROOTHDF5" && ' installopts = "PREFIX='%(installdir)s'" ``` +(again an imaginary example as on LUMI we advise to use the Cray-provided HDF5 modules.) Here we are: @@ -406,7 +468,13 @@ Finally, you will usually see the `moduleclass` easyconfig parameter to be defin moduleclass = 'lib' ``` -This is done to categorize software, and it is used to group the generated module files into smaller sets ([remember what we saw when installing software earlier](../basic_usage/#using-installed-software)). +This is done to categorize software, and it is used to group the generated module files into smaller sets +([remember what we saw when installing software earlier](../1_08_basic_usage/#using-installed-software)). + +This is currently not used on LUMI since we feel that (a) it is not easy to explain to users how they can +then only make certain classes that are useful to them visible and more importantly (b) since it is not +always intuitive to decide which moduleclass should be used for a package, or from a user 's perspective, +in which category to look for a package. ## Generating tweaked easyconfigs @@ -428,7 +496,7 @@ eb example-1.2.3.eb --try-software-version 1.2.4 Or, to try using a different compiler toolchain you can use `--try-toolchain`: ```shell -eb example-1.2.3-foss-2020b.eb --try-toolchain intel,2020b +eb example-1.2.3-foss-2020b.eb --try-toolchain cpeCray,21.12 ``` It is important to keep in mind the *"try"* aspect here: while easyconfigs that @@ -477,20 +545,18 @@ You can consult the unpacked sources at [https://github.com/easybuilders/easybui ### Preparation -Make sure EasyBuild is properly configured before you start: +Make sure that a proper version of the `LUMI` software stack is loaded +(we recommend `LUMI/21.12` for the session for which this tutorial was designed) +and also `EasyBuild-user` is loaded to configure EasyBuild to install in +`$HOME/EasyBuild` or `$EBU_USER_PREFIX`. -```shell -export EASYBUILD_PREFIX=$HOME/easybuild -export EASYBUILD_BUILDPATH=/tmp/$USER +``` shell +module load LUMI/21.12 +module load EasyBuild-user` ``` -and that the installed software in `/easybuild` is available: -```shell -module use /easybuild/modules/all -``` - -### Mandatory parameters +### Easyblock Let's start by getting the mandatory easyconfig parameters defined in the easyconfig file: @@ -499,48 +565,26 @@ name = 'eb-tutorial' version = '1.0.1' homepage = 'https://easybuilders.github.io/easybuild-tutorial' -description = "EasyBuild tutorial example" -``` -If we try using this (very) minimal easyconfig file, EasyBuild will inform us that we failed to specify one of the mandatory easyconfig parameters: `toolchain`: +whatis = [ 'Description: EasyBuild tutorial example'] -``` -$ eb example.eb -== Temporary log file in case of crash /tmp/eb-90j723rl/easybuild-q21plqvx.log -== found valid index for /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs, so using it... -ERROR: Failed to process easyconfig /home/example/example.eb: mandatory parameters not provided in pyheader: toolchain +description = """ +This is a short C++ example program that can be buid using CMake. +""" ``` -We will use `GCC/10.2.0` as toolchain, since we know it is already installed in `/easybuild` in the prepared environment, so we also define the `toolchain` easyconfig parameter: - -```python -toolchain = {'name': 'GCC', 'version': '10.2.0'} -``` - -In addition, we'll also specify the `moduleclass`. -This is not required, but it is usually set to a sensible value: - -```python -moduleclass = 'tools' -``` - -The default value is '`base`', at least '`tools`' has *some* meaning. - -### Easyblock - -Let us see what happens if we take our current easyconfig file for a spin: +Let's see what EasyBuild does with this very minimal easyconfig file: ```shell -$ eb example.eb -== temporary log file in case of crash /tmp/eb-8_vxjfn7/easybuild-k3aaoan2.log -ERROR: Failed to process easyconfig /home/example/example.eb: +$ eb eb-tutorial.eb +== Temporary log file in case of crash /run/user/XXXXXXXXX/easybuild/tmp/eb-k_82wotb/easybuild-wg0k_reb.log +ERROR: Failed to process easyconfig /pfs/lustrep4/users/XXXXXXXX/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v1.eb: No software-specific easyblock 'EB_eb_minus_tutorial' found for eb-tutorial ``` -That didn't get us very far... - -The error shows that there is no software-specific easyblock available for installing the software with the name '`eb-tutorial`'. -Does that mean we have to implement an easyblock? +It is not mandatory to specify an easyblock in the easyconfig. However, in the absense of that +specification, EasyBuild goes looking for an application-specific easyblock with the standard name, +in this case `EB_eb_minus_tutorial`, which it does not have. Does that mean we have to implement an easyblock? In this simple case it doesn't, since we can leverage one of the available *generic easyblocks*. But, which one? @@ -559,81 +603,90 @@ easyblock = 'CMakeMake' The "`easyblock =`" line is usually at the top of the easyconfig file, but strictly speaking the order of the parameter definitions doesn't matter (unless one is defined in terms of another one). -### CMake build dependency -Does using the `CMakeMake` generic easyblock help at all? +### Mandatory parameters + +When trying this improved easyconfig file, EasyBuild will inform us that we failed to specify +one of the mandatory easyconfig parameters: `toolchain`: ``` $ eb example.eb -== temporary log file in case of crash /tmp/eb-yutbor1p/easybuild-4jc9v1u9.log -== found valid index for /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs, so using it... -== processing EasyBuild easyconfig /home/example/example.eb -== building and installing eb-tutorial/1.0.1-GCC-10.2.0... -== fetching files... -== creating build dir, resetting environment... -== unpacking... -== patching... -== preparing... -== configuring... -== FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/ebtutorial/1.0.1/GCC-10.2.0): -build failed (first 300 chars): cmd " cmake -DCMAKE_INSTALL_PREFIX=/home/example/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER='gcc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno' -DCMAKE_CXX_COMPILER='g++' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=native (took 0 sec) +== Temporary log file in case of crash /run/user/XXXXXXXXX/easybuild/tmp/eb-22om7hut/easybuild-ipozjipf.log +ERROR: Failed to process easyconfig /pfs/lustrep4/users/XXXXXXXX/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v2.eb: +mandatory parameters not provided in pyheader: toolchain ``` -It did help: EasyBuild made an attempt to configure the build using the `cmake` command, but that failed almost -instantly. We need to dive into the log file to see the actual reason. By starting at the end of the log file and -scrolling up, you should be able to locate the following error message: +We will use `cpeCray/21.12` as toolchain, so we also define the `toolchain` easyconfig parameter: +```python +toolchain = {'name': 'cpeCray', 'version': '21.12'} ``` -/bin/bash: cmake: command not found + +In addition, we'll also specify the `moduleclass`. +This is not required, but it is usually set to a sensible value: + +```python +moduleclass = 'tools' ``` -Ah, that explains it, `cmake` isn't even installed on this system. Or is it? +The default value is '`base`', at least '`tools`' has *some* meaning. -```shell -$ module avail CMake +### CMake build dependency ---------------------------- /easybuild/modules/all ---------------------------- - CMake/3.18.4-GCCcore-10.2.0 -``` +The `CMakeMake` easyblock needs the `cmake` command. On LUMI we are lucky as +`cmake` is already installed in the OS. It may be a somewhat older version, but for this +program is is probably enough. Yet in general it is better to use sufficiently recent +build tools, and `cmake` is one of those tools that is typically entered as a build +dependency. After all, the less you rely on the OS, the more likely it becomes that +your easyconfig is useful for other sites also. -Since a module is available for `CMake` that is compatible with the toolchain we are using (GCC 10.2.0), -we can use it as a dependency for the installation. -It is only needed for building the software, not for running it, so it's only a *build* dependency: +In the [section on Lmod](../1_02_Lmod#module-extensions) we've already seen that on LUMI +the `cmake` command is available through the `buildtools` modules, and as discussed in +other examples on this page, LUMI has one for every `LUMI` software stack with its version +number the same as the stack and corresponding toolchains. It is a good practice to +add this module as a build dependency: ```python -builddependencies = [('CMake', '3.18.4')] +builddependencies = [ + ('buildtools', '%(toolchain_version)s', '', True) +] ``` -There is usually no need to specify toolchain for (build) dependencies, EasyBuild will automatically consider -[subtoolchains](../terminology/#toolchains) compatible with the specified toolchain to locate module for the dependencies. +In a more traditional EasyBuild setup with the common toolchains, there is usually no need to specify +the toolchain for (build) dependencies. EasyBuild will automatically consider +[subtoolchains](../1_05_terminology/#toolchains) compatible with the specified toolchain to locate +modules for the dependencies. However, the Cray PE toolchains on LUMI are currently not part of +such a hierarchy and the `SYSTEM` toolchain we used for `buildtools` is not automatically considered +which is why we need the 4-element version of the dependency specification. -You can verify this via `eb -D` (equivalent with `eb --dry-run`): +You can verify that EasyBuild now locates the dependency via `eb -D` (equivalent with `eb --dry-run`): ``` -$ eb example.eb -D +$ eb eb-tutorial.eb -D ... - * [x] /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs/g/GCC/GCC-10.2.0.eb (module: GCC/10.2.0) - * [x] /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs/c/CMake/CMake-3.18.4-GCCcore-10.2.0.eb (module: CMake/3.18.4-GCCcore-10.2.0) - * [ ] /home/example/example.eb (module: eb-tutorial/1.0.1-GCC-10.2.0) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeCray/cpeCray-21.12.eb (module: cpeCray/21.12) + * [ ] /pfs/lustrep4/users/kulust/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v4.eb (module: eb-tutorial/1.0.-cpeCray-21.12) ``` ### Sources -If you try again after adding `CMake` as a build dependency, you will see the installation fail again in the +If you try again after adding `buildtools` as a build dependency, you will see the installation fail again in the configuration step. Inspecting the log file reveals this: ``` -CMake Error: The source directory "/tmp/example/ebtutorial/1.0.1/GCC-10.2.0" does not appear to contain CMakeLists.txt. +CMake Error: The source directory "/run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12" does not appear to contain CMakeLists.txt. ``` -Wait, but there *is* a `CMakeLists.txt`, we can see it in the [unpacked sources](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1)! +Wait, but there *is* a `CMakeLists.txt`, we can see it in the +[unpacked sources](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1)! Let's inspect the build directory: ``` -$ ls /tmp/$USER/ebtutorial/1.0.1/GCC-10.2.0 +$ ls /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12 easybuild_obj -$ ls /tmp/$USER/ebtutorial/1.0.1/GCC-10.2.0/easybuild_obj +$ ls /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12/easybuild_obj $ ``` @@ -672,7 +725,8 @@ If now we try installing the easyconfig file again, EasyBuild complains that it can't find the specified source file anywhere: ``` -Couldn't find file eb-tutorial-1.0.1.tar.gz anywhere, and downloading it didn't work either... +Couldn't find file eb-tutorial-1.0.1.tar.gz anywhere, and downloading it didn't work either... +Paths attempted (in order):... ``` To let EasyBuild automatically download the source file if it is not available yet, @@ -716,11 +770,13 @@ expanded when the configure command is run (see [Exercise 7.1](#exercises)). Hopefully that brings us closer to getting the installation to work... ``` -$ eb example.eb +$ eb eb-tutorial.eb .... == sanity checking... -== FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/ebtutorial/1.0.1/GCC-10.2.0): build failed (first 300 chars): -Sanity check failed: no (non-empty) directory found at 'lib' or 'lib64' in /home/easybuild/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 (took 2 sec) +== ... (took 2 secs) +== FAILED: Installation ended unsuccessfully (build directory: /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12): +build failed (first 300 chars): Sanity check failed: no (non-empty) directory found at 'lib' or 'lib64' in +/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 (took 7 secs) ``` It got all the way to the sanity check step, that's great! @@ -729,9 +785,9 @@ The sanity check failed because no '`lib`' or `'lib64'` directory was found. Indeed: ``` -$ ls $HOME/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 +$ ls /users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 bin -$ ls $HOME/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0/bin +$ ls /users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 eb-tutorial ``` @@ -757,10 +813,11 @@ a module, we know it will work as expected. By enabling trace mode via `--trace` we can get some more information too: ```shell -$ eb example.eb --module-only --trace +$ eb eb-tutorial.eb --module-only --trace ... == sanity checking... >> file 'bin/eb-tutorial' found: OK + >> loading modules: eb-tutorial/1.0.1-cpeCray-21.12... >> running command 'eb-tutorial' ... >> result for command 'eb-tutorial': OK ... @@ -773,7 +830,6 @@ To convince yourself that the installation works as intended, try to load the `e run the `eb-tutorial` command yourself: ``` -$ module use $HOME/easybuild/modules/all $ module load eb-tutorial $ eb-tutorial Hello from the EasyBuild tutorial! @@ -820,12 +876,73 @@ You can let EasyBuild determine *and* inject these SHA256 checksums automatically via `eb --inject-checksums`: ``` -$ eb example.eb --inject-checksums +$ eb eb-tutorial.eb --inject-checksums ... == injecting sha256 checksums for sources & patches in example.eb... == * eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 ``` +Finally, we should consider changing the name of the easyconfig that we just developed +to align with the EasyBuild conventions as otherwise it would not be found when used as +a dependency of another package. In this case, the name should be +`eb-tutorial-1.0.0-cpeCray-21.12.eb`. In fact, EasyBuild stored a processed version +of our easyconfig with that name in the repository: + +``` +$ ls $EASYBUILD_REPOSITORYPATH +eb-tutorial +$ ls $EASYBUILD_REPOSITORYPATH/eb-tutorial +eb-tutorial-1.0.1-cpeCray-21.12.eb +$ cat $EASYBUILD_REPOSITORYPATH/eb-tutorial/eb-tutorial-1.0.1-cpeCray-21.12.eb +... +``` + +Note that EasyBuild has added an additional parameter to the easyconfig file, `buildstats`, +with a lot of information about how easybuild was called and properties of the node on which +it was run. This file is still a valid easyconfig file though from which we can build the +program again. + +Let's also inspect the installation directory when the whole build process has finished successfully. +This is very easy after loading the module, as EasyBuild-generated modules define a number of environment +variables for each module: + +``` +$ module load eb-tutorial +$ env | grep TUTORIAL +EBVERSIONEBMINTUTORIAL=1.0.1 +EBDEVELEBMINTUTORIAL=/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12/easybuild/eb-tutorial-1.0.1-cpeCray-21.12-easybuild-devel +EBROOTEBMINTUTORIAL=/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 +``` +The most interesting one of those variables is the `EBROOT` variable which points to the installation directory. +As variable names cannot contain minus signs, the minus in the module name is replaced with `MIN` in the name +of the variable (which is not the most consistent thing however as in the name of an easyblock it is replaced +with `minus`). + +Let's have a look in that directory: + +``` +$ ls $EBROOTEBMINTUTORIAL +bin easybuild +$ ls EBROOTEBMINTUTORIAL/easybuild +easybuild-eb-tutorial-1.0.1-20220401.184518.log +easybuild-eb-tutorial-1.0.1-20220401.184518_test_report.md +eb-tutorial-1.0.1-cpeCray-21.12-easybuild-devel +eb-tutorial-1.0.1-cpeCray-21.12.eb +reprod +$ ls EBROOTEBMINTUTORIAL/easybuild/reprod +easyblocks +eb-tutorial-1.0.1-cpeCray-21.12.eb +eb-tutorial-1.0.1-cpeCray-21.12.env +hooks +ls $EBROOTEBMINTUTORIAL/easybuild/reprod/easyblocks +cmakemake.py configuremake.py +``` + +As you can see, EasyBuild has also created the `easybuild` subdirectory (and it actually told us about that +at the end of the installation) which contains a lot of information about the build, also to make it easier +to reproduce a build process afterwards. + + --- ## Exercises @@ -854,6 +971,9 @@ a bit more personal: include the username of the account that was used to instal ``` To re-install the `example.eb` easyconfig, you will need to use `eb --rebuild` or `eb --force`. + EasyBuild will also complain about modules that are loaded already if you tested the program above. + A good way to get rid of all those modules on LUMI is to simply use `module purge`. You don't need + to reload the software stack, but you will need to load `EasyBuild-user` again. --- @@ -874,125 +994,34 @@ You can leverage the `eb-tutorial` easyconfig file we have composed in the examp ??? success "(click to show solution)" You can use the `--try-software-version` option for this: ```shell - $ eb example.eb --try-software-version 1.1.0 + $ eb eb-tutorial.eb --try-software-version 1.1.0 ... - == building and installing eb-tutorial/1.1.0-GCC-10.2.0... + == building and installing eb-tutorial/1.1.0-cpeCray-21.12... ... == COMPLETED: Installation ended successfully (took 4 sec) ``` To test: ``` - $ module use $HOME/easybuild/modules/all - $ module load eb-tutorial/1.1.0-GCC-10.2.0 + $ module load eb-tutorial/1.1.0-cpeCray-21.12 $ eb-tutorial I have a message for you: - Hello from the EasyBuild tutorial! + Hello from the EasyBuild tutorial! I was installed by XXXXXXXX. ``` (`eb-tutorial` version 1.0.1 doesn't print "`I have a message for you:`") ---- - -***Exercise E.3**** - Installing py-eb-tutorial 1.0.0* - -Try composing an easyconfig file for the `py-eb-tutorial` example software, which is a tiny Python package. -The source tarball can be downloaded from this link: [py-eb-tutorial-1.0.0.tar.gz](https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/py-eb-tutorial-1.0.0.tar.gz). - -A couple of tips: - -* There is a [generic easyblock](#easyblock) available for installing Python packages, which will come in useful here. - -* By default EasyBuild performs an `import` check when install Python packages, using a Python module name that is derived from the software name by default, which will be incorrect in this case. You can specify the correct name to use in the import check by specifying it via the `options` -easyconfig parameter in your easyconfig file: - ```python - options = {'modulename': 'example'} - ``` - (you will need to change '`example`' here, of course) - -* Leverage the software that is already pre-installed in `/easybuild` in the prepared environment. - Remember that some already installed modules may be a *bundle* of a couple of other software packages. - -Please also take this into account: - -* Unfortunately this software doesn't come with documentation. That is done to make it an example that - is representative for software that you may run into in the wild (it's *not* because - we were lazy when preparing the exercises, really!). - You can inspect the sources of this software [here](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/py-eb-tutorial-1.0.0). Definitely take a look at the `setup.py` file, it includes some clues - about the requirements to get this software installed. - -* Make sure the installation actually *works*, by checking that the `py-eb-tutorial` command runs correctly. - Maybe you will need to make sure other required software is available as well, for it to work correctly... - - -??? success "(click to show solution)" - - Here is a complete working easyconfig file for `py-eb-tutorial`: - ```python - - easyblock = 'PythonPackage' - - name = 'py-eb-tutorial' - version = '1.0.0' - versionsuffix = '-Python-%(pyver)s' - - homepage = 'https://easybuilders.github.io/easybuild-tutorial' - description = "EasyBuild tutorial Python example" - - source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] - sources = [SOURCE_TAR_GZ] - checksums = ['fcf73a9efc65527a210b993e8889d41ebf05977eef1f6a65ebac3188152cd496'] - - toolchain = {'name': 'foss', 'version': '2020b'} - - dependencies = [ - ('Python', '3.8.6'), - ('SciPy-bundle', '2020.11'), - ('eb-tutorial', '1.0.1'), - ] - - use_pip = True - - options = {'modulename': 'eb_tutorial'} - - sanity_check_paths = { - 'files': ['bin/py-eb-tutorial'], - 'dirs': ['lib/python%(pyshortver)s/site-packages'], - } - - sanity_check_commands = ["py-eb-tutorial"] - - moduleclass = 'tools' + EasyBuild has also created a new easyconfig for this configuration and stored + in the repository and the `easybuild` subdirectory from the installation + directory. As on LUMI the repository is in the search path we can actually copy + the file back to the current directory: + ``` + eb --copy-ec eb-tutorial-1.1.0-cpeCray-21.12. ``` + Some of the formatting is lost though and the checksum is still missing, so you may want + to do some cleaning up. ``` + eb eb-tutorial-1.1.0-cpeCray-21.12.eb --inject-checksum + ``` + - Some remarks: - - * We used the `PythonPackage` generic easyblock. There is also a `PythonBundle` easyblock for installing - bundles of Python packages, which is used for `SciPy-bundle` for example. But we don't need that here, - since we are only dealing with a single Python package. - - * The `versionsuffix` is not strictly needed, but it's common to tag Python packages with the Python version - for which they were installed. - * The SHA256 checksum for the source tarball was added automatically via `eb py-eb-tutorial.eb --inject-checksums`. - - * `py-eb-tutorial` only wants to be installed with `pip install`, so we had to set `use_pip = True`. - You can consult the custom easyconfig parameters supported by the `PythonPackage` easyblock via - "`eb -a -e PythonPackage`", see the `EASYBLOCK-SPECIFIC` part of the output. - Even when the default installation mechanism used by `PythonPackage` - (which consists of running `python setup.py install`) works fine, - it is recommended to instruct EasyBuild to use `pip install` instead. - - * By default EasyBuild will try to import `py_eb_tutorial`, while the actual name of the Python package - provided by `py-eb-tutorial` is just `eb_tutorial`. We fixed this by specifying the correct Python module name to - use via `options`. - - * Strictly speaking we don't need to specify a custom `sanity_check_paths`, since the default used - by Python package is already pretty decent (it will check for a non-empty `lib/python3.8/site-packages` - directory in the installation). We also want to make sure the `py-eb-tutorial` command is available in - the `bin` subdirectory however. Hardcoding to `python3.8` can be avoided using the `%(pyshortver)s` - template value. - - * A good way to check whether the `py-eb-tutorial` command works correctly is by running it as a sanity check - command. If the `eb-tutorial` command is not available the `py-eb-tutorial` command will fail, - since it basically just runs the `eb-tutorial` command. So we need to include `eb-tutorial` as a (runtime) - dependency in the `py-eb-tutorial` easyconfig file. +--- *[[next: Implementing easyblocks]](2_03_implementing_easyblocks.md)* From c14df7a2b96e6843a75d0c30893c1d9c54bc1192 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 1 Apr 2022 19:12:05 +0200 Subject: [PATCH 024/159] Adding in an additional section about external modules. --- docs/2022-CSC_and_LO/2_00_part2_using.md | 3 ++- docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md | 2 +- ...lementing_easyblocks.md => 2_04_implementing_easyblocks.md} | 2 +- docs/2022-CSC_and_LO/index.md | 3 ++- mkdocs.yml | 3 ++- 5 files changed, 8 insertions(+), 5 deletions(-) rename docs/2022-CSC_and_LO/{2_03_implementing_easyblocks.md => 2_04_implementing_easyblocks.md} (99%) diff --git a/docs/2022-CSC_and_LO/2_00_part2_using.md b/docs/2022-CSC_and_LO/2_00_part2_using.md index e033499c..c50036be 100644 --- a/docs/2022-CSC_and_LO/2_00_part2_using.md +++ b/docs/2022-CSC_and_LO/2_00_part2_using.md @@ -4,4 +4,5 @@ * [Troubleshooting](2_01_troubleshooting.md) *(hands-on)* * [Creating easyconfig files](2_02_creating_easyconfig_files.md) *(hands-on)* -* [Implementing easyblocks](2_03_implementing_easyblocks.md) *(hands-on)* +* [Using external modules from the Cray PE](2_03_external_modules.md) +* [Implementing easyblocks](2_04_implementing_easyblocks.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md index 99ac7d88..f7d6a646 100644 --- a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md @@ -1024,4 +1024,4 @@ You can leverage the `eb-tutorial` easyconfig file we have composed in the examp --- -*[[next: Implementing easyblocks]](2_03_implementing_easyblocks.md)* +*[[next: Using external modules from the Cray PE]](2_03_external_modules.md)* diff --git a/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md similarity index 99% rename from docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md rename to docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md index 5355548e..e0c07510 100644 --- a/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md +++ b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md @@ -1,6 +1,6 @@ # Implementing easyblocks -*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* +*[[back: Using external modules from the Cray PE]](2_03_external_modules.md)* --- diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md index 69ab40e8..4ae6e976 100644 --- a/docs/2022-CSC_and_LO/index.md +++ b/docs/2022-CSC_and_LO/index.md @@ -70,7 +70,8 @@ freely available, so unless you have access to a system with this environment yo - [Part II: **Using EasyBuild**](2_00_part2_using.md) *(Tue March 23rd 2021, 9am-12 CET)* - [Troubleshooting](2_01_troubleshooting.md) *(hands-on)* - [Creating easyconfig files](2_02_creating_easyconfig_files.md) *(hands-on)* - - [Implementing easyblocks](2_03_implementing_easyblocks.md) *(hands-on)* + - [Using external modules from the Cray PE](2_03_external_modules.md) + - [Implementing easyblocks](2_04_implementing_easyblocks.md) *(hands-on)* diff --git a/mkdocs.yml b/mkdocs.yml index 3200ba55..232b088f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -49,7 +49,8 @@ nav: - (overview): 2022-CSC_and_LO/2_00_part2_using.md - Troubleshooting: 2022-CSC_and_LO/2_01_troubleshooting.md - Creating easyconfig files: 2022-CSC_and_LO/2_02_creating_easyconfig_files.md - - Implementing easyblocks: 2022-CSC_and_LO/2_03_implementing_easyblocks.md + - Using external modules from the Cray PE: 2022-CSC_and_LO/2_03_external_modules.md + - Implementing easyblocks: 2022-CSC_and_LO/2_04_implementing_easyblocks.md - Advanced topics: - (overview): 2022-CSC_and_LO/3_00_part3_advanced.md - Using EasyBuild as a library: 2022-CSC_and_LO/3_01_easybuild_library.md From f89ee1f647eadf0949b35f37d2842692d2c1aaf8 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 1 Apr 2022 19:12:42 +0200 Subject: [PATCH 025/159] Adding in a section taken from a previous tutorial and integrating some of the material in an earlier section on the Cray PE. --- docs/2022-CSC_and_LO/1_03_CPE.md | 222 ++++++++++++++---- docs/2022-CSC_and_LO/2_03_external_modules.md | 81 +++++++ 2 files changed, 259 insertions(+), 44 deletions(-) create mode 100644 docs/2022-CSC_and_LO/2_03_external_modules.md diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md index 077fbc36..d3a2ff84 100644 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -15,76 +15,207 @@ tools and to configure the environment in a flexible way. ## Cray PE components -- **Cray Compiling Environment (CCE)**: +### Cray compiler environments - CCE consists of Cray compilers performing code analysis during compilation to generate highly optimized code. - Supported languages include Fortran, C and C++, and UPC (Unified Parallel C). - The Cray C/C++ compiler is basically the Clang/LLVM compiler with a back-end configured by HPE Cray that also - adds some Cray-specific options while the Cray Fortran compiler uses an HPE-Cray specific front-end - with a back-end based on LLVM. The Fortran compiler supports most of Fortran 2018 and tends to be - considerably stricter than the GNU or Intel Fortran compilers. +Cray PE supports multiple compilers, Cray and third party compilers as well: AOCC, Cray, Intel, GNU. - The CCE compilers will also support OpenMP offload to NVIDIA and AMD GPUs but that is still work in progress - at the time of the development of this tutorial. The Fortran compiler (but not the C/C++ compiler) also supports - OpenACC for offloading to GPU. +Users can access the compilers loading a programming environment module (see below). - The classic pre-LLVM Cray C/C++ compiler is not available on Cray EX systems. +Compilers in the Cray Programming Environment are almost always used through the Cray compiler wrappers. In fact, +until recently, some compilers could not be used without the wrappers as they failed to even detect their own +include files and some components. The behaviour of the wrapper will depend on the other modules that are +loaded in the environment. E.g., there is no separate wrapper for MPI as the MPI header files and libraries are +automatically activated through the regular wrappers as soon as a Cray MPI module is loaded. -- **Cray Scientific and Mathematical Libraries (CSML)**: +The commands to invoke compiler wrappers are `ftn` (Fortran), `cc` (C), `CC` (C++). They wrap automatically +to the right compilers based on the programming environment module that is loaded and the compiler module. - A set of high performance libraries providing portability for scientific applications, sparse and dense linear - algebra (BLAS, LAPACK, ScaLAPACK) and fast Fourier transforms (FFTW). +The online help can be accessed with the `-help` option. E.g.: `cc -help`, `CC -help`. +One of the most frequently used options of the compiler wrappers is `-craype-verbose`: +``` + ftn -help | grep verbose -A 1 + -craype-verbose Print the command which is forwarded + to compiler invocation +``` +More information is available with the `info` or `man` commands. +E.g.: both `info cc` or `man cc` will display the man page of the C compiler wrapper. - There is a special (more limited) version of the library with GPU-optimized routines. +The compiler wrappers call the correct compiler in the currently loaded programming environment, +with appropriate options to build and link applications with relevant libraries, as required by the modules loaded. +Besides a number of generic options, they also will pass all other options to the underlying compile, +so you can still use the regular compiler options also. - CSML also includes the HDF5 and netCDF libraries, in versions with and without MPI. +The compiler wrappers should replace direct calls to compiler drivers in Makefiles and build scripts +to ensure that the proper libraries are used. -- **Cray Message Passing Toolkit**: +!!! Note + For system libraries, only dynamic linking is supported by compiler wrappers on the Cray EX system - A collection of software libraries used to perform data transfers between nodes running in parallel applications. - It includes the Message Passing Interface (MPI) and OpenSHMEM parallel programming models. +--- - The HPE Cray MPI implementation is based on the MPICH3 sources from ANL. However, some algorithms are tweaked - specifically for HPE Cray hardware and the Cray interconnects. +#### Cray Compiling Environment (CCE) -- **Cray Environment Setup and Compiling Support**: - - Libraries supporting code compilation and setting up the development environment, including compiler drivers. +The Cray Compiling Environment is set by the module `PrgEnv-cray`, which is loaded by default at login. +The compiler itself is contained in the `cce` module. -- **Cray Performance and Analysis Tools**: +CCE consists of Cray compilers performing code analysis during compilation to generate highly optimized code. +Supported languages include Fortran, C and C++, and UPC (Unified Parallel C). +The Cray C/C++ compiler is basically the Clang/LLVM compiler with a back-end configured by HPE Cray that also +adds some Cray-specific options while the Cray Fortran compiler uses an HPE-Cray specific front-end +with a back-end based on LLVM. The Fortran compiler supports most of Fortran 2018 and tends to be +considerably stricter than the GNU or Intel Fortran compilers. - Tools to analyze the performance and behavior of programs that are run on Cray systems, and a Performance API (PAPI). +The CCE compilers will also support OpenMP offload to NVIDIA and AMD GPUs but that is still very immature and +very much work in progress at the time of the development of this tutorial. +The Fortran compiler (but not the C/C++ compiler) also supports +OpenACC for offloading to GPU and is being updated to the newest versions of this standard. -- **Cray Debugging Support Tools**: +The classic pre-LLVM Cray C/C++ compiler is not available on Cray EX systems. - Debugging tools, including `gdb4hpc` and `Valgrind4hpc`. +Compiler-specific manpages can be accessed on the system with `man crayftn`, `man craycc` or `man crayCC`. + +More details are given in the +[Cray Fortran Reference Manual](https://support.hpe.com/connect/s/search?language=en_US#q=Cray%20Fortran%20Reference%20manual&t=All&sort=relevancy&numberOfResults=25) + and the + [Cray Compiling Environment Release](https://support.hpe.com/connect/s/search?language=en_US#q=Cray%20Compiling%20Environment%20Release&t=All&sort=relevancy&numberOfResults=25) +that unfortunately are hidden deep in the new [support pages of HPE](https://support.hpe.com/). +The [Clang Compiler User’s Manual](https://clang.llvm.org/docs/UsersManual.html) +is another source of information for the Cray C and C++ Clang compilers. + +For more information about compiler pragmas and directives, see `man intro_directives` on the system. + +--- + +### Third-Party Compilers + +#### GNU + +The GNU C/C++ and Fortran compilers are probably the best supported third party +compiler in the Cray PE. + +Compiler-specific manpages can be accessed on the system with `man gfortran`, `man gcc` or `man g++`. + +More details are provided by the [GCC online documentation](https://gcc.gnu.org/onlinedocs). + + +#### AOCC + +There AOCC compilers, the AMD Optimizing C/C++ Compiler and matching fortran compilers, +AMD's compiler offering for CPU-only systems, have a matching programming environment module +and are a full citizen of the Cray PE. + +Compiler-specific documentation is available in the [AOCC User Guide](https://developer.amd.com/amd-aocc/#userguide). + +Cray provides a bundled package of support libraries to install into the PE environment to enable AOCC, +and Cray PE utilities such as debuggers and performance tools work with AOCC. + + +#### AMD ROCm compilers + +The AMD ROCm compilers (in the AMD world sometimes known as AOMP) are supported on systems +with AMD CPUs. However, at the time of writing of this tutorial they are not yet available in the +LUMI environment and the integration with the Cray environment still seems to be work in progress. + + +#### Intel (not on LUMI) + +The Cray PE also provides a programming environment module to enable +the Intel® oneAPI compiler and tools. + +The documentation is available in the [Intel® oneAPI Programming Guide](https://software.intel.com/content/www/us/en/develop/documentation/oneapi-programming-guide/top/oneapi-development-environment-setup.html) + +Cray provides a bundled package of support libraries to install into the Cray PE to enable the Intel compiler, allowing +utilities such as debuggers and performance tools to work with it. + + +#### NVIDIA HPC toolkit (not on LUMI) -- **Third-party compilers**: +The NVIDIA HPC Toolkit compilers (formerly PGI) are supported on systems with NVIDIA GPUs. - The Cray PE also supports several third-party compilers. Not all compilers are supported on all systems: - - The GNU C/C++ and Fortran compilers will be present on most systems with the Cray PE. +### Cray Scientific and Math Library - - The AMD AOCC C/C++ and Fortran compilers are supported on systems with AMD CPUs. +* Manpages: `intro_libsci`, `intro_fftw3` - - The AMD ROCm compilers (sometimes known as AOMP) are supported on systems with AMD GPU technology. +The Cray Scientific and Math Libraries (CSML, also known as LibSci) are a collection of numerical routines +optimized for best performance on Cray systems. - - The Intel C/C++ and Fortran compilers (also the oneAPI versions) are supported on systems with Intel - processors. HPE Cray does not include them in licenses for AMD CPUs. +These libraries satisfy dependencies for many commonly used applications on Cray systems for a wide variety of domains. - - The NVIDIA HPC Toolkit compilers are supported on systems with NVIDIA GPUs. +When the module for a CSML package (such as `cray-libsci` or `cray-fftw`) is loaded, +all relevant headers and libraries for these packages are added to the compile +and link lines of the `cc`, `ftn`, and `CC` compiler wrappers, so linking with them is +completely transparant (to the extent that users wonder where the libraries are). + +The CSML collection contains the following Scientific Libraries: + +* BLAS (Basic Linear Algebra Subroutines) +* BLACS (Basic Linear Algebra Communication Subprograms) +* CBLAS (Collection of wrappers providing a C interface to the Fortran BLAS library) +* IRT (Iterative Refinement Toolkit) +* LAPACK (Linear Algebra Routines) +* LAPACKE (C interfaces to LAPACK Routines) +* ScaLAPACK (Scalable LAPACK) +* `libsci_acc` (library of Cray-optimized BLAS, LAPACK, and ScaLAPACK routines) +* HDF5 +* NetCDF (Network Common Data Format) +* FFTW3 (the Fastest Fourier Transforms in the West, release 3) + + +### Cray Message Passing Toolkit + +* Manpages: `intro_mpi` + +* Website: [http://www.mpi-forum.org](http://www.mpi-forum.org) + +MPI is a widely used parallel programming model that establishes a practical, portable, efficient, +and flexible standard for passing messages between ranks in parallel processes. + +Cray MPI is derived from Argonne National Laboratory MPICH and implements the MPI-3.1 standard +as documented by the MPI Forum in MPI: A Message Passing Interface Standard, Version 3.1. + +Support for MPI varies depending on system hardware. To see which functions and environment variables the +system supports, please have a look at the corresponding man pages with `man intro_mpi` on the system. + +Note that though on LUMI at the time of the tutorial both the OpenFabric Interface (OFI) based and UCX-based +versions of the library are supported, the finalised LUMI system will only support OFI. + + +### DSMML + +* Modules: `cray-dsmml` +* +* Manpages: `intro_dsmml` +* +* Website: [https://pe-cray.github.io/cray-dsmml](https://pe-cray.github.io/cray-dsmml) + +Distributed Symmetric Memory Management Library (DSMML) is a HPE Cray proprietary memory management library. + +DSMML is a standalone memory management library for maintaining distributed shared symmetric memory +heaps for top-level PGAS languages and libraries like Coarray Fortran, UPC, and OpenSHMEM. + +DSMML allows user libraries to create multiple symmetric heaps and share information with other libraries. + +Through DSMML, interoperability can be extracted between PGAS programming models. + +Further details are available in the man page on the system with `man intro_dsmml`. + + +### Other components + +- **Cray Performance and Analysis Tools**: + + Tools to analyze the performance and behavior of programs that are run on Cray systems, and a Performance API (PAPI). + +- **Cray Debugging Support Tools**: + + Debugging tools, including `gdb4hpc` and `Valgrind4hpc`. --- ## Configuring the Cray PE through modules -Compilers in the Cray Programming Environment are almost always used through the Cray compiler wrappers. In fact, -until recently, some compilers could not be used without the wrappers as they failed to even detect their own -include files and some components. The behaviour of the wrapper will depend on the other modules that are -loaded in the environment. E.g., there is no separate wrapper for MPI as the MPI header files and libraries are -automatically activated through the regular wrappers as soon as a Cray MPI module is loaded. - Multiple releases of the HPE Cray PE can be installed simultaneously on the system and users can mix-and-match components. However, it is also possible to load only components for a specific release of the PE. Cray PE versions have version numbers of the form ``yy.mm``, e.g., ``22.02`` for the version released in @@ -95,7 +226,7 @@ Below we only discuss those modules that are important when building software wi profilers, etc., are not included in the list. -### The PrgEnv-* modules +### The programming environment modules The usual way to initialise the Cray PE is by loading one of the ``PrgEnv-*`` modules. This module will then load the appropriate compiler, compiler wrappers and other libraries. Some of the components that will be @@ -117,7 +248,10 @@ All ``PrgEnv-*`` modules belong to the same module family. Hence Lmod will autom already loaded ``PrgEnv-*`` module when you load a different one. -### Selecting the version through the cpe module +### Selecting the version through the cpe meta-module + +The Cray PE on the EX system provides the meta-module `cpe`: the purpose of the meta-module is +similar to the scope of the `cdt` and `cdt-cuda` meta-modules available on the XC systems. Loading one of the ``cpe/yy.mm`` modules (e.g., ``cpe/22.02``) has the following effects: diff --git a/docs/2022-CSC_and_LO/2_03_external_modules.md b/docs/2022-CSC_and_LO/2_03_external_modules.md new file mode 100644 index 00000000..5d6632c0 --- /dev/null +++ b/docs/2022-CSC_and_LO/2_03_external_modules.md @@ -0,0 +1,81 @@ +# Using external modules from the Cray PE + +*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* + +--- + +## What are external modules? + +EasyBuild supports the use of modules that were not installed via EasyBuild. +We refer to such modules as [external modules](https://docs.easybuild.io/en/latest/Using_external_modules.html). +These modules do not define the `EBROOT*` and `EBVERSION*` environment variables that EasyBuild would define +and uses internally in several easyblocks and some easyconfig files. + +This feature is used extensively on Cray systems to interface with the Cray PE (which comes with its own +modules and cannot be installed via EasyBuild): +[external modules can be used as dependencies](https://docs.easybuild.io/en/latest/Using_external_modules.html#using-external-modules-as-dependencies), +by including the module name in the dependencies list, +along with the `EXTERNAL_MODULE` constant marker. + +For example, to specify the module `cray-fftw` as a dependency, you should write the following in your easyconfig file: +``` python +dependencies = [('cray-fftw', EXTERNAL_MODULE)] +``` + +For such dependencies, EasyBuild will: + +* load the module before initiating the software build and install procedure + +* include a `module load` statement in the generated module file (for runtime dependencies) + +!!! Note + The default version of the external module will be loaded unless a specific version is given as dependency, + and here that version needs to be given as part of the name of the module and not as the second element in the + tuple. + + ```python + dependencies = [('cray-fftw/3.3.8.12', EXTERNAL_MODULE)] + ``` + +If the specified module is not available, EasyBuild will exit with an error message stating that the dependency +can not be resolved because the module could not be found, without searching for a matching easyconfig file +from which it could generate the module. + + +--- + +## EasyBuild Metadata for external modules + +[Metadata](https://docs.easybuild.io/en/latest/Using_external_modules.html#metadata-for-external-modules) + can be supplied to EasyBuild for external modules: using the `--external-modules-metadata` +configuration option, the location of one or more metadata files can be specified. + +The files are expected to be in INI format, with a section per module name +and key-value assignments specific to that module. + +The external modules metadata file can be also defined with the corresponding environment variable: +``` +echo $EASYBUILD_EXTERNAL_MODULES_METADATA +/apps/common/UES/jenkins/production/easybuild/cpe_external_modules_metadata-21.04.cfg +``` + +The following keys are +[supported by EasyBuild](https://docs.easybuild.io/en/latest/Using_external_modules.html#supported-metadata-values): + +* name: software name(s) provided by the module +* version: software version(s) provided by the module +* prefix: installation prefix of the software provided by the module + +For instance, the external module version loaded by the dependency `cray-fftw` can be specified as follows: +```ini +[cray-fftw] +name = FFTW +prefix = FFTW_DIR/.. +version = 3.3.8.10 +``` + +The environment variable `$EBROOTFFTW` will also be defined according to the `prefix` specified in the metadata file. + +--- + +*[[next: Implementing easyblocks]](2_04_implementing_easyblocks.md)* From 28741b4d8996733ea3af00b23ed92d852ebe277e Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 5 Apr 2022 15:54:42 +0200 Subject: [PATCH 026/159] Implementing EasyBlocks adapted for LUMI. --- .../2_04_implementing_easyblocks.md | 113 ++++++++++++++++-- 1 file changed, 101 insertions(+), 12 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md index e0c07510..a91b0dc8 100644 --- a/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md +++ b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md @@ -27,7 +27,10 @@ by most GNU software packages. *Software-specific* easyblocks implement the build and installation procedure for a particular software package. Typically this involves highly customised steps, for example specifying dedicated configuration options, creating or adjusting specific files, executing non-standard shell commands, etc. Usually a custom implementation of the -sanity check is also included. +sanity check is also included. Much of the work done in software-specific easyblocks can often also be done +in generic easyblocks using parameters such as ``confdigopts`` etc., but a software-specific easyblock can +hide some of that complexity from the user. Other software-specific easyblocks implement very specific +installation procedures that do not fit in one of the generic ones. Using a generic easyblock requires specifying the ``easyblock`` parameter in the easyconfig file. If it is not specified, EasyBuild will try and find the software-specific easyblock derived from the software name. @@ -38,7 +41,8 @@ for an easyblock (see below). ## Naming -Easyblocks need to follow a strict naming scheme, to ensure that EasyBuild can pick them up as needed. This involves two aspects: +Easyblocks need to follow a strict naming scheme, to ensure that EasyBuild can pick them up automatically as needed. +This involves two aspects: * the name of the Python class; * the name and location of the Python module file. @@ -52,7 +56,7 @@ Because of limitations in Python on characters allowed in names of Python classe only alphanumeric characters and underscores (``_``) are allowed. Any other characters are replaced following an encoding scheme: * spaces are replaced by underscores (``_``); -* dashes ``-`` are replaced by ``_minus_``; +* dashes ``-`` are replaced by ``_minus_`` (note the inconsistency with the naming of ``EBROOT`` and ``EBVERSION`` variables); * underscores are replaced by ``_underscore_``; The ``encode_class_name`` function provided in ``easybuild.tools.filetools`` returns the expected class name @@ -117,6 +121,11 @@ To let EasyBuild pick up one or more new or customized easyblocks, you can use t configuration option. As long as both the filename of the Python module and the name of the Python class are correct, EasyBuild will use these easyblocks when needed. +On LUMI, the EasyBuild configuration modules take care of setting this parameter (using the corresponding environment +variable), pointing to custom easyblocks in the LUMI software stack itself and a repo (with a fixed name) that users +can create themselves. At this moment it does not yet include possible other easyblock repositories in other repositories. + + ## Structure of an easyblock The example below shows the overal structure of an easyblock: @@ -232,7 +241,7 @@ self.cfg['some_list'].append('example') ### Custom parameters -Additional custom easyconfig parameters can be defined in an easyblock to allowing steering its behaviour. +Additional custom easyconfig parameters can be defined in an easyblock to steer its behaviour. This is done via the ``extra_options`` *static* method. Custom parameters can be specified to be mandatory. The example below shows how this can be implemented: @@ -256,7 +265,9 @@ class EB_Example(ConfigureMake): ``` The first element in the list of a defined custom parameter corresponds to the default value for that parameter -(both ``None`` in the example above). The second element provides some informative help text, and the last element +(both ``None`` in the example above). The second element provides some informative help text +(which can then be displayed with ``eb -a -e ``, eg, ``eb -a -e EB_GCC``), +and the last element indicates whether the parameter is mandatory (``MANDATORY``) or just an optional custom parameter (``CUSTOM``). ## Easyblock constructor @@ -519,6 +530,29 @@ class EB_Example(EasyBlock): super(EB_Example, self).sanity_check_step(custom_commands=[self.command]) ``` + +## Easyblocks in the Cray ecosystem + +The generic easyblocks are usually rather independent of compilers etc. and tend to work well with all toolchains. +However, software-specific easyblocks may contain code that is specific for certain toolchains and are often only +tested with the common toolchains (foss and intel and their subtoolchains). Many of those easyblocks will fail +on Cray systems (or any system that uses other toolchains) as they don't recognise the compiler and rather than +implementing some generic behaviour that may or may not work, produce an error message instead that the compiler +toolchain is not supported. + +Several packages on LUMI therefore use generic easyblocks rather than the software-specific easyblocks that may +exist for those applications. Adapting those software-specific easyblocks for LUMI poses an interesting maintenance +problem. Either one could decide to not contribute back to the community, but this implies then that all modifications +made to the corresponding easyblocks in the EasyBuild distribution should be monitored and implemented in the custom +easyblocks for Cray also. On the other hand, contributing back to the community also poses two problems. First it +would also require to implement the Cray toolchains as used on LUMI in the core of EasyBuild (which already contains +a different set of toolchains targeted more at how the Cray PE works with the regular environment modules), and that +only makes sense if these toolchains are first extended to not only cover the programming environments supported on +LUMI but also the Intel and NVIDIA programming environments. Second, the EasyBuild community has no easy way of testing +any modification made to such an easyblock on a Cray PE system. Hence every update made in the community may break +the Cray PE support again. + + ## Exercise ### Exercise I.1 @@ -570,9 +604,9 @@ Your easyblock should: cmd = ' '.join([ self.cfg['preconfigopts'], - "cmake", - "-DCMAKE_INSTALL_PREFIX='%s'" % self.installdir, - "-DEBTUTORIAL_MSG='%s'" % self.cfg['message'], + 'cmake', + '-DCMAKE_INSTALL_PREFIX=\'%s\'' % self.installdir, + '-DEBTUTORIAL_MSG="%s"' % self.cfg['message'], self.cfg['configopts'], ]) run_cmd(cmd) @@ -582,7 +616,7 @@ Your easyblock should: cmd = ' '.join([ self.cfg['prebuildopts'], - "make", + 'make', self.cfg['buildopts'], ]) run_cmd(cmd) @@ -592,7 +626,7 @@ Your easyblock should: cmd = ' '.join([ self.cfg['preinstallopts'], - "make install", + 'make install', self.cfg['installopts'], ]) run_cmd(cmd) @@ -607,6 +641,58 @@ Your easyblock should: custom_commands=custom_commands) ``` + We also need to adapt our easyconfig file for ``eb-tutorial``: + + - The ``easyblock`` line is no longer needed as we will rely on the automatic selection of the + software-specific easyblock. + - We don't need to define the message through ``configopts`` but via the easyblock-specific + configuration parameter ``message``. In fact, we were so careful when implementing the ``configure_step`` + that even variable expansion will still work so we can still include ``$USER`` in the message. + - The sanity check is also no longer needed as it is done by the software-specific easyblock. + + So the easyconfig file simplifies to: + + ```python + name = 'eb-tutorial' + version = "1.1.0" + + homepage = 'https://easybuilders.github.io/easybuild-tutorial' + + whatis = [ 'Description: EasyBuild tutorial example'] + + description = """ + This is a short C++ example program that can be buid using CMake. + """ + + toolchain = {'name': 'cpeCray', 'version': '21.12'} + + builddependencies = [ + ('buildtools', '%(toolchain_version)s', '', True) + ] + + source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] + sources = [SOURCE_TAR_GZ] + checksums = ['def18b69b11a3ec34ef2a81752603b2118cf1a57e350aee41de9ea13c2e6a7ef'] + + message = 'Hello from the EasyBuild tutorial! I was installed by $USER.' + + moduleclass = 'tools' + + ``` + + Running this example on LUMI is a little tricky as using ``--include-easyblocks`` to point EasyBuild to + our new easyblock interfers with settings already made by the EasyBuild configuration modules (``EasyBuild-user``) + and causes error messages about the toolchains. So either the easyblock needs to be copied to the user location + that can be found by looking at the output of ``eb --show-config`` or we simply need to extend the list of + easyblocks that EasyBuild searches with the easyblocks in the current directory: + + ``` shell + EASYBUILD_INCLUDE_EASYBLOCKS="$EASYBUILD_INCLUDE_EASYBLOCKS,./*.py" + ``` + + + + ### Exercise I.2 Try implementing another new custom easyblock for ``eb-tutorial``, @@ -648,7 +734,7 @@ Your easyblock should only: def configure_step(self): """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" - self.cfg.update('configopts', "-DEBTUTORIAL_MSG='%s'" % self.cfg['message']) + self.cfg.update('configopts', '-DEBTUTORIAL_MSG="%s"'% self.cfg['message']) super(EB_eb_minus_tutorial, self).configure_step() @@ -662,4 +748,7 @@ Your easyblock should only: custom_commands=custom_commands) ``` -*[[next: PArt 3: OVerview]](3_00_part3_advanced.md)* + This is a much simpler easyblock as we already use all the logic that has been written for us to build + with CMake. + +*[[next: Part 3: Overview]](3_00_part3_advanced.md)* From 5f5898e5a70d8c8370c17b5e39912306642f81d9 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 5 Apr 2022 15:54:55 +0200 Subject: [PATCH 027/159] Correction of typos. --- docs/2022-CSC_and_LO/2_03_external_modules.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_03_external_modules.md b/docs/2022-CSC_and_LO/2_03_external_modules.md index 5d6632c0..b19313ff 100644 --- a/docs/2022-CSC_and_LO/2_03_external_modules.md +++ b/docs/2022-CSC_and_LO/2_03_external_modules.md @@ -8,10 +8,10 @@ EasyBuild supports the use of modules that were not installed via EasyBuild. We refer to such modules as [external modules](https://docs.easybuild.io/en/latest/Using_external_modules.html). -These modules do not define the `EBROOT*` and `EBVERSION*` environment variables that EasyBuild would define +External modules do not define the `EBROOT*` and `EBVERSION*` environment variables that EasyBuild would define and uses internally in several easyblocks and some easyconfig files. -This feature is used extensively on Cray systems to interface with the Cray PE (which comes with its own +External modules are used extensively on Cray systems to interface with the Cray PE (which comes with its own modules and cannot be installed via EasyBuild): [external modules can be used as dependencies](https://docs.easybuild.io/en/latest/Using_external_modules.html#using-external-modules-as-dependencies), by including the module name in the dependencies list, @@ -47,7 +47,7 @@ from which it could generate the module. ## EasyBuild Metadata for external modules [Metadata](https://docs.easybuild.io/en/latest/Using_external_modules.html#metadata-for-external-modules) - can be supplied to EasyBuild for external modules: using the `--external-modules-metadata` +can be supplied to EasyBuild for external modules: using the `--external-modules-metadata` configuration option, the location of one or more metadata files can be specified. The files are expected to be in INI format, with a section per module name @@ -76,6 +76,10 @@ version = 3.3.8.10 The environment variable `$EBROOTFFTW` will also be defined according to the `prefix` specified in the metadata file. +On LUMI, users in generally don't need to be too concerned about the metadata file as the EasyBuild-user (and other hidden +EasyBuild configuration modules) take care of pointing to the right metadata file, which is specific for each version of the +Cray PE and hence each version of the LUMI software stack. + --- *[[next: Implementing easyblocks]](2_04_implementing_easyblocks.md)* From bbe836e07315d5a39cca035968136e24783a528e Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 5 Apr 2022 15:55:13 +0200 Subject: [PATCH 028/159] Using EasyBuild as a library corrected for LUMI. --- .../2022-CSC_and_LO/3_01_easybuild_library.md | 26 ++++++++++++------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_01_easybuild_library.md b/docs/2022-CSC_and_LO/3_01_easybuild_library.md index c037f49f..f0c255eb 100644 --- a/docs/2022-CSC_and_LO/3_01_easybuild_library.md +++ b/docs/2022-CSC_and_LO/3_01_easybuild_library.md @@ -254,10 +254,12 @@ def parse_easyconfig(ec_fn): # parse easyconfig file; # the 'parse_easyconfigs' function expects a list of tuples, - # where the second item indicates whether or not the easyconfig file was automatically generated or not + # where the second item indicates whether or not the easyconfig file was + # automatically generated or not ec_dicts, _ = parse_easyconfigs([(ec_path, False)]) - # only retain first parsed easyconfig, ignore any others (which are unlikely anyway) + # only retain first parsed easyconfig, ignore any others (which are unlikely + # anyway) return ec_path, ec_dicts[0]['ec'] @@ -278,15 +280,21 @@ for key in keys: print("%s: %s" % (key, ec[key])) ``` -Example usage: +Example usage (ensure that ``EasyBuild-user`` is loaded as EasyBuild should be +configured properly!): -```shell -$ ./inspect_easyconfig.py Subread-2.0.0-GCC-8.3.0.eb name version sources sanity_check_paths -name: Subread -version: 2.0.0 -sources: ['subread-2.0.0-source.tar.gz'] -sanity_check_paths: {'files': ['bin/exactSNP', 'bin/featureCounts', 'bin/subindel', 'bin/subjunc', 'bin/sublong', 'bin/subread-align', 'bin/subread-buildindex'], 'dirs': ['bin/utilities']} ``` +$ ./inspect_easyconfig.py GMP-6.2.1-cpeCray-21.12.eb name version sources sanity_check_paths +Inspecting /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/GMP/GMP-6.2.1-cpeCray-21.12.eb ... +name: GMP +version: 6.2.1 +sources: ['gmp-6.2.1.tar.bz2'] +sanity_check_paths: {'files': ['include/gmp.h', 'include/gmpxx.h', 'lib/libgmp.a', 'lib/libgmp.la', 'lib/libgmp.so', 'lib/libgmpxx.a', 'lib/libgmpxx.la', 'lib/libgmpxx.so', 'lib/pkgconfig/gmp.pc', 'lib/pkgconfig/gmpxx.pc'], 'dirs': ['share/info']} +``` + +This feature can be used, e.g., to extract information from easyconfig files to generate documentation +for the library of easyconfig files. + --- From 272f2fece65487a8f0608957d6d7b0acb8085104 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 5 Apr 2022 15:55:33 +0200 Subject: [PATCH 029/159] Section about hooks extended with references to additional examples. --- docs/2022-CSC_and_LO/3_02_hooks.md | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_02_hooks.md b/docs/2022-CSC_and_LO/3_02_hooks.md index a2663ff0..f91f7d7c 100644 --- a/docs/2022-CSC_and_LO/3_02_hooks.md +++ b/docs/2022-CSC_and_LO/3_02_hooks.md @@ -19,9 +19,32 @@ would normally execute. This is a very powerful feature, but it is important to aware of some details in order to make good use of it. -Support for hooks is also covered in detail in the EasyBuild documentation, see -[here](https://docs.easybuild.io/en/latest/Hooks.html). - +Support for hooks is also +[covered in detail in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Hooks.html). + +## Why hooks? + +Hooks can be implemented for various reasons. Some examples used by some sites in the EasyBuild +community: + +- Enforce some site policies on easyconfig files. A practical example for LUMI could be imposing + the use of the `whatis` parameter and ensuring that it contains a `Description` line to have a + more consistent format for module files. +- Ensuring that some information is always in the module file. E.g., on LUMI the `site_contacts` + parameter is added automatically when installing in the central stack if it is not already in + the easyconfig file to refer to the LUST support pages. +- Modify the behaviour of standard easyconfig files that come with EasyBuild to adapt them to the system + while allowing users to simply use the standard easyconfig files. E.g., we could consider this option + to offer the common EasyBuild FOSS toolchain and its subtoolchains to users on LUMI-C, while using hooks to + adapt, e.g., the easyconfig files for Open MPI to work on LUMI. + +Some site have really large hook files to implement policies and modify standard EasyBuild build recipes, e.g., +- [The hooks file from Jülich Supercomputing Centre](https://github.com/easybuilders/JSC/blob/2022/Custom_Hooks/eb_hooks.py) +- ComputeCanada [cc_hooks_common.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_common.py) + and [cc_hooks_gento.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_gentoo.py) +- [The hooks file from the EESSI software stack](https://github.com/EESSI/software-layer/blob/main/eb_hooks.py). + [EESSI](https://www.eessi-hpc.org/) is an effort to build a software stack distributed via + [CernVM-FS](https://cernvm.cern.ch/fs/) using EasyBuild to build all software. ## Implementing and using hooks From 3234fd9bb5932f6bd3512a4d31462191a9d554ff Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 13:57:15 +0200 Subject: [PATCH 030/159] Slurm job submission from EasyBuild a bit reworked. --- docs/2022-CSC_and_LO/3_03_slurm_jobs.md | 109 +++++++++++++++--------- 1 file changed, 69 insertions(+), 40 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md index 0ffa12a9..567f8abd 100644 --- a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md +++ b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md @@ -6,12 +6,30 @@ EasyBuild can submit jobs to different backends including Slurm to install software, to *distribute* the often time-consuming installation of a set of software applications and -the dependencies they require to a cluster. +the dependencies they require to a cluster. Each individual package is installed in a separate +job and job dependencies are used to manage the dependencies between package so that no build +is started before the dependencies are in place. This is done via the ``--job`` command line option. It is important to be aware of some details before you start using this, which we'll cover here. +!!! Warning "This section is not supported on LUMI, use at your own risk" + + EasyBuild on LUMI is currently not fully configured to support job submission via Slurm. Several + changes would be needed to the configuration of EasyBuild, including the location of the + temporary files and build directory. Those have to be made by hand. + + Due to the setup of the central software stack, this feature is currently useless to install + the central stack. For user installations, there are also limitations as the enviornment + on the compute nodes is different from the login nodes so, e.g., different locations for + temporary files are being used. These would only be refreshed if the EasyBuild configuration + modules are reloaded on the compute nodes which cannot be done currently in the way Slurm + job submission is set up in EasyBuild. + + Use material in this section with care; it has not been completely tested. + + ## Configuration The EasyBuild configuration that is active at the time that ``eb --job`` is used @@ -25,6 +43,8 @@ that are specified via an [EasyBuild configuration file](configuration.md#config This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables that are in place in the job environment are most likely *irrelevant*, since configuration settings they specify they will most likely be overruled by the corresponding command line options. +It does also imply however that the EasyBuild configuration that is in place when ``eb --job`` is used +does also work on the compute nodes to which the job is submitted. ## Using ``eb --job`` @@ -39,6 +59,9 @@ to ``Slurm``, for example by setting the corresponding environment variable: export EASYBUILD_JOB_BACKEND='Slurm' ``` +On LUMI this is taken care of in the EasyBuild configuration modules such as ``EasyBuild-user``. + + ### Job resources To submit an installation as a job, simply use ``eb --job``: @@ -73,13 +96,13 @@ For example, to specify a particular account that should be used for the jobs su (equivalent with using the ``-A`` or ``--account`` command line option for ``sbatch``): ```shell -export SBATCH_ACCOUNT='example_project' +export SBATCH_ACCOUNT='project_XXXXXXXXX' ``` Or to submit to a particular Slurm partition (equivalent with the ``-p`` or ``--partition`` option for ``sbatch``): ```shell -export SBATCH_PARTITION='example_partition' +export SBATCH_PARTITION='small' ``` For more information about supported ``$SBATCH_*`` environment variables, @@ -113,24 +136,29 @@ as jobs, to avoid that they fail almost instantly due to a lack of disk space. Keep in mind that the active EasyBuild configuration is passed down into the submitted jobs, so any configuration that is present on the workernodes may not have any effect. -For example, if you commonly use `/tmp/$USER` for build directories on a login node, -you may need to tweak that when submitting jobs to use a different location: +For example, on LUMI it is possible to use ``$XDG_RUNTIME_DIR`` on the login nodes which has +the advantage that any leftovers of failed builds will be cleaned up when the user ends their last +login session on that node, but it is not possible to do so on the compute nodes. ```shell # EasByuild is configured to use /tmp/$USER on the login node -login01 $ eb --show-config | grep buildpath -buildpath (E) = /tmp/example +uan01 $ eb --show-config | grep buildpath +buildpath (E) = /run/user/XXXXXXXX/easybuild/build -# use /localdisk/$USER for build directories when submitting installations as jobs -login01 $ eb --job --buildpath /localdisk/$USER example.eb --robot +# use /dev/shm/$USER for build directories when submitting installations as jobs +login01 $ eb --job --buildpath /dev/shm/$USER/easybuild example.eb --robot ``` + ### Temporary log files and build directories -The temporary log file that EasyBuild creates is most likely going to end up on the local disk -of the workernode on which the job was started (by default in `$TMPDIR` or `/tmp`). -If an installation fails, the job will finish and temporary files will likely be cleaned up instantly, -which may leave you wondering about the actual cause of the failing installation... +The problems for the temporary log files are twofold. First, they may end up in a place +that is not available on the compute nodes. E.g., for the same reasons as for the build +path, the LUMI EasyBuild configuration will place the temporary files in a subdirectory of +``$XDG_RUNTIME_DIR`` on the loginnodes but a subdirectory of ``/dev/shm/$USER`` on the +compute nodes. The second problem however is that if an installation fails, those log files are +not even accessible anymore which may leave you wondering about the actual cause of the failing +installation... To remedy this, there are a couple of EasyBuild configuration options you can use: @@ -139,6 +167,7 @@ To remedy this, there are a couple of EasyBuild configuration options you can us ```shell $ eb --job example.eb --tmp-logdir $HOME/eb_tmplogs ``` + This will move at least the log file to a suitable place. * If you prefer having the entire log file stored in the Slurm job output files, you can use ``--logtostdout`` when submitting the jobs. This will result in extensive logging @@ -146,11 +175,13 @@ To remedy this, there are a couple of EasyBuild configuration options you can us log to ``stdout`` when the installation is running in the job, and hence the log messages will be captured in the job output files. -The same remark applies to build directories: they should be on a local filesystem (to avoid problems -that often occur when building software on a parallel filesystem like GPFS or Lustre), -which will probably be cleaned up automatically when a job fails. Here it is less easy to provide -general advice on how to deal with this, but one thing you can consider is retrying the installation -in an interactive job, so you can inspect the build directory after the installation fails. +The build directory of course also suffers from the problem of being no longer accessible if the +installation fails, but there it is not so easy to find a solution. Building on a shared file system +is not only much slower, but in particular on parallel file systems like GPFS/SpectrumScale, Lustre +or BeeGFS buiding sometimes fails in strange ways. One thing you can consider if you cannot do the +build on a login node (e.g., because the code is not suitable for cross-compiling or the configure +system does tests that would fail on the login node), is to rety the installation in an +interactive job, so you can inspect the build directory after the installation fails. ### Lock files @@ -171,37 +202,37 @@ subdirectory of ``installpath``) manually, or re-submit the job with ``eb --job As an example, we will let EasyBuild submit jobs to install ``AUGUSTUS`` with the ``foss/2020b`` toolchain. +!!! Warning "This example does not work on LUMI" + + Note that this is an example using the FOSS common toolchain. For this reason it does not work on + LUMI. + ### Configuration Before using ``--job``, let's make sure that EasyBuild is properly configured: ```shell -# use $HOME/easybuild for software, modules, sources, etc. -export EASYBUILD_PREFIX=$HOME/easybuild +# Load the EasyBuild-user module (central installations will not work at all +# using job submission) +module load LUMI/21.12 +module load partition/C +module load EasyBuild-user # use ramdisk for build directories -export EASYBUILD_BUILDPATH=/dev/shm/$USER +export EASYBUILD_BUILDPATH=/dev/shm/$USER/build +export EASYBUILD_TMPDIR=/dev/shm/$USER/tmp # use Slurm as job backend export EASYBUILD_JOB_BACKEND=Slurm ``` -In addition, add the path to the centrally installed software to ``$MODULEPATH`` via ``module use``: -```shell -module use /easybuild/modules/all -``` - -Load the EasyBuild module: - -```shell -module load EasyBuild -``` - -Let's assume that we also need to inform Slurm that jobs should be submitted into a particular account: +We will also need to inform Slurm that jobs should be submitted into a particular account, and +in a particular partition: ```shell -export SBATCH_ACCOUNT=example_project +export SBATCH_ACCOUNT=project_XXXXXXXXX +export SBATCH_PARTITION='small' ``` This will be picked up by the ``sbatch`` commands that EasyBuild will run to submit the software installation jobs. @@ -234,14 +265,14 @@ $ eb AUGUSTUS-3.4.0-foss-2020b.eb --missing Several dependencies are not installed yet, so we will need to use ``--robot`` to ensure that EasyBuild also submits jobs to install these first. -To speed up the installations a bit, we will request 10 cores for each submitted job (via ``--job-cores``). +To speed up the installations a bit, we will request 8 cores for each submitted job (via ``--job-cores``). That should be sufficient to let each installation finish in (well) under 1 hour, so we only request 1 hour of walltime per job (via ``--job-max-walltime``). In order to have some meaningful job output files, we also enable trace mode (via ``--trace``). ``` -$ eb AUGUSTUS-3.4.0-foss-2020b.eb --job --job-cores 10 --job-max-walltime 1 --robot --trace +$ eb AUGUSTUS-3.4.0-foss-2020b.eb --job --job-cores 8 --job-max-walltime 1 --robot --trace ... == resolving dependencies ... ... @@ -278,7 +309,7 @@ these jobs will be able to start. After about 20 minutes, AUGUSTUS and all missing dependencies should be installed: ``` -$ ls -lrt $HOME/easybuild/modules/all/*/*.lua | tail -11 +$ ls -lrt $HOME/EasyBuild/modules/.../*.lua | tail -11 -rw-rw----. 1 example example 1634 Mar 29 10:13 /users/example/easybuild/modules/all/HTSlib/1.11-GCC-10.2.0.lua -rw-rw----. 1 example example 1792 Mar 29 10:13 /users/example/easybuild/modules/all/SAMtools/1.11-GCC-10.2.0.lua -rw-rw----. 1 example example 1147 Mar 29 10:13 /users/example/easybuild/modules/all/BamTools/2.5.1-GCC-10.2.0.lua @@ -291,11 +322,9 @@ $ ls -lrt $HOME/easybuild/modules/all/*/*.lua | tail -11 -rw-rw----. 1 example example 1365 Mar 29 10:28 /users/example/easybuild/modules/all/SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0.lua -rw-rw----. 1 example example 2233 Mar 29 10:30 /users/example/easybuild/modules/all/AUGUSTUS/3.4.0-foss-2020b.lua -$ module use $HOME/easybuild/modules/all - $ module avail AUGUSTUS --------- /users/hkenneth/easybuild/modules/all -------- +-- EasyBuild managed user software for software stack ... -- AUGUSTUS/3.4.0-foss-2020b ``` From ad2a7f512c73b2ccd0efb026c58fb4765de33cc1 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 15:42:35 +0200 Subject: [PATCH 031/159] Some explanation in the overview of part 3 --- docs/2022-CSC_and_LO/1_02_Lmod.md | 82 +++++++- docs/2022-CSC_and_LO/3_00_part3_advanced.md | 6 + .../3_04_module_naming_schemes.md | 192 +++++++----------- 3 files changed, 152 insertions(+), 128 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md index 0caa8a47..d6d73a6c 100644 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -73,12 +73,54 @@ to a hierarchy with 3 levels: 3. The ``MPI`` level, containing libraries and applications that depend on the compiler used and the MPI implementation. +??? Example "A simple Lmod hierarchy with a single compiler" + + Here is a simple example of such a 3-level module hierarchy + (that almost could have been generated by EasyBuild): + +
+ + In this example the ``Core`` level only includes a single module `GCC/9.3.0`, + while the ``Compiler`` level includes two modules: `OpenMPI/4.0.3` and `MPICH/3.3.2`. + In the ``MPI `` level, three modules are available: one for `FFTW`, one for `ScaLAPACK`, + and one for `HDF5`. + + Initially only the modules on the top level of a module hierarchy are available for loading. + If you run "`module avail`", the command that is used to view all modules that are available + for loading, with this example module hierarchy, you will only see the `GCC/9.3.0` module. + + Some modules in the top level of the hierarchy act as a "gateway" to modules in the + next level below. + To make additional modules available for loading one of these gateway modules has to be loaded. + In our example, loading the `GCC/9.3.0` module results in two additional modules coming into + view from the ``Compiler`` level, as indicated by the arrows: the modules for `OpenMPI` and `MPICH`. + These correspond to installations of `OpenMPI` + and `MPICH` that were built using `GCC/9.3.0`. + + Similarly, the `OpenMPI/4.0.3` module serves as a gateway to the three modules in the ``MPI`` level. + Only by loading the `OpenMPI` module will these additional three modules become + available for loading. They correspond to software installations built using the ``GCC/9.3.0`` + compiler with ``OpenMPI/4.0.3``. + Now assume that we have two compilers in the hierarchy, Compiler_A and Compiler_B. Their modules would reside -at the ``Core`` level. Both copilers provide the same MPI implementation, MPI_C. So there would be two modules +at the ``Core`` level. Both compilers provide the same MPI implementation, MPI_C. So there would be two modules for ``MPI_C`` in two different subdirectories at the ``Compiler`` level. And further assume that we have an application, Appl_E, compiled with both Compiler_A and Compiler_B and using MPI_C. For that application there would also be two module files at the ``MPI`` level, one in a subdirectory corresponding ao Compiler_A and MPI_C and one -in a subdirectory corresponding to Compiler_B and MPI_C. To be able to load the module for Appl_E, a user should +in a subdirectory corresponding to Compiler_B and MPI_C. + +```mermaid +graph TD; +A[Compiler_A] --> AC[MPI_C]; +A --> AD[MPI_D] +B[Compiler_B] --> BC[MPI_C]; +AC --> ACE[Appl_E]; +AD --> ADE[Appl_E] +BC --> BCE[Appl_E]; +``` + + +To be able to load the module for Appl_E, a user should first load Compiler_A, then load MPI_C and only then is it possible to load the module for Appl_E: ```bash @@ -102,6 +144,23 @@ and depending on those adapt the path to the binaries, several very simple modul logic, and one could add an Appl_E module for a different compiler or MPI implementation without touching any of the already existing module files for that application. +Similarly, if after + +```bash +module load Compiler_A MPI_C Appl_E +``` + +one does + +```bash +module load MPI_D +``` + +thent MPI_C gets unloaded, Lmod notices that it also has to unload/deactivate Appl_E, then will load MPI_D for Compiler_A and +finally will notice that there is an equivalent Appl_E module available again, and Lmod will load that one also. However, +now loading Compiler_B will cause a warning that MPI_D and Appl_E have been deactivated as there is no module name MPI_D in +any version for Compiler_B. + ### Building blocks @@ -126,8 +185,9 @@ Some mechanisms in Lmod make implementing a hierarchy fairly easy (though there ### Implementation details -The above example could be implemented using 6 module files: One for each compiler, two for the MPI module and -two for the application module. +The above example could be implemented using 8 module files: One for each compiler, three for the MPI modules +(two for MPI_C and one for MPI_D) and +three for the application modules. ``` moduleroot @@ -139,8 +199,10 @@ moduleroot ├── Compiler │ ├── Compiler_A │ │ └── version_A -│ │ └── MPI_C -│ │ └── version_C.lua +│ │ ├── MPI_C +│ │ │ └── version_C.lua +│ │ └── MPI_D +│ │ └── version_D.lua │ └── Compiler_B │ └── version_B │ └── MPI_C @@ -148,8 +210,12 @@ moduleroot └── MPI ├── Compiler_A │ └── version_A - │ └── MPI_C - │ └── version_C + │ ├── MPI_C + │ │ └── version_C + │ │ └── Appl_E + │ │ └── version_E.lua + │ └── MPI_D + │ └── version_D │ └── Appl_E │ └── version_E.lua └── Compiler_B diff --git a/docs/2022-CSC_and_LO/3_00_part3_advanced.md b/docs/2022-CSC_and_LO/3_00_part3_advanced.md index 708c57f8..28a24990 100644 --- a/docs/2022-CSC_and_LO/3_00_part3_advanced.md +++ b/docs/2022-CSC_and_LO/3_00_part3_advanced.md @@ -2,6 +2,12 @@ *[[back to start page]](index.md)* +In this section we mostly cover "good to know that they exist" features as they are not used +on LUMI, or not really accessible to regular user installations that are performed with the +LUMI ``EasyBuild-user`` module. Hooks are used on LUMI, but it is not really advised to +overwrite the centrally defined hooks with a local file. And the whole structure of the +EasyBuild integration is also set up to make use of the GitHub integration in the future. + * [Using EasyBuild as a library](3_01_easybuild_library.md) *(hands-on)* * [Using hooks to customise EasyBuild](3_02_hooks.md) *(hands-on)* * [Submitting installations as Slurm jobs](3_03_slurm_jobs.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md index 14229d97..d0b4ebe8 100644 --- a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md +++ b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md @@ -4,21 +4,24 @@ --- -Up until now we have used EasyBuild's default module naming scheme (`EasyBuildMNS`), -which produces module files with names that closely resemble to the names of the +Up until now we have used the naming scheme in use on LUMI which is for many practical purposes +is indistinguishable from the default EasyBuild naming scheme (`EasyBuildMNS`). +It produces module files with names that closely resemble to the names of the corresponding easyconfig files. -For example, when installing `h5py-3.1.0-foss-2020b.eb` the generated module was named `h5py/3.1.0-foss-2020b`. +For example, when installing `zlib-1.2.11-cpeGNU-21.12.eb` the generated module was named +`zlib/1.2.11-cpeGNU-21.12`. EasyBuild supports several different module naming schemes: ```shell $ eb --avail-module-naming-schemes List of supported module naming schemes: - CategorizedHMNS - CategorizedModuleNamingScheme - EasyBuildMNS - HierarchicalMNS - MigrateFromEBToHMNS + EasyBuildMNS + LUMI_FlatMNS + MigrateFromEBToHMNS + HierarchicalMNS + CategorizedHMNS + CategorizedModuleNamingScheme ``` In this part of the tutorial we will take a closer look at `HierarchicalMNS`, @@ -28,52 +31,54 @@ We will also take a quick look at implementing our own custom module naming sche ## Flat vs hierarchical -The default module naming scheme `EasyBuildMNS` is an example of regular *"flat"* module naming scheme, which is characterized by: +!!! Note -* all module files are directly available for loading; -* each module name uniquely identifies a particular installation; + This text is very much based on the generic EasyBuild tutorials as we cannot demonstrate + all aspects on LUMI as it is configured today. -In contrast, a *hierarchical* module naming scheme -consists of a *hierarchy* of module files. - -The typical module hierarchy has 3 levels: + On LUMI, the complete module scheme is partly hierarchical, but EasyBuild uses a flat naming + scheme. The two levels in the hierarchy that are present to deal with different versions of + the software stack and the various types of compute nodes, are not managed by EasyBuild. -* a **core** level, where module files for software that was installed using the - [`system` toolchain](../introduction/#system-toolchain) are kept; -* a **compiler** level, where module files for software that was installed using a -*compiler-only toolchain* are stored; -* and an **MPI** level, which houses module files for software that was installed using a toolchain that includes (at least) a compiler and MPI component; +The default module naming scheme `EasyBuildMNS` +and the custom `LUMI_FlatMNS` naming scheme are both examples of regular *"flat"* +module naming schemes, which are characterized by: -Here is a simple example of such a 3-level module hierarchy: +- all module files are directly available for loading; +- each module name uniquely identifies a particular installation; -
+In contrast, a *hierarchical* module naming scheme +consists of a *hierarchy* of module files. +A fairly typical 3-level schme (``Core``, ``Compiler`` and ``MPI``) has been +discussed in the [section on Lmod](../1_02_Lmod#lmod-hierarchy). +This typical Lmod hierarcny would map very well on the EasyBuild common toolchains. -In this example the core level only includes a single module `GCC/9.3.0`, -while the compiler level includes two modules: `OpenMPI/4.0.3` and `MPICH/3.3.2`. -In the MPI level, three modules are available: one for `FFTW`, one for `ScaLAPACK`, and one for `HDF5`. +In fact, for the example -Initially only the modules on the top level of a module hierarchy are available for loading. -If you run "`module avail`" with the example module hierarchy, you will only see the `GCC/9.3.0` module. +
-Some modules in the top level of the hierarchy act as a "gateway" to modules in the -next level below. -To make additional modules available for loading one of these gateway modules has to be loaded. In our exampe, loading the `GCC/9.3.0` module results in two additional modules coming into view from the compiler level, as indicated by the arrows: the modules for `OpenMPI` and `MPICH`. These correspond to installations of `OpenMPI` -and `MPICH` that were built using `GCC/9.3.0` as a (compiler-only) toolchain. +software at the ``Core`` level would be installed with the ``GCCcore`` and ``SYSTEM`` +toolchains. Software at the ``Compiler`` level would be installed with the ``GCC`` +toolchain, while software at the ``MPI`` level after loading the ``OpenMPI`` module +would be installed using the regular ``foss`` toolchain or the ``gompi`` toolchain +(see the diagram in the +["Common toolchains" section of the page on terminology](../1_05_terminology/#common-toolchains)). -Similarly, the `OpenMPI/4.0.3` module serves as a gateway to the three modules in the MPI level. Only by loading the `OpenMPI` module will these additional three modules become -available for loading. They correspond to software installations built using the `gompi/2020a` toolchain that -consists of the `GCC/9.3.0` compiler module and the `OpenMPI/4.0.3` MPI module. Software installed using -`foss/2020a` (which is a full toolchain that also includes OpenBLAS, FFTW and ScaLAPACK) would also be stored -in this level of the module hierarchy. +On LUMI, where software is installed through the Cray Programming Environment with no real choice of +MPI implementation, a two-level arrangement would still make a lot of sense, with at the ``Core`` level +all software compiled with the SYSTEM toolchain while there could be a ``PrgEnv`` level for software +compiled with a particula programming enviornment aka cepGNU/cpeCray/cpeAOCC toolchain. Such a scheme +is used on the Cray systems at CSCS. -The characteristics of a module hierarchy are: +To recap, the characteristics of a module hierarchy are: * not all module files are directly available for loading; * some modules serve as a gateway to more modules; * to access some software installations you will first need to load one or more gateway modules in order to use them; -You can probably think of other ways to organize module files in a hierarchical module tree, but here we will stick to the standard core / compiler / MPI hierarchy. +You can probably think of other ways to organize module files in a hierarchical module tree, but here +we will stick to the standard core / compiler / MPI hierarchy. ### Pros & cons @@ -81,22 +86,24 @@ So why go through all this trouble of organizing modules hierarchically? There are a couple of advantages to this approach: -* shorter module names; +* shorter module names (or at least for the version part of the name); * less overwhelming list of available modules; * only compatible modules can be loaded together; However, the are some minor disadvantages too: -* not all existing modules are directly visible; +* not all existing modules are directly visible, so users have to learn how to find + modules using ``module spider`` etc.; * gateway modules may have little meaning to end users; #### Length of module names -When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For our `HDF5` installation for example, -we have `HDF5/1.10.6-gompi-2020a` as module name. The `-gompi-2020a` part of the name refers to the toolchain that was +When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For a `HDF5` installation +with the EasyBuild common toolchains for example, +one might have `HDF5/1.10.6-gompi-2020a` as module name. The `-gompi-2020a` part of the name refers to the toolchain that was used for this installation, but it may be confusing to some people (what kind of Pokémon is a "gompi"?!). -In the example module hierarchy shown above, the module for `HDF5` is named `HDF5/1.10.6` which is basically the bare +In the example module hierarchy shown above, the module for `HDF5` could simply be named `HDF5/1.10.6` which is basically the bare essentials: software name and version. That's way better, nice and clean! #### Amount of available modules @@ -130,8 +137,10 @@ and so these modules can not be loaded together easily. One downside of a module hierarchy is that not all existing modules are directly available for loading or are even *visible* to the user, since the output of "`module avail`" only shows a subset of all modules. -[Lmod](https://lmod.readthedocs.io) has a solution for this though: it provides a separate "`module spider`" -command to search for module files throughout the entire module hierarchy. So as long as the end users are +As we have discussed in [the Lmod section](../LMOD). +[Lmod](https://lmod.readthedocs.io) provides a powerful solution to search for modules through the +``module spider`` and to some extent also the ``module keyword`` commands. +So as long as the end users are aware of this additional command, it should not be difficult to discover which software installations exist and how they can be accessed. The "`module spider`" command will inform the user which of the gateway modules need to be loaded in order to load a specific module file. @@ -140,7 +149,7 @@ need to be loaded in order to load a specific module file. An additional potential problem of a module hierarchy is that the semantics of the gateway modules may not be clear to end users. They may wonder why they need to pick a specific compiler and MPI library, or which of the -available options is the best one. Maybe there are not even be aware what exactly a "compiler" is, or how it is +available options is the best one. Maybe they are not even aware what exactly a "compiler" is, or how it is relevant to the software they need in their bioinformatics pipeline... This can be partially resolved by loading a default compiler and MPI module so a particular set of modules @@ -148,10 +157,15 @@ is available right after login, which could be the ones used in the most recent recommended versions. More experienced users could then leverage the "`module spider`" command to navigate the module hierarchy. +On LUMI this would mean loading a default software stack, but due to the the default modules are currently +loaded on LUMI this was not possible to accomplish without losing other functionality of the module tree. + ## Using a custom module naming scheme Next to the module naming schemes that are included with EasyBuild, -you can also define your own module naming scheme (MNS), and configure EasyBuild to use it. +you can also define your own module naming scheme (MNS), and configure EasyBuild to use it +(which is eactly what has been done on LUMI to remove a feature of the default ``EasyBuildMNS`` scheme +that we do not use). ### Implementation @@ -254,7 +268,9 @@ class ExampleMNS(ModuleNamingScheme): return short_modname.startswith(name.lower().replace('-', '_') + '/') ``` -We can see what the module names with this module naming scheme would like like via ``eb -D``: +We can see what the module names with this module naming scheme would like like via ``eb -D``. +E.g., for the common toolchains (the example will not work on LUMI as the default easyconfig +files are not in the robot- and the search path): ``` $ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D @@ -270,9 +286,10 @@ $ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D ## Example module hierarchy: HDF5 -!!! Warning - **This exercise will only work if you have write access to the software installation prefix**, - because EasyBuild will try to copy the installation log file to each installlation directory. +!!! Warning "Example not suitable for LUMI" + **This exercise is meant for a system where the common toolchains can be used and requires an + indpendent EasyBuild installation in your personal file space**, + because EasyBuild will try to copy the installation log file to each installation directory. Now that we know more about hierarchical module naming schemes, let us see how EasyBuild can help us with generating a hierarchical module tree. @@ -308,7 +325,7 @@ a module. We have just made all modules unavailable, so we would have to first install EasyBuild again in our hierarchical module tree before we can continue. **We strongly recommend using an EasyBuild installation that was [installed via "`pip install`" -or "`pip3 install`"](../installation/#method-1-using-pip-recommended) in this part of the tutorial.** +or "`pip3 install`"](../1_06_installation#method-1-using-pip) in this part of the tutorial.** An easy way to do this is in the prepared environment is to run: @@ -376,7 +393,9 @@ The steps we will have to go through are: * Tell EasyBuild we want to "install" the `HDF5-1.10.7-gompi-2020b.eb` easyconfig file; * Enable dependency resolution via `--robot`; -* Instruct EasyBuild to only generate the module files, not to install the software (since it is +* Assuming the software would have been installed already with the default naming scheme + in a different module directory, instruct EasyBuild to only generate the module files, + not to install the software (since it is there already in `/easybuild/software`), via the `--module-only` option. These steps translate to this single `eb` command: @@ -544,73 +563,6 @@ Currently Loaded Modules: 4) GCC/10.2.0 8) libpciaccess/0.16 12) libfabric/1.11.0 16) HDF5/1.10.7 ``` -## Exercise - -Now it is your turn! - -Try to get a feeling for how a hierarchical module tree works by: - -* installing the missing modules for the `SciPy-bundle-2020.11-foss-2020b.eb` in the module hierarchy we - generated for HDF5; -* figure out where the `SciPy-bundle` module is located in the hierarchy, and then also load it; - -You can verify your work by running this command (since `pandas` is one of the Python packages included -in the `SciPy-bundle` installation): - -```shell -python -c 'import pandas; print(pandas.__version__)' -``` - -Start from a clean slate, by first running: - -```shell -module purge -module unuse $MODULEPATH -``` - -??? success "(click to show solution)" - - * Step 0: check which modules are still missing, using `--missing` or `-M`: - ```shell - eb SciPy-bundle-2020.11-foss-2020b.eb -M - ``` - The output should tell you that 15 out of 50 required modules are still missing. - - * Install the missing modules in the module hierarchy we have generated in `$HOME/hmns/modules`: - ```shell - eb SciPy-bundle-2020.11-foss-2020b.eb --robot --module-only - ``` - Don't forget to use both `--robot` (to enable dependency resolution) and `--module-only` - (to only run the sanity check and generate module files, not install the software again). - - * Start at the top of the module hierarchy (the `Core` level), - and run module spider to check which gateway modules to load to make `SciPy-bundle` available: - ```shell - module use $HOME/hmns/modules/all/Core - module spider SciPy-bundle/2020.11 - ``` - * Load the gateway modules: - ```shell - module load GCC/10.2.0 OpenMPI/4.0.5 - ``` - * Check that the `SciPy-bundle` module is available, and load it: - ```shell - $ module avail SciPy-bundle - ----- /home/example/hmns/modules/all/MPI/GCC/10.2.0/OpenMPI/4.0.5 ------ - SciPy-bundle/2020.11 - ``` - ``` - module load SciPy-bundle/2020.11 - ``` - * Run the test command: - ``` - $ python -c 'import pandas; print(pandas.__version__)' - 1.1.4 - ``` - -!!! Warning - **This exercise will only work if you have write access to the software installation prefix**, - because EasyBuild will try to copy the installation log file to each installlation directory. --- From 4f1dcabdbf23b32191b5c361776fbc37c05d0ade Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 16:22:58 +0200 Subject: [PATCH 032/159] GitHub integration section, mostly old text and not all suited for LUMI. --- .../3_05_github_integration.md | 28 +++++++++++++++---- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_05_github_integration.md index 59dcbb67..6d04ce7f 100644 --- a/docs/2022-CSC_and_LO/3_05_github_integration.md +++ b/docs/2022-CSC_and_LO/3_05_github_integration.md @@ -4,6 +4,20 @@ --- +!!! Warning "Preliminary not for LUMI and many other Cray systems" + + Not everything in this section is as useful for EasyBuild installations that build + on the Cray Programming Environment. E.g., the default easyconfig repository does + not contain easyconfigs for the Cray PE. + + Moreover, due to the restricted operating system on the compute nodes of big + Cray systems such as the EX series (COS instead of the full SUSE linux), + the GitHub integration does not work on the compute nodes of a typical setup as + one of the required Python packages fails to work as it needs the DBus daemon. + + Since GitHub integration is not yet really used on LUMI, this section is not fully + tested but left in the tutorial for completeness. + To contribute changes to the EasyBuild code (framework or easyblocks) or easyconfigs, you will need to be a bit familiar with Git and GitHub. Or maybe not? @@ -94,11 +108,6 @@ and configuration. **Additional dependencies** Both the `GitPython` and `keyring` Python packages as well as the `keyrings.cryptfile` add-on package must be installed. -In the prepared environment, you can do this via: - -```shell -pip3 install --user GitPython keyring keyrings.cryptfile -``` !!! Note You may experiences problems installing the ``cryptography`` Python packages, @@ -294,6 +303,15 @@ will include information about your system (OS, processor, etc.) and your EasyBu ## Demo +!!! Warning "Not currently suited for LUMI" + + This is text from a previous (non-Cray) version of the tutorial. The demo does require + a number of files not installed on LUMI after going through this tutorial, + and someone who can master the repository used to clean up again. + However, having a look at the output (which was generated in the spring of 2021 on + the CSC system puhti) still offers some information, so the demo was left in this + version of the tutorial. + That is a lot to digest, so let us make this a bit more concrete with an example: we will open a pull request for the [`eb-tutorial` example software](../adding_support_software/#example) to *a fork* of the [`easybuild-easyconfigs` repository](https://github.com/easybuilders/easybuild-easyconfigs) using the `eb` command, and submit a test report in it. From 83d63d8504615b32e6ccd0e2d78a84daf90b27cc Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 16:34:33 +0200 Subject: [PATCH 033/159] Adapt the structure and include an additional reading section. --- .../3_05_github_integration.md | 2 +- .../4_00_additional_reading.md | 13 ++++++ docs/2022-CSC_and_LO/index.md | 43 ++++--------------- mkdocs.yml | 1 + 4 files changed, 23 insertions(+), 36 deletions(-) create mode 100644 docs/2022-CSC_and_LO/4_00_additional_reading.md diff --git a/docs/2022-CSC_and_LO/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_05_github_integration.md index 6d04ce7f..6e19cb9a 100644 --- a/docs/2022-CSC_and_LO/3_05_github_integration.md +++ b/docs/2022-CSC_and_LO/3_05_github_integration.md @@ -457,4 +457,4 @@ The gist linked from this comment provides more detailed information: --- -*[[next: Overview]](index.md)* +*[[next: Additional reading]](4_00_additional_reading.md)* diff --git a/docs/2022-CSC_and_LO/4_00_additional_reading.md b/docs/2022-CSC_and_LO/4_00_additional_reading.md new file mode 100644 index 00000000..8f9d54d9 --- /dev/null +++ b/docs/2022-CSC_and_LO/4_00_additional_reading.md @@ -0,0 +1,13 @@ +# Additional reading + +*[[back: GitHub integration]](3_05_github_integration.md)* + +--- + + + + + +--- + +*[[next: Overview]](index.md)* diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md index 4ae6e976..7161a112 100644 --- a/docs/2022-CSC_and_LO/index.md +++ b/docs/2022-CSC_and_LO/index.md @@ -67,42 +67,15 @@ freely available, so unless you have access to a system with this environment yo - [Installation](1_06_installation.md) - [Configuration](1_07_configuration.md) - [Basic usage](1_08_basic_usage.md) *(hands-on)* -- [Part II: **Using EasyBuild**](2_00_part2_using.md) *(Tue March 23rd 2021, 9am-12 CET)* +- [Part II: **Using EasyBuild**](2_00_part2_using.md) - [Troubleshooting](2_01_troubleshooting.md) *(hands-on)* - [Creating easyconfig files](2_02_creating_easyconfig_files.md) *(hands-on)* - [Using external modules from the Cray PE](2_03_external_modules.md) - [Implementing easyblocks](2_04_implementing_easyblocks.md) *(hands-on)* - - - -TODO - - - - - - - -## OLD TEXT - -- [Part I: **Introduction to EasyBuild**](part1_intro.md) *(Tue March 9th 2021, 9am-12 CET)* - * [What is EasyBuild?](what_is_easybuild.md) - * [Terminology](terminology.md) - * [Installation](installation.md) *(hands-on)* - * [Configuration](configuration.md) *(hands-on)* - * [Basic usage](basic_usage.md) *(hands-on)* -- [Part II: **Using EasyBuild**](part2_using.md) *(Tue March 23rd 2021, 9am-12 CET)* - * [Troubleshooting](troubleshooting.md) *(hands-on)* - * [Creating easyconfig files](creating_easyconfig_files.md) *(hands-on)* - * [Implementing easyblocks](implementing_easyblocks.md) *(hands-on)* -- [Part III: **Advanced topics**](part3_advanced.md) *(Tue March 30th 2021, 9am-12 CEST)* - * [Using EasyBuild as a library](easybuild_library.md) *(hands-on)* - * [Using hooks to customise EasyBuild](hooks.md) *(hands-on)* - * [Submitting installations as Slurm jobs](slurm_jobs.md) *(hands-on)* - * [Module naming schemes (incl. hierarchical)](module_naming_schemes.md) *(hands-on)* - * [GitHub integration to facilitate contributing to EasyBuild](github_integration.md) *(hands-on)* -- [Part IV: **EasyBuild on Cray systems**](part4_cray.md) *(Friday June 18th 2021, 09-12 CEST)* - * [Introduction to Cray Programming Environment](cray/introduction.md) *(hands-on)* - * [Cray External Modules](cray/external_modules.md) *(hands-on)* - * [Cray Custom Toolchains](cray/custom_toolchains.md) *(hands-on)* - * [EasyBuild at CSCS](cray/easybuild_at_cscs.md) *(hands-on)* +- [Part III: **Advanced topics**](3_00_part3_advanced.md) + - [Using EasyBuild as a library](3_01_easybuild_library.md) + - [Using hooks to customise EasyBuild](3_02_hooks.md) + - [Submitting installations as Slurm jobs](3_03_slurm_jobs.md) + - [Module naming schemes (incl. hierarchical)](3_04_module_naming_schemes.md) + - [GitHub integration to facilitate contributing to EasyBuild](3_05_github_integration.md) +- [**Additional reading**](4_00_additional_reading.md) diff --git a/mkdocs.yml b/mkdocs.yml index 232b088f..203bc62b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -58,6 +58,7 @@ nav: - Submitting installations as Slurm jobs: 2022-CSC_and_LO/3_03_slurm_jobs.md - Module naming schemes (incl. hierarchical): 2022-CSC_and_LO/3_04_module_naming_schemes.md - GitHub integration to facilitate contributing to EasyBuild: 2022-CSC_and_LO/3_05_github_integration.md + - Additional reading: 2022-CSC_and_LO/4_00_additional_reading.md - "ISC'21 (25 June 2021)": - (overview): 2021-isc21/index.md - Practical info: 2021-isc21/practical_info.md From d1aee2d0fb5951531c2677270a30e04a4b80eec8 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 17:08:35 +0200 Subject: [PATCH 034/159] Additional reading section. --- .../4_00_additional_reading.md | 49 +++++++++++++++++-- 1 file changed, 46 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/4_00_additional_reading.md b/docs/2022-CSC_and_LO/4_00_additional_reading.md index 8f9d54d9..87264f0d 100644 --- a/docs/2022-CSC_and_LO/4_00_additional_reading.md +++ b/docs/2022-CSC_and_LO/4_00_additional_reading.md @@ -4,9 +4,52 @@ --- - - - +## EasyBuild + +- [EasyBuild documentation](https://docs.easybuild.io/en/latest/#) +- [EasyBuild web site](https://easybuild.io/) containing links to much of the recent information on + EasyBuild. +- [EasyBuild YouTube channel](https://www.youtube.com/c/EasyBuilders) +- This tutorial is an evolution of the + [EasyBuild tutorial prepared for the LUMI User Suppport Team, spring '21](https://easybuilders.github.io/easybuild-tutorial/2021-lust/) + given by Kenneth Hoste (UGent, EasyBuild lead developer) and Luca Marsella (CSCS) + - [Recordings are available on YouTube](https://www.youtube.com/watch?v=JTRw8hqi6x0&list=PLhnGtSmEGEQh573bk3BeOj_KCRBBiA5OT) +- The EasyBuild setup on LUMI is partly insprired on the setup used at CSCS on their Cray systems + - [Section on the CSCS setup](https://easybuilders.github.io/easybuild-tutorial/2021-lust/cray/easybuild_at_cscs/) + from the LUST EasyBuild tutorial spring '21. + - [CSCS site presentation](https://easybuild.io/eum21/#eb-site-talk-cscs) + during the [6th EasyBuild User Meeting in 2021](https://easybuild.io/eum21/) + - CSCS site presentation ([video](https://www.youtube.com/watch?v=rhoHmUhzSGw&list=PLhnGtSmEGEQidEM8MZKkOaVutgt9WmqI0)/[slides](https://users.ugent.be/~kehoste/eum20/eum20_11_luca_cscs.pdf)) + during the [5h EasyBuild User Meeting in 2020](https://github.com/easybuilders/easybuild/wiki/5th-EasyBuild-User-Meeting) + - CSCS site presentation ([video](https://youtu.be/qLuHeaFmydM?list=PLhnGtSmEGEQhEjG5hD70tRwL3n58aWNHc&t=1451)) + during the [4h EasyBuild User Meeting in 2020](https://github.com/easybuilders/easybuild/wiki/4th-EasyBuild-User-Meeting) + + +## HPE Cray Programming Environment + +- [LUMI documentation: "Developing" section](https://docs.lumi-supercomputer.eu/development/) +- The Cray PE is mostly documented through man pages. There used to be some documentation on the + Cray web site also but the documentation system got reworked after the merger with HPE. + The documentation is now in the [HPE Support Centre](https://support.hpe.com/) where it + is very difficult to find the right version of the documents. +- The [PE-Cray](https://github.com/PE-Cray) GitHub project also provides some additional + documentation, including + - [some whitepapers](https://github.com/PE-Cray/whitepapers), + - [documentation for Cray OpenSMHEMX](https://github.com/PE-Cray/cray-openshmemx) and + - [information about Cray DSMML](https://github.com/PE-Cray/cray-dsmml) + though much of that information ia also not updated regularly anymore. + + +## LUMI + +- [LUMI web site](https://lumi-supercomputer.eu/) +- [LUMI User Documentation](https://docs.lumi-supercomputer.eu/) + - [Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/) + - [Organisation of the software stack](https://docs.lumi-supercomputer.eu/computing/softwarestacks/) + - [EasyBuild on LUMI](https://docs.lumi-supercomputer.eu/software/installing/easybuild/) +- [EasyBuild on LUMI presentation](https://easybuild.io/eum22/#lumi) + ([video](https://www.youtube.com/watch?v=hZezVG6lJNk&list=PLhnGtSmEGEQgCneeSQvYoIZrbv7wIKlo2)/[slides](https://easybuild.io/eum22/001_eum22_EasyBuild_on_LUMI.pdf)) + during the [7th EasyBuild User Meeting in 2022](https://easybuild.io/eum22/) --- From 3cf8a3c41927d55c3db516d7be6dc16e512ffddc Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 17:43:28 +0200 Subject: [PATCH 035/159] Correcting a number of links. --- docs/2022-CSC_and_LO/1_06_installation.md | 4 ++-- docs/2022-CSC_and_LO/1_07_configuration.md | 6 +++--- docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md | 6 +++--- docs/2022-CSC_and_LO/2_03_external_modules.md | 2 +- docs/2022-CSC_and_LO/3_03_slurm_jobs.md | 6 +++--- 5 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_06_installation.md b/docs/2022-CSC_and_LO/1_06_installation.md index 6e68b500..24c71d5e 100644 --- a/docs/2022-CSC_and_LO/1_06_installation.md +++ b/docs/2022-CSC_and_LO/1_06_installation.md @@ -424,7 +424,7 @@ We do so only for the framework and easyblocks files as the easconfig files are install EasyBuild. Instead we create our own EasyConfig file for EasyBuild which contains some additional packages that enable extra features in EasyBuild and also provide more information to Lmod. Next the configuration module for EasyBuild (see the next section, -["Configuring EasyBuild"](1_06_configuring.md), for more information) and use the temporary +["Configuring EasyBuild"](1_07_configuration), for more information) and use the temporary installation of EasyBuild with our own EasyConfig file to do a proper installation of EasyBuild with module in the final location. @@ -468,7 +468,7 @@ eb --show-config ``` This should tell you that EasyBuild (ab)uses `$HOME/.local/easybuild` as a default location. -More on configuring EasyBuild in the [next part of the tutorial](configuration.md). +More on configuring EasyBuild in the [next part of the tutorial](configuration). #### System information diff --git a/docs/2022-CSC_and_LO/1_07_configuration.md b/docs/2022-CSC_and_LO/1_07_configuration.md index 17e60afe..a1c2f3de 100644 --- a/docs/2022-CSC_and_LO/1_07_configuration.md +++ b/docs/2022-CSC_and_LO/1_07_configuration.md @@ -280,7 +280,7 @@ The default `EasyBuildMNS` module naming scheme roughly corresponds to the filen and consists of the software name followed by a combination of the software version, toolchain and an optional label (which corresponds to the value of the `versionsuffix` easyconfig parameter): `/<-toolchain>`. Just like with [names of easyconfig -files](terminology.md#easyconfig-files), the `<-toolchain>` part is omitted when the `system` toolchain is used, +files](1_05_terminology#easyconfig-files), the `<-toolchain>` part is omitted when the `system` toolchain is used, and the `` value is empty by default. !!! Note "Module naming scheme on LUMI" @@ -366,9 +366,9 @@ across different software installations. They define *configuration settings*, such as the location where software should be installed, or the syntax that should be used when generating module files. -An [easyconfig file](terminology.md#easyconfig-files) (`*.eb`) on the other hand *specifies the details for one particular software installation*. +An [easyconfig file](1_05_terminology#easyconfig-files) (`*.eb`) on the other hand *specifies the details for one particular software installation*. It does this by defining a set of *easyconfig parameters*, which tell EasyBuild the name and version -of the software to install, which [toolchain](terminology.md#toolchains) and [easyblock](terminology.md#easyblocks) to use, etc. +of the software to install, which [toolchain](1_05_terminology#toolchains) and [easyblock](1_05_terminology#easyblocks) to use, etc. For each software installation performed by EasyBuild, there is a corresponding easyconfig file. There typically are only a handful of configuration files used however, for example a system-level diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md index f7d6a646..f9bec25a 100644 --- a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md @@ -1,6 +1,6 @@ # Creating easyconfig files -*[[back: Troubleshooting]](2_02_troubleshooting.md)* +*[[back: Troubleshooting]](2_02_troubleshooting)* --- @@ -58,7 +58,7 @@ Reasons to consider implementing a software-specific easyblock rather than using bootstrapping with the system compiler and then re-installing with itself.* For implementing easyblocks we refer to the -["Implementing easyblocks" section of this tutorial](2_03_implementing_easyblocks.md) +["Implementing easyblocks" section of this tutorial](2_03_implementing_easyblocks) and the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). ## Writing easyconfig files @@ -1024,4 +1024,4 @@ You can leverage the `eb-tutorial` easyconfig file we have composed in the examp --- -*[[next: Using external modules from the Cray PE]](2_03_external_modules.md)* +*[[next: Using external modules from the Cray PE]](2_03_external_modules)* diff --git a/docs/2022-CSC_and_LO/2_03_external_modules.md b/docs/2022-CSC_and_LO/2_03_external_modules.md index b19313ff..80a29712 100644 --- a/docs/2022-CSC_and_LO/2_03_external_modules.md +++ b/docs/2022-CSC_and_LO/2_03_external_modules.md @@ -1,6 +1,6 @@ # Using external modules from the Cray PE -*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* +*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files)* --- diff --git a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md index 567f8abd..283a53a0 100644 --- a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md +++ b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md @@ -37,8 +37,8 @@ will be *passed down* into the submitted job automatically, via command line opt command that is run in the job script. This includes not only command line options used next to ``--job``, but also configuration settings -that are specified via an [EasyBuild configuration file](configuration.md#configuration-files) or through -[``$EASYBUILD_*`` environment variables](configuration.md#easybuild_-environment-variables). +that are specified via an [EasyBuild configuration file](1_07_configuration#configuration-files) or through +[``$EASYBUILD_*`` environment variables](1_07_configuration#easybuild_-environment-variables). This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables that are in place in the job environment are most likely *irrelevant*, since configuration settings @@ -330,4 +330,4 @@ $ module avail AUGUSTUS --- -*[[next: Module naming schemes]](3_04_module_naming_scheme.md)* +*[[next: Module naming schemes]](3_04_module_naming_scheme)* From 7b1f47064189fab41241c9d7721e2bcd436967a2 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 7 Apr 2022 17:26:56 +0200 Subject: [PATCH 036/159] Restructuring for nicer navigation bar on the left. --- .../{ => 1_Intro}/1_01_what_is_easybuild.md | 0 .../{ => 1_Intro}/1_02_Lmod.md | 0 .../2022-CSC_and_LO/{ => 1_Intro}/1_03_CPE.md | 0 .../{ => 1_Intro}/1_04_LUMI_software_stack.md | 0 .../{ => 1_Intro}/1_05_terminology.md | 0 .../{ => 1_Intro}/1_06_installation.md | 0 .../{ => 1_Intro}/1_07_configuration.md | 0 .../{ => 1_Intro}/1_08_basic_usage.md | 2 +- .../{1_00_part1_intro.md => 1_Intro/index.md} | 0 .../{ => 2_Using}/2_01_troubleshooting.md | 2 +- .../2_02_creating_easyconfig_files.md | 16 ++++---- .../{ => 2_Using}/2_03_external_modules.md | 0 .../2_04_implementing_easyblocks.md | 2 +- .../{2_00_part2_using.md => 2_Using/index.md} | 0 .../3_01_easybuild_library.md | 2 +- .../{ => 3_Advanced}/3_02_hooks.md | 0 .../{ => 3_Advanced}/3_03_slurm_jobs.md | 4 +- .../3_04_module_naming_schemes.md | 8 ++-- .../3_05_github_integration.md | 0 .../index.md} | 0 docs/2022-CSC_and_LO/index.md | 40 +++++++++---------- 21 files changed, 38 insertions(+), 38 deletions(-) rename docs/2022-CSC_and_LO/{ => 1_Intro}/1_01_what_is_easybuild.md (100%) rename docs/2022-CSC_and_LO/{ => 1_Intro}/1_02_Lmod.md (100%) rename docs/2022-CSC_and_LO/{ => 1_Intro}/1_03_CPE.md (100%) rename docs/2022-CSC_and_LO/{ => 1_Intro}/1_04_LUMI_software_stack.md (100%) rename docs/2022-CSC_and_LO/{ => 1_Intro}/1_05_terminology.md (100%) rename docs/2022-CSC_and_LO/{ => 1_Intro}/1_06_installation.md (100%) rename docs/2022-CSC_and_LO/{ => 1_Intro}/1_07_configuration.md (100%) rename docs/2022-CSC_and_LO/{ => 1_Intro}/1_08_basic_usage.md (99%) rename docs/2022-CSC_and_LO/{1_00_part1_intro.md => 1_Intro/index.md} (100%) rename docs/2022-CSC_and_LO/{ => 2_Using}/2_01_troubleshooting.md (99%) rename docs/2022-CSC_and_LO/{ => 2_Using}/2_02_creating_easyconfig_files.md (97%) rename docs/2022-CSC_and_LO/{ => 2_Using}/2_03_external_modules.md (100%) rename docs/2022-CSC_and_LO/{ => 2_Using}/2_04_implementing_easyblocks.md (99%) rename docs/2022-CSC_and_LO/{2_00_part2_using.md => 2_Using/index.md} (100%) rename docs/2022-CSC_and_LO/{ => 3_Advanced}/3_01_easybuild_library.md (99%) rename docs/2022-CSC_and_LO/{ => 3_Advanced}/3_02_hooks.md (100%) rename docs/2022-CSC_and_LO/{ => 3_Advanced}/3_03_slurm_jobs.md (98%) rename docs/2022-CSC_and_LO/{ => 3_Advanced}/3_04_module_naming_schemes.md (98%) rename docs/2022-CSC_and_LO/{ => 3_Advanced}/3_05_github_integration.md (100%) rename docs/2022-CSC_and_LO/{3_00_part3_advanced.md => 3_Advanced/index.md} (100%) diff --git a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md similarity index 100% rename from docs/2022-CSC_and_LO/1_01_what_is_easybuild.md rename to docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md similarity index 100% rename from docs/2022-CSC_and_LO/1_02_Lmod.md rename to docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md similarity index 100% rename from docs/2022-CSC_and_LO/1_03_CPE.md rename to docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md diff --git a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md b/docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md similarity index 100% rename from docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md rename to docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md diff --git a/docs/2022-CSC_and_LO/1_05_terminology.md b/docs/2022-CSC_and_LO/1_Intro/1_05_terminology.md similarity index 100% rename from docs/2022-CSC_and_LO/1_05_terminology.md rename to docs/2022-CSC_and_LO/1_Intro/1_05_terminology.md diff --git a/docs/2022-CSC_and_LO/1_06_installation.md b/docs/2022-CSC_and_LO/1_Intro/1_06_installation.md similarity index 100% rename from docs/2022-CSC_and_LO/1_06_installation.md rename to docs/2022-CSC_and_LO/1_Intro/1_06_installation.md diff --git a/docs/2022-CSC_and_LO/1_07_configuration.md b/docs/2022-CSC_and_LO/1_Intro/1_07_configuration.md similarity index 100% rename from docs/2022-CSC_and_LO/1_07_configuration.md rename to docs/2022-CSC_and_LO/1_Intro/1_07_configuration.md diff --git a/docs/2022-CSC_and_LO/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md similarity index 99% rename from docs/2022-CSC_and_LO/1_08_basic_usage.md rename to docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md index f5031e48..97e87fdf 100644 --- a/docs/2022-CSC_and_LO/1_08_basic_usage.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md @@ -1213,4 +1213,4 @@ and it's a lot to take in at once, take your time... --- -*[[next: Part 2: Using EasyBuild]](2_00_part2_using.md)* +*[[next: Troubleshooting]](2_Advanced/2_01_troubleshooting.md)* diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_Intro/index.md similarity index 100% rename from docs/2022-CSC_and_LO/1_00_part1_intro.md rename to docs/2022-CSC_and_LO/1_Intro/index.md diff --git a/docs/2022-CSC_and_LO/2_01_troubleshooting.md b/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md similarity index 99% rename from docs/2022-CSC_and_LO/2_01_troubleshooting.md rename to docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md index 30f2f5cc..453c2615 100644 --- a/docs/2022-CSC_and_LO/2_01_troubleshooting.md +++ b/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md @@ -1,6 +1,6 @@ # Troubleshooting -*[[back: Overview]](index.md)* +*[[back: BAsic usage]](../1_Intro/1_08_basic_usage)* --- diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md similarity index 97% rename from docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md rename to docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md index f9bec25a..0db1dedd 100644 --- a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md @@ -15,10 +15,10 @@ In this part of the tutorial we will look at the guts of easyconfig files and ev ## Easyconfigs vs easyblocks -Before we dive into writing [easyconfig files](../1_05_terminology/#easyconfig-files), -let us take a brief look at how they relate to [easyblocks](../1_05_terminology/#easyblocks). +Before we dive into writing [easyconfig files](../../1_Intro/1_05_terminology/#easyconfig-files), +let us take a brief look at how they relate to [easyblocks](../../1_Intro/1_05_terminology/#easyblocks). -As we discussed [earlier](../1_05_terminology/#terminology), an easyconfig file (`*.eb`) is required +As we discussed [earlier](../../1_Intro/1_05_terminology/#terminology), an easyconfig file (`*.eb`) is required for each installation performed by EasyBuild which specifies the details of the installation (which software version, toolchain, etc.), while the installation procedure is implemented @@ -139,10 +139,10 @@ Note that in this case the word "Description:" has to be explicitly specified. #### `toolchain` -EasyBuild also requires that the [compiler toolchain](../1_05_terminology/#toolchains) is specified, via the `toolchain` +EasyBuild also requires that the [compiler toolchain](../../1_Intro/1_05_terminology/#toolchains) is specified, via the `toolchain` easyconfig parameter. -This can either be the [`system` toolchain](../1_05_terminology/#system-toolchain), for which a constant named `SYSTEM` is available: +This can either be the [`system` toolchain](../../1_Intro/1_05_terminology/#system-toolchain), for which a constant named `SYSTEM` is available: ```python toolchain = SYSTEM @@ -271,7 +271,7 @@ tar_config_opts* Override tar settings as determined by configure. [defau #### Dependencies -You will often need to list one or more [dependencies](../1_05_terminology/#dependencies) that are required +You will often need to list one or more [dependencies](../../1_Intro/1_05_terminology/#dependencies) that are required to install or run the software. We distinguish between two main different types of dependencies: runtime dependencies and build dependencies. @@ -640,7 +640,7 @@ build tools, and `cmake` is one of those tools that is typically entered as a bu dependency. After all, the less you rely on the OS, the more likely it becomes that your easyconfig is useful for other sites also. -In the [section on Lmod](../1_02_Lmod#module-extensions) we've already seen that on LUMI +In the [section on Lmod](../../1_Intro/1_02_Lmod#module-extensions) we've already seen that on LUMI the `cmake` command is available through the `buildtools` modules, and as discussed in other examples on this page, LUMI has one for every `LUMI` software stack with its version number the same as the stack and corresponding toolchains. It is a good practice to @@ -654,7 +654,7 @@ builddependencies = [ In a more traditional EasyBuild setup with the common toolchains, there is usually no need to specify the toolchain for (build) dependencies. EasyBuild will automatically consider -[subtoolchains](../1_05_terminology/#toolchains) compatible with the specified toolchain to locate +[subtoolchains](../../1_Intro/1_05_terminology/#toolchains) compatible with the specified toolchain to locate modules for the dependencies. However, the Cray PE toolchains on LUMI are currently not part of such a hierarchy and the `SYSTEM` toolchain we used for `buildtools` is not automatically considered which is why we need the 4-element version of the dependency specification. diff --git a/docs/2022-CSC_and_LO/2_03_external_modules.md b/docs/2022-CSC_and_LO/2_Using/2_03_external_modules.md similarity index 100% rename from docs/2022-CSC_and_LO/2_03_external_modules.md rename to docs/2022-CSC_and_LO/2_Using/2_03_external_modules.md diff --git a/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md similarity index 99% rename from docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md rename to docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md index a91b0dc8..9373a1e8 100644 --- a/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md +++ b/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md @@ -751,4 +751,4 @@ Your easyblock should only: This is a much simpler easyblock as we already use all the logic that has been written for us to build with CMake. -*[[next: Part 3: Overview]](3_00_part3_advanced.md)* +*[[next: EasyBuild as a library]](../3_Advanced/3_01_easybuild_library.md)* diff --git a/docs/2022-CSC_and_LO/2_00_part2_using.md b/docs/2022-CSC_and_LO/2_Using/index.md similarity index 100% rename from docs/2022-CSC_and_LO/2_00_part2_using.md rename to docs/2022-CSC_and_LO/2_Using/index.md diff --git a/docs/2022-CSC_and_LO/3_01_easybuild_library.md b/docs/2022-CSC_and_LO/3_Advanced/3_01_easybuild_library.md similarity index 99% rename from docs/2022-CSC_and_LO/3_01_easybuild_library.md rename to docs/2022-CSC_and_LO/3_Advanced/3_01_easybuild_library.md index f0c255eb..6a0c8810 100644 --- a/docs/2022-CSC_and_LO/3_01_easybuild_library.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_01_easybuild_library.md @@ -1,6 +1,6 @@ # Using EasyBuild as a library -*[[back: Configuring EasyBuild]](1_07_configuration.md)* +*[[back: Implementing EasyBlocks]](../2_Using/2_04_implementing_easyblocks.md)* --- diff --git a/docs/2022-CSC_and_LO/3_02_hooks.md b/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md similarity index 100% rename from docs/2022-CSC_and_LO/3_02_hooks.md rename to docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md diff --git a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md similarity index 98% rename from docs/2022-CSC_and_LO/3_03_slurm_jobs.md rename to docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md index 283a53a0..6a5f7243 100644 --- a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md @@ -37,8 +37,8 @@ will be *passed down* into the submitted job automatically, via command line opt command that is run in the job script. This includes not only command line options used next to ``--job``, but also configuration settings -that are specified via an [EasyBuild configuration file](1_07_configuration#configuration-files) or through -[``$EASYBUILD_*`` environment variables](1_07_configuration#easybuild_-environment-variables). +that are specified via an [EasyBuild configuration file](../1_Intro/1_07_configuration#configuration-files) or through +[``$EASYBUILD_*`` environment variables](../1_Intro/1_07_configuration#easybuild_-environment-variables). This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables that are in place in the job environment are most likely *irrelevant*, since configuration settings diff --git a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md similarity index 98% rename from docs/2022-CSC_and_LO/3_04_module_naming_schemes.md rename to docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md index d0b4ebe8..7da4d4ca 100644 --- a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md @@ -50,19 +50,19 @@ module naming schemes, which are characterized by: In contrast, a *hierarchical* module naming scheme consists of a *hierarchy* of module files. A fairly typical 3-level schme (``Core``, ``Compiler`` and ``MPI``) has been -discussed in the [section on Lmod](../1_02_Lmod#lmod-hierarchy). +discussed in the [section on Lmod](../1_Intro/1_02_Lmod#lmod-hierarchy). This typical Lmod hierarcny would map very well on the EasyBuild common toolchains. In fact, for the example -
+
software at the ``Core`` level would be installed with the ``GCCcore`` and ``SYSTEM`` toolchains. Software at the ``Compiler`` level would be installed with the ``GCC`` toolchain, while software at the ``MPI`` level after loading the ``OpenMPI`` module would be installed using the regular ``foss`` toolchain or the ``gompi`` toolchain (see the diagram in the -["Common toolchains" section of the page on terminology](../1_05_terminology/#common-toolchains)). +["Common toolchains" section of the page on terminology](../1_Intro/1_05_terminology/#common-toolchains)). On LUMI, where software is installed through the Cray Programming Environment with no real choice of MPI implementation, a two-level arrangement would still make a lot of sense, with at the ``Core`` level @@ -325,7 +325,7 @@ a module. We have just made all modules unavailable, so we would have to first install EasyBuild again in our hierarchical module tree before we can continue. **We strongly recommend using an EasyBuild installation that was [installed via "`pip install`" -or "`pip3 install`"](../1_06_installation#method-1-using-pip) in this part of the tutorial.** +or "`pip3 install`"](../1_Intro/1_06_installation#method-1-using-pip) in this part of the tutorial.** An easy way to do this is in the prepared environment is to run: diff --git a/docs/2022-CSC_and_LO/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md similarity index 100% rename from docs/2022-CSC_and_LO/3_05_github_integration.md rename to docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md diff --git a/docs/2022-CSC_and_LO/3_00_part3_advanced.md b/docs/2022-CSC_and_LO/3_Advanced/index.md similarity index 100% rename from docs/2022-CSC_and_LO/3_00_part3_advanced.md rename to docs/2022-CSC_and_LO/3_Advanced/index.md diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md index 7161a112..58831541 100644 --- a/docs/2022-CSC_and_LO/index.md +++ b/docs/2022-CSC_and_LO/index.md @@ -58,24 +58,24 @@ freely available, so unless you have access to a system with this environment yo ## Contents -- [Part I: **Introduction to EasyBuild on Cray systems**](1_00_part1_intro.md) - - [What is EasyBuild?](1_01_what_is_easybuild.md) - - [The Lmod module system](1_02_Lmod.md) - - [The HPE Cray Programming Environment](1_03_CPE.md) - - [LUMI software stacks](1_04_LUMI_software_stack.md) - - [Terminology](1_05_terminology.md) - - [Installation](1_06_installation.md) - - [Configuration](1_07_configuration.md) - - [Basic usage](1_08_basic_usage.md) *(hands-on)* -- [Part II: **Using EasyBuild**](2_00_part2_using.md) - - [Troubleshooting](2_01_troubleshooting.md) *(hands-on)* - - [Creating easyconfig files](2_02_creating_easyconfig_files.md) *(hands-on)* - - [Using external modules from the Cray PE](2_03_external_modules.md) - - [Implementing easyblocks](2_04_implementing_easyblocks.md) *(hands-on)* -- [Part III: **Advanced topics**](3_00_part3_advanced.md) - - [Using EasyBuild as a library](3_01_easybuild_library.md) - - [Using hooks to customise EasyBuild](3_02_hooks.md) - - [Submitting installations as Slurm jobs](3_03_slurm_jobs.md) - - [Module naming schemes (incl. hierarchical)](3_04_module_naming_schemes.md) - - [GitHub integration to facilitate contributing to EasyBuild](3_05_github_integration.md) +- [Part I: **Introduction to EasyBuild on Cray systems**](1_Intro/index.md) + - [What is EasyBuild?](1_Intro/1_01_what_is_easybuild.md) + - [The Lmod module system](1_Intro/1_02_Lmod.md) + - [The HPE Cray Programming Environment](1_Intro/1_03_CPE.md) + - [LUMI software stacks](1_Intro/1_04_LUMI_software_stack.md) + - [Terminology](1_Intro/1_05_terminology.md) + - [Installation](1_Intro/1_06_installation.md) + - [Configuration](1_Intro/1_07_configuration.md) + - [Basic usage](1_Intro/1_08_basic_usage.md) *(hands-on)* +- [Part II: **Using EasyBuild**](2_Using/index.md) + - [Troubleshooting](2_Using/2_01_troubleshooting.md) *(hands-on)* + - [Creating easyconfig files](2_Using/2_02_creating_easyconfig_files.md) *(hands-on)* + - [Using external modules from the Cray PE](2_Using/2_03_external_modules.md) + - [Implementing easyblocks](2_Using/2_04_implementing_easyblocks.md) *(hands-on)* +- [Part III: **Advanced topics**](3_Advanced/index.md) + - [Using EasyBuild as a library](3_Advanced/3_01_easybuild_library.md) + - [Using hooks to customise EasyBuild](3_Advanced/3_02_hooks.md) + - [Submitting installations as Slurm jobs](3_Advanced/3_03_slurm_jobs.md) + - [Module naming schemes (incl. hierarchical)](3_Advanced/3_04_module_naming_schemes.md) + - [GitHub integration to facilitate contributing to EasyBuild](3_Advanced/3_05_github_integration.md) - [**Additional reading**](4_00_additional_reading.md) From 724e059c5aabdb39ecd9250771c4508e4d966392 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 7 Apr 2022 17:32:46 +0200 Subject: [PATCH 037/159] Link corrections. --- docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md | 2 +- docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md | 2 +- docs/2022-CSC_and_LO/4_00_additional_reading.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md index 97e87fdf..ade394ce 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md @@ -1213,4 +1213,4 @@ and it's a lot to take in at once, take your time... --- -*[[next: Troubleshooting]](2_Advanced/2_01_troubleshooting.md)* +*[[next: Troubleshooting]](../2_Using/2_01_troubleshooting.md)* diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md index 6e19cb9a..17360c58 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md @@ -457,4 +457,4 @@ The gist linked from this comment provides more detailed information: --- -*[[next: Additional reading]](4_00_additional_reading.md)* +*[[next: Additional reading]](../4_00_additional_reading.md)* diff --git a/docs/2022-CSC_and_LO/4_00_additional_reading.md b/docs/2022-CSC_and_LO/4_00_additional_reading.md index 87264f0d..f393c097 100644 --- a/docs/2022-CSC_and_LO/4_00_additional_reading.md +++ b/docs/2022-CSC_and_LO/4_00_additional_reading.md @@ -1,6 +1,6 @@ # Additional reading -*[[back: GitHub integration]](3_05_github_integration.md)* +*[[back: GitHub integration]](3_Advanced/3_05_github_integration.md)* --- From 64775be85cdbed37557dc8b6a767d8b01ba6d178 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 7 Apr 2022 21:14:52 +0200 Subject: [PATCH 038/159] Correction of links. --- .../1_Intro/1_01_what_is_easybuild.md | 54 +++++++++---------- .../2_Using/2_01_troubleshooting.md | 2 +- .../2_Using/2_02_creating_easyconfig_files.md | 4 +- .../2_Using/2_03_external_modules.md | 2 +- .../3_Advanced/3_03_slurm_jobs.md | 2 +- .../3_Advanced/3_04_module_naming_schemes.md | 2 +- .../3_Advanced/3_05_github_integration.md | 8 +-- docs/2022-CSC_and_LO/3_Advanced/index.md | 10 ++-- mkdocs.yml | 40 +++++++------- 9 files changed, 62 insertions(+), 62 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md index 979c6e90..6f169754 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md @@ -1,8 +1,8 @@ # What is EasyBuild? -*[[back: Overview]](index.md)* +*[[back: Course overview]](../index.md)* -

EasyBuild logo

+

EasyBuild logo

**[EasyBuild](http://easybuild.io)** is a **software build and installation framework** that manages (scientific) software in an efficient way, without compromising on performance. @@ -12,7 +12,7 @@ and provides a **command-line interface** to facilitate the installation and man --- - + ## Elevator pitch @@ -173,7 +173,7 @@ Since EasyBuild is primarily intended for installing (scientific) software on HP systems running some flavor of **GNU/Linux**. EasyBuild also works on *macOS*, but significant effort would be required to use it extensively there. - + EasyBuild is implemented in [**Python**](https://www.python.org), and is compatible with Python 2.7 and 3.5+. Since [Python 2 is no longer maintained](https://blog.python.org/2020/04/python-2718-last-release-of-python-2.html), @@ -189,7 +189,7 @@ On LUMI Lmod is used as the module tool. ## Development - + EasyBuild is **actively developed** by a worldwide community under the [**GPLv2 open source license**](https://www.gnu.org/licenses/old-licenses/gpl-2.0.html) via the [**``easybuilders``** organisation on GitHub](https://github.com/easybuilders), in the following repositories: @@ -227,26 +227,26 @@ sites leverage it, and integrate it with other tools like [Jenkins](https://www. HPC sites, consortia, and companies that use EasyBuild include (but are not limited to): -VSC logo -JSC logo -HPCNow! logo -Fred Hutch logo -Vienna BioCenter logo -CÉCI logo -Compute Canada logo -UiO logo -BEAR logo -SNIC logo -National University of Singapore logo -IT4Innovations logo -Texas A&M logo -TU Dresden logo -University of Melbourne logo -University of Luxembourg logo -EPCC logo -Cyprus Institute logo -University of Basel logo -Microsoft logo +VSC logo +JSC logo +HPCNow! logo +Fred Hutch logo +Vienna BioCenter logo +CÉCI logo +Compute Canada logo +UiO logo +BEAR logo +SNIC logo +National University of Singapore logo +IT4Innovations logo +Texas A&M logo +TU Dresden logo +University of Melbourne logo +University of Luxembourg logo +EPCC logo +Cyprus Institute logo +University of Basel logo +Microsoft logo --- @@ -262,7 +262,7 @@ The [search box](https://docs.easybuild.io/en/latest/search.html) on the top lef ### GitHub - + You can open an issue in one of the EasyBuild repositories in [the `easybuilders` GitHub organisation](https://github.com/easybuilders/) to ask questions, report problems or share suggestions and ideas for additional @@ -289,7 +289,7 @@ to the EasyBuild Slack channel. ### Slack -Slack logo +Slack logo The [EasyBuild Slack](https://easybuild.slack.com/) is a quick and easy way to get in touch with the EasyBuild community. diff --git a/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md b/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md index 453c2615..88509b3d 100644 --- a/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md +++ b/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md @@ -1,6 +1,6 @@ # Troubleshooting -*[[back: BAsic usage]](../1_Intro/1_08_basic_usage)* +*[[back: Basic usage]](../1_Intro/1_08_basic_usage.md)* --- diff --git a/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md index 0db1dedd..7ccd4389 100644 --- a/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md @@ -1,6 +1,6 @@ # Creating easyconfig files -*[[back: Troubleshooting]](2_02_troubleshooting)* +*[[back: Troubleshooting]](2_01_troubleshooting.md)* --- @@ -1024,4 +1024,4 @@ You can leverage the `eb-tutorial` easyconfig file we have composed in the examp --- -*[[next: Using external modules from the Cray PE]](2_03_external_modules)* +*[[next: Using external modules from the Cray PE]](2_03_external_modules.md)* diff --git a/docs/2022-CSC_and_LO/2_Using/2_03_external_modules.md b/docs/2022-CSC_and_LO/2_Using/2_03_external_modules.md index 80a29712..b19313ff 100644 --- a/docs/2022-CSC_and_LO/2_Using/2_03_external_modules.md +++ b/docs/2022-CSC_and_LO/2_Using/2_03_external_modules.md @@ -1,6 +1,6 @@ # Using external modules from the Cray PE -*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files)* +*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* --- diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md index 6a5f7243..0a5db233 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md @@ -330,4 +330,4 @@ $ module avail AUGUSTUS --- -*[[next: Module naming schemes]](3_04_module_naming_scheme)* +*[[next: Module naming schemes]](3_04_module_naming_schemes.md)* diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md index 7da4d4ca..6c66b130 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md @@ -55,7 +55,7 @@ This typical Lmod hierarcny would map very well on the EasyBuild common toolchai In fact, for the example -
+
software at the ``Core`` level would be installed with the ``GCCcore`` and ``SYSTEM`` toolchains. Software at the ``Compiler`` level would be installed with the ``GCC`` diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md index 17360c58..3a9732e0 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md @@ -72,7 +72,7 @@ git push my_fork example * the changes are indeed the ones you want to propose; * clicking the (correct) green button; -
Pfft
+
Pfft
That didn't exactly motivate you to contribute, did it... @@ -82,7 +82,7 @@ Over the years we noticed that some people were keen on contributing to EasyBuil but they were not very familiar with Git or GitHub. That meant they had to overcome a relatively steep learning curve before they could contribute... -
Gandalf vs Git
+
Gandalf vs Git
In addition, the contribution workflow can be a bit daunting and time consuming, even if you're already familiar with the procedure. You will have dozens of @@ -449,11 +449,11 @@ Note that we may need to use `--rebuild` here since `eb-tutorial` may already be This results in a comment being added to the pull request: -
Test report comment
+
Test report comment
The gist linked from this comment provides more detailed information: -
Test report gist
+
Test report gist
--- diff --git a/docs/2022-CSC_and_LO/3_Advanced/index.md b/docs/2022-CSC_and_LO/3_Advanced/index.md index 28a24990..d7831506 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/index.md +++ b/docs/2022-CSC_and_LO/3_Advanced/index.md @@ -8,8 +8,8 @@ LUMI ``EasyBuild-user`` module. Hooks are used on LUMI, but it is not really adv overwrite the centrally defined hooks with a local file. And the whole structure of the EasyBuild integration is also set up to make use of the GitHub integration in the future. -* [Using EasyBuild as a library](3_01_easybuild_library.md) *(hands-on)* -* [Using hooks to customise EasyBuild](3_02_hooks.md) *(hands-on)* -* [Submitting installations as Slurm jobs](3_03_slurm_jobs.md) *(hands-on)* -* [Module naming schemes (incl. hierarchical)](3_04_module_naming_schemes.md) *(hands-on)* -* [GitHub integration to facilitate contributing to EasyBuild](3_05_github_integration.md) *(hands-on)* +* [Using EasyBuild as a library](3_01_easybuild_library.md) +* [Using hooks to customise EasyBuild](3_02_hooks.md) +* [Submitting installations as Slurm jobs](3_03_slurm_jobs.md) +* [Module naming schemes (incl. hierarchical)](3_04_module_naming_schemes.md) +* [GitHub integration to facilitate contributing to EasyBuild](3_05_github_integration.md) diff --git a/mkdocs.yml b/mkdocs.yml index 203bc62b..1e7528db 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -36,28 +36,28 @@ nav: - "CSC (spring '22)": - (overview): 2022-CSC_and_LO/index.md - Introduction to EasyBuild: - - (overview): 2022-CSC_and_LO/1_00_part1_intro.md - - What is EasyBuild?: 2022-CSC_and_LO/1_01_what_is_easybuild.md - - The Lmod module system: 2022-CSC_and_LO/1_02_Lmod.md - - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_03_CPE.md - - LUMI software stacks: 2022-CSC_and_LO/1_04_LUMI_software_stack.md - - Terminology: 2022-CSC_and_LO/1_05_terminology.md - - Installation: 2022-CSC_and_LO/1_06_installation.md - - Configuration: 2022-CSC_and_LO/1_07_configuration.md - - Basic usage: 2022-CSC_and_LO/1_08_basic_usage.md + - (overview): 2022-CSC_and_LO/1_Intro/index.md + - What is EasyBuild?: 2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md + - The Lmod module system: 2022-CSC_and_LO/1_Intro/1_02_Lmod.md + - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_Intro/1_03_CPE.md + - LUMI software stacks: 2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md + - Terminology: 2022-CSC_and_LO/1_Intro/1_05_terminology.md + - Installation: 2022-CSC_and_LO/1_Intro/1_06_installation.md + - Configuration: 2022-CSC_and_LO/1_Intro/1_07_configuration.md + - Basic usage: 2022-CSC_and_LO/1_Intro/1_08_basic_usage.md - Using EasyBuild: - - (overview): 2022-CSC_and_LO/2_00_part2_using.md - - Troubleshooting: 2022-CSC_and_LO/2_01_troubleshooting.md - - Creating easyconfig files: 2022-CSC_and_LO/2_02_creating_easyconfig_files.md - - Using external modules from the Cray PE: 2022-CSC_and_LO/2_03_external_modules.md - - Implementing easyblocks: 2022-CSC_and_LO/2_04_implementing_easyblocks.md + - (overview): 2022-CSC_and_LO/2_Using/index.md + - Troubleshooting: 2022-CSC_and_LO/2_Using/2_01_troubleshooting.md + - Creating easyconfig files: 2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md + - Using external modules from the Cray PE: 2022-CSC_and_LO/2_Using/2_03_external_modules.md + - Implementing easyblocks: 2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md - Advanced topics: - - (overview): 2022-CSC_and_LO/3_00_part3_advanced.md - - Using EasyBuild as a library: 2022-CSC_and_LO/3_01_easybuild_library.md - - Using hooks to customise EasyBuild: 2022-CSC_and_LO/3_02_hooks.md - - Submitting installations as Slurm jobs: 2022-CSC_and_LO/3_03_slurm_jobs.md - - Module naming schemes (incl. hierarchical): 2022-CSC_and_LO/3_04_module_naming_schemes.md - - GitHub integration to facilitate contributing to EasyBuild: 2022-CSC_and_LO/3_05_github_integration.md + - (overview): 2022-CSC_and_LO/3_Advanced/index.md + - Using EasyBuild as a library: 2022-CSC_and_LO/3_Advanced/3_01_easybuild_library.md + - Using hooks to customise EasyBuild: 2022-CSC_and_LO/3_Advanced/3_02_hooks.md + - Submitting installations as Slurm jobs: 2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md + - Module naming schemes (incl. hierarchical): 2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md + - GitHub integration to facilitate contributing to EasyBuild: 2022-CSC_and_LO/3_Advanced/3_05_github_integration.md - Additional reading: 2022-CSC_and_LO/4_00_additional_reading.md - "ISC'21 (25 June 2021)": - (overview): 2021-isc21/index.md From d915944fd2581d1f14c0284eeec79a59abe0fb4b Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 8 Apr 2022 18:35:45 +0200 Subject: [PATCH 039/159] Corrected a number of spelling mistakes. --- docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md | 14 +++++++------- docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md | 10 +++++----- .../1_Intro/1_04_LUMI_software_stack.md | 2 +- docs/2022-CSC_and_LO/1_Intro/1_07_configuration.md | 4 ++-- docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md | 6 +++--- .../2_Using/2_02_creating_easyconfig_files.md | 6 +++--- .../2_Using/2_04_implementing_easyblocks.md | 4 ++-- docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md | 4 ++-- .../3_Advanced/3_04_module_naming_schemes.md | 8 ++++---- docs/2022-CSC_and_LO/4_00_additional_reading.md | 2 +- 10 files changed, 30 insertions(+), 30 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md index d6d73a6c..bc1ed006 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md @@ -23,7 +23,7 @@ Unloading the environment module will restore the shell environment to its previ Environment module files are processed via a **modules tool**, of which there are several conceptually similar yet slightly different implementations. The oldest module tool still in use today is Environment Modules 3.2, implemented in C and -supporting module files written in Tcl. After a gap in developement, Xavier Delaruelle of CEA +supporting module files written in Tcl. After a gap in development, Xavier Delaruelle of CEA developed [Environment Modules 4 and 5](https://sourceforge.net/projects/modules/) which is fully implemented on Tcl. An alternative module tool is [Lmod](https://lmod.readthedocs.io), developed by Robert McLay at TACC and implemented in LUA. This tool supports natively LUA @@ -39,7 +39,7 @@ Lmod also has some powerful features that are lacking in Environment Modules 3.2 On LUMI, Lmod was selected as the module tool. One area where there are significant differences between Environment Modules 3.2 (and also the newer versions) and Lmod is in the commands for discovering modules on the system. If you are not familiar with Lmod - and its commands for users, it is worthwile to read the + and its commands for users, it is worthwhile to read the [LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). Some of those commands are also discussed on this page. @@ -244,7 +244,7 @@ family('MPI') prepend_path('MODULEPATH', 'moduleroot/MPI/Compiler_A/version_A/MPI_C/version_C') ``` -Finally two vesions of the ``version_E.lua`` file are needed, one to prepare the environment for using the +Finally two versions of the ``version_E.lua`` file are needed, one to prepare the environment for using the package with Compiler_A anmd MPI_C and one for using the package with Compiler_B and MPI_C. However, these are just regular modules and no additions are needed to work for the hierarchy. @@ -375,7 +375,7 @@ e.g., NumPy and SciPy for Python. Installing all those packages as separate modu to make it easy to see if they are installed or not on a system would lead to an overload of modules on the system. -Similary, admins of a software stack may chose to bundle several libraries or tools +Similarly, admins of a software stack may chose to bundle several libraries or tools that are often used together in a single module (and single installation directory), e.g., to reduce module clutter but also to reduce the length of the search paths for binaries, libraries or manual pages to speed up loading of applications. @@ -550,7 +550,7 @@ indicated version. Lmod works by executing the module file. However, the actions of all Lmod-defined functions will depend upon the mode in which Lmod is executing the module function, and the module file can also detect in which mode it is executing. -Modes include "load", "unload"but also "spider". E.g., when te mode is "load", the +Modes include "load", "unload" but also "spider". E.g., when the mode is "load", the ``setenv`` function will set an environment variable to the indicated value while in "unload" mode that environment variable will be unset, and in "spider" mode the environment variable is left untouched. The working of ``prepend_path``, a function @@ -564,7 +564,7 @@ in that PATH-style variable). When the mode is "spider", the function has specia if it is used to change the ``MODULEPATH``. It will then note the change and add that directory to the list of directories that has to be searched for module files. This makes ``module spider`` a very expensive command as it may have to traverse a lot -of directories and has to execute all module files in there. Therefor Lmod will build +of directories and has to execute all module files in there. Therefore Lmod will build a so-called spider cache which can be pre-built in the system for certain directories and otherwise will be build in the user's home directory (in the ``.lmod.d/.cache`` subdirectory). Our experience is that this cache tends to be rather fragile, @@ -661,7 +661,7 @@ module load MyPythonPackage/1.0 Because of the *"one name rule"* this will again trigger an unload followed by a load of the module. The problem is in the unload. One would expect that first unloading ``MyPythonPackage`` would remove the 2.7 directory from the ``PYTHON_PATH`` but it -will not. Lmod does not remeber that last time it loaded ``MyPythonPackage`` it added +will not. Lmod does not remember that last time it loaded ``MyPythonPackage`` it added the 2.7 directory to ``PythonPath``. Instead it will execute the commands in the modulefile and reverse certain commands. Since ``PYTHON_API_VERSION`` has now the value ``3.6``, it will try to remove the directory for version ``3.6`` which is not in the diff --git a/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md b/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md index d3a2ff84..9502afaa 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md @@ -6,9 +6,9 @@ On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated as Cray PE). The environment provides several tools, including compilers, communication libraries, -optmized math libraries and various other libraries, analyzers and debuggers. +optimised math libraries and various other libraries, analyzers and debuggers. -The Cray PE is made available through *environment modules* tha allow to select particular versions of +The Cray PE is made available through *environment modules* that allow to select particular versions of tools and to configure the environment in a flexible way. --- @@ -145,7 +145,7 @@ These libraries satisfy dependencies for many commonly used applications on Cray When the module for a CSML package (such as `cray-libsci` or `cray-fftw`) is loaded, all relevant headers and libraries for these packages are added to the compile and link lines of the `cc`, `ftn`, and `CC` compiler wrappers, so linking with them is -completely transparant (to the extent that users wonder where the libraries are). +completely transparent (to the extent that users wonder where the libraries are). The CSML collection contains the following Scientific Libraries: @@ -326,7 +326,7 @@ can be set through target modules: ``craype-network-ucx`` module. - The ``craype-hugepages*`` modules enable Cray Huge Pages support. To fully enable this support they have to - be used at link-time and at run-time. At link time, support is compiled into the binary wile at run-time they + be used at link-time and at run-time. At link time, support is compiled into the binary while at run-time they are used to set the actual size of the huge pages. The ``craype-hugepages*`` modules are not supported by all compilers. E.g., the AOCC compiler does not support @@ -375,7 +375,7 @@ is loaded first. ## Some unexpected behaviour of the Cray PE -On Cray EX systems, dynamic linking is the preferred way of linking applications. Whith that comes some +On Cray EX systems, dynamic linking is the preferred way of linking applications. With that comes some unexpected behaviour of the Cray modules. EasyBuild users expect that at run time the versions of the libraries that are used, are the ones from the modules that are loaded. This is not always the case for the runtime libraries of the Cray PE. By default the Cray PE will use a default set of libraries diff --git a/docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md b/docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md index 3bfae70c..4d91a9fd 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md @@ -5,7 +5,7 @@ --- The user-facing documentation on how to use the LUMI software stacks is -avialable in [the LUMI documentation](https://docs.lumi-supercomputer.eu/computing/softwarestacks/). +available in [the LUMI documentation](https://docs.lumi-supercomputer.eu/computing/softwarestacks/). On this page we focus more on the technical implementation behind it. --- diff --git a/docs/2022-CSC_and_LO/1_Intro/1_07_configuration.md b/docs/2022-CSC_and_LO/1_Intro/1_07_configuration.md index a1c2f3de..7e8e6858 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_07_configuration.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_07_configuration.md @@ -14,7 +14,7 @@ and to configure EasyBuild according to your preferences and the system on which !!! Note "EasyBuild configuration on LUMI" - On LUMI serveral configurations of EasyBuild are already available. + On LUMI several configurations of EasyBuild are already available. The ``EasyBuild-user`` module is the most important one. It will configure EasyBuild to install software in either a default location in the user's home directory @@ -104,7 +104,7 @@ and the default `software` and `modules/all` names for the subdirectories are us This makes it slightly easier to organise the module tree with user-friendly labeling, but above all also makes the synchronisation process of the 4 instances of the software directory more robust - as it is now easy to synchonise all modules in the last step, which is a much quicker process than + as it is now easy to synchronise all modules in the last step, which is a much quicker process than syncrhonising the software installations. We also use short paths for software installations (to avoid overrunning the maximum length of a diff --git a/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md index ade394ce..f292bf0c 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md @@ -524,7 +524,7 @@ Dry run: printing build status of easyconfigs and dependencies == Temporary directory /run/user/10012026/easybuild/tmp/eb-wm_bk3j6 has been removed. ``` -and now instal the library: +and now install the library: ```shell $ eb libdap-3.20.9-cpeGNU-21.12.eb @@ -782,7 +782,7 @@ sourcepath (E) = /users/XXXXXXXX/LUMI-user/sources:/appl/lumi/sources ... ``` -This is sligthly different from the default EasyBuild setup, where the modules, software, +This is slightly different from the default EasyBuild setup, where the modules, software, repository and sources would be installed in respectively the subdirectories `modules`, `software`, `ebfiles_repo` and `sources` of the directory pointed to by the `installpath` line. @@ -932,7 +932,7 @@ Currently Loaded Modules: S: Module is Sticky, requires --force to unload or purge ``` -Runnin `module --force purge` instead will remove all modules, including the `init-lumi` +Running `module --force purge` instead will remove all modules, including the `init-lumi` module which does part of the initialisation. You will not be able to use the software stacks completely as before without first loading `init-lumi` in its most recent (or default) version again! diff --git a/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md index 7ccd4389..66a357b6 100644 --- a/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md @@ -318,7 +318,7 @@ specific parameter is less useful on LUMI as we currently try to build on top of The `buildtools` build dependency shows that there is a fourth parameter specifying the toolchain used for that dependency and is needed if that toolchain is different from the one used in the example. -As it is not possible to load several Cray toolchains together (they are not in a hierachical relation) +As it is not possible to load several Cray toolchains together (they are not in a hierarchical relation) the only useful value on LUMI is `True` which tells that `buildtools` is build with the `SYSTEM` toolchain. Here also we use a template, `%(toolchain_version)s` which - as its name suggests - expands to the version of the toolchain, as we version our `buildtools` modules after the version of the Cray @@ -569,7 +569,7 @@ homepage = 'https://easybuilders.github.io/easybuild-tutorial' whatis = [ 'Description: EasyBuild tutorial example'] description = """ -This is a short C++ example program that can be buid using CMake. +This is a short C++ example program that can be build using CMake. """ ``` @@ -582,7 +582,7 @@ ERROR: Failed to process easyconfig /pfs/lustrep4/users/XXXXXXXX/easybuild-tutor No software-specific easyblock 'EB_eb_minus_tutorial' found for eb-tutorial ``` -It is not mandatory to specify an easyblock in the easyconfig. However, in the absense of that +It is not mandatory to specify an easyblock in the easyconfig. However, in the absence of that specification, EasyBuild goes looking for an application-specific easyblock with the standard name, in this case `EB_eb_minus_tutorial`, which it does not have. Does that mean we have to implement an easyblock? diff --git a/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md index 9373a1e8..d811778b 100644 --- a/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md +++ b/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md @@ -107,7 +107,7 @@ module location for a particular software name or easyblock class name. For exam ``` The location of the Python module is determined by whether the easyblock is generic or software-specific. -Generic easyblocks are located in the ``easybuid.easyblocks.generic`` namespace, while software-specific easyblocks +Generic easyblocks are located in the ``easybuild.easyblocks.generic`` namespace, while software-specific easyblocks live in the ``easybuild.easyblocks`` namespace directly. To keep things organised, the actual Python module files @@ -661,7 +661,7 @@ Your easyblock should: whatis = [ 'Description: EasyBuild tutorial example'] description = """ - This is a short C++ example program that can be buid using CMake. + This is a short C++ example program that can be build using CMake. """ toolchain = {'name': 'cpeCray', 'version': '21.12'} diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md index 0a5db233..aca3f0eb 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md @@ -21,7 +21,7 @@ It is important to be aware of some details before you start using this, which w temporary files and build directory. Those have to be made by hand. Due to the setup of the central software stack, this feature is currently useless to install - the central stack. For user installations, there are also limitations as the enviornment + the central stack. For user installations, there are also limitations as the environment on the compute nodes is different from the login nodes so, e.g., different locations for temporary files are being used. These would only be refreshed if the EasyBuild configuration modules are reloaded on the compute nodes which cannot be done currently in the way Slurm @@ -178,7 +178,7 @@ To remedy this, there are a couple of EasyBuild configuration options you can us The build directory of course also suffers from the problem of being no longer accessible if the installation fails, but there it is not so easy to find a solution. Building on a shared file system is not only much slower, but in particular on parallel file systems like GPFS/SpectrumScale, Lustre -or BeeGFS buiding sometimes fails in strange ways. One thing you can consider if you cannot do the +or BeeGFS building sometimes fails in strange ways. One thing you can consider if you cannot do the build on a login node (e.g., because the code is not suitable for cross-compiling or the configure system does tests that would fail on the login node), is to rety the installation in an interactive job, so you can inspect the build directory after the installation fails. diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md index 6c66b130..e60b885d 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md @@ -49,7 +49,7 @@ module naming schemes, which are characterized by: In contrast, a *hierarchical* module naming scheme consists of a *hierarchy* of module files. -A fairly typical 3-level schme (``Core``, ``Compiler`` and ``MPI``) has been +A fairly typical 3-level scheme (``Core``, ``Compiler`` and ``MPI``) has been discussed in the [section on Lmod](../1_Intro/1_02_Lmod#lmod-hierarchy). This typical Lmod hierarcny would map very well on the EasyBuild common toolchains. @@ -67,7 +67,7 @@ would be installed using the regular ``foss`` toolchain or the ``gompi`` toolcha On LUMI, where software is installed through the Cray Programming Environment with no real choice of MPI implementation, a two-level arrangement would still make a lot of sense, with at the ``Core`` level all software compiled with the SYSTEM toolchain while there could be a ``PrgEnv`` level for software -compiled with a particula programming enviornment aka cepGNU/cpeCray/cpeAOCC toolchain. Such a scheme +compiled with a particula programming environment aka cepGNU/cpeCray/cpeAOCC toolchain. Such a scheme is used on the Cray systems at CSCS. To recap, the characteristics of a module hierarchy are: @@ -164,7 +164,7 @@ loaded on LUMI this was not possible to accomplish without losing other function Next to the module naming schemes that are included with EasyBuild, you can also define your own module naming scheme (MNS), and configure EasyBuild to use it -(which is eactly what has been done on LUMI to remove a feature of the default ``EasyBuildMNS`` scheme +(which is exactly what has been done on LUMI to remove a feature of the default ``EasyBuildMNS`` scheme that we do not use). ### Implementation @@ -288,7 +288,7 @@ $ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D !!! Warning "Example not suitable for LUMI" **This exercise is meant for a system where the common toolchains can be used and requires an - indpendent EasyBuild installation in your personal file space**, + independent EasyBuild installation in your personal file space**, because EasyBuild will try to copy the installation log file to each installation directory. Now that we know more about hierarchical module naming schemes, diff --git a/docs/2022-CSC_and_LO/4_00_additional_reading.md b/docs/2022-CSC_and_LO/4_00_additional_reading.md index f393c097..83988f8e 100644 --- a/docs/2022-CSC_and_LO/4_00_additional_reading.md +++ b/docs/2022-CSC_and_LO/4_00_additional_reading.md @@ -11,7 +11,7 @@ EasyBuild. - [EasyBuild YouTube channel](https://www.youtube.com/c/EasyBuilders) - This tutorial is an evolution of the - [EasyBuild tutorial prepared for the LUMI User Suppport Team, spring '21](https://easybuilders.github.io/easybuild-tutorial/2021-lust/) + [EasyBuild tutorial prepared for the LUMI User Support Team, spring '21](https://easybuilders.github.io/easybuild-tutorial/2021-lust/) given by Kenneth Hoste (UGent, EasyBuild lead developer) and Luca Marsella (CSCS) - [Recordings are available on YouTube](https://www.youtube.com/watch?v=JTRw8hqi6x0&list=PLhnGtSmEGEQh573bk3BeOj_KCRBBiA5OT) - The EasyBuild setup on LUMI is partly insprired on the setup used at CSCS on their Cray systems From 22772b63d11aefcb80370862dca44ab01a5d71dc Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 8 Apr 2022 18:40:07 +0200 Subject: [PATCH 040/159] Removed a new line from a module file as it caused problems with the codespell check that is run automatically. --- docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md index bc1ed006..70f116c2 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md @@ -708,7 +708,7 @@ still fail to remove the 2.7 directory from ``PYTHONPATH``. 1. The files ``level1/MyPython/2.7.18.lua`` and ``level1/MyPython/3.6.10.lua``, both with the same contents: ``` lua - LmodMessage( '\nIn ' .. myModuleFullName() .. ' in mode ' .. mode() ) + LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) local api_version = myModuleVersion():match( '(%d+%.%d+)%..*' ) @@ -724,7 +724,7 @@ still fail to remove the 2.7 directory from ``PYTHONPATH``. 2. The files ``level2/PythonAPI/2.7/MyPythonPackage/1.0.lua`` and ``level2/PythonAPI/3.6/MyPythonPackage/1.0.lua``, both with the contents: ``` lua - LmodMessage( '\nIn ' .. myFileName() .. ' in mode ' .. mode() ) + LmodMessage( 'In ' .. myFileName() .. ' in mode ' .. mode() ) local python_api_version = myFileName():match( '.*/level2/PythonAPI/([^/]+)/.*' ) LmodMessage( 'Detected Python API version from hierarchy: ' .. python_api_version ) From 6ea43a8c3c4ba3b04e86c4a0fa6fecf97f908e11 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 21 Apr 2022 11:08:18 +0200 Subject: [PATCH 041/159] Removed some TODOs to complete the tutorial. --- .../1_Intro/1_01_what_is_easybuild.md | 6 +-- .../1_Intro/1_04_LUMI_software_stack.md | 46 +++++++++++++++---- .../1_Intro/1_06_installation.md | 12 ++--- .../1_Intro/1_08_basic_usage.md | 1 - 4 files changed, 45 insertions(+), 20 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md index 6f169754..87e25f79 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md @@ -103,16 +103,14 @@ cannot easily overwrite those. In addition to performance, **reproducibility of installations** is a core aspect of EasyBuild. -TODO correct link in the next lines. - -Most software installations performed with EasyBuild use a **particular compiler toolchain**, +Most software installations performed with EasyBuild use a **particular compiler toolchain**, with which we aim to be in control over the build environment and avoid relying on tools and libraries provided by the operating system. For similar reasons, we try to **provide all required dependencies through EasyBuild** as well, with a few notable exceptions, like ``OpenSSL`` for security reasons, and Infiniband and GPU drivers which are too closely intertwined with the operating system. For both toolchains and dependencies, **fixed software versions** are specified in the -easyconfig files. That way, easyconfig files can easily be shared with others: +easyconfig files. That way, easyconfig files can easily be shared with others: if they worked for you it is very likely that they will work for others too, because the vast majority of the software stack is controlled by EasyBuild. diff --git a/docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md b/docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md index 4d91a9fd..90ff7032 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md @@ -15,7 +15,34 @@ On this page we focus more on the technical implementation behind it. LUMI has different node types providing compute resources: - LUMI has 16 login nodes, though many of those are reserved for special purposes and not - available to all users. TODO + available to all users. These login nodes have a zen2 CPU. These nodes have a SlingShot 10 + interconnect. +- There are 1536 regular CPU compute nodes in a partition denoted as LUMI-C. These + compute nodes have a zen3 CPU and run a reduced version of SUSE Linux optimised + by Cray to reduce OS jitter. These nodes will in the future be equipped with a + SlingShot 11 interconnect card. +- There are 2560 GPU compute nodes in a partition denoted as LUMI-G. These nodes have + a single zen3-based CPU with optimised I/O die linked to 4 AMD MI250X GPUs. Each node + has 4 SlingShot 11 interconnect cards, one attached to each GPU. +- The interactive data analytics and visualisation partition is really two different partitions + from the software point-of-view: + - 8 nodes are CPU-only but differ considerably from the regular compute nodes, + not only in the amount of memory. These nodes are equipped with zen2 CPUs + and in that sense comparable to the login nodes. They also have local SSDs + and are equipped with SlingShot 10 interconnect cards (2 each???) + - 8 nodes have zen2 CPUs and 8 NVIDIA A40 GPUs each, and have 2 SlingShot 10 + interconnect cards each. +- The early access platform (EAP) has 14 nodes equiped with a single 64-core + zen2 CPU and 4 AMD MI100 GPUS. Each node has a single SlingShot 10 interconnect + and also local SSDs. + +SlingShot 10 and SlingShot 11 are different software-wise. SlingShot 10 uses a +Mellanox CX5 NIC that support both OFI and UCX, and hence can also use the +UCX version of Cray MPICH. SlingShot 11 uses a NIC code-named Cassini and +supports only OFI with an OFI provider specific for the Cassini NIC. However, +given that the nodes that are equipped with SlingShot 10 cards are not meant +to be used for big MPI jobs, we build our software stack solely on top of +libfabric and Cray MPICH. --- @@ -46,7 +73,9 @@ On LUMI, two types of software stacks are currently offered: A deliberate choice was made to only offer a limited number of software packages in the globally installed stack as the setup of redundancy on LUMI makes it difficult to update the stack in a way that is guaranteed to not - affect running jobs and as a large central stack is also hard to manage. + affect running jobs and as a large central stack is also hard to manage, especially + as we expect frequent updates to the OS and compiler infrastructure in + the first years of operation. However, the EasyBuild setup is such that users can easily install additional software in their home or project directory using EasyBuild build recipes that we provide or they develop, and that software will fully @@ -72,12 +101,13 @@ set of target modules after loading the ``CrayEnv`` module. The ``LUMI`` module currently supports four partition modules, but that number may be reduced in the future: -| Partition | CPU target | Accelerator | -|:----------------|-----------------------|:----------------------------| -| ``partition/L`` | ``craype-x86-rome`` | ``craype-accel-host`` | -| ``partition/C`` | ``craype-x86-milan`` | ``craype-accel-host`` | -| ``partition/G`` | ``craype-x86-trento`` | ``craype-accel-amd-gfx90a`` | -| ``partition/D`` | ``craype-x86-rome`` | ``craype-accel-nvidia80`` | +| Partition | CPU target | Accelerator | +|:------------------|-----------------------|:----------------------------| +| ``partition/L`` | ``craype-x86-rome`` | ``craype-accel-host`` | +| ``partition/C`` | ``craype-x86-milan`` | ``craype-accel-host`` | +| ``partition/G`` | ``craype-x86-trento`` | ``craype-accel-amd-gfx90a`` | +| ``partition/D`` | ``craype-x86-rome`` | ``craype-accel-nvidia80`` | +| ``partition/EAP`` | ``craype-x86-rome`` | ``craype-accel-amd-gfx908`` | All ``partition`` modules also load `craype-network-ofi``. diff --git a/docs/2022-CSC_and_LO/1_Intro/1_06_installation.md b/docs/2022-CSC_and_LO/1_Intro/1_06_installation.md index 24c71d5e..10ab889a 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_06_installation.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_06_installation.md @@ -417,6 +417,8 @@ export EB_PYTHON=python3 To keep the different versions of the LUMI software stack as independent from one another as possible, EasyBuild is bootstrapped for each software stack by the ``prepare_LUMI_stack.sh`` script. We use the system Python 3 for running EasyBuild. That Python version currently doesn't have ``pip`` installed. +Our procedure is based on the ["Installing EasyBuild with EasyBuild" procedure](#method-2-installing-easybuild-with-easybuild) +but works around the lack of `pip`. EasyBuild is first installed in a temporary work directory from files downloaded from PyPi. These are each untarred and then installed into their location by running ``python3 setup.py install --prefix=...`` in their unpacked directory. @@ -424,16 +426,11 @@ We do so only for the framework and easyblocks files as the easconfig files are install EasyBuild. Instead we create our own EasyConfig file for EasyBuild which contains some additional packages that enable extra features in EasyBuild and also provide more information to Lmod. Next the configuration module for EasyBuild (see the next section, -["Configuring EasyBuild"](1_07_configuration), for more information) and use the temporary +["Configuring EasyBuild"](../1_07_configuration), for more information) and use the temporary installation of EasyBuild with our own EasyConfig file to do a proper installation of EasyBuild with module in the final location. -TODO: follow development setup but with downloading rather than cloning, then use that one with the -EasyBuild configuration module to install EasyBuild with itself. - - - ## Verifying the installation Regardless of how EasyBuild was installed, you can now run a couple of basic commands to verify the installation: @@ -447,7 +444,8 @@ To check which EasyBuild version you have installed, run: eb --version ``` -The output should match with the latest EasyBuild version. +The latest version (and an overview of previous versions) can always be +[found on PyPi](https://pypi.org/project/easybuild/). #### Consulting the help output diff --git a/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md index f292bf0c..e502c47a 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md @@ -965,7 +965,6 @@ won't go into those here). ## Exercises -TODO: Rework for LUMI and provide the necessary EasyConfig files. ***Guidelines*** From d1f6c7bf39f5685ff1e7bcc2ed519515540a794b Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 21 Apr 2022 12:38:27 +0200 Subject: [PATCH 042/159] Corrected and updated several links. --- .../1_Intro/1_01_what_is_easybuild.md | 16 ++++++++-------- docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md | 4 ++-- docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md | 5 +++-- docs/2022-CSC_and_LO/1_Intro/1_05_terminology.md | 7 +++++-- .../2022-CSC_and_LO/1_Intro/1_06_installation.md | 6 +++--- .../1_Intro/1_07_configuration.md | 12 ++++++------ docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md | 4 ++-- .../2_Using/2_01_troubleshooting.md | 5 +++-- .../2_Using/2_02_creating_easyconfig_files.md | 12 ++++++------ .../2_Using/2_04_implementing_easyblocks.md | 4 ++-- .../3_Advanced/3_03_slurm_jobs.md | 4 ++-- .../3_Advanced/3_04_module_naming_schemes.md | 8 ++++---- .../3_Advanced/3_05_github_integration.md | 13 +++++++------ docs/2022-CSC_and_LO/4_00_additional_reading.md | 3 +-- 14 files changed, 54 insertions(+), 49 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md index 87e25f79..4e125bb6 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md @@ -4,14 +4,14 @@

EasyBuild logo

-**[EasyBuild](http://easybuild.io)** is a **software build and installation framework** +**[EasyBuild](https://easybuild.io)** is a **software build and installation framework** that manages (scientific) software in an efficient way, without compromising on performance. Easybuild is especially tailored towards [**High Performance Computing (HPC)**](https://en.wikipedia.org/wiki/Supercomputer) systems, and provides a **command-line interface** to facilitate the installation and management of a scientific software stack. --- - + @@ -68,7 +68,7 @@ as it is performed. EasyBuild supports using a **custom module naming scheme**, allows for *hierarchical* module naming schemes, and **integrates with various other tools** ranging from resource managers ([Slurm](https://slurm.schedmd.com) and [GC3Pie](https://github.com/gc3pie/gc3pie)), -container tools ([Singularity](https://github.com/hpcng/singularity) and [Docker](https://www.docker.com)), +container tools ([Singularity](https://github.com/apptainer/singularity) and [Docker](https://www.docker.com)), packaging tools ([FPM](https://fpm.readthedocs.io)), and so on. --- @@ -127,7 +127,7 @@ and to [**contribute back**](https://docs.easybuild.io/en/latest/Contributing.ht by opening pull requests to the [GitHub repositories](https://github.com/easybuilders) or the [documentation](https://docs.easybuild.io). The EasyBuild community also provides packages for two standard toolchains, called the -[**common toolchains**](https://easybuild.readthedocs.io/en/latest/Common-toolchains.html). +[**common toolchains**](https://docs.easybuild.io/en/latest/Common-toolchains.html). The so-called ``foss`` is based on free and open software (GNU compilers, FlexiBlas with OpenBLAS backend, LaPACK, FFTW and Open MPI) while the ``intel`` toolchain uses the full stack of Intel compilers, libraries and MPI. Currently these toolchains cannot be supported on LUMI though as getting Open MPI to run on LUMI is challenging and as the Intel libraries and @@ -137,7 +137,7 @@ Focusing the efforts on those two popular common toolchains makes it possible to collected in the [central repository](https://github.com/easybuilders/easybuild-easyconfigs). They can still serve as a basis to develop build recipes for the HPE Cray Programming Environment. -Last but not least, EasyBuild provides various [**GitHub integration features**](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) +Last but not least, EasyBuild provides various [**GitHub integration features**](https://docs.easybuild.io/en/latest/Integration_with_GitHub.html) that greatly facilitate the contribution process: opening, updating, and testing pull requests, reviewing incoming contributions, and much more can all be done directly from the EasyBuild command line. This not only saves time, effort, brain cycles, and mouse clicks for contributors, @@ -225,12 +225,12 @@ sites leverage it, and integrate it with other tools like [Jenkins](https://www. HPC sites, consortia, and companies that use EasyBuild include (but are not limited to): -VSC logo +VSC logo JSC logo HPCNow! logo Fred Hutch logo Vienna BioCenter logo -CÉCI logo +CÉCI logo Compute Canada logo UiO logo BEAR logo @@ -242,7 +242,7 @@ HPC sites, consortia, and companies that use EasyBuild include (but are not limi University of Melbourne logo University of Luxembourg logo EPCC logo -Cyprus Institute logo +Cyprus Institute logo University of Basel logo Microsoft logo diff --git a/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md index 70f116c2..fed386d1 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md @@ -78,7 +78,7 @@ to a hierarchy with 3 levels: Here is a simple example of such a 3-level module hierarchy (that almost could have been generated by EasyBuild): -
+
In this example the ``Core`` level only includes a single module `GCC/9.3.0`, while the ``Compiler`` level includes two modules: `OpenMPI/4.0.3` and `MPICH/3.3.2`. @@ -268,7 +268,7 @@ is included in the module files to improve discoverability of software. !!! Note "Documentation in the LUMI documentation" Extensive information on search commands with examples of how to use them on LUMI can be found - in the [LUMI documentation]([https://)](https://docs.lumi-supercomputer.eu/), in + in the [LUMI documentation](https://docs.lumi-supercomputer.eu/), in [the computing section, "Module environment page", "Finding modules" section](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/#finding-modules). diff --git a/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md b/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md index 9502afaa..5930e2cb 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md @@ -122,7 +122,8 @@ LUMI environment and the integration with the Cray environment still seems to be The Cray PE also provides a programming environment module to enable the Intel® oneAPI compiler and tools. -The documentation is available in the [Intel® oneAPI Programming Guide](https://software.intel.com/content/www/us/en/develop/documentation/oneapi-programming-guide/top/oneapi-development-environment-setup.html) +The documentation is available in the +[Intel® oneAPI Programming Guide](https://www.intel.com/content/www/us/en/develop/documentation/oneapi-programming-guide/top/oneapi-development-environment-setup.html) Cray provides a bundled package of support libraries to install into the Cray PE to enable the Intel compiler, allowing utilities such as debuggers and performance tools to work with it. @@ -166,7 +167,7 @@ The CSML collection contains the following Scientific Libraries: * Manpages: `intro_mpi` -* Website: [http://www.mpi-forum.org](http://www.mpi-forum.org) +* Website: [http://www.mpi-forum.org](https://www.mpi-forum.org) MPI is a widely used parallel programming model that establishes a practical, portable, efficient, and flexible standard for passing messages between ranks in parallel processes. diff --git a/docs/2022-CSC_and_LO/1_Intro/1_05_terminology.md b/docs/2022-CSC_and_LO/1_Intro/1_05_terminology.md index 8e05012f..dbb7b8ff 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_05_terminology.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_05_terminology.md @@ -85,8 +85,11 @@ The `foss` toolchain consists of all open source components (hence the name: In recent versions (since 2021a), [FlexiBLAS](https://www.mpi-magdeburg.mpg.de/projects/flexiblas) is used as the BLAS library, with OpenBLAS and LAPACK as the backend. -The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version -controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. +The `intel` toolchain consists of the +[Intel C, C++ and Fortran compilers](https://www.intel.com/content/www/us/en/developer/tools/oneapi/dpc-compiler.html) +(on top of a `GCC` version controlled through EasyBuild) alongside the +[Intel MPI](https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html) and +[Intel MKL](https://www.intel.com/content/www/us/en/developer/tools/oneapi/onemkl.html) libraries. Roughly every 6 months, a new version of these common toolchains is agreed upon in the EasyBuild community, after extensive testing. diff --git a/docs/2022-CSC_and_LO/1_Intro/1_06_installation.md b/docs/2022-CSC_and_LO/1_Intro/1_06_installation.md index 10ab889a..4d1ac6c5 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_06_installation.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_06_installation.md @@ -85,11 +85,11 @@ If you see output that starts with a line like "`VERSION=3.2.10`" or "`Modules R you have the original Tcl-based environment modules tool installed, either in its original C-based imnplementation (``VERSION=3.2.10``) or the new Tcl-based implementation developed at CEA (``Modules Release 4.5.0``) -and *EasyBuild will need to be configured to use it after installation*. +and *EasyBuild will need to be configured to use it after installation*. If the `module` function is not defined either you do not have a modules tool installed or your environment is not properly set up to use it. In this case, -please refer to the EasyBuild documentation here for more information. +please refer to the EasyBuild documentation here for more information. HPE Cray supports both the old-style Environments Modules (based on version 3.2.10) and Lmod. On LUMI, we use Lmod as the default module system for users, and EasyBuild is configured to use that one. @@ -466,7 +466,7 @@ eb --show-config ``` This should tell you that EasyBuild (ab)uses `$HOME/.local/easybuild` as a default location. -More on configuring EasyBuild in the [next part of the tutorial](configuration). +More on configuring EasyBuild in the [next part of the tutorial](../1_07_configuration). #### System information diff --git a/docs/2022-CSC_and_LO/1_Intro/1_07_configuration.md b/docs/2022-CSC_and_LO/1_Intro/1_07_configuration.md index 7e8e6858..6d8e6acb 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_07_configuration.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_07_configuration.md @@ -186,7 +186,7 @@ archive, the location of which is specified by the `repositorypath` configuratio By default the specified path is assumed to be a regular directory, but using a Git repository as easyconfigs archive is also supported (for more details, see -[the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Configuration.html#easyconfigs-repository-repository-repositorypath)). +[the EasyBuild documentation](https://docs.easybuild.io/en/latest/Configuration.html#easyconfigs-repository-repository-repositorypath)). --- @@ -204,7 +204,7 @@ are using a different modules tool; see the output of `eb --avail-modules-tools` modules tools. Note that for anything other than Lmod, you *must* make sure that the actual modules tool binary command is available through `$PATH` (more information -on this [in the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Configuration.html#modules-tool-modules-tool)). +on this [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Configuration.html#modules-tool-modules-tool)). If you prefer that EasyBuild generates module files in [Tcl](https://tcl.tk/) syntax, you can instruct it to do so via the `module-syntax` configuration setting. Note that this is *required* if you are using @@ -246,7 +246,7 @@ software.*** There are ways around this however, which are outside of the scope of this tutorial. For more information, - [see the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Using_the_EasyBuild_command_line.html#controlling-robot-search-path). + [see the EasyBuild documentation](https://docs.easybuild.io/en/latest/Using_the_EasyBuild_command_line.html#controlling-robot-search-path). *On LUMI the included easyconfig files cannot be used without changes as they are for the common toolchains which are currently not supported in the LUMI software stacks.* @@ -280,7 +280,7 @@ The default `EasyBuildMNS` module naming scheme roughly corresponds to the filen and consists of the software name followed by a combination of the software version, toolchain and an optional label (which corresponds to the value of the `versionsuffix` easyconfig parameter): `/<-toolchain>`. Just like with [names of easyconfig -files](1_05_terminology#easyconfig-files), the `<-toolchain>` part is omitted when the `system` toolchain is used, +files](../1_05_terminology#easyconfig-files), the `<-toolchain>` part is omitted when the `system` toolchain is used, and the `` value is empty by default. !!! Note "Module naming scheme on LUMI" @@ -366,9 +366,9 @@ across different software installations. They define *configuration settings*, such as the location where software should be installed, or the syntax that should be used when generating module files. -An [easyconfig file](1_05_terminology#easyconfig-files) (`*.eb`) on the other hand *specifies the details for one particular software installation*. +An [easyconfig file](../1_05_terminology#easyconfig-files) (`*.eb`) on the other hand *specifies the details for one particular software installation*. It does this by defining a set of *easyconfig parameters*, which tell EasyBuild the name and version -of the software to install, which [toolchain](1_05_terminology#toolchains) and [easyblock](1_05_terminology#easyblocks) to use, etc. +of the software to install, which [toolchain](../1_05_terminology#toolchains) and [easyblock](../1_05_terminology#easyblocks) to use, etc. For each software installation performed by EasyBuild, there is a corresponding easyconfig file. There typically are only a handful of configuration files used however, for example a system-level diff --git a/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md index e502c47a..dda28b97 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md @@ -189,7 +189,7 @@ operations involving lots of (small) files can be slow. For the easyconfig files included with an EasyBuild release, a search index is readily provided. For other directories, you can create a search index using `eb --create-index `. -[See the EasyBuild documentation for more information](https://easybuild.readthedocs.io/en/latest/Easyconfigs_index.html). +[See the EasyBuild documentation for more information](https://docs.easybuild.io/en/latest/Easyconfigs_index.html). --- @@ -307,7 +307,7 @@ are already installed, and which ones are still missing. This can be helpful to ensure that your EasyBuild configuration is set up correctly, and to prevent from accidentally installing an entirely new software stack from scratch in an unintended location (like `$HOME/.local/easybuild`, the -[default installation path](../configuration/#install-path)). +[default installation path](../1_07_configuration/#install-path)). --- diff --git a/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md b/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md index 88509b3d..0f1b5c3c 100644 --- a/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md +++ b/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md @@ -28,7 +28,8 @@ Things that could go wrong during an installation include: * required dependencies that are not specified in the easyconfig file; * failing shell commands; * running out of available memory or disk space; -* a segmentation fault caused by a flipped bit triggered by a cosmic ray ([really, it happens!](https://blogs.oracle.com/linux/attack-of-the-cosmic-rays-v2)); +* a segmentation fault caused by a flipped bit triggered by a cosmic ray + ([really, it happens!](https://blogs.oracle.com/linux/post/attack-of-the-cosmic-rays)); Unfortunately this is not an exhaustive list, there are plenty of other potential problems that could result in a failing installation... @@ -219,7 +220,7 @@ Let's work our way through a less smooth software installation, using the easyconfig file that is provided below. Don't worry if most of this is still unclear to you, we'll get -to writing easyconfig files from scratch [later in this tutorial](2_02_creating_easyconfig_files). +to writing easyconfig files from scratch [later in this tutorial](../2_02_creating_easyconfig_files). ```python easyblock = 'MakeCp' diff --git a/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md index 66a357b6..2da0fdd0 100644 --- a/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md @@ -58,7 +58,7 @@ Reasons to consider implementing a software-specific easyblock rather than using bootstrapping with the system compiler and then re-installing with itself.* For implementing easyblocks we refer to the -["Implementing easyblocks" section of this tutorial](2_03_implementing_easyblocks) +["Implementing easyblocks" section of this tutorial](../2_04_implementing_easyblocks) and the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). ## Writing easyconfig files @@ -66,7 +66,7 @@ and the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementi Writing an easyconfig file boils down to defining a set of easyconfig parameters in a text file, which we give a '`.eb`' extension by convention. The name of an easyconfig file doesn't matter when using it directly to install software, but it does matter when EasyBuild needs to find it -to resolve a dependency for example (as we [discussed earlier](../1_08_basic_usage/#easyconfig-filenames)). +to resolve a dependency for example (as we [discussed earlier](../../1_Intro/1_08_basic_usage/#easyconfig-filenames)). The syntax for easyconfig files is *Python syntax*: you are basically defining a bunch of Python variables that correspond to easyconfig parameters. @@ -469,7 +469,7 @@ moduleclass = 'lib' ``` This is done to categorize software, and it is used to group the generated module files into smaller sets -([remember what we saw when installing software earlier](../1_08_basic_usage/#using-installed-software)). +([remember what we saw when installing software earlier](../../1_Intro/1_08_basic_usage/#using-installed-software)). This is currently not used on LUMI since we feel that (a) it is not easy to explain to users how they can then only make certain classes that are useful to them visible and more importantly (b) since it is not @@ -590,9 +590,9 @@ In this simple case it doesn't, since we can leverage one of the available *gene But, which one? Build instructions are usually included in a `README` file, or in the documentation. -In this case, there's indeed a minimal [`README` -file](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1/README) available, -which tells us that we should use the `cmake` command to configure the installation, followed by `make` and `make install`. +In this case, there's indeed a minimal +[`README` file](https://github.com/easybuilders/easybuild-tutorial/blob/main/docs/files/eb-tutorial-1.0.1/README) +available, which tells us that we should use the `cmake` command to configure the installation, followed by `make` and `make install`. [We briefly discussed](#easyblock) a generic easyblock that does exactly this: `CMakeMake`. diff --git a/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md index d811778b..1a9cb29f 100644 --- a/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md +++ b/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md @@ -114,7 +114,7 @@ To keep things organised, the actual Python module files for software-specific easyblocks are kept in 'letter' subdirectories, rather than in one large '``easyblocks``' directory (see -[https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/](https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/)), +[https://github.com/easybuilders/easybuild-easyblocks/tree/main/easybuild/easyblocks](https://github.com/easybuilders/easybuild-easyblocks/tree/main/easybuild/easyblocks)), but this namespace is collapsed transparently by EasyBuild (you don't need to import from letter subpackages). To let EasyBuild pick up one or more new or customized easyblocks, you can use the [``--include-easyblocks``](https://docs.easybuild.io/en/latest/Including_additional_Python_modules.html#including-additional-easyblocks-include-easyblocks) @@ -448,7 +448,7 @@ class EB_Example(EasyBlock): ## Compatibility with ``--extended-dry-run`` and ``--module-only`` Some special care must be taken to ensure that an easyblock is fully compatible with ``--extended-dry-run`` / ``-x`` -(see [Inspecting install procedures](../basic_usage/#inspecting-install-procedures)) and ``--module-only``. +(see [Inspecting install procedures](../../1_Intro/1_08_basic_usage/#inspecting-install-procedures)) and ``--module-only``. For compatibility with ``--extended-dry-run``, you need to take into account that specified operations like manipulating files or running shell commands will not actually be executed. You can check diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md index aca3f0eb..ec7a36a5 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md @@ -37,8 +37,8 @@ will be *passed down* into the submitted job automatically, via command line opt command that is run in the job script. This includes not only command line options used next to ``--job``, but also configuration settings -that are specified via an [EasyBuild configuration file](../1_Intro/1_07_configuration#configuration-files) or through -[``$EASYBUILD_*`` environment variables](../1_Intro/1_07_configuration#easybuild_-environment-variables). +that are specified via an [EasyBuild configuration file](../../1_Intro/1_07_configuration#configuration-files) or through +[``$EASYBUILD_*`` environment variables](../../1_Intro/1_07_configuration#easybuild_-environment-variables). This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables that are in place in the job environment are most likely *irrelevant*, since configuration settings diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md index e60b885d..4a6346c5 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md @@ -50,7 +50,7 @@ module naming schemes, which are characterized by: In contrast, a *hierarchical* module naming scheme consists of a *hierarchy* of module files. A fairly typical 3-level scheme (``Core``, ``Compiler`` and ``MPI``) has been -discussed in the [section on Lmod](../1_Intro/1_02_Lmod#lmod-hierarchy). +discussed in the [section on Lmod](../../1_Intro/1_02_Lmod#lmod-hierarchy). This typical Lmod hierarcny would map very well on the EasyBuild common toolchains. In fact, for the example @@ -62,7 +62,7 @@ toolchains. Software at the ``Compiler`` level would be installed with the ``GCC toolchain, while software at the ``MPI`` level after loading the ``OpenMPI`` module would be installed using the regular ``foss`` toolchain or the ``gompi`` toolchain (see the diagram in the -["Common toolchains" section of the page on terminology](../1_Intro/1_05_terminology/#common-toolchains)). +["Common toolchains" section of the page on terminology](../../1_Intro/1_05_terminology/#common-toolchains)). On LUMI, where software is installed through the Cray Programming Environment with no real choice of MPI implementation, a two-level arrangement would still make a lot of sense, with at the ``Core`` level @@ -137,7 +137,7 @@ and so these modules can not be loaded together easily. One downside of a module hierarchy is that not all existing modules are directly available for loading or are even *visible* to the user, since the output of "`module avail`" only shows a subset of all modules. -As we have discussed in [the Lmod section](../LMOD). +As we have discussed in [the Lmod section](../../1_Intro/1_02_Lmod). [Lmod](https://lmod.readthedocs.io) provides a powerful solution to search for modules through the ``module spider`` and to some extent also the ``module keyword`` commands. So as long as the end users are @@ -325,7 +325,7 @@ a module. We have just made all modules unavailable, so we would have to first install EasyBuild again in our hierarchical module tree before we can continue. **We strongly recommend using an EasyBuild installation that was [installed via "`pip install`" -or "`pip3 install`"](../1_Intro/1_06_installation#method-1-using-pip) in this part of the tutorial.** +or "`pip3 install`"](../../1_Intro/1_06_installation#method-1-using-pip) in this part of the tutorial.** An easy way to do this is in the prepared environment is to run: diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md index 3a9732e0..7495343a 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md @@ -44,7 +44,7 @@ git checkout -b example ``` 3) Stage the changes you want to contribute, after you make sure that your easyconfig file has the -[correct filename](../basic_usage/#easyconfig-filenames), and that it's located in the appropriate directory. +[correct filename](../../1_Intro/1_08_basic_usage/#easyconfig-filenames), and that it's located in the appropriate directory. ```shell mkdir -p easybuild/easyconfigs/e/example/ @@ -95,13 +95,13 @@ a project that aims to *automate* tedious software installation procedures. At the end of 2015 efforts were made to tackle this issue by implementing GitHub integration features in EasyBuild, which automate the contribution -workflow by running `git` commands and interacting with the [GitHub API](https://developer.github.com/v3/). +workflow by running `git` commands and interacting with the [GitHub API](https://docs.github.com/en/rest). -We will briefly go over some of these features here, but they are also covered in detail [in the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html). +We will briefly go over some of these features here, but they are also covered in detail [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Integration_with_GitHub.html). ### Requirements & configuration -First of all, the GitHub integration features impose a couple of additional [requirements](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) +First of all, the GitHub integration features impose a couple of additional [requirements](https://docs.easybuild.io/en/latest/Integration_with_GitHub.html) and configuration. @@ -223,7 +223,8 @@ to try out the GitHub integration features! If your SSH private key is protected with a password, you may need to enter your password a couple of times when running "`eb --check-github`". - You can avoid this by [using an SSH agent](https://help.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent). + You can avoid this by + [using an SSH agent](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent). !!! Note If you see the ``push access`` check fail with ``Failed to fetch branch 'main'``, @@ -313,7 +314,7 @@ will include information about your system (OS, processor, etc.) and your EasyBu version of the tutorial. That is a lot to digest, so let us make this a bit more concrete with an example: -we will open a pull request for the [`eb-tutorial` example software](../adding_support_software/#example) to *a fork* of the [`easybuild-easyconfigs` repository](https://github.com/easybuilders/easybuild-easyconfigs) using the `eb` command, +we will open a pull request for the [`eb-tutorial` example software](../../2_Using/2_02_creating_easyconfig_files/#example) to *a fork* of the [`easybuild-easyconfigs` repository](https://github.com/easybuilders/easybuild-easyconfigs) using the `eb` command, and submit a test report in it. !!! Note diff --git a/docs/2022-CSC_and_LO/4_00_additional_reading.md b/docs/2022-CSC_and_LO/4_00_additional_reading.md index 83988f8e..f14ba6fb 100644 --- a/docs/2022-CSC_and_LO/4_00_additional_reading.md +++ b/docs/2022-CSC_and_LO/4_00_additional_reading.md @@ -21,10 +21,9 @@ during the [6th EasyBuild User Meeting in 2021](https://easybuild.io/eum21/) - CSCS site presentation ([video](https://www.youtube.com/watch?v=rhoHmUhzSGw&list=PLhnGtSmEGEQidEM8MZKkOaVutgt9WmqI0)/[slides](https://users.ugent.be/~kehoste/eum20/eum20_11_luca_cscs.pdf)) during the [5h EasyBuild User Meeting in 2020](https://github.com/easybuilders/easybuild/wiki/5th-EasyBuild-User-Meeting) - - CSCS site presentation ([video](https://youtu.be/qLuHeaFmydM?list=PLhnGtSmEGEQhEjG5hD70tRwL3n58aWNHc&t=1451)) + - CSCS site presentation ([video](https://www.youtube.com/watch?list=PLhnGtSmEGEQhEjG5hD70tRwL3n58aWNHc&t=1451&v=qLuHeaFmydM&feature=youtu.be)) during the [4h EasyBuild User Meeting in 2020](https://github.com/easybuilders/easybuild/wiki/4th-EasyBuild-User-Meeting) - ## HPE Cray Programming Environment - [LUMI documentation: "Developing" section](https://docs.lumi-supercomputer.eu/development/) From 8e0353a73eabaa1bdea329a628e2c1dd1600a17a Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 9 May 2022 18:26:57 +0200 Subject: [PATCH 043/159] Various corrections (typos etc.) and minor additions. --- .../1_Intro/1_01_what_is_easybuild.md | 2 +- docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md | 4 +- .../1_Intro/1_04_LUMI_software_stack.md | 2 +- .../1_Intro/1_08_basic_usage.md | 28 +++++++------ .../2_Using/2_02_creating_easyconfig_files.md | 39 +++++++++++++++---- 5 files changed, 51 insertions(+), 24 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md index 4e125bb6..3059cd82 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md @@ -95,7 +95,7 @@ By default EasyBuild tries to run tests if the build process of a package provid functionality, or can run simple tests of its own, and of course these tests will fail if the host CPU does not support the full instruction set of the target CPU. Secondly, more and more software has broken build processes where developers thought that it was a good -idea to hard-code options such as ``-march=native`` or ``--xHost`` in a way that EasyBuild +idea to hard-code options such as ``-march=native`` or ``-xHost`` in a way that EasyBuild cannot easily overwrite those. diff --git a/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md b/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md index 5930e2cb..8eb10e2c 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md @@ -13,7 +13,7 @@ tools and to configure the environment in a flexible way. --- -## Cray PE components +## HPE Cray PE components ### Cray compiler environments @@ -185,9 +185,7 @@ versions of the library are supported, the finalised LUMI system will only suppo ### DSMML * Modules: `cray-dsmml` -* * Manpages: `intro_dsmml` -* * Website: [https://pe-cray.github.io/cray-dsmml](https://pe-cray.github.io/cray-dsmml) Distributed Symmetric Memory Management Library (DSMML) is a HPE Cray proprietary memory management library. diff --git a/docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md b/docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md index 90ff7032..7439177e 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md @@ -98,7 +98,7 @@ set of target modules after loading the ``CrayEnv`` module. ## The ``partition`` module -The ``LUMI`` module currently supports four partition modules, but that number may +The ``LUMI`` module currently supports five partition modules, but that number may be reduced in the future: | Partition | CPU target | Accelerator | diff --git a/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md index dda28b97..c78afa0a 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md @@ -143,18 +143,18 @@ while `eb -S` produces a more concise output. For example, let's check which easyconfig files are available for OpenFOAM 8 with a `foss` toolchain: ```shell -$ eb --search openfoam-8-foss - * /home/example/.local/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-8-foss-2020a.eb - * /home/example/.local/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-8-foss-2020b.eb +$ eb --search openfoam-9 + * /appl/lumi/LUMI-EasyBuild-contrib/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-9-cpeGNU-21.08.eb + * /appl/lumi/LUMI-EasyBuild-contrib/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-9-cpeGNU-21.12.eb ``` The output is a bit more condensed when using `eb -S`: ```shell -$ eb -S openfoam-8-foss -CFGS1=/home/example/.local/easybuild/easyconfigs/o/OpenFOAM - * $CFGS1/OpenFOAM-8-foss-2020a.eb - * $CFGS1/OpenFOAM-8-foss-2020b.eb +$ eb -S openfoam-9 +CFGS1=/appl/lumi/LUMI-EasyBuild-contrib/easybuild/easyconfigs/o/OpenFOAM + * $CFGS1/OpenFOAM-9-cpeGNU-21.08.eb + * $CFGS1/OpenFOAM-9-cpeGNU-21.12.eb ``` Note that the search is performed *case-insensitive*. @@ -164,13 +164,17 @@ or other special characters like `^` or `$` to mark the start/end of the filenam but you need to be careful that `bash` does not expand these before the `eb` command is started, so it is recommended to wrap the search pattern in single quotes (`'...'`) when using wildcards. -For example, to check which easyconfigs are available to install TensorFlow 2.4.1 with the `2020b` version of a toolchain: +For example, to check which easyconfigs are available to install GROMACS 2021 and subversions with +the `cpeGNU` toolchains: ```shell -$ eb -S '^tensorflow-2.4.1.*2020b' -CFGS1=/home/example/.local/easybuild/easyconfigs/t/TensorFlow - * $CFGS1/TensorFlow-2.4.1-foss-2020b.eb - * $CFGS1/TensorFlow-2.4.1-fosscuda-2020b.eb +$ eb -S '^gromacs-2021.*cpeGNU.*' +CFGS1=/appl/lumi/LUMI-EasyBuild-contrib/easybuild/easyconfigs/g/GROMACS + * $CFGS1/GROMACS-2021-cpeGNU-21.08-PLUMED-2.7.2-CPU.eb + * $CFGS1/GROMACS-2021.3-cpeGNU-21.08-CPU.eb + * $CFGS1/GROMACS-2021.4-cpeGNU-21.12-PLUMED-2.7.4-CPU.eb + * $CFGS1/GROMACS-2021.4-cpeGNU-21.12-PLUMED-2.8.0-CPU.eb + * $CFGS1/GROMACS-2021.5-cpeGNU-21.12-CPU.eb ``` --- diff --git a/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md index 2da0fdd0..628f51d9 100644 --- a/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md @@ -44,16 +44,24 @@ specific for a certain compiler and don't recognise the Cray compilers. Reasons to consider implementing a software-specific easyblock rather than using a generic easyblock include: - 'critical' values for easyconfig parameters required to make installation succeed; -- toolchain-specific aspects of the build and installation procedure (e.g., configure options); -- interactive commands that need to be run; - *For example: The easyblock to [install maple](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/m/maple.py) - responds to a number of interactive questions.* -- custom (configure) options for dependencies; + *For example, the [easyblock for bowtie2](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/b/bowtie2.py) + defines a number of variables used in the Makefile are specified on the make command line to ensure that the right + compilers are used.* +- toolchain-specific aspects of the build and installation procedure (e.g., configure options); + *For example, the [easyblock for CP2K](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/c/cp2k.py) + will add several compiler options when compiling with gcc and gfortran, including the infamous + `-fallow-argument-mismatch` which is required from gfortran 10 on for many older codes.* +- custom (configure) options for dependencies; *For example, the [easyblock for VMD](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/v/vmd.py) will add configure options depending on the dependency list of the package.* -- having to create or adjust specific (configuration) files; +- interactive commands that need to be run; + *For example: The [easyblock for maple](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/m/maple.py) + responds to a number of interactive questions.* +- having to create or adjust specific (configuration) files; + *For example, the [easyblock for Siesta](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/s/siesta.py) + edits one of the configuration files through regular expressions.* - 'hackish' usage of a generic easyblock; -- complex or very non-standard installation procedure; +- complex or very non-standard installation procedure; *For example, the [easyblock to install the gcc compilers from source](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/g/gcc.py), bootstrapping with the system compiler and then re-installing with itself.* @@ -499,6 +507,20 @@ Or, to try using a different compiler toolchain you can use `--try-toolchain`: eb example-1.2.3-foss-2020b.eb --try-toolchain cpeCray,21.12 ``` +There is also a `--try` option to change an arbitrary parameter in an easyconfig: + +```shell +eb example-1.2.3-foss-2020b.eb --try-amend versionsuffix='-test' +``` + +Its general syntax is + +``` +--try-amend=VAR=VALUE[,VALUE] +``` + +This option can be used multiple times also to make multiple changes. + It is important to keep in mind the *"try"* aspect here: while easyconfigs that are generated by EasyBuild via a `--try-*` option often do work fine, there is no strong guarantee they will. Newer software versions may come with changes to @@ -506,6 +528,9 @@ the installation procedure, additional dependencies that are required, etc. Using a different compiler toolchain may be as simple as just switching one for another, but it may require additional changes to be made to configure options, for example. +The easyconfig files modified in this way will be stored in the `easybuild` subdirectory +of the software installation directory and in the easyconfig archive that EasyBuild creates. + ## Copying easyconfigs One additional handy command line option we want to highlight is `--copy-ec`, which can be used to From dc0895fde8b36f7583a012cc416b6e647ea3aded Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 9 May 2022 18:27:18 +0200 Subject: [PATCH 044/159] Reworked the exercises of the troubleshooting section. --- .../2_Using/2_01_troubleshooting.md | 280 +++++++++++++++--- 1 file changed, 247 insertions(+), 33 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md b/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md index 0f1b5c3c..ee6716fd 100644 --- a/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md +++ b/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md @@ -28,6 +28,7 @@ Things that could go wrong during an installation include: * required dependencies that are not specified in the easyconfig file; * failing shell commands; * running out of available memory or disk space; +* compiler errors and compiler crashes ; * a segmentation fault caused by a flipped bit triggered by a cosmic ray ([really, it happens!](https://blogs.oracle.com/linux/post/attack-of-the-cosmic-rays)); @@ -239,7 +240,7 @@ checksums = ['d808eb5b1823c572cb45a97c95a3c5acb3d8e29aa47ec74e3ca1eb345787c17b'] start_dir = 'src' -# -fcommon is required to compile Subread 2.0.1 with GCC 10, +# -fcommon is required to compile Subread 2.0.1 with GCC 10/11, # which uses -fno-common by default (see https://www.gnu.org/software/gcc/gcc-10/porting_to.html) buildopts = '-f Makefile.Linux CFLAGS="-fast -fcommon"' @@ -287,10 +288,60 @@ Remember though: *no peeking* before you tried to solve each step yourself! --- -***Exercise T.1**** - Sources* +***Exercise T.1**** - Toolchain* Try to install the `subread.eb` easyconfig file, see what happens. +Take into account that we just want to get this software package installed, +we don't care too much about details like the version of the dependencies or +the toolchain here... + + +??? success "(click to show solution)" + + The installation fails because the easyconfig specifies that `PrgEnv-gnu/21.10` + should be used as toolchain: + + ``` + $ eb subread.eb + ... + ERROR: Failed to process easyconfig /pfs/lustrep3/users/kurtlust/easybuild-tutorial/Troubleshooting/subread.eb: Toolchain PrgEnv-gnu not found, + available toolchains: ... + ... + ``` + + `PrgEnv-gnu` is an HPE Cray PE module that may look like a toolchain - it certainly has + the same function: provide compiler, MPI and basic math libraries - but it is not + recognised as a toolchain by EasyBuild. EasyBuild prefers to manage its own modules so that it knows + well what is in it which is not the case with the `PrgEnv-*` modules from the Cray PE + as the content may differ between systems and as the versions of the compilers etc. that + are loaded differ on other modules that are loaded. Hence we created Cray-specific toolchains. + You'll actually find two series of Cray toolchains in the list of available toolchains. + + A more readable list of toolchains supported by EasyBuild can be generated using + + ```shell + eb --list-toolchains + ``` + + The `CrayGNU`, `CrayIntel`, `CrayPGI` and `CrayCCE` are included with the EasyBuild distribution + and where developed by CSCS for their systems using Environment Modules. These were not compatible + with the initial releases of the Cray PE with Lmod modules so new ones were developed on which we + also built for the LUMI toolchains. Those are called `cpeCray`, `cpeGNU`, `cpeAOCC` and `cpeAMD` + and are maintained by LUST and available via the LUMI repositories. + +Note: Depending on how you use EasyBuild you may now first run into the problem of Exercise T.2 or +first run into the problem covered by Exercise T.3. + + +--- + +***Exercise T.2**** - Sources* + +After fixing the problem with the name of the toolchain, try running `eb` again. + +What's wrong now? How can you fix it quickly? + Can you fix the problem you run into, perhaps without even changing the easyconfig file? @@ -323,6 +374,9 @@ the easyconfig file? mv subread-2.0.1-source.tar.gz $EBU_USER_PREFIX/sources/s/Subread/ ``` + (assuming you have set `EBU_USER_PREFIX`, otherwise replace `$EBU_USER_PREFIX` with + `$HOME/EasyBuild`). + Or, we can change the easyconfig file to specify the location where the easyconfig file can be downloaded from: ```python @@ -344,7 +398,7 @@ the easyconfig file? --- -***Exercise T.2**** - Toolchain* +***Exercise T.3**** - Toolchain revisited* After fixing the problem with missing source file, try the installation again. @@ -357,33 +411,10 @@ the toolchain here... ??? success "(click to show solution)" - The installation fails because the easyconfig specifies that GCC 8.5.0 + The installation fails because the easyconfig specifies that `PrgEnv-gnu/21.12` should be used as toolchain: - ``` - $ eb subread.eb - ... - ERROR: Failed to process easyconfig /pfs/lustrep3/users/kurtlust/easybuild-tutorial/Troubleshooting/subread.eb: Toolchain PrgEnv-gnu not found, - available toolchains: ... - ... - ``` - - `PrgEnv-gnu` is an HPE Cray PE module that may look like a toolchain - it certainly has - the same function: provide compiler, MPI and basic math libraries - but it is not - recognised as a toolchain by EasyBuild. EasyBuild prefers to manage its own modules so that it knows - well what is in it which is not the case with the `PrgEnv-*` modules from the Cray PE - as the content may differe between systems and as the versions of the compilers etc. that - are loaded differ on other modules that are loaded. Hence we created Cray-specific toolchains. - You'll actually find two series of Cray toolchains in the list of available toolchains. The - `CrayGNU`, `CrayIntel`, `CrayPGI` and `CrayCCE` are included with the EasyBuild distribution - and where developed by CSCS for their systems using Environment Modules. These were not compatible - with the initial releases of the Cray PE with Lmod modules so new ones were developed on which we - also built for the LUMI toolchains. Those are called `cpeCray`, `cpeGNU`, `cpeAOCC` and `cpeAMD` - and are maintained by LUST and available via the LUMI repositories. - - Changing the toolchain name to `cpeGNU` is not enough to solve all problems though: - - ``` + ```shell $ eb subread.eb ... ERROR: Build of /pfs/lustrep3/users/kurtlust/easybuild-tutorial/Troubleshooting/subread.eb failed (err: 'build failed (first 300 chars): @@ -391,7 +422,7 @@ the toolchain here... ... ``` - We don't have this `cpeGNU` version installed, but we do have GCC 21.12: + We don't have this `cpeGNU` version installed, but we do have `cpeGNU/21.12`: ```shell $ module avail cpeGNU/ @@ -407,11 +438,12 @@ the toolchain here... ```python toolchain = {'name': 'cpeGNU', 'version': '21.12'} ``` + --- -***Exercise T.3**** - Build step* +***Exercise T.4**** - Build step* -With the first two problems fixed, now we can actually try to build the software. +With the first three problems fixed, now we can actually try to build the software. Can you fix the next problem you run into? @@ -489,9 +521,10 @@ Can you fix the next problem you run into? defaults in EasyBuild, settings in the EasyBuild configuration and settings in the easyconfig file that we shall discuss later). + --- -***Exercise T.4**** - Sanity check* +***Exercise T.5**** - Sanity check* After fixing the compilation issue, you're really close to getting the installation working, we promise! @@ -529,7 +562,12 @@ Don't give up now, try one last time and fix the last problem that occurs... --- -In the end, you should be able to install Subread 2.0.1 with the cpeGNU 21.12 toolchain by fixing the problems with the `subread.eb` easyconfig file. +--- + +***Exercise T.6**** - Post-install check of the log file* + +In the end, you should be able to install Subread 2.0.1 with the cpeGNU 21.12 toolchain by +fixing the problems with the `subread.eb` easyconfig file. Check your work by manually loading the module and checking the version via the `featureCounts` command, which should look like this: @@ -541,6 +579,182 @@ $ featureCounts -v featureCounts v2.0.1 ``` +So all is well know, or is it? + +Unfortunately we don't have a complete log file of the last build (at least if you only re-installed +the module) as most of the steps were skipped in the last build. + +Let's do the build again and check the full log file, just to be sure. But we'll first need to +clean up a bit as EasyBuild doesn't like to build in a shell in which the modules which are +used for the build are already loaded: + +```shell +module unload Subread cpeGNU +``` + +Now look at the output of an extended dry run and then rebuild to have a full log file so +that we can expect if EasyBuild really did what we expected: + +```shell +eb subread.eb -x +eb subread.eb -f +``` + +(the last line to force a rebuild). + +Now go to the `$EBU_USER_PREFIX/SW/LUMI-21.12/L/Subread/2.0.1-cpeGNU-21.12/easybuild` +(or `$HOME/EasyBuild/SW/LUMI-21.12/L/Subread/2.0.1-cpeGNU-21.12/easybuild`, depending on your configuration,) directory and open +the log file in your favourite editor. Search for the build step by searching for the string +`INFO Starting build` and look carefully at how the program was actually build... + +You'll very likely have to look at the solution to understand how to correct the +problems as that requires more advanced knowlege than we have at this point in +the tutorial, but try to figure out what could be wrong first though... + +??? hint "(Click for a hint)" + Check the compiler that has been used and the compiler flags. Are these really + what you would like to see and what you would expect from running `eb subread.eb -x` + as we did before? + + +??? success "(click to show solution)" + According to the output of `eb subread.eb -x`, the build should be done using + `cc` as the compiler as that is the value assigned to the `CC` environment which + by convention points to the C compiler. Moreover, EasyBuild sets `CFLAGS` to + `-O2 -ftree-vectorize -fno-math-errno`, and then the `make` command line adds + `-fcommon` to those flags. + + However, this is not what we see in the build log. It turns out that Subread + is one of those horror packages that follows no established convention for + build procedures. + + One of the first lines we + run into (yours may differ since this is a parallel build) is + + ``` + gcc -mtune=core2 -O3 -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\""2.0.1"\" -D_FILE_OFFSET_BITS=64 -fmessage-length=0 -ggdb -O2 -ftree-vectorize -fno-math-errno -fcommon -I/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/include -c -o core.o core.c + ``` + + The flags that we added via `CFLAGS` are in there but only after some other flags. + The build process didn't pick up our C compiler either! And o horror, it even defines + the processor architecture! So it will not run on older architectures than the Intel Sandy + Bridge family, but it will not exploit newer architectures either (well, it it could, the code + may not benefit at all from newer vectorisation instructions, but at least the compiler might + do a better job optimising). + Scrolling down a bit we see some lines that generate executables from a single + C file and a list of already generated object files, and there we don't even see our + compiler flags at all! + + The problem is truly in the makefiles of Subread. We could now untar the source file + that was saved by EasyBuild in a temporary work directory and inspect the sources, or we could + retry the build and stop after the build step. Let's take the latter option. The command to + do this is + + ``` + eb subread.eb -f --stop build + ``` + + We'll need to search for the build directory now as it is not printed when EasyBuild stops in + a regular way. + + ``` + pushd $EASYBUILD_BUILDPATH/Subread/2.0.1/cpeGNU-21.12 + cd subread-2.0.1-source + cd src + ``` + + The EasyConfig uses the makefile `Makefile.Linux` so let's check that one. Some of the crucial + lines are: + + ``` + CC_EXEC = gcc + OPT_LEVEL = 3 + + CCFLAGS = -mtune=core2 ${MACOS} -O${OPT_LEVEL} -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\"${SUBREAD_VERSION}\" -D_FILE_OFFSET_BITS=64 ${WARNING_LEVEL} + CC = ${CC_EXEC} ${CCFLAGS} -fmessage-length=0 -ggdb + ``` + + We see several problems at once + + - `CC` is defined in the Makefile in a way that we do not want to redefine it on the `make`` command + line as it also already includes all compiler options. It turns out we need to redefine `CC_EXEC` + instead to use a different compiler. + - `CCFLAGS` includes several options that should enter through `CFLAGS` and should not be imposed in + a proper build process. The most dangerous one is the `-mtune=core2`, but in general we prefer to + leave the choice of the optimisation level to EasyBuild also unless there are good reasons to use + a very specific optimisation level. + - One may wonder why at least some of the compiles did pick up `CFLAGS` then. This is because these + files were compiled using an implicit rule that used the `CC` command as defined in `Makefile.Linux` + so with a lot of compiler flags already added to it and then adds `CFLAGS` as defined on the `make` + command line generated by EasyBuild. Those compile commands that were generated from an explicit rule + don't pick up `CFLAGS` though. + + There are two ways to fix this in EasyBuild (besides teaching the developer of this software package how + to write a proper Makefile following the usual conventions). + + 1. The approach which is usually followed is to make a patch file for `Makefile.Linux` that changes the line + + ``` + CCFLAGS = -mtune=core2 ${MACOS} -O${OPT_LEVEL} -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\"${SUBREAD_VERSION}\" -D_FILE_OFFSET_BITS=64 ${WARNING_LEVEL} + ``` + + to, e.g., + + ``` + CCFLAGS = ${CFLAGS} -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\"${SUBREAD_VERSION}\" -D_FILE_OFFSET_BITS=64 ${WARNING_LEVEL} + ``` + + combined with changing the `buildopts` line to also overwrite `CC_EXEC`: + + ``` + buildopts = '-f Makefile.Linux CC_EXEC="$CC" CFLAGS="-fast -fcommon"' + ``` + + (or you could also change the `CC_EXEC` line in `Makefile.Linux` with the same patch to use the `cc` command, + but that would also make the patch file Cray-only.) + + 2. The other option is to simply edit `Makefile.Linux` using `sed` to replace + `-mtune=core2 ${MACOS} -O${OPT_LEVEL}` with + `${CFLAGS}`. This can be done by executing a `sed` command before calling `make`. + As we shall see later in this tutorial, this can be done with `prebuildopts`: + + ```python + prebuildopts = "sed -e 's/-mtune=core2 ${MACOS} -O${OPT_LEVEL}/${CFLAGS}/' -i Makefile.Linux && " + ``` + + and as in the previous case we also still need to overwrite `CC_EXEC` with the + correct compiler on the `make` command line: + + ``` + buildopts = '-f Makefile.Linux CC_EXEC="$CC" CFLAGS="-fast -fcommon"' + ``` + + Now check the output of `eb subread.eb -x` to see what will happen during the build phase. + + Let's implement the second approach, then do a full rebuild: + + ```shell + eb subread.eb -f + ``` + + and then open the log file (again in the `easybuild` subdirectory of the software installation + directory) and check what happened now during the build step. + + As we scroll through the output of the build step, we still see a few lines mentioning + `gcc`... It turns out there is a second Makefile hidden in the subdirectory `longread-one` so we + need to edit that one too... So following the second approach we can do this with + + ```python + prebuildopts = "sed -e 's/-mtune=core2 ${MACOS} -O${OPT_LEVEL}/${CFLAGS}/' -i Makefile.Linux && " + prebuildopts += "sed -e 's/-mtune=core2 ${MACOS} -O${OPT_LEVEL}/${CFLAGS}/' -i longread-one/Makefile && " + ``` + + Now we can build once more and check the log file and finally we can be satisfied... + + This exercise also show how tedious developing an easyconfig can be. And it also shows mistakes that + are sometimes overlooked in easyconfigs that come with EasyBuild. + + --- *[[next: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* From a9e0d3e530bddf3bdd8588dcb13a37c19e4fff9a Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 9 May 2022 19:21:09 +0200 Subject: [PATCH 045/159] Reworked the exercises for the basic usage section. --- .../1_Intro/1_08_basic_usage.md | 279 ++++++++---------- 1 file changed, 129 insertions(+), 150 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md index c78afa0a..7bc1adc2 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_08_basic_usage.md @@ -976,80 +976,94 @@ Do yourself a favor: don't peek at the solution until you have made an attempt t Please do not spoil solutions to others before they have been discussed by the tutorial organisers. -The exercises are based on the easyconfig files included with EasyBuild 4.3.3. +The exercises are based on the easyconfig files available on LUMI during the tutorial of +May 2022. --- -***Exercise U.0**** - Making installed software available* - -Before working on the exercises for this part of the tutorial, -make sure that the software that is already installed in the prepared environment is available. - -We will assume that you have a small software stack installed using the `2020b` version of the `foss` toolchain. +***Exercise U.1**** - Searching easyconfigs* -**Tip:** execute a "`module use`" command, and verify with "`module avail`" that a bunch of software modules -are available for loading. +See if EasyBuild on LUMI provides any easyconfig files for installing PLUMED in one of the 2.7 versions. ??? success "(click to show solution)" + To check for available easyconfig files, we can use `eb --search` or `eb -S`: - Use the following command to make the modules for the software available that is pre-installed - in the prepared environment: ```shell - module use /easybuild/modules/all + eb -S PLUMED-2.7 + CFGS1=/appl/lumi/LUMI-EasyBuild-contrib/easybuild/easyconfigs + * $CFGS1/g/GROMACS/GROMACS-2020.6-cpeCray-21.08-PLUMED-2.7.2-CPU.eb + * $CFGS1/g/GROMACS/GROMACS-2020.6-cpeGNU-21.08-PLUMED-2.7.2-CPU.eb + * $CFGS1/g/GROMACS/GROMACS-2021-cpeCray-21.08-PLUMED-2.7.2-CPU.eb + * $CFGS1/g/GROMACS/GROMACS-2021-cpeGNU-21.08-PLUMED-2.7.2-CPU.eb + * $CFGS1/g/GROMACS/GROMACS-2021.4-cpeAOCC-21.12-PLUMED-2.7.4-CPU.eb + * $CFGS1/g/GROMACS/GROMACS-2021.4-cpeCray-21.12-PLUMED-2.7.4-CPU.eb + * $CFGS1/g/GROMACS/GROMACS-2021.4-cpeGNU-21.12-PLUMED-2.7.4-CPU.eb + * $CFGS1/p/PLUMED/PLUMED-2.7.2-cpeAMD-21.08.eb + * $CFGS1/p/PLUMED/PLUMED-2.7.2-cpeCray-21.08.eb + * $CFGS1/p/PLUMED/PLUMED-2.7.2-cpeGNU-21.08.eb + * $CFGS1/p/PLUMED/PLUMED-2.7.2-cpeGNU-21.12.eb + * $CFGS1/p/PLUMED/PLUMED-2.7.4-cpeAOCC-21.12.eb + * $CFGS1/p/PLUMED/PLUMED-2.7.4-cpeCray-21.12.eb + * $CFGS1/p/PLUMED/PLUMED-2.7.4-cpeGNU-21.12.eb ``` - If software is installed in a different location than `/easybuild/` in your environment, - you should adjust the command accordingly. - - ---- + We do get more output than we were hoping for as it also shows the GROMACS + versions using PLUMED. One way around this is to use regular expressions: -***Exercise U.1**** - Searching easyconfigs* - -See if EasyBuild provides any easyconfig files for installing GROMACS version 2020/5. - -??? success "(click to show solution)" - To check for available easyconfig files, we can use `eb --search` or `eb -S`: ```shell - $ eb -S gromacs-2020.5 - == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... - CFGS1=/home/example/.local/easybuild/easyconfigs/g/GROMACS - * $CFGS1/GROMACS-2020.5-fosscuda-2020a-Python-3.8.2.eb - * $CFGS1/GROMACS-2020.5_fix_threads_gpu_Gmxapitests.patch + $ eb -S ^PLUMED-2.7 + CFGS1=/appl/lumi/LUMI-EasyBuild-contrib/easybuild/easyconfigs/p/PLUMED + * $CFGS1/PLUMED-2.7.2-cpeAMD-21.08.eb + * $CFGS1/PLUMED-2.7.2-cpeCray-21.08.eb + * $CFGS1/PLUMED-2.7.2-cpeGNU-21.08.eb + * $CFGS1/PLUMED-2.7.2-cpeGNU-21.12.eb + * $CFGS1/PLUMED-2.7.4-cpeAOCC-21.12.eb + * $CFGS1/PLUMED-2.7.4-cpeCray-21.12.eb + * $CFGS1/PLUMED-2.7.4-cpeGNU-21.12.eb ``` - This actually shows one easyconfig file but also a patch file. We can also search specifically - for GROMACS 2020.5 in the `foss` and `fosscuda` toolchains using - ```shell - $ eb -S gromacs-2020.5-foss - == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... - CFGS1=/home/example/.local/easybuild/easyconfigs/g/GROMACS - * $CFGS1/GROMACS-2020.5-fosscuda-2020a-Python-3.8.2.eb + + It is also possible to check for all 2.7 variants for the cpeGNU toolchain but + this requires a bit more care in composing the regular expression to ensure that + the shell does not try to expand the expression: + ``` - and now we find a single easyconfig file. + eb -S '^PLUMED-2.7.*-cpeGNU' + CFGS1=/appl/lumi/LUMI-EasyBuild-contrib/easybuild/easyconfigs/p/PLUMED + * $CFGS1/PLUMED-2.7.2-cpeGNU-21.08.eb + * $CFGS1/PLUMED-2.7.2-cpeGNU-21.12.eb + * $CFGS1/PLUMED-2.7.4-cpeGNU-21.12.eb + ``` --- ***Exercise U.2**** - Checking dependencies* -Check which dependencies are missing to install QuantumESPRESSO version 6.6 with the `2020b` version of the `foss` toolchain. +Check which dependencies are missing to install GROMACS 2021.4 with the +`cpeGNU` toolchain and with the PLUMED plugin in the +most recent version available in the LUMI recipes. ??? success "(click to show solution)" - First, we need to determine the name of the easyconfig file for QuantumESPRESSO version 6.6: + First, we need to determine the name of the easyconfig file for the required + version of GROMACS. Easyconfigs would start with `GROMACS-2021.4-cpeGNU` so + let's simply search for that: + ```shell - $ eb -S 'QuantumESPRESSO-6.6.*foss-2020b' - == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... - CFGS1=/home/example/.local/easybuild/easyconfigs/q/QuantumESPRESSO - * $CFGS1/QuantumESPRESSO-6.6-foss-2020b.eb + $ eb -S GROMACS-2021.4-cpeGNU + CFGS1=/appl/lumi/LUMI-EasyBuild-contrib/easybuild/easyconfigs/g/GROMACS + * $CFGS1/GROMACS-2021.4-cpeGNU-21.12-PLUMED-2.7.4-CPU.eb + * $CFGS1/GROMACS-2021.4-cpeGNU-21.12-PLUMED-2.8.0-CPU.eb ``` - To determine which dependencies are missing to install this QuantumESPRESSO easyconfig file, we can use `--missing`: - ```shell - $ eb QuantumESPRESSO-6.6-foss-2020b.eb --missing - 3 out of 58 required modules missing: + We want the one for the most recent `PLUMED` toolchian, hence the second one. + + To determine which dependencies are missing to install this GROMACS easyconfig file, we can use `--missing`: + ```shell + $ eb GROMACS-2021.4-cpeGNU-21.12-PLUMED-2.8.0-CPU.eb --missing - * libxc/4.3.4-GCC-10.2.0 (libxc-4.3.4-GCC-10.2.0.eb) - * ELPA/2020.11.001-foss-2020b (ELPA-2020.11.001-foss-2020b.eb) - * QuantumESPRESSO/6.6-foss-2020b (QuantumESPRESSO-6.6-foss-2020b.eb) + 2 out of 15 required modules missing: + + * PLUMED/2.8.0-cpeGNU-21.12 (PLUMED-2.8.0-cpeGNU-21.12.eb) + * GROMACS/2021.4-cpeGNU-21.12-PLUMED-2.8.0-CPU (GROMACS-2021.4-cpeGNU-21.12-PLUMED-2.8.0-CPU.eb) ``` (some nonessential output removed). @@ -1058,154 +1072,119 @@ Check which dependencies are missing to install QuantumESPRESSO version 6.6 with ***Exercise U.3**** - Performing a dry run* Figure out which command EasyBuild would use to compile -the software provided by the `Bowtie2-2.4.2-GCC-9.3.0.eb` easyconfig file, -without actually installing `Bowtie2`. +the software provided by the `SAMtools-1.14-cpeGNU-21.12.eb` easyconfig file, +without actually installing `SAMtools`. Also, which binaries will EasyBuild check for to sanity check the installation? ??? success "(click to show solution)" - To inspect the installation procedure, we can use `eb -x Bowtie2-2.4.2-GCC-9.3.0.eb`. + To inspect the installation procedure, we can use `SAMtools-1.14-cpeGNU-21.12.eb`. The output for the build step shows the actual compilation command that would be performed (`make ...`): ```shell [build_step method] - >> running command: - [started at: 2021-03-08 20:15:08] - [working dir: /local_scratch/hkenneth/eb-1wodfohg/__ROOT__/local_scratch/hkenneth/Bowtie2/2.4.2/GCC-9.3.0/Bowtie2-2.4.2] - [output logged in /local_scratch/hkenneth/eb-1wodfohg/easybuild-run_cmd-haojzisn.log] - make -j 48 CC="gcc" CPP="g++" CXX="g++" RELEASE_FLAGS="-O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -std=gnu++98" - (in /local_scratch/hkenneth/Bowtie2/2.4.2/GCC-9.3.0/Bowtie2-2.4.2) + running command "make -j 256 CC="cc" CXX="CC" CFLAGS="-O2 -ftree-vectorize -fno-math-errno -fPIC" CXXFLAGS="-O2 -ftree-vectorize -fno-math-errno -fPIC"" + (in /XXXX/build/SAMtools/1.14/cpeGNU-21.12/SAMtools-1.14) ``` - If the output you get is less detailed, you may not have set `export EASYBUILD_TRACE=1`. + (And if you also add `--trace` the output will even be a bit more detailed). The output for the sanity check step shows which binaries are expected to be installed: ``` [sanity_check_step method] Sanity check paths - file ['files'] - * bin/bowtie2 - * bin/bowtie2-align-l - * bin/bowtie2-align-s - * bin/bowtie2-build - * bin/bowtie2-build-l - * bin/bowtie2-build-s - * bin/bowtie2-inspect - * bin/bowtie2-inspect-l - * bin/bowtie2-inspect-s + * bin/ace2sam + * bin/blast2sam.pl + * bin/bowtie2sam.pl + * bin/export2sam.pl + * bin/interpolate_sam.pl + * bin/maq2sam-long + * bin/maq2sam-short + * bin/md5fa + * bin/md5sum-lite + * bin/novo2sam.pl + * bin/plot-bamstats + * bin/psl2sam.pl + * bin/r2plot.lua + * bin/sam2vcf.pl + * bin/samtools + * bin/samtools.pl + * bin/seq_cache_populate.pl + * bin/soap2sam.pl + * bin/vcfutils.lua + * bin/wgsim + * bin/wgsim_eval.pl + * bin/zoom2sam.pl + * include/bam/bam.h + * include/bam/bam2bcf.h + * include/bam/sample.h ``` --- ***Exercise U.4**** - Installing software* -Install version 3.1.0 of the `h5py` Python package and all missing dependencies, -using the `foss/2020b` toolchain, into `/tmp/$USER/easybuild`, -while leveraging the already installed software available from `/easybuild`. +Install version 2.0.5 of CDO using the cpeGNU toolchain (21.12 version) +including all missing dependencies. Enable trace output so you can see which parts of the installation take a while. ??? success "(click to show solution)" First, determine the easyconfig file we can use for this: ```shell - $ eb -S 'h5py-3.1.0.*foss-2020b' - CFGS1=/home/example/.local/easybuild/easyconfigs/h/h5py - * $CFGS1/h5py-3.1.0-foss-2020b.eb - ``` - - Make sure the pre-install software in `/easybuild/` is available: - ```shell - module use /easybuild/modules/all + $ eb -S '^CDO.*cpeGNU-21.12' + CFGS1=/appl/lumi/LUMI-EasyBuild-contrib/easybuild/easyconfigs/c/CDO + * $CFGS1/CDO-2.0.5-cpeGNU-21.12.eb ``` - Check which dependencies are missing to install this `h5py` easyconfig: + Check which dependencies are missing to install this CDO easyconfig: ```shell - $ eb h5py-3.1.0-foss-2020b.eb --missing + $ eb CDO-2.0.5-cpeGNU-21.12.eb - 2 out of 63 required modules missing: + 4 out of 32 required modules missing: - * pkgconfig/1.5.1-GCCcore-10.2.0-python (pkgconfig-1.5.1-GCCcore-10.2.0-python.eb) - * h5py/3.1.0-foss-2020b (h5py-3.1.0-foss-2020b.eb) + * json-c/0.16-cpeGNU-21.12 (json-c-0.16-cpeGNU-21.12.eb) + * ecCodes/2.25.0-cpeGNU-21.12 (ecCodes-2.25.0-cpeGNU-21.12.eb) + * CMOR/3.6.1-cpeGNU-21.12 (CMOR-3.6.1-cpeGNU-21.12.eb) + * CDO/2.0.5-cpeGNU-21.12 (CDO-2.0.5-cpeGNU-21.12.eb) ``` - Install `h5py` by specifying the easyconfig file and enabling dependency resolution via `--robot`, - while indicating that we want to install the software into `/tmp/$USER/easybuild` using the `--installpath` - option. Also make sure that trace mode is enabled by defining the `$EASYBUILD_TRACE` environment variable. + Install CDO by specifying the easyconfig file and enabling dependency resolution via `--robot`, + Also make sure that trace mode is enabled by defining the `$EASYBUILD_TRACE` environment variable. ```shell $ export EASYBUILD_TRACE=1 - $ eb h5py-3.1.0-foss-2020b.eb --robot --installpath /tmp/$USER/easybuild + $ eb CDO-2.0.5-cpeGNU-21.12.eb --robot + ... + == building and installing ecCodes/2.25.0-cpeGNU-21.12... + ... + == building and installing json-c/0.16-cpeGNU-21.12... ... - == building and installing pkgconfig/1.5.1-GCCcore-10.2.0-python... + == building and installing CMOR/3.6.1-cpeGNU-21.12... ... - == building and installing h5py/3.1.0-foss-2020b... + == building and installing CDO/2.0.5-cpeGNU-21.12... ... + == configuring... + >> running command: + ... + >> command completed: exit 0, ran in 00h01m52s + == ... (took 1 min 52 secs) == building... >> running command: - [started at: 2020-06-10 21:47:32] - [working dir: /tmp/example/h5py/3.1.0/foss-2020b/h5py-3.1.0] - [output logged in /tmp/eb-rjjkbqe1/easybuild-run_cmd-d_dkc4iz.log] - python setup.py configure --mpi --hdf5=$EBROOTHDF5 && /easybuild/software/Python/3.8.6-GCCcore-10.2.0/bin/python setup.py build - >> command completed: exit 0, ran in 00h01m27s + [started at: 2022-05-09 20:07:44] + [working dir: /run/user/10012026/easybuild/build/CDO/2.0.5/cpeGNU-21.12/cdo-2.0.5] + [output logged in /run/user/10012026/easybuild/tmp/eb-0ihsqw7j/easybuild-run_cmd-pnuot3pi.log] + make -j 256 + >> command completed: exit 0, ran in 00h00m57s ... - == COMPLETED: Installation ended successfully (took 2 min 46 sec) + == COMPLETED: Installation ended successfully (took 3 mins 14 secs) ... - == Build succeeded for 2 out of 2 + == Build succeeded for 4 out of 4 ``` - The trace output shows that most time is spent in the build command, - which runs both `python setup.py configure` and `python setup.py build`. - ---- - -***Exercise U.5**** - Using installed software* - -Using the `h5py` installation from the previous exercise to create an empty HDF5 file, -using the following Python statements: - -```python -import h5py -f = h5py.File("empty.hdf5", "w") -f.close() -``` - -Check the resulting file using the `h5stat` command. - -??? success "(click to show solution)" - First, we need to make the modules tool aware of the module files that were installed into `/tmp/$USER/easybuild`: - ```shell - module use /tmp/$USER/easybuild/modules/all - ``` + The trace output shows that most time is spent in the configure phase (and this is also the + case for some of the dependencies). - Then we can check the `h5py` module is available, and load it: - ```shell - $ module avail h5py - ------------ /tmp/example/easybuild/modules/all ------------ - h5py/3.1.0-foss-2020b - ``` - - ```shell - module load h5py/3.1.0-foss-2020b - ``` - - The Python code snippet can be run directly on the command line using "`python -c '...'`", since it's tiny: - ```shell - python -c 'import h5py; f = h5py.File("empty.hdf5", "w"); f.close()' - ``` - Of course you can also copy the Python code snippet in a file named `test_h5py.py`, - and then run it with `python test_h5py.py`. - - Checking with the `h5stat` command shows that the resulting `empty.hdf5` is indeed a valid HDF5 file: - ```shell - $ ls -l empty.hdf5 - -rw-rw-r-- 1 example example 800 Jun 10 21:54 empty.hdf5 - - $ h5stat empty.hdf5 - Filename: empty.hdf5 - File information - # of unique groups: 1 - # of unique datasets: 0 - ... - ``` --- From d851ae12917d018d9c658ae7f04a81843ce6d3cf Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 10 May 2022 17:50:47 +0200 Subject: [PATCH 046/159] Multiple minor corrections. --- .../2_Using/2_02_creating_easyconfig_files.md | 149 ++++++++++-------- .../2_Using/2_04_implementing_easyblocks.md | 8 +- docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md | 4 +- .../3_Advanced/3_03_slurm_jobs.md | 16 +- .../3_Advanced/3_04_module_naming_schemes.md | 4 +- .../3_Advanced/3_05_github_integration.md | 2 +- 6 files changed, 107 insertions(+), 76 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md index 628f51d9..f9cb4577 100644 --- a/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md @@ -45,8 +45,8 @@ Reasons to consider implementing a software-specific easyblock rather than using - 'critical' values for easyconfig parameters required to make installation succeed; *For example, the [easyblock for bowtie2](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/b/bowtie2.py) - defines a number of variables used in the Makefile are specified on the make command line to ensure that the right - compilers are used.* + defines a number of variables used in the Makefile on the make command line to ensure that the right + compilers are used (look for the `build_step` in the easyblock).* - toolchain-specific aspects of the build and installation procedure (e.g., configure options); *For example, the [easyblock for CP2K](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/c/cp2k.py) will add several compiler options when compiling with gcc and gfortran, including the infamous @@ -172,58 +172,6 @@ We will cover the most commonly used ones here, but keep in mind that these are A full overview of all known easyconfig parameters can be obtained via "`eb --avail-easyconfig-params`" or just "`eb -a`" for short, or can be consulted in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_parameters.html). -#### Sources, patches, and checksums - -In most easyconfig files you will see that a list of source files is specified via the `sources` -easyconfig parameter, usually combined -with one or more URLs where these sources can be downloaded specified via `source_urls`. -There also may be patch files listed (specified via `patches`), -and checksums for both the source files and patches (specified via `checksums`). - -The `sources` easyconfig parameter is commonly defined but it is *not* mandatory, -because some easyconfig files only specify bundles of software packages and hence only -serve to generate a module file. - -Here is an example of how these easyconfig parameters can be specified: - -```python -source_urls = [ - 'https://example.org/download/', - 'https://example.org/download/archive/', -] -sources = ['example-1.0-src.tar.gz'] -patches = ['example-fix.patch'] -checksums = [ - '9febae18533d035ac688d977cb2ca050e6ca8379311d7a14490ad1ef948d45fa', - '864395d648ad9a5b75d1a745c8ef82b78421d571584037560a22a581ed7a261c', -] -``` - -Each of these require a *list* of values, so even if there is only a single source file or download URL -you must use square brackets as shown in the example. The default value for each of these is an empty list (`[]`). - -Some things worth pointing out here: - -* The download URLs specified via `source_urls` do *not* include the name of the file, that is added - automatically by EasyBuild when it tries to download the file (only if it's not available already.) -* If multiple download URLs are specified, they are each tried once in order until the download of the source file was - successful. This can be useful to include backup locations where source files can be downloaded from. -* Names of source files and patches should not include hardcoded software versions, they usually use a - template value like `%(version)s` instead: - ```python - sources = ['example-%(version)s-src.tar.gz'] - ``` - EasyBuild will use the value of the `version` easyconfig parameter to determine the actual name of the source - file. This way the software version is only specified in one place and the easyconfig file is easier to - update to other software versions. A list of template values can be consulted via the EasyBuild command - line via the `--avail-easyconfig-templates` option, or in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_templates.html). -* Source files can also be specified in ways other than just using a filename, see the - [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-param-sources-alt) for more information. - It is also possible to download a given commit from a GitHub repository. -* Specified checksums are usually SHA256 checksum values, but - [other types are also supported](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html?highlight=checksums#checksums). - - #### Easyblock The easyblock that should be used for the installation can be specified via the `easyblock` easyconfig parameter. @@ -277,6 +225,58 @@ prefix_opt* Prefix command line option for configure script ('--pref tar_config_opts* Override tar settings as determined by configure. [default: False] ``` +#### Sources, patches, and checksums + +In most easyconfig files you will see that a list of source files is specified via the `sources` +easyconfig parameter, usually combined +with one or more URLs where these sources can be downloaded specified via `source_urls`. +There also may be patch files listed (specified via `patches`), +and checksums for both the source files and patches (specified via `checksums`). + +The `sources` easyconfig parameter is commonly defined but it is *not* mandatory, +because some easyconfig files only specify bundles of software packages and hence only +serve to generate a module file. + +Here is an example of how these easyconfig parameters can be specified: + +```python +source_urls = [ + 'https://example.org/download/', + 'https://example.org/download/archive/', +] +sources = ['example-1.0-src.tar.gz'] +patches = ['example-fix.patch'] +checksums = [ + '9febae18533d035ac688d977cb2ca050e6ca8379311d7a14490ad1ef948d45fa', + '864395d648ad9a5b75d1a745c8ef82b78421d571584037560a22a581ed7a261c', +] +``` + +Each of these require a *list* of values, so even if there is only a single source file or download URL +you must use square brackets as shown in the example. The default value for each of these is an empty list (`[]`). + +Some things worth pointing out here: + +* The download URLs specified via `source_urls` do *not* include the name of the file, that is added + automatically by EasyBuild when it tries to download the file (only if it's not available already.) +* If multiple download URLs are specified, they are each tried once in order until the download of the source file was + successful. This can be useful to include backup locations where source files can be downloaded from. +* Names of source files and patches should not include hardcoded software versions, they usually use a + template value like `%(version)s` instead: + ```python + sources = ['example-%(version)s-src.tar.gz'] + ``` + EasyBuild will use the value of the `version` easyconfig parameter to determine the actual name of the source + file. This way the software version is only specified in one place and the easyconfig file is easier to + update to other software versions. A list of template values can be consulted via the EasyBuild command + line via the `--avail-easyconfig-templates` option, or in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_templates.html). +* Source files can also be specified in ways other than just using a filename, see the + [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-param-sources-alt) for more information. + It is also possible to download a given commit from a GitHub repository. +* Specified checksums are usually SHA256 checksum values, but + [other types are also supported](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html?highlight=checksums#checksums). + + #### Dependencies You will often need to list one or more [dependencies](../../1_Intro/1_05_terminology/#dependencies) that are required @@ -306,9 +306,10 @@ builddependencies = [ ] dependencies = [ - ('Python', '3.8.2'), - ('HDF5', '1.10.6'), - ('SciPy-bundle', '2020.03', '-Python-%(pyver)s'), + ('cray-hdf5', EXTERNAL_MODULE), + ('cray-netcdf', EXTERNAL_MODULE), + ('GSL', '2.7''), + ('ANTLR', '2.7.7', '-python3'), ] ``` @@ -316,14 +317,10 @@ Both `builddependencies` and `dependencies` require a list of tuples, each of which specifying one dependency. The name and version of a dependency is specified with a 2-tuple (a tuple with two string values). -In some cases additional information may have to be provided, as is shown in the example above for the `SciPy-bundle` +In some cases additional information may have to be provided, as is shown in the example above for the `ANTLR` dependency where a 3rd value is specified corresponding to the `versionsuffix` value of this dependency. If this is not specified, it is assumed to be the empty string (`''`). -Note how we use the '`%(pyver)s'` template value in the `SciPy-bundle` dependency -specification, to avoid hardcoding the Python version in different places. (Though this -specific parameter is less useful on LUMI as we currently try to build on top of `cray-python`.) - The `buildtools` build dependency shows that there is a fourth parameter specifying the toolchain used for that dependency and is needed if that toolchain is different from the one used in the example. As it is not possible to load several Cray toolchains together (they are not in a hierarchical relation) @@ -332,10 +329,34 @@ toolchain. Here also we use a template, `%(toolchain_version)s` which - as its n to the version of the toolchain, as we version our `buildtools` modules after the version of the Cray toolchains for which they are intended. +When using the HPE Cray PE based toolchains, another type of dependency comes in: +[external modules](../2_03_external_modules) (discussed in the next section) that +are used to interface with modules provided by the HPE Cray PE but could also be +used to interfact with other modules that do not contain the metadata that EasyBuild +includes in module files that it generates. (EasyBuild sets a number of EasyBuild-specific +environment variables in each module, including one pointing to the installation directory +and one specifying the version of the packages.) + +Another example (with modules taken from the EasyBuild common toolchains, not from a repository +on LUMI) is + +```python +dependencies = [ + ('Python', '3.9.6'), + ('HDF5', '1.12.1'), + ('SciPy-bundle', '2021.10', '-Python-%(pyver)s'), +] +``` + +Note how we use the '`%(pyver)s'` template value in the `SciPy-bundle` dependency +specification, to avoid hardcoding the Python version in different places. (Though this +specific parameter is less useful on LUMI as we currently try to build on top of `cray-python`.) + See also the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#dependencies) for additional options on specifying dependencies. That page specifies two more dependency types: -* `hiddendependencies` are currently not used on LUMI, and if we would use them in the future, it will likely +* `hiddendependencies` are currently not used on LUMI nor in the easyconfigs included with EasyBuild, + and if we would use them in the future, it will likely be through a way that does not require this parameter. * `osdependencies` can be used to let EasyBuild check if certain needed OS packages are installed. See, e.g., the [easyconfigs for the `buildtools` package](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/easybuild/easyconfigs/b/buildtools) @@ -910,7 +931,7 @@ $ eb eb-tutorial.eb --inject-checksums Finally, we should consider changing the name of the easyconfig that we just developed to align with the EasyBuild conventions as otherwise it would not be found when used as a dependency of another package. In this case, the name should be -`eb-tutorial-1.0.0-cpeCray-21.12.eb`. In fact, EasyBuild stored a processed version +`eb-tutorial-1.0.1-cpeCray-21.12.eb`. In fact, EasyBuild stored a processed version of our easyconfig with that name in the repository: ``` diff --git a/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md index 1a9cb29f..5322678f 100644 --- a/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md +++ b/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md @@ -231,14 +231,18 @@ If want to *add* to the existing value however, you *must* use the ``self.cfg.up self.cfg.update('some_list', 'example') ``` -This will ***not*** work (because ``self.cfg['some_list']`` does not return a reference to the original value, -but to a *temporary copy* thereof): +One could be tempted to use ```python # anti-pattern, this does NOT work as expected! self.cfg['some_list'].append('example') ``` +instead, but this will ***not*** work because ``self.cfg['some_list']`` does not return a reference to the original value, +but to a *temporary copy* thereof. + + + ### Custom parameters Additional custom easyconfig parameters can be defined in an easyblock to steer its behaviour. diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md b/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md index f91f7d7c..74f3f82a 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md @@ -29,7 +29,8 @@ community: - Enforce some site policies on easyconfig files. A practical example for LUMI could be imposing the use of the `whatis` parameter and ensuring that it contains a `Description` line to have a - more consistent format for module files. + more consistent format for module files (or, alternatively, imposing this if the mandatory + `description` parameter is to long to be good for a `whatis` line). - Ensuring that some information is always in the module file. E.g., on LUMI the `site_contacts` parameter is added automatically when installing in the central stack if it is not already in the easyconfig file to refer to the LUST support pages. @@ -39,6 +40,7 @@ community: adapt, e.g., the easyconfig files for Open MPI to work on LUMI. Some site have really large hook files to implement policies and modify standard EasyBuild build recipes, e.g., + - [The hooks file from Jülich Supercomputing Centre](https://github.com/easybuilders/JSC/blob/2022/Custom_Hooks/eb_hooks.py) - ComputeCanada [cc_hooks_common.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_common.py) and [cc_hooks_gento.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_gentoo.py) diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md index ec7a36a5..d49a7925 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md @@ -44,7 +44,8 @@ This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environm that are in place in the job environment are most likely *irrelevant*, since configuration settings they specify they will most likely be overruled by the corresponding command line options. It does also imply however that the EasyBuild configuration that is in place when ``eb --job`` is used -does also work on the compute nodes to which the job is submitted. +should also work on the compute nodes to which the job is submitted as EasyBuild will generate the command +line options used in the job script based on that configuration. ## Using ``eb --job`` @@ -90,7 +91,7 @@ using the ``sbatch`` command. EasyBuild currently doesn't provide away to customize the Slurm submission options, for example to submit to a particular partition, or to use a particular account, -build you can set the corresponding ``$SBATCH_*`` environment variables prior to running ``eb --job``. +build you can set the corresponding ``SBATCH_*`` environment variables prior to running ``eb --job``. For example, to specify a particular account that should be used for the jobs submitted by EasyBuild (equivalent with using the ``-A`` or ``--account`` command line option for ``sbatch``): @@ -105,14 +106,14 @@ Or to submit to a particular Slurm partition (equivalent with the ``-p`` or ``-- export SBATCH_PARTITION='small' ``` -For more information about supported ``$SBATCH_*`` environment variables, +For more information about supported ``SBATCH_*`` environment variables, see the [Slurm documentation](https://slurm.schedmd.com/sbatch.html#lbAJ). ## Combining ``--job`` and ``--robot`` If one or more dependencies are still missing for the software you want to install, you can combine ``--job`` and ``--robot`` to get EasyBuild to submit a *separate* job -for each of the installations. These jobs will *not* ``--robot``, they will each only +for each of the installations. These jobs will *not* use ``--robot``, they will each only perform a single installation. Dependencies between jobs will be "registered" at submission time, so Slurm will put jobs @@ -218,9 +219,10 @@ module load LUMI/21.12 module load partition/C module load EasyBuild-user -# use ramdisk for build directories -export EASYBUILD_BUILDPATH=/dev/shm/$USER/build -export EASYBUILD_TMPDIR=/dev/shm/$USER/tmp +# use /tmp for build directories and temporary directories as we have those +# on login and compute nodes. +export EASYBUILD_BUILDPATH=/tmp/$USER/build +export EASYBUILD_TMPDIR=/tmp/$USER/tmp # use Slurm as job backend export EASYBUILD_JOB_BACKEND=Slurm diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md index 4a6346c5..0a6b712d 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md @@ -93,7 +93,9 @@ There are a couple of advantages to this approach: However, the are some minor disadvantages too: * not all existing modules are directly visible, so users have to learn how to find - modules using ``module spider`` etc.; + modules using ``module spider`` etc. (but that is not an issue on LUMI as users + also need to use this command anyway to discover software that may not be in all + software stacks); * gateway modules may have little meaning to end users; #### Length of module names diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md index 7495343a..9900e1b3 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_05_github_integration.md @@ -4,7 +4,7 @@ --- -!!! Warning "Preliminary not for LUMI and many other Cray systems" +!!! Warning "Preliminary note for LUMI and many other Cray systems" Not everything in this section is as useful for EasyBuild installations that build on the Cray Programming Environment. E.g., the default easyconfig repository does From 8f29ae4074c12834300601cf7dc97e2c2b1b7119 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 10 May 2022 17:52:45 +0200 Subject: [PATCH 047/159] Corrected two typos. --- docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md | 2 +- docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md b/docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md index 7439177e..99250db2 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md @@ -32,7 +32,7 @@ LUMI has different node types providing compute resources: and are equipped with SlingShot 10 interconnect cards (2 each???) - 8 nodes have zen2 CPUs and 8 NVIDIA A40 GPUs each, and have 2 SlingShot 10 interconnect cards each. -- The early access platform (EAP) has 14 nodes equiped with a single 64-core +- The early access platform (EAP) has 14 nodes equipped with a single 64-core zen2 CPU and 4 AMD MI100 GPUS. Each node has a single SlingShot 10 interconnect and also local SSDs. diff --git a/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md b/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md index ee6716fd..6849061e 100644 --- a/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md +++ b/docs/2022-CSC_and_LO/2_Using/2_01_troubleshooting.md @@ -608,7 +608,7 @@ the log file in your favourite editor. Search for the build step by searching fo `INFO Starting build` and look carefully at how the program was actually build... You'll very likely have to look at the solution to understand how to correct the -problems as that requires more advanced knowlege than we have at this point in +problems as that requires more advanced knowledge than we have at this point in the tutorial, but try to figure out what could be wrong first though... ??? hint "(Click for a hint)" From 8f4dd154d0d05b766960015e4e5342a23b928c2b Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 11 May 2022 13:49:40 +0200 Subject: [PATCH 048/159] Corrections to the example --- .../2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md index f9cb4577..9acc9c89 100644 --- a/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md @@ -894,13 +894,13 @@ version = '1.0.1' homepage = 'https://easybuilders.github.io/easybuild-tutorial' description = "EasyBuild tutorial example" -toolchain = {'name': 'GCC', 'version': '10.2.0'} +toolchain = {'name': 'cpeCray', 'version': '21.12'} source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] sources = [SOURCE_TAR_GZ] checksums = ['d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9'] -builddependencies = [('CMake', '3.18.4')] +builddependencies = [('buildtools', '%(toolchain_version)s', '', True)] configopts = "-DEBTUTORIAL_MSG='Hello from the EasyBuild tutorial!' " From 430f584e5a9a963b661b56660762c5c174e2fd6d Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 11 May 2022 13:56:58 +0200 Subject: [PATCH 049/159] Removed an unnecessary accent. --- docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md index 9acc9c89..6c2dc02e 100644 --- a/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md @@ -598,7 +598,7 @@ and also `EasyBuild-user` is loaded to configure EasyBuild to install in ``` shell module load LUMI/21.12 -module load EasyBuild-user` +module load EasyBuild-user ``` From 10a3be848299a857d033d5536f58756bb76082e4 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 11 May 2022 14:02:41 +0200 Subject: [PATCH 050/159] Corrected wrong termination of code block. --- docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md index 6c2dc02e..8faf07b6 100644 --- a/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md @@ -1060,7 +1060,8 @@ You can leverage the `eb-tutorial` easyconfig file we have composed in the examp directory. As on LUMI the repository is in the search path we can actually copy the file back to the current directory: ``` - eb --copy-ec eb-tutorial-1.1.0-cpeCray-21.12. ``` + eb --copy-ec eb-tutorial-1.1.0-cpeCray-21.12. + ``` Some of the formatting is lost though and the checksum is still missing, so you may want to do some cleaning up. ``` From 728e972270a4dc19c458a23d2c72ea9dc973b980 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 7 Mar 2022 17:29:52 +0100 Subject: [PATCH 051/159] Initial commit, already contains the structure to work on the first tutorial. --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index e84c0d54..f7ed0418 100644 --- a/README.md +++ b/README.md @@ -22,11 +22,11 @@ which makes it very easy to preview the result of the changes you make locally. make preview - or + or mkdocs serve - Visit http://127.0.0.1:8000 to see the local live preview of the changes you make. + Visit http://127.0.0.1:8000 to see the local live preview of the changes you make. * If you prefer building a static preview you can use ``make`` or ``mkdocs build``, which should result in a ``site/`` subdirectory that contains the rendered documentation. From f20177b5fa73c2aa699b6c062df62c6970f9bf44 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 7 Mar 2022 18:30:05 +0100 Subject: [PATCH 052/159] Updated the "What is EasyBuild" page of the tutorial for LUMI." --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 12 + .../2022-CSC_and_LO/1_01_what_is_easybuild.md | 321 ++++++++++++++++++ 2 files changed, 333 insertions(+) create mode 100644 docs/2022-CSC_and_LO/1_00_part1_intro.md create mode 100644 docs/2022-CSC_and_LO/1_01_what_is_easybuild.md diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md new file mode 100644 index 00000000..0bfd0ad5 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -0,0 +1,12 @@ +# Part I: Introduction to EasyBuild on Cray systems + +*[[back to start page]](index.md)* + +*(Tue March 4th 2021, 9am-12 CET)* + +- [What is EasyBuild?](1_01_what_is_easybuild.md) + +- [Terminology](terminology.md) +- [Installation](installation.md) *(hands-on)* +- [Configuration](configuration.md) *(hands-on)* +- [Basic usage](basic_usage.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md new file mode 100644 index 00000000..678c8c4a --- /dev/null +++ b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md @@ -0,0 +1,321 @@ +# What is EasyBuild? + +*[[back: Overview]](index.md)* + +

EasyBuild logo

+ +**[EasyBuild](http://easybuild.io)** is a **software build and installation framework** +that manages (scientific) software in an efficient way, without compromising on performance. +Easybuild is especially tailored towards [**High Performance Computing (HPC)**](https://en.wikipedia.org/wiki/Supercomputer) systems, +and provides a **command-line interface** to facilitate the installation and management of a scientific software stack. + +--- + + + + + +### *Elevator pitch* + +EasyBuild intends to relieve HPC support teams from manually managing +software installations while at the same time **providing a consistent and well performing +scientific software stack** to end users of HPC systems. + +It serves as a **uniform interface for installing scientific software** +and saves valuable time (and frustration) by the automation of tedious, boring and repetitive tasks. + +In addition, EasyBuild can **empower scientific researchers to self-manage their software stack**, +and it can serve as a tool that can be leveraged for **building optimized container images**. + +The project has grown to become a **platform for collaboration** among HPC sites worldwide. + +--- + +On LUMI we pay a lot of attention to empowering users to create their custom software stack which +is best suited for their project without being confronted with hundreds of modules that are not +useful to them. It also enables them to easily customize the build instructions to create a +project-specific build or even to install software with custom patches to adapt to their needs. +For this we offer both [EasyBuild](https://easybuild.io/) and [Spack](https://spack.io/) +as tools but obviously this tutorial concentrates on EasyBuild alone. + +EasyBuild can also be used to [generate container recipes](https://docs.easybuild.io/en/latest/Containers.html) +that will then use EasyBuild when building +the container to install platform-optimised software in the container. However, as that functionality +is still under development on LUMI it is not yet covered in this LUMI-specific version of the tutorial. + + +--- + +### *Key features* + +EasyBuild is capable of **fully autonomously installing (scientific) software**, +including making sure that all necessary dependencies are installed, +and automatically generating environment module files. + +***No*** **admin privileges are required**: it is sufficient to have write permissions +to the preferred software installation prefix. + +EasyBuild is [**highly configurable**](https://docs.easybuild.io/en/latest/Configuration.html) via configuration files, +environment variables, and command line options. Various elements of the functionality can be +[**dynamically extended**](https://docs.easybuild.io/en/latest/Including_additional_Python_modules.html) via plugins, +and [**hooks**](https://docs.easybuild.io/en/latest/Hooks.html) are available for further site-specific customizations. + +The installation procedure executed by EasyBuild is thoroughly +[**logged**](https://docs.easybuild.io/en/latest/Logfiles.html), and is fully transparent via support for +[**dry runs**](https://docs.easybuild.io/en/latest/Extended_dry_run.html) and +[**tracing**](https://docs.easybuild.io/en/latest/Tracing_progress.html) the software installation procedure +as it is performed. + +EasyBuild supports using a **custom module naming scheme**, allows for +*hierarchical* module naming schemes, and **integrates with various other tools** ranging from resource managers ([Slurm](https://slurm.schedmd.com) and [GC3Pie](https://github.com/gc3pie/gc3pie)), +container tools ([Singularity](https://github.com/hpcng/singularity) and [Docker](https://www.docker.com)), +packaging tools ([FPM](https://fpm.readthedocs.io)), and so on. + +--- + +### *Focus points* + +EasyBuild was created specifically for **installing scientific software on HPC systems**, +which is reflected in some of the design choices that were made. + + +#### Performance + +EasyBuild strongly prefers to **build software from source code**, whenever possible. + +This is important to ensure that the binaries that are installed can maximally exploit +the capabilities of the system architecture on which the software will be run. + +For that same reason, EasyBuild out-of-the-box **optimizes software for the processor architecture of the build host** +by default, via compiler options like ``-march=native`` (GCC), ``-xHost`` (Intel compilers), etc. +This behaviour [can be changed via the ``--optarch`` configuration setting](https://docs.easybuild.io/en/latest/Controlling_compiler_optimization_flags.html). +When using the HPE Cray Programming Environment, the target modules are used instead. +In principle it is possible to cross-compile with EasyBuild, but this comes with some pitfalls. +By default EasyBuild tries to run tests if the build process of a package provides that +functionality, or can run simple tests of its own, and of course these tests will fail if +the host CPU does not support the full instruction set of the target CPU. Secondly, more +and more software has broken build processes where developers thought that it was a good +idea to hard-code options such as ``-march=native`` or ``--xHost`` in a way that EasyBuild +cannot easily overwrite those. + + +#### Reproducibility + +In addition to performance, **reproducibility of installations** is a core aspect of EasyBuild. + +TODO correct link in the next lines. + +Most software installations performed with EasyBuild use a **particular compiler toolchain**, +with which we aim to be in control over the build environment and avoid relying on tools and libraries +provided by the operating system. For similar reasons, we try to **provide all required dependencies through EasyBuild** as well, +with a few notable exceptions, like ``OpenSSL`` for security reasons, and Infiniband and GPU drivers which +are too closely intertwined with the operating system. + +For both toolchains and dependencies, **fixed software versions** are specified in the +easyconfig files. That way, easyconfig files can easily be shared with others: +if they worked for you it is very likely that they will work for others too, because the vast majority of the +software stack is controlled by EasyBuild. + +As we shall see when discussing the HPE Cray Programming Environment, the reproducibility is a bit more limited +in that environment due to the way the programming environment selects shared libraries at runtime. + + +#### Community effort + +In a number of different ways, we try to encourage EasyBuild users to **collaborate** and help each other out. + +We actively recommend people to report problems and bugs, to submit ideas for additional features and improvements, +and to [**contribute back**](https://docs.easybuild.io/en/latest/Contributing.html) when possible, be it +by opening pull requests to the [GitHub repositories](https://github.com/easybuilders) or the [documentation](https://docs.easybuild.io). + +The EasyBuild community also provides packages for two standard toolchains, called the +[**common toolchains**](https://easybuild.readthedocs.io/en/latest/Common-toolchains.html). +The so-called ``foss`` is based on free and open software (GNU compilers, FlexiBlas with OpenBLAS backend, LaPACK, FFTW and Open MPI) +while the ``intel`` toolchain uses the full stack of Intel compilers, libraries and MPI. Currently these toolchains +cannot be supported on LUMI though as getting Open MPI to run on LUMI is challenging and as the Intel libraries and +some MPI versions also produce slow or incorrect results on AMD CPUs (and it is not clear if Intel MPI is compatible +with SlingShot 11). +Focusing the efforts on those two popular common toolchains makes it possible to support a lot of build recipes that are +collected in the [central repository](https://github.com/easybuilders/easybuild-easyconfigs). +They can still serve as a basis to develop build recipes for the HPE Cray Programming Environment. + +Last but not least, EasyBuild provides various [**GitHub integration features**](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) +that greatly facilitate the contribution process: opening, updating, and testing pull requests, +reviewing incoming contributions, and much more can all be done directly from the EasyBuild +command line. This not only saves time, effort, brain cycles, and mouse clicks for contributors, +but it also makes the review process for *maintainers* significantly easier. +All together this leads to improved stability and consistency. + +--- + +### *What EasyBuild is* ***not*** + +EasyBuild is ***not*** **YABT (Yet Another Build Tool)**: it does *not* replace established build +tools like CMake or ``make``, it wraps around them. +If the installation procedure of a software package involves running some unholy trinity of tools while whispering +the correct magic incantations, EasyBuild automates this process for you. + +It is ***not*** **a replacement for traditional Linux package managers** like ``yum``, ``dnf`` or ``apt``. +EasyBuild relies on certain tools and libraries provided by the operating system. This includes glibc, OpenSSL, +drivers for Infiniband and GPUs, and so on. It is required that these tools are installed and managed by other means. +This is typically done via the package management tool that comes with your Linux distribution. + +Finally, EasyBuild is ***not*** **a magic solution to all your (software installation) problems**. +You may, and probably still will occasionally, run into compiler errors unless somebody has already taken care of the problem for you. +This is even more likely to happen in the Cray environment as builds are not often tested with the Cray compilers +or even with the AMD compilers. + +--- + +### *Requirements* + +Since EasyBuild is primarily intended for installing (scientific) software on HPC systems, the main target is +systems running some flavor of **GNU/Linux**. EasyBuild also works on *macOS*, +but significant effort would be required to use it extensively there. + + + +EasyBuild is implemented in [**Python**](https://www.python.org), and is compatible with Python 2.7 and 3.5+. +Since [Python 2 is no longer maintained](https://blog.python.org/2020/04/python-2718-last-release-of-python-2.html), +it is recommended to use a recent version of Python 3 for running EasyBuild. + +An [**environment modules**](https://en.wikipedia.org/wiki/Environment_Modules_(software)) tool is required for running EasyBuild. +The modern Lua-based implementation **[Lmod](https://lmod.readthedocs.io)** is strongly recommended, +but other (Tcl-based) implementations are also supported. +Both Tcl and Lua are supported as syntax for environment module files. +On LUMI Lmod is used as the module tool. + +--- + +### *Development* + + +EasyBuild is **actively developed** by a worldwide community under the +[**GPLv2 open source license**](https://www.gnu.org/licenses/old-licenses/gpl-2.0.html) +via the [**``easybuilders``** organisation on GitHub](https://github.com/easybuilders), in the following repositories: + +- [easybuild-framework](https://github.com/easybuilders/easybuild-framework) +- [easybuild-easyblocks](https://github.com/easybuilders/easybuild-easyblocks) +- [easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs) +- [easybuild](https://github.com/easybuilders/easybuild) (website, documentation) +- [easybuild-tutorial](https://github.com/easybuilders/easybuild-tutorial) (the generic EasyBuild tutorials) + +On LUMI the generic EasyBuild installation is used but extended and adapted using LUMI-specifics +plugins and hooks maintained in the [LUMI software stack repository](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack). + +Stable versions of EasyBuild have been released roughly every 6-8 weeks since 2012, +and are published via the *Python Package Index (PyPI)*, +see [https://pypi.org/project/easybuild](https://pypi.org/project/easybuild/). + +**Comprehensive testing** practices are applied throughout the +development cycle, with extensive suites of unit and integration tests being run in a CI environment (GitHub Actions), +consistent testing of incoming contributions (pull requests), and thorough regression testing before every release. + +--- + +### *A worldwide community* + +EasyBuild has grown into a **community project** since it was first released publicy in 2012, +and is used and developed by dozens (hundreds?) of HPC centres and consortia worldwide. + +Since 2016 the EasyBuild community has gathered for a yearly [**EasyBuild User Meeting**](https://easybuild.io/eum), +which is an open and highly interactive event that provides a great opportunity to meet fellow EasyBuild users, +developers, and maintainers, and to discuss related topics, learn about new aspects of the tool, and how other HPC +sites leverage it, and integrate it with other tools like [Jenkins](https://www.youtube.com/watch?v=bzW45cfRUMU), +[Ansible](https://www.youtube.com/watch?v=Qr2udsAjkG0), [CernVM-FS](https://www.youtube.com/watch?v=_0j5Shuf2uE), +[ReFrame](https://www.youtube.com/watch?v=KN9QwQIrso4) etc. + +HPC sites, consortia, and companies that use EasyBuild include (but are not limited to): + +VSC logo +JSC logo +HPCNow! logo +Fred Hutch logo +Vienna BioCenter logo +CÉCI logo +Compute Canada logo +UiO logo +BEAR logo +SNIC logo +National University of Singapore logo +IT4Innovations logo +Texas A&M logo +TU Dresden logo +University of Melbourne logo +University of Luxembourg logo +EPCC logo +Cyprus Institute logo +University of Basel logo +Microsoft logo + +--- + +### *Getting help* + +#### *Documentation* + +If you have any questions about EasyBuild, make sure to check the EasyBuild documentation first, +which is available at [**https://docs.easybuild.io**](https://docs.easybuild.io). + +It is not unlikely that an answer to your question is available there already. +The [search box](https://docs.easybuild.io/en/latest/search.html) on the top left may be helpful. + +#### GitHub + + + +You can open an issue in one of the EasyBuild repositories in [the `easybuilders` GitHub organisation](https://github.com/easybuilders/) to +ask questions, report problems or share suggestions and ideas for additional +improvements. + +Please try to post the issue in the most suitable repository. If you are +not sure which repository is most suited just go ahead and open the issue anyway, +the EasyBuild maintainers can move your issue if deemed appropriate. + +Also here you can use the search box on the top left, to see if somebody else +has already asked a similar question or reported a similar problem. + +#### Mailing list + +Feel free to ask any questions or provide feedback via the EasyBuild mailing list +`easybuild@lists.ugent.be`. + +Keep in mind that **you must be subscribed to the mailing list in order to +post messages there!** You can subscribe via [this link](https://lists.ugent.be/wws/subscribe/easybuild). + +In 2016-2017 over 1,000 messages were posted to the EasyBuild mailing list. +More recently this has decreased to about 700 messages per year, mostly due +to the EasyBuild Slack channel. + +#### Slack + +Slack logo + +The [EasyBuild Slack](https://easybuild.slack.com/) is a quick and easy way +to get in touch with the EasyBuild community. +Over 400 people have joined it so far, and there is usually somebody awake in there +who may be able to help you out. + +**To join the EasyBuild Slack request an invitation via +[https://easybuild-slack.herokuapp.com](https://easybuild-slack.herokuapp.com/).** + +If you are not a big fan of Slack, you can also consider joining the `#easybuild` +IRC channel (available on the [Freenode IRC network](https://freenode.net/)). +A bot relays messages between this IRC channel and the `#general` channel in the EasyBuild Slack. + + +#### Conference calls + +Every other week on Wednesday an EasyBuild conference call is set up to discuss recent +developments, provide an outlook to the next EasyBuild release, and +provide answers to questions that can be raised by the people attending the call. + +These conference calls are open to anyone interested in the EasyBuild project, +and have been organised since November 2013. + +More information, including the dates and times of the next calls, is available +[on the EasyBuild Wiki](https://github.com/easybuilders/easybuild/wiki/Conference-calls). + +--- + +*[[next: Terminology]](terminology.md)* From df7f0f036cdbb13837b675eaeb82f2f2c6582aa0 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 9 Mar 2022 10:58:10 +0100 Subject: [PATCH 053/159] Updated the structure, initial updates of terminology and a new page about the Cray PE and Lmod. --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 4 +- .../2022-CSC_and_LO/1_01_what_is_easybuild.md | 34 +- docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md | 71 ++++ docs/2022-CSC_and_LO/1_03_terminology.md | 348 ++++++++++++++++++ 4 files changed, 438 insertions(+), 19 deletions(-) create mode 100644 docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md create mode 100644 docs/2022-CSC_and_LO/1_03_terminology.md diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md index 0bfd0ad5..e381f45d 100644 --- a/docs/2022-CSC_and_LO/1_00_part1_intro.md +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -5,8 +5,8 @@ *(Tue March 4th 2021, 9am-12 CET)* - [What is EasyBuild?](1_01_what_is_easybuild.md) - -- [Terminology](terminology.md) +- [Terminology](1_02_terminology.md) + - [Installation](installation.md) *(hands-on)* - [Configuration](configuration.md) *(hands-on)* - [Basic usage](basic_usage.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md index 678c8c4a..5ccdb1db 100644 --- a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md @@ -15,7 +15,7 @@ and provides a **command-line interface** to facilitate the installation and man -### *Elevator pitch* +## Elevator pitch EasyBuild intends to relieve HPC support teams from manually managing software installations while at the same time **providing a consistent and well performing @@ -46,7 +46,7 @@ is still under development on LUMI it is not yet covered in this LUMI-specific v --- -### *Key features* +## Key features EasyBuild is capable of **fully autonomously installing (scientific) software**, including making sure that all necessary dependencies are installed, @@ -73,13 +73,13 @@ packaging tools ([FPM](https://fpm.readthedocs.io)), and so on. --- -### *Focus points* +## Focus points EasyBuild was created specifically for **installing scientific software on HPC systems**, which is reflected in some of the design choices that were made. -#### Performance +### Performance EasyBuild strongly prefers to **build software from source code**, whenever possible. @@ -99,7 +99,7 @@ idea to hard-code options such as ``-march=native`` or ``--xHost`` in a way that cannot easily overwrite those. -#### Reproducibility +### Reproducibility In addition to performance, **reproducibility of installations** is a core aspect of EasyBuild. @@ -120,7 +120,7 @@ As we shall see when discussing the HPE Cray Programming Environment, the reprod in that environment due to the way the programming environment selects shared libraries at runtime. -#### Community effort +### Community effort In a number of different ways, we try to encourage EasyBuild users to **collaborate** and help each other out. @@ -148,7 +148,7 @@ All together this leads to improved stability and consistency. --- -### *What EasyBuild is* ***not*** +## What EasyBuild is ***not*** EasyBuild is ***not*** **YABT (Yet Another Build Tool)**: it does *not* replace established build tools like CMake or ``make``, it wraps around them. @@ -167,7 +167,7 @@ or even with the AMD compilers. --- -### *Requirements* +## Requirements Since EasyBuild is primarily intended for installing (scientific) software on HPC systems, the main target is systems running some flavor of **GNU/Linux**. EasyBuild also works on *macOS*, @@ -187,7 +187,7 @@ On LUMI Lmod is used as the module tool. --- -### *Development* +## Development EasyBuild is **actively developed** by a worldwide community under the @@ -213,7 +213,7 @@ consistent testing of incoming contributions (pull requests), and thorough regre --- -### *A worldwide community* +## A worldwide community EasyBuild has grown into a **community project** since it was first released publicy in 2012, and is used and developed by dozens (hundreds?) of HPC centres and consortia worldwide. @@ -250,9 +250,9 @@ HPC sites, consortia, and companies that use EasyBuild include (but are not limi --- -### *Getting help* +## Getting help -#### *Documentation* +### *Documentation* If you have any questions about EasyBuild, make sure to check the EasyBuild documentation first, which is available at [**https://docs.easybuild.io**](https://docs.easybuild.io). @@ -260,7 +260,7 @@ which is available at [**https://docs.easybuild.io**](https://docs.easybuild.io) It is not unlikely that an answer to your question is available there already. The [search box](https://docs.easybuild.io/en/latest/search.html) on the top left may be helpful. -#### GitHub +### GitHub @@ -275,7 +275,7 @@ the EasyBuild maintainers can move your issue if deemed appropriate. Also here you can use the search box on the top left, to see if somebody else has already asked a similar question or reported a similar problem. -#### Mailing list +### Mailing list Feel free to ask any questions or provide feedback via the EasyBuild mailing list `easybuild@lists.ugent.be`. @@ -287,7 +287,7 @@ In 2016-2017 over 1,000 messages were posted to the EasyBuild mailing list. More recently this has decreased to about 700 messages per year, mostly due to the EasyBuild Slack channel. -#### Slack +### Slack Slack logo @@ -304,7 +304,7 @@ IRC channel (available on the [Freenode IRC network](https://freenode.net/)). A bot relays messages between this IRC channel and the `#general` channel in the EasyBuild Slack. -#### Conference calls +### Conference calls Every other week on Wednesday an EasyBuild conference call is set up to discuss recent developments, provide an outlook to the next EasyBuild release, and @@ -318,4 +318,4 @@ More information, including the dates and times of the next calls, is available --- -*[[next: Terminology]](terminology.md)* +*[[next: The HPE Cray Programming Environment and Lmod]](1_02_CPE_and_Lmod.md)* diff --git a/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md b/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md new file mode 100644 index 00000000..01812057 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md @@ -0,0 +1,71 @@ +# The HPE Cray Programming Environment and Lmod + +*[[back: What is EasyBuild?]](1_01_what_is_easybuild.md)* + +--- + +On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated +as Cray PE). The environment provides several tools, including compilers, communication libraries, +optmized math libraries and various other libraries, analyzers and debuggers. + +The Cray PE is made available through *environment modules* tha allow to select particular versions of +tools and to configure the environment in a flexible way. + +--- + +## Modules + +*Module* is a massively overloaded term in (scientific) software and IT in general +(kernel modules, Python modules, and so on). +In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. + +[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept +on HPC systems: it is a way to specify changes that should be made to one or more +[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a +[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file +is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or +[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, +and specifies which environment variables should be updated, and how (append, +prepend, (re)define, undefine, etc.) upon loading the environment module. +Unloading the environment module will restore the shell environment to its previous state. + +Environment module files are processed via a **modules tool**, of which there +are several conceptually similar yet slightly different implementations. +The oldest module tool still in use today is Environment Modules 3.2, implemented in C and +supporting module files written in Tcl. After a gap in developement, Xavier Delaruelle of CEA +developed [Environment Modules 4 and 5](https://sourceforge.net/projects/modules/) which is +fully implemented on Tcl. An alternative module tool is [Lmod](https://lmod.readthedocs.io), +developed by Robert McLay at TACC and implemented in LUA. This tool supports natively LUA +module files but also offers a high degree of compatibility with Tcl-based module files +developed for Environment Modules fia a translation layer and some API translation. + +The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no +packages or official support for Environment Modules 4 or 5. On LUMI, Lmod was selected as +the module tool. At the user level, Lmod and the various versions of Emvironment Modules +have very simmilar commands for managing the environment, but with different options. +The commands for searching for modules are very different though so if you are not familiar +with Lmod and its commands for users, it is worthwile to read the +[LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). + +--- + +## Lmod features + + +--- + +## Cray PE components + + +--- + +## Configuring the Cray PE through modules + +--- + +## Further reading + + +--- + +*[[next: Terminology]](1_03_terminology.md)* diff --git a/docs/2022-CSC_and_LO/1_03_terminology.md b/docs/2022-CSC_and_LO/1_03_terminology.md new file mode 100644 index 00000000..ec085b0c --- /dev/null +++ b/docs/2022-CSC_and_LO/1_03_terminology.md @@ -0,0 +1,348 @@ +# Terminology + +*[[back: The HPE Cray Programming Environment and Lmod]](1_02_CPE_and_Lmod.md)* + +--- + +Over the years, we have come up with some terminology specific to EasyBuild +to refer to particular components, which we use alongside established terminology relevant to the context +of building and installing software. + +It is important to be familiar with these terms, so we'll briefly cover them one by one. + +--- + +## *Toolchains* + +A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, +which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. + +We refer to the different parts of a toolchain as **toolchain components**. + +The *compiler component* typically consists of [C](https://en.wikipedia.org/wiki/C_(programming_language)), +[C++](https://en.wikipedia.org/wiki/C%2B%2B), and [Fortran](https://en.wikipedia.org/wiki/Fortran) +compilers in the context of HPC, but additional compilers (for example, +a [CUDA](https://developer.nvidia.com/cuda-zone) compiler for +[GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units) software) +can also be included. + +Additional toolchain components are usually special-purpose libraries: + +- an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/)); + +- libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), + [LAPACK](http://performance.netlib.org/lapack/)); + + These libraries are included because they tend to provide a common API, so mixing them in a single + executable would cause name conflicts when linking. + +- a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); + +A toolchain that includes all of these libraries is referred to as a **full toolchain**, while +a **subtoolchain** is a toolchain that is missing one or more of these libraries. +A **compiler-only toolchain** only consists of compilers (no additional libraries). + +Many EasBuild toolchains are organised in a hierarchy. Each toolchain can use components compiled +with itself or with one of the subtoolchains. + + +### System toolchain + +The **`system` toolchain** is a special case which corresponds to using the compilers and libraries +*provided by the operating system*, rather than using toolchain components that were installed using EasyBuild. + +It used sparingly, mostly to install software where no actual compilation is done or +to build a set of toolchain compilers and its dependencies, since the versions of the system compilers +and libraries are beyond the control of EasyBuild, which could affect the reproducibility of the installation. + +On LUMI however it is used a bit more and it takes some of the functions of the ``GCCcore`` toolchain in +other EasyBuild toolchain hierarchies. + + +### Common toolchains + +The `foss` and `intel` toolchains are also known as the *common toolchains*, +because they are widely adopted by the EasyBuild community. + +The `foss` toolchain consists of all open source components (hence the name: +"FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), +[ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). + +The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version +controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. + +Roughly every 6 months, a new version of these common toolchains is agreed upon +in the EasyBuild community, after extensive testing. + +More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). + +--- + +## *Modules* + +*Module* is a massively overloaded term in (scientific) software and IT in general +(kernel modules, Python modules, and so on). +In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. + +[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept +on HPC systems: it is a way to specify changes that should be made to one or more +[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a +[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file +is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or +[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, +and specifies which environment variables should be updated, and how (append, +prepend, (re)define, undefine, etc.) upon loading the environment module. +Unloading the environment module will restore the shell environment to its previous state. + +Environment module files are processed via a **modules tool**, of which there +are several conceptually similar yet slightly different implementations. +The Tcl-based [Environment Modules](https://sourceforge.net/projects/modules/) implementation, and +[Lmod](https://lmod.readthedocs.io), a more recent Lua-based implementation (which also supports module +files written in Tcl syntax), are the most commonly used ones. + +**Environment module files are automatically generated for each software installation** by EasyBuild, +and *loading* a module results in changes being made to the environment of the current shell +session such that the corresponding software installation can be used. + +--- + +## *EasyBuild framework* + +The EasyBuild *framework* consists of a set of Python modules organised in packages (``easybuild.framework``, +``easybuild.toolchains``, ``easybuild.tools``, etc.) that collectively form **the core of EasyBuild**, +and is developed in the [``easybuild-framework`` repository on GitHub](https://github.com/easybuilders/easybuild-framework). + +It implements the **common functionality that you need when building software from source**, +providing functions for unpacking source files, applying patch files, collecting the output produced +by shell commands that are being run and checking their exit code, generating environment module files, etc. + +The EasyBuild framework does *not* implement any specific installation procedure, it only provides +the necessary functionality to facilitate this. + + +--- + +## *Easyblocks* + +An *easyblock* is **a Python module that implements a specific software installation procedure**, +and can be viewed as a plugin to the EasyBuild framework. Easyblocks can be either *generic* or *software-specific*. + +A **generic easyblock** implements an installation procedure that can be used for +multiple different software packages. Commonly used examples include the ``ConfigureMake`` easyblock +which implements the ubiquitous ``configure``-``make``-``make install`` procedure, and the +``PythonPackage`` easyblock that can be used to install a Python package. + +A **software-specific easyblock** implements an installation procedure that is specific to a particular software packages. +Infamous examples include the easyblocks we have for ``GCC``, ``OpenFOAM``, ``TensorFlow``, ``WRF``, ... + +The installation procedure performed by an easyblock can be controlled by defining +**easyconfig parameters** in an easyconfig file. + +A collection of (generic and software-specific) easyblocks is developed by the EasyBuild community +in the [``easybuild-easyblocks`` repository on GitHub](https://github.com/easybuilders/easybuild-easyblocks). + +--- + +## *Easyconfig parameters* + +An **easyconfig parameter** specifies a particular aspect of a software installation that should be performed by +EasyBuild. + +Some easyconfig parameters are **mandatory**. The following parameters *must* be defined in *every* easyconfig file: + +* ``name`` and ``version``, which specify the name and version of the software to install; +* ``homepage`` and ``description``, which provide key metadata for the software; +* ``toolchain``, which specifies the compiler toolchain to use to install the software (see + ``toolchains`` tab); + +Other easyconfig parameters are **optional**: they can be used to provide required information, +or to control specific aspects of the installation procedure performed by the easyblock. + +Some commonly used optional easyconfig parameters include: + +* ``easyblock``, which specifies which (generic) easyblock should be used for the installation; +* ``sources`` and ``source_urls``, which specify the list of source files and where to download them; +* ``dependencies`` and ``builddependencies``, which specify the list of (build) dependencies; +* ``configopts``, ``buildopts``, and ``installopts``, which specify options for the configuration/build/install commands, respectively; + +If no value is specified for an optional easyconfig parameter, the corresponding default value will be used. + +There are two groups of easyconfig parameters. *General* easyconfig parameters can be defined for any software +package, and (usually) control a specific aspect of the installation. *Custom* easyconfig parameters are +only supported by certain easyblocks, and only make sense for particular (types of) software. + +--- + +## *Easyconfig files* + +*Easyconfig files* (or *easyconfigs* for short), are **simple text files written in Python syntax +that specify what EasyBuild should install**. +Each easyconfig file defines the set of **easyconfig parameters** that collectively form a complete specification +for a particular software installation. + +The **filename** of an easyconfig file usually ends with the ``.eb`` extension. +In some contexts the filename is expected to be determined by the value of a handful of key +easyconfig parameters: ``name``, ``version``, ``toolchain`` and ``versionsuffix``. The general format for +the filename of an easyconfig file is: ``--.eb``, +where the toolchain part is omitted when a ``system`` toolchain is used, and the `` can be empty. + +The filename of easyconfig files is particularly relevant when EasyBuild is searching for easyconfig files to resolve dependencies, +since it does this purely based on filenames: interpreting the contents of every (potential) easyconfig file +it encounters would be too expensive. + +In the [``easybuild-easyconfigs`` repository on GitHub](https://github.com/easybuilders/easybuild-easyconfigs), +the EasyBuild community maintains a large (and growing) collection of easyconfig files, for a wide range of +(scientific) software. + +--- + +## *Easystack files* + +[**Easystack files**](https://docs.easybuild.io/en/latest/Easystack-files.html) +are a new concept in EasyBuild, providing a way to define a *software stack* +that should be installed by EasyBuild. + +They are written in [YAML syntax](https://yaml.org/), and include a list of *software specifications* +which correspond to a list of easyconfig files, with support for providing specific EasyBuild +configuration options for particular software packages, and including or excluding specific software +packages based on labels. + +The support for using easystack files is currently marked as *experimental*, +which means it is subject to change in future EasyBuild releases, and may be prone to errors. + +--- + +## *Extensions* + +*Extensions* is the collective term we use for **additional software packages that can be installed +on top of another software package**. Common examples are *Python packages*, *R libraries*, and *Perl modules*. + +As you can tell the common terminology here is a bit messy, so we came up with a unifying term... + +Extensions can be installed in different ways: + +* *stand-alone*, as a separate installation on top of one or more other installations; +* as a part of a *bundle* of extensions that collectively form a separate installation; +* or as an actual *extension* to a specific installation to yield a "batteries included" + type of installation (for examples by adding a bunch of Python packages from PyPI into + a Python installation); + +--- + +## *Dependencies* + +A *dependency* is a common term in the context of software. It refers to **a software +package that is either strictly required by other software, or that can be leveraged to +enhance other software** (for example to support specific features). + +There are three main types of dependencies for computer software: + +* a **build dependency** is only required when building/installing a software package; + once the software package is installed, it is no longer needed to *use* that software + (examples: `CMake`, `pkg-config`); +* a **run-time dependency** (often referred to simply as *dependency*) is a software package that is + required to *use* (or *run*) another software package (example: `Python`); +* a **link-time dependency** is somewhere in between a build and runtime dependency: + it is only needed when *linking* a software package; it can become either a build or runtime + dependency, depending on exactly how the software is installed (example: `OpenBLAS`); + +The distinction between link-time and run-time dependencies is mostly irrelevant for this tutorial, +but we will discriminate build-only dependencies. + +--- + +## *Toolchains* + +A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, +which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. + +We refer to the different parts of a toolchain as **toolchain components**. + +The *compiler component* typically consists of [C](https://en.wikipedia.org/wiki/C_(programming_language)), +[C++](https://en.wikipedia.org/wiki/C%2B%2B), and [Fortran](https://en.wikipedia.org/wiki/Fortran) +compilers in the context of HPC, but additional compilers (for example, +a [CUDA](https://developer.nvidia.com/cuda-zone) compiler for +[GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units) software) +can also be included. + +Additional toolchain components are usually special-purpose libraries: + +* an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/)); +* libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), + [LAPACK](http://performance.netlib.org/lapack/)); +* a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); + +A toolchain that includes all of these libraries is referred to as a **full toolchain**, while +a **subtoolchain** is a toolchain that is missing one or more of these libraries. +A **compiler-only toolchain** only consists of compilers (no additional libraries). + +### System toolchain + +The **`system` toolchain** is a special case which corresponds to using the compilers and libraries +*provided by the operating system*, rather than using toolchain components that were installed using EasyBuild. + +It used sparingly, mostly to install software where no actual compilation is done or +to build a set of toolchain compilers and its dependencies, since the versions of the system compilers +and libraries are beyond the control of EasyBuild, which could affect the reproducibility of the installation. + +### Common toolchains + +The `foss` and `intel` toolchains are also known as the *common toolchains*, +because they are widely adopted by the EasyBuild community. + +The `foss` toolchain consists of all open source components (hence the name: +"FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), +[ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). + +The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version +controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. + +Roughly every 6 months, a new version of these common toolchains is agreed upon +in the EasyBuild community, after extensive testing. + +More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). + +--- + +## *Modules* + +*Module* is a massively overloaded term in (scientific) software and IT in general +(kernel modules, Python modules, and so on). +In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. + +[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept +on HPC systems: it is a way to specify changes that should be made to one or more +[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a +[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file +is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or +[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, +and specifies which environment variables should be updated, and how (append, +prepend, (re)define, undefine, etc.) upon loading the environment module. +Unloading the environment module will restore the shell environment to its previous state. + +Environment module files are processed via a **modules tool**, of which there +are several conceptually similar yet slightly different implementations. +The Tcl-based [Environment Modules](https://sourceforge.net/projects/modules/) implementation, and +[Lmod](https://lmod.readthedocs.io), a more recent Lua-based implementation (which also supports module +files written in Tcl syntax), are the most commonly used ones. + +**Environment module files are automatically generated for each software installation** by EasyBuild, +and *loading* a module results in changes being made to the environment of the current shell +session such that the corresponding software installation can be used. + +--- + +## *Bringing it all together* + +The EasyBuild **framework** leverages **easyblocks** to automatically build and install +(scientific) software, potentially including additional **extensions**, using a particular compiler **toolchain**, +as specified in **easyconfig files** which each define a set of **easyconfig parameters**. + +EasyBuild ensures that the specified **(build) dependencies** are in place, +and automatically generates a set of **(environment) modules** that facilitate access to the installed software. + +An **easystack file** can be used to specify a collection of software to install with EasyBuild. + +--- + +*[[next: Installation]](installation.md)* From d873a2030c860a66224148061d0ceefcf956636c Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 10 Mar 2022 14:37:51 +0100 Subject: [PATCH 054/159] Additions to the Lmod section and some restructuring, and better information in the README for the repository. --- .../2022-CSC_and_LO/1_01_what_is_easybuild.md | 2 +- docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md | 71 ------ docs/2022-CSC_and_LO/1_02_Lmod.md | 211 ++++++++++++++++++ docs/2022-CSC_and_LO/1_03_CPE.md | 30 +++ ..._03_terminology.md => 1_04_terminology.md} | 2 +- 5 files changed, 243 insertions(+), 73 deletions(-) delete mode 100644 docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md create mode 100644 docs/2022-CSC_and_LO/1_02_Lmod.md create mode 100644 docs/2022-CSC_and_LO/1_03_CPE.md rename docs/2022-CSC_and_LO/{1_03_terminology.md => 1_04_terminology.md} (99%) diff --git a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md index 5ccdb1db..979c6e90 100644 --- a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md @@ -318,4 +318,4 @@ More information, including the dates and times of the next calls, is available --- -*[[next: The HPE Cray Programming Environment and Lmod]](1_02_CPE_and_Lmod.md)* +*[[next: The Lmod module system]](1_02_Lmod.md)* diff --git a/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md b/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md deleted file mode 100644 index 01812057..00000000 --- a/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md +++ /dev/null @@ -1,71 +0,0 @@ -# The HPE Cray Programming Environment and Lmod - -*[[back: What is EasyBuild?]](1_01_what_is_easybuild.md)* - ---- - -On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated -as Cray PE). The environment provides several tools, including compilers, communication libraries, -optmized math libraries and various other libraries, analyzers and debuggers. - -The Cray PE is made available through *environment modules* tha allow to select particular versions of -tools and to configure the environment in a flexible way. - ---- - -## Modules - -*Module* is a massively overloaded term in (scientific) software and IT in general -(kernel modules, Python modules, and so on). -In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. - -[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept -on HPC systems: it is a way to specify changes that should be made to one or more -[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a -[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file -is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or -[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, -and specifies which environment variables should be updated, and how (append, -prepend, (re)define, undefine, etc.) upon loading the environment module. -Unloading the environment module will restore the shell environment to its previous state. - -Environment module files are processed via a **modules tool**, of which there -are several conceptually similar yet slightly different implementations. -The oldest module tool still in use today is Environment Modules 3.2, implemented in C and -supporting module files written in Tcl. After a gap in developement, Xavier Delaruelle of CEA -developed [Environment Modules 4 and 5](https://sourceforge.net/projects/modules/) which is -fully implemented on Tcl. An alternative module tool is [Lmod](https://lmod.readthedocs.io), -developed by Robert McLay at TACC and implemented in LUA. This tool supports natively LUA -module files but also offers a high degree of compatibility with Tcl-based module files -developed for Environment Modules fia a translation layer and some API translation. - -The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no -packages or official support for Environment Modules 4 or 5. On LUMI, Lmod was selected as -the module tool. At the user level, Lmod and the various versions of Emvironment Modules -have very simmilar commands for managing the environment, but with different options. -The commands for searching for modules are very different though so if you are not familiar -with Lmod and its commands for users, it is worthwile to read the -[LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). - ---- - -## Lmod features - - ---- - -## Cray PE components - - ---- - -## Configuring the Cray PE through modules - ---- - -## Further reading - - ---- - -*[[next: Terminology]](1_03_terminology.md)* diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md new file mode 100644 index 00000000..91cb7571 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -0,0 +1,211 @@ +# The Lmod module system + +*[[back: What is EasyBuild?]](1_01_what_is_easybuild.md)* + +--- + +On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated +as Cray PE). The environment provides several tools, including compilers, communication libraries, +optmized math libraries and various other libraries, analyzers and debuggers. + +The Cray PE is made available through *environment modules* tha allow to select particular versions of +tools and to configure the environment in a flexible way. + +--- + +## Modules + +*Module* is a massively overloaded term in (scientific) software and IT in general +(kernel modules, Python modules, and so on). +In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. + +[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept +on HPC systems: it is a way to specify changes that should be made to one or more +[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a +[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file +is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or +[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, +and specifies which environment variables should be updated, and how (append, +prepend, (re)define, undefine, etc.) upon loading the environment module. +Unloading the environment module will restore the shell environment to its previous state. + +Environment module files are processed via a **modules tool**, of which there +are several conceptually similar yet slightly different implementations. +The oldest module tool still in use today is Environment Modules 3.2, implemented in C and +supporting module files written in Tcl. After a gap in developement, Xavier Delaruelle of CEA +developed [Environment Modules 4 and 5](https://sourceforge.net/projects/modules/) which is +fully implemented on Tcl. An alternative module tool is [Lmod](https://lmod.readthedocs.io), +developed by Robert McLay at TACC and implemented in LUA. This tool supports natively LUA +module files but also offers a high degree of compatibility with Tcl-based module files +developed for Environment Modules fia a translation layer and some API translation. + +The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no +packages or official support for Environment Modules 4 or 5. On LUMI, Lmod was selected as +the module tool. At the user level, Lmod and the various versions of Emvironment Modules +have very simmilar commands for managing the environment, but with different options. +The commands for searching for modules are very different though so if you are not familiar +with Lmod and its commands for users, it is worthwile to read the +[LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). + +--- + +## Lmod hierarchy + +### User view + +Lmod supports a module hierarchy. In a hierarchy, there is a distinction between the *installed +modules* and the *available modules*. Available modules are those that can be loaded directly +without first loading any other module, while the installed modules is the complete set of +modules that one could load one way or another. A typical use case +is a hierarchy to deal with different compilers on a system and different MPI implementations. +After all, it is a common practice to only link libraries and application code compiled with the +same compiler to avoid compatibility problems between compilers (and to be able to use advanced +features such as link time optimization). This is even more important for MPI, as Open MPI and +MPCIH-derived MPI implementations have incompatible Application Binary Interfaces. This would lead +to a hierarchy with 3 levels: + +1. The ``Core`` level containing the modules for the compilers themselves, e.g., one or more versions + of the GNU compiler suite and one or more versions of LLVM-based compilers. + + Loading a compiler module would then make the next level available: + +2. The ``Compiler`` level, containing modules for libraries and packages that only rely on the compilers + but do not use MPI, as well as the MPI modules, e.g., a version of Open MPI and a version of MPICH. + + Loading one of the MPI modules would then make the next level available: + +3. The ``MPI`` level, containing libraries and applications that depend on the compiler used and the MPI + implementation. + +Now assume that we have two compilers in the hierarchy, Compiler_A and Compiler_B. Their modules would reside +at the ``Core`` level. Both copilers provide the same MPI implementation, MPI_C. So there would be two modules +for ``MPI_C`` in two different subdirectories at the ``Compiler`` level. And further assume that we have an +application, Appl_E, compiled with both Compiler_A and Compiler_B and using MPI_C. For that application there would +also be two module files at the ``MPI`` level, one in a subdirectory corresponding ao Compiler_A and MPI_C and one +in a subdirectory corresponding to Compiler_B and MPI_C. To be able to load the module for Appl_E, a user should +first load Compiler_A, then load MPI_C and only then is it possible to load the module for Appl_E: + +```bash +module load Compiler_A MPI_C Appl_E +``` + +What is interesting is what happens if the user now loads Compiler_B: + +```bash +module load Compiler_B +``` + +In a properly designed and implemented hierarchy, Lmod will unload Compiler_A which will also trigger the unloading/deactivation +of MPI_C and Appl_E. It will then load the module for Compiler_B and proceed with looking if it can find another module for +MPI_C. That will then be loaded which now makes a different module for Appl_E available, which Lmod will proceed to load. If it +cannot find an exact match for the version, Lmod will even try to locate a different version. Hence the situation after loading +Compiler_B is that now modules are loaded for Compiler_B, MPI_C for Compiler_B and Appl_E for Compiler_A with MPI_C. +All this requires very little effort from the module file programmer and very little logic in the module files. E.g., rather +then implementing a single module file for Appl_E that would require logic to see which compiler and MPI implementation is loaded +and depending on those adapt the path to the binaries, several very simple modules need to be written with very little +logic, and one could add an Appl_E module for a different compiler or MPI implementation without touching any of the already +existing module files for that application. + + +### Building blocks + +Some mechanisms in Lmod make implementing a hierarchy fairly easy (though there are a lot of hidden pitfalls) + +- The *MODULEPATH* environment variable determines which modules are available. MODULEPATH is different from any other + path-style variable in Lmod in that any change will immediately trigger a re-evaluation of which modules are available + and trigger deactivating modules that are no longer available when a directory is removed from the MODULEPATH or + looking for alternatives for deactivated modules when a directory is added to the MODULEPATH. + +- The *"one name rule"*: Lmod cannot have two modules loaded with the same name (but a different version). By default, when loading + a module with the name of an already loaded module, Lmod will automatically swap the old one with the new one, i.e., unload the + already loaded module and load the new one. + +- The *family* concept: It is possible to declare a module to be part of a family using a command in the module file. No two modules + of the same family can be loaded at the same time, and Lmod will again by default auto-swap the already loaded one with the one + being loaded. The procedure is different though as Lmod now first has to read the new module file to discover the family, and this + may lead to more side effects. But that discussion is outside the scope of this tutorial. + + The family concept was for a long time a unique feature of Lmod, but it has been added now also to Environment Modules version 5.1. + + +### Implementation details + +The above example could be implemented using 6 module files: One for each compiler, two for the MPI module and +two for the application module. + +``` +moduleroot +├── Core +│ ├── Compiler_A +│ │ └── version_A.lua +│ └── Compiler_B +│ └── version_B.lua +├── Compiler +│ ├── Compiler_A +│ │ └── version_A +│ │ └── MPI_C +│ │ └── version_C.lua +│ └── Compiler_B +│ └── version_B +│ └── MPI_C +│ └── version_C.lua +└── MPI + ├── Compiler_A + │ └── version_A + │ └── MPI_C + │ └── version_C + │ └── Appl_E + │ └── version_E.lua + └── Compiler_B + └── version_B + └── MPI_C + └── version_C + └── Appl_E + └── version_E.lua +``` + +Besides the module functions needed to create the environment needed to run the compiler, the module file for +Compiler_A would need only two lines to implement the hierarchy: + +```Lua +family('Compiler') +prepend_path('MODULEPATH', 'moduleroot/Compiler/Compiler_A/version_A') +``` + +There are now two different ``version_C.lua`` files. One contains the necessary calls to module functions to +initialise the environment to use the version compiled with Compiler_A/version_A while the other contains the +necessary functions to do that for Compiler_B/version_B. Again, two more lines are needed to implement the hierarchy. +E.g., for ``moduleroot/Compiler/Compiler_A/version_A/MPI_C/version_C.lua``: + +```Lua +family('MPI') +prepend_path('MODULEPATH', 'moduleroot/MPI/Compiler_A/version_A/MPI_C/version_C') +``` + +Finally two vesions of the ``version_E.lua`` file are needed, one to prepare the environment for using the +package with Compiler_A anmd MPI_C and one for using the package with Compiler_B and MPI_C. However, these +are just regular modules and no additions are needed to work for the hierarchy. + +Both EasyBuild and Spack support Lmod hierarchies and with these tools it is also fairly automatic to create +different versions of the module files for each compiler and MPI library used to build the application. When +hand-writing modules it may be more interesting to have a generic module which would work for all those cases +and that is also possible with Lmod. Lmod does have a range of *introspection functions* that a module can use +to figure out its name, version and place in the module tree. All that would be needed is that the various +instances of the module file are at the correct location in the module tree and link to the generic file which +can be outside the module tree. In fact, this feature is used on LUMI to implement the modules that load a +particular version of the hardware for a particular section of LUMI. + +--- + +## Finding modules + +TODO: module spider, module help, module whatis, module keyword and where do they get their information. + +--- + +## Further reading + + +--- + +*[[next: The Cray Programming Environment]](1_03_CPE.md)* diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md new file mode 100644 index 00000000..ff054066 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -0,0 +1,30 @@ +# The HPE Cray Programming Environment + +*[[back: The Lmod module system]](1_02_Lmod.md)* + +--- + +On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated +as Cray PE). The environment provides several tools, including compilers, communication libraries, +optmized math libraries and various other libraries, analyzers and debuggers. + +The Cray PE is made available through *environment modules* tha allow to select particular versions of +tools and to configure the environment in a flexible way. + +--- + +## Cray PE components + + +--- + +## Configuring the Cray PE through modules + +--- + +## Further reading + + +--- + +*[[next: Terminology]](1_04_terminology.md)* diff --git a/docs/2022-CSC_and_LO/1_03_terminology.md b/docs/2022-CSC_and_LO/1_04_terminology.md similarity index 99% rename from docs/2022-CSC_and_LO/1_03_terminology.md rename to docs/2022-CSC_and_LO/1_04_terminology.md index ec085b0c..3450c8f0 100644 --- a/docs/2022-CSC_and_LO/1_03_terminology.md +++ b/docs/2022-CSC_and_LO/1_04_terminology.md @@ -1,6 +1,6 @@ # Terminology -*[[back: The HPE Cray Programming Environment and Lmod]](1_02_CPE_and_Lmod.md)* +*[[back: The HPE Cray Programming Environment]](1_03_CPE.md)* --- From 78d54a5243464f70f3da75cacbe362d784ba46cd Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 10 Mar 2022 14:40:29 +0100 Subject: [PATCH 055/159] Corrections to the README file. --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index f7ed0418..e84c0d54 100644 --- a/README.md +++ b/README.md @@ -22,11 +22,11 @@ which makes it very easy to preview the result of the changes you make locally. make preview - or + or mkdocs serve - Visit http://127.0.0.1:8000 to see the local live preview of the changes you make. + Visit http://127.0.0.1:8000 to see the local live preview of the changes you make. * If you prefer building a static preview you can use ``make`` or ``mkdocs build``, which should result in a ``site/`` subdirectory that contains the rendered documentation. From 48d6d029a98a9a3516cb6d451bc0e931899d327d Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 10 Mar 2022 14:42:32 +0100 Subject: [PATCH 056/159] Corrections to the overview for part I. --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md index e381f45d..bdfddfc5 100644 --- a/docs/2022-CSC_and_LO/1_00_part1_intro.md +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -2,10 +2,10 @@ *[[back to start page]](index.md)* -*(Tue March 4th 2021, 9am-12 CET)* - - [What is EasyBuild?](1_01_what_is_easybuild.md) -- [Terminology](1_02_terminology.md) +- [The Lmod module system](1_02_Lmod.md) +- [The HPE Cray Programming Environment and Lmod](1_03_CPE.md) +- [Terminology](1_04_terminology.md) - [Installation](installation.md) *(hands-on)* - [Configuration](configuration.md) *(hands-on)* From 04b642ea97ea5430e045cc4e3c8f412bd8234671 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 11 Mar 2022 14:16:34 +0100 Subject: [PATCH 057/159] Further work on the Lmod section. --- docs/2022-CSC_and_LO/1_02_Lmod.md | 314 ++++++++++++++++++++++++++++-- 1 file changed, 299 insertions(+), 15 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md index 91cb7571..04ccff8e 100644 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -4,15 +4,6 @@ --- -On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated -as Cray PE). The environment provides several tools, including compilers, communication libraries, -optmized math libraries and various other libraries, analyzers and debuggers. - -The Cray PE is made available through *environment modules* tha allow to select particular versions of -tools and to configure the environment in a flexible way. - ---- - ## Modules *Module* is a massively overloaded term in (scientific) software and IT in general @@ -40,12 +31,16 @@ module files but also offers a high degree of compatibility with Tcl-based modul developed for Environment Modules fia a translation layer and some API translation. The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no -packages or official support for Environment Modules 4 or 5. On LUMI, Lmod was selected as -the module tool. At the user level, Lmod and the various versions of Emvironment Modules -have very simmilar commands for managing the environment, but with different options. -The commands for searching for modules are very different though so if you are not familiar -with Lmod and its commands for users, it is worthwile to read the -[LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). +packages or official support for Environment Modules 4 or 5. At the user level, +Environment Modules 3.2 and Lmod have many commands in common, but with different options. +Lmod also has some powerful features that are lacking in Environment Modules 3.2. + +!!! Note "The Cray PE on LUMI" + On LUMI, Lmod was selected as the module tool. One area where there are significant + differences between Environment Modules 3.2 (and also the newer versions) and Lmod is + in the commands for discovering modules on the system. If you are not familiar with Lmod + and its commands for users, it is worthwile to read the + [LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). --- @@ -199,8 +194,297 @@ particular version of the hardware for a particular section of LUMI. ## Finding modules +In a hierarchical setup, not all modules are available at login. This implies that a user cannot use +``module avail`` to discover which software is available on the system. To this end Lmod has powerful +search commands. It is important to understand how these commands work to ensure that the proper information +is included in the module files to improve discoverability of software. + +!!! Note "Documentation in the LUMI documentation" + Extensive information on search commands with examples of how to use them on LUMI can be found + in the [LUMI documentation]([https://)](https://docs.lumi-supercomputer.eu/), in + [the computing section, "Module environment page", "Finding modules" section](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/#finding-modules). + TODO: module spider, module help, module whatis, module keyword and where do they get their information. +### module spider command + +The available modules at any point in time are often only a subset of all installed modules on a +system. However, Lmod provides the ``module spider`` command to search for a module with a given name +among all installed modules and to tell you how this module can be loaded (i.e., which other modules +need to be loaded to make the module available). + +The ``module spider`` command has three levels, producing different outputs: + + 1. ``module spider`` without further arguments will produce a list of all + installed software and show some basic information about those packages. + Some packages may have an ``(E)`` behind their name and will appear in blue + (in the default colour scheme) which means that they are part of a different + package. These are called *extensions* of packages or modules. + This is explained a little further in this page. + + Note that ``module spider`` will also search in packages that are hidden from + being displayed. These packages can be loaded and used. However administrators + may have decided to hide them + either because they are not useful to regular users or because they think that + they will rarely or never be directly loaded by a user and want to avoid + overloading the module display. + + 2. ``module spider `` will search for the specific package. This + can be the name of a module, but it will also search some other information + that can be included in the modules. The search is also case-insensitive. + E.g., on LUMI + ```bash + module spider GNUplot + ``` + will show something along the lines of + ``` + ------------------------------------------------------------------ + gnuplot: + ------------------------------------------------------------------ + Description: + Gnuplot is a portable command-line driven graphing utility + + Versions: + gnuplot/5.4.2-cpeCray-21.08 + gnuplot/5.4.2-cpeGNU-21.08 + ``` + so even though the capitalisation of the name was wrong, it can tell us that + there are two versions of gnuplot. The ``cpeGNU-21.08`` and ``cpeCray-21.08`` + tell that the difference is the compiler that was used to install gnuplot, + being the GNU compiler (PrgEnv-gnu) and the Cray compiler (PrgEnv-cray) + respectively. + + In some cases, if there is no ambiguity, `module spider` will actually + already produce help about the package, which is the next level. + + 3. `module spider /` will show more help information + about the package, including information on which other modules need to be + loaded to be able to load the package. E.g., + ```bash + module spider git/2.35.1 + ``` + will return something along the lines of + ``` + ------------------------------------------------------------------- + git: git/2.35.1 + ------------------------------------------------------------------- + Description: + Git is a free and open source distributed version control + system + + You will need to load all module(s) on any one of the lines below + before the "git/2.35.1" module is available to load. + + CrayEnv + LUMI/21.12 partition/C + LUMI/21.12 partition/D + LUMI/21.12 partition/G + LUMI/21.12 partition/L + + Help: + ``` + (abbreviated output). Note that it also tells you which other modules need + to be loaded. You need to choose the line which is appropriate for you and + load all modules on that line, not the whole list of in this case 9 + modules. + +!!! failure "Known issue" + The Cray PE uses Lmod in an unconventional manner with the hierarchy not + build fully in the way Lmod expects. As a consequence Lmod is not always + able to generate the correct list of modules that need to be loaded to make + a package available, and the list of ways to make a module available may + also be incomplete. + + The problem is somewhat aggrevated on LUMI because the Cray PE hierarchy sits + next to the hierarchy of the software stack as the Cray PE is installed + separately and hence cannot be integrated in the way the Lmod developer had + in mind. + + +#### Module extensions + +Certain packages, e.g., Python, Perl or R, get a lot of their functionality through +other packages that are installed together with them and extend the functionity, +e.g., NumPy and SciPy for Python. Installing all those packages as separate modules +to make it easy to see if they are installed or not on a system would lead to an +overload of modules on the system. + +Similary, admins of a software stack may chose to bundle several libraries or tools +that are often used together in a single module (and single installation directory), +e.g., to reduce module clutter but also to reduce the length of the search paths for +binaries, libraries or manual pages to speed up loading of applications. + +Lmod offers a way to make those individual packages installed in a module discoverable +by declaring them as *extensions* of the module. The ``module spider`` command will +search for those too. + + 1. ``module spider`` without further arguments: The output may contain lines similar + to + ``` + ----------------------------------------------------------------------- + The following is a list of the modules and extensions currently available: + ----------------------------------------------------------------------- + Autoconf: Autoconf/2.71 (E) + + CMake: CMake/3.21.2 (E), CMake/3.22.2 (E) + ``` + which tells that ``Autoconf`` and ``CMake`` are not available as modules themselves + but as extensions of another module, and it also tells the versions that are available, + though that list may not be complete (and is not always complete for modules either + as it is limited to one line of output). + +2. ``module spider `` will search for extensions also. E.g., + ``` + module spider CMake + ``` + on LUMI will return something along the lines of + ``` + ----------------------------------------------------------------------- + CMake: + ----------------------------------------------------------------------- + Versions: + CMake/3.21.2 (E) + CMake/3.22.2 (E) + ``` + (output abbreviated). + This tells that there is no ``CMake`` module on the system but that two versions + of ``CMake`` are provided in another module. + + 3. `module spider /` will show more information on the + extension, including which module provides the extension and which other modules + have to be loaded to make that module available. E.g., on LUMI, + ``` + module spider CMake/3.22.2 + ``` + will output something along the lines of + ``` + ----------------------------------------------------------------------- + CMake: CMake/3.22.2 (E) + ----------------------------------------------------------------------- + This extension is provided by the following modules. To access the + extension you must load one of the following modules. Note that any + module names in parentheses show the module location in the software + hierarchy. + + buildtools/21.12 (LUMI/21.12 partition/L) + buildtools/21.12 (LUMI/21.12 partition/G) + buildtools/21.12 (LUMI/21.12 partition/D) + buildtools/21.12 (LUMI/21.12 partition/C) + buildtools/21.12 (CrayEnv) + ``` + (output abbreviated and slightly reformatted for readability). This tells that + ``CMake/3.22.2`` is provided by the ``bvuildtools/21.12`` module and that there + are 5 different ways to make that package available. + +??? Bug "Restrictions with older Lmod versions" + At the time of development of this tutorial, Cray is still using the pretty old + 8.3.1 version of Lmod. Even though extensions were supported since Lmod version 8.2.5, + Lmod 8.3.1 has several problems: + + - It is not possible to hide extensions in the output of ``module avail``, a feature + that only became available in version 8.5. This may be annoying to many users as + the extension list of packages such as Python, R and Perl can be very long (the + default EasyBuild installation of R contains on the order of 600 packages). + + For that reason on LUMI extensions are only used for some modules. + + - ``module avail`` also shows extensions for modules that are not available which + makes no sense. This bug was only corrected in Lmod 8.6.13 and 8.6.14. + + +### module keyword + +Another search command that is sometimes useful is `module keyword`. It really +just searches for the given word in the short descriptions that are included in +most module files and in the name of the module. The output is not always +complete since not all modules may have a complete enough short description. + +Consider we are looking for a library or package that supports MP3 audio +encoding. +```bash +module keyword mp3 +``` +will return something along the lines of +``` +---------------------------------------------------------------- + +The following modules match your search criteria: "mp3" +---------------------------------------------------------------- + + LAME: LAME/3.100-cpeCray-21.08, LAME/3.100-cpeGNU-21.08 + LAME is a high quality MPEG Audio Layer III (mp3) encoder +``` +though the output will depend on the version of Lmod. This may not be the most +useful example on a supercomputer, but the library is in fact needed to be able +to install some other packages even though the sound function is not immediately +useful. + +??? bug "Know issue: Irrelevant output" + At the moment of the development of this tutorial, this command actually + returns a lot more output, referring to completely irrelevant extensions. + This is a bug in the HPE-Cray-provided version of Lmod (8.3.1 at the time + of development of this tutorial) that was only solved in more recent versions. + + +### module avail + +The `module avail` command is used to show only available modules, i.e., modules +that can be loaded directly without first loading other modules. It can be used +in two ways: + + 1. Without a further argument it will show an often lengthy list of all + available modules. Some modules will be marked with `(D)` which means that + they are the default module that would be loaded should you load the module + using only its name. + + 2. With the name of a module (or a part of the name) it will show all modules + that match that (part of) a name. E.g., + ```bash + module avail gnuplot + ``` + will show something along the lines of + ``` + ------ EasyBuild managed software for software stack LUMI/21.08 on LUMI-L ------ + gnuplot/5.4.2-cpeCray-21.08 gnuplot/5.4.2-cpeGNU-21.08 (D) + + Where: + D: Default Module + (output abbreviated). + ``` + but + ```bash + module avail gnu + ``` + will show you an often lengthy list that contains all packages with gnu + (case insensitive) in their name or version. + + +### Getting help + +One way to get help on a particular module has already been discussed on this +page: ``module spider /`` will produce help about the package as +soon as it can unambiguously determine the package. It is the only command that +can produce help for all installed packages. The next two commands can only +produce help about available packages. + +A second command is ``module whatis`` with the name or name and version of a +module. It will show the brief description of the module that is included in +most modules on the system. If the full version of the module is not given, it +will display the information for the default version of that module. + +The third command is ``module help``. Without any further argument it will display +some brief help about the module command. However, when used as +``module help `` or ``module help /`` it will produce help for either the +default version of the package (if the version is not specified) or the +indicated version. + + +### Implementation details + +**TODO** Tell about the extensions, whatis and help functions in module file and maybe a +little bit about how module spider works. + + --- ## Further reading From 71b0dcbfd43c15504e0550a482d4ea8b42e54a72 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 14 Mar 2022 14:38:34 +0100 Subject: [PATCH 058/159] Finished the Lmod section. --- docs/2022-CSC_and_LO/1_02_Lmod.md | 74 +++++++++++++++++++++++++++++-- 1 file changed, 71 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md index 04ccff8e..2d1d5121 100644 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -41,6 +41,7 @@ Lmod also has some powerful features that are lacking in Environment Modules 3.2 in the commands for discovering modules on the system. If you are not familiar with Lmod and its commands for users, it is worthwile to read the [LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). + Some of those commands are also discussed on this page. --- @@ -204,7 +205,6 @@ is included in the module files to improve discoverability of software. in the [LUMI documentation]([https://)](https://docs.lumi-supercomputer.eu/), in [the computing section, "Module environment page", "Finding modules" section](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/#finding-modules). -TODO: module spider, module help, module whatis, module keyword and where do they get their information. ### module spider command @@ -481,14 +481,82 @@ indicated version. ### Implementation details -**TODO** Tell about the extensions, whatis and help functions in module file and maybe a -little bit about how module spider works. +Lmod works by executing the module file. However, the actions of all Lmod-defined +functions will depend upon the mode in which Lmod is executing the module function, +and the module file can also detect in which mode it is executing. +Modes include "load", "unload"but also "spider". E.g., when te mode is "load", the +``setenv`` function will set an environment variable to the indicated value while in +"unload" mode that environment variable will be unset, and in "spider" mode the +environment variable is left untouched. The working of ``prepend_path``, a function +that modifies PATH-style variables, depends a bit on how Lmod is configured (as it is +possible to work with reference counts), but in its most basic mode, ``prepend_path`` +will add a given directory to a given PATH-style environment variable (or move it to +the front of the PATH-style variable if the directory is already in there), while in +"unload" mode that specific directory will be removed from the PATH (but no error will +be generated should the directory that is used as the argument not be part of the path +in that PATH-style variable). When the mode is "spider", the function has special behaviour +if it is used to change the ``MODULEPATH``. It will then note the change and add that +directory to the list of directories that has to be searched for module files. +This makes ``module spider`` a very expensive command as it may have to traverse a lot +of directories and has to execute all module files in there. Therefor Lmod will build +a so-called spider cache which can be pre-built in the system for certain directories +and otherwise will be build in the user's home directory (in the ``.lmod.d/.cache`` +subdirectory). Our experience is that this cache tends to be rather fragile, +in particular on Cray systems (and that has been confirmed in discussions with +people with access to some other Cray systems) so from time to time Lmod fails to +note changes to the modules, at least when using commands such as ``module spider``. +The actual loading and unloading of the module is not based on cached information. + +Lmod has several functions that can be used in module files to provide the information +that Lmod needs for the search-related and help commands. + +The ``help`` function defines the long help text used by ``module help`` and by +``module spider`` as soon as there is no ambiguity anymore about which module is +being searched for. + +The ``whatis`` function is used to provide short information about a module. That +information is then used by ``module whatis`` and ``module keyword`` , but also +for brief information shown by ``module spider`` when multiple modules or versions +of modules are found by the command. A module file can contain multiple ``whatis`` +commands and the Lmod manuel suggests to use those lines as a kind of database +record. See, e.g., +[the Lmod manual page with module file examples](https://lmod.readthedocs.io/en/latest/100_modulefile_examples.html?highlight=whatis). +One such example is +```Lua +whatis("Name: valgrind") +whatis("Version: 3.7.0") +whatis("Category: tools") +whatis("URL: http://www.valgrind.org") +whatis("Description: memory usage tester") +``` +It is not all that important to include all those lines in a module file, but some of +those lines get a special treatment from Lmod. The line starting with ``Description`` +is used by ``module spider`` to provide some brief information about the module if it +is not totally resolved. This comes with a limitation though: It is not show for each +version of the module, so ideally all "GROMACS" modules should contain the same +description line and use other lines to provide further information about what +distinguished a particular version. +Likewise the ``Category:`` line is used by the ``spider_decoration`` hook that can be +used to add decoration to the spider level 1 output. +All in all the ``whatis`` function if often overlooked in Lmod-based module functionx +but it is a very useful function to include in the proper way in module files. + +A third function that provides information to the search commands is ``extensions``. +It can be used to list up the extensions supported by the module. The argument list +may seem strange as it takes only a single argument, a string of comma-separated ``extension/version`` +elements, but that is because the number of arguments to a function is limited in +Lua and that limit can actually be met easily by modules for Python, Perl or R packages. --- ## Further reading +- [Lmod documentation](https://lmod.readthedocs.io/en/latest/index.html) +- [Lmod on LUMI in the LUMI documentation](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/) +- [Documentation of Environment Modules 5](https://modules.readthedocs.io/en/latest/), + an alternative to Lmod (though not currently supported by HPE Cray) + --- From 46fe516d3656f8a0c417ef34c510fceeb9e077d3 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 14 Mar 2022 15:36:44 +0100 Subject: [PATCH 059/159] Continued merging of the new CSC tutorial in the structure of the repository. --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md index bdfddfc5..7b138670 100644 --- a/docs/2022-CSC_and_LO/1_00_part1_intro.md +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -4,7 +4,7 @@ - [What is EasyBuild?](1_01_what_is_easybuild.md) - [The Lmod module system](1_02_Lmod.md) -- [The HPE Cray Programming Environment and Lmod](1_03_CPE.md) +- [The HPE Cray Programming Environment](1_03_CPE.md) - [Terminology](1_04_terminology.md) - [Installation](installation.md) *(hands-on)* From ec860bda7785d95f7d09bd2672652a8cce4fb16a Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 14 Mar 2022 18:27:32 +0100 Subject: [PATCH 060/159] Tutorial page about the Cray PE, and correction of a typo. --- docs/2022-CSC_and_LO/1_03_CPE.md | 251 +++++++++++++++++++++++++++++++ 1 file changed, 251 insertions(+) diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md index ff054066..8de332a4 100644 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -15,16 +15,267 @@ tools and to configure the environment in a flexible way. ## Cray PE components +- **Cray Compiling Environment (CCE)**: + + CCE consists of Cray compilers performing code analysis during compilation to generate highly optimized code. + Supported languages include Fortran, C and C++, and UPC (Unified Parallel C). + The Cray C/C++ compiler is basically the Clang/LLVM compiler with a back-end configured by HPE Cray that also + adds some Cray-specific options while the Cray Fortran compiler uses an HPE-Cray specific front-end + with a back-end based on LLVM. The Fortran compiler supports most of Fortran 2018 and tends to be + considerably stricter than the GNU or Intel Fortran compilers. + + The CCE compilers will also support OpenMP offload to NVIDIA and AMD GPUs but that is still work in progress + at the time of the development of this tutorial. The Fortran compiler (but not the C/C++ compiler) also supports + OpenACC for offloading to GPU. + + The classic pre-LLVM Cray C/C++ compiler is not available on Cray EX systems. + +- **Cray Scientific and Mathematical Libraries (CSML)**: + + A set of high performance libraries providing portability for scientific applications, sparse and dense linear + algebra (BLAS, LAPACK, ScaLAPACK) and fast Fourier transforms (FFTW). + + There is a special (more limited) version of the library with GPU-optimized routines. + + CSML also includes the HDF5 and netCDF libraries, in versions with and without MPI. + +- **Cray Message Passing Toolkit**: + + A collection of software libraries used to perform data transfers between nodes running in parallel applications. + It includes the Message Passing Interface (MPI) and OpenSHMEM parallel programming models. + + The HPE Cray MPI implementation is based on the MPICH3 sources from ANL. However, some algorithms are tweaked + specifically for HPE Cray hardware and the Cray interconnects. + +- **Cray Environment Setup and Compiling Support**: + + Libraries supporting code compilation and setting up the development environment, including compiler drivers. + +- **Cray Performance and Analysis Tools**: + + Tools to analyze the performance and behavior of programs that are run on Cray systems, and a Performance API (PAPI). + +- **Cray Debugging Support Tools**: + + Debugging tools, including `gdb4hpc` and `Valgrind4hpc`. + +- **Third-party compilers**: + + The Cray PE also supports several third-party compilers. Not all compilers are supported on all systems: + + - The GNU C/C++ and Fortran compilers will be present on most systems with the Cray PE. + + - The AMD AOCC C/C++ and Fortran compilers are supported on systems with AMD CPUs. + + - The AMD ROCm compilers (sometimes known as AOMP) are supported on systems with AMD GPU technology. + + - The Intel C/C++ and Fortran compilers (also the oneAPI versions) are supported on systems with Intel + processors. HPE Cray does not include them in licenses for AMD CPUs. + + - The NVIDIA HPC Toolkit compilers are supported on systems with NVIDIA GPUs. + --- ## Configuring the Cray PE through modules +Compilers in the Cray Programming Environment are almost always used through the Cray compiler wrappers. In fact, +until recently, some compilers could not be used without the wrappers as they failed to even detect their own +include files and some components. The behaviour of the wrapper will depend on the other modules that are +loaded in the environment. E.g., there is no separate wrapper for MPI as the MPI header files and libraries are +automatically activated through the regular wrappers as soon as a Cray MPI module is loaded. + +Multiple releases of the HPE Cray PE can be installed simultaneously on the system and users can mix-and-match +components. However, it is also possible to load only components for a specific release of the PE. +Cray PE versions have version numbers of the form ``yy.mm``, e.g., ``22.02`` for the version released in +February of 2022. However, each of the components have their own version number and it is not easy to see +which version of a component came with which version(s) of the Cray PE. + +Below we only discuss those modules that are important when building software with EasyBuild. Debuggers, +profilers, etc., are not included in the list. + + +### The PrgEnv-* modules + +The usual way to initialise the Cray PE is by loading one of the ``PrgEnv-*`` modules. This module will then +load the appropriate compiler, compiler wrappers and other libraries. Some of the components that will be +loaded are configured through the ``/etc/cray-pe.d/cray-pe-configuration.sh`` file, so the list of modules +may depend on the actual system that you are using. + +The Cray PE supports the following ``PrgEnv-*`` modules. On LUMI, only the first three are currently available: + +| Module | Compiler module | What? | +|:---------------|-----------------|:--------------------------------------------| +| PrgEnv-cray | cce | The Cray Compiling Environment compilers | +| PrgEnv-gnu | gcc | The GNU compilers | +| PrgEnv-aocc | aocc | AMD compilers for CPU-only systems | +| PrgEnv-amd | rocm | AMD ROCm compilers for GPU systems | +| PrgEnv-intel | intel | The Intel compilers | +| PrgEnv-nvidia | nvidia | NVIDIA HPC toolkit compilers (formerly PGI) | + +All ``PrgEnv-*`` modules belong to the same module family. Hence Lmod will automatically unload any +already loaded ``PrgEnv-*`` module when you load a different one. + + +### Selecting the version through the cpe module + +Loading one of the ``cpe/yy.mm`` modules (e.g., ``cpe/22.02``) has the following effects: + +- It sets the default versions of each of the Cray PE modules to the version that comes with that + particular release of the Cray PE. E.g., + + ``` bash + module load cpe/22.02 + module load cce + ``` + + would load that version of the ``cce`` compiler that comes with the 22.02 release of the Cray PE. + +- It will reload all already loaded Cray PE modules and switch them over to the version corresponding + to that particular release of the Cray PE. + +???+ Bug "Limitations and bugs" + Due to the way Lmod works and implementation bugs in the ``cpe`` modules, loading the ``cpe`` module + does not always have the desired effect. + + - The Cray PE sets the default version of each module by adding a file to the list of files in the + ``LMOD_MODULERCFILE`` environment variable. This is because Lmod does not re-evaluate the visibility + of modules and the internal list of default version immediately when the value of ``LMOD_MODULERCFILE`` + is changed but only the next time the ``module`` command is executed. Hence + ``` bash + module load cpe/22.02 ; module load cce + ``` + and + ``` + module load cpe/22.02 cce + ``` + do not have the same effect. In the first version, the version of ``cce`` loaded is the version that + corresponds to the 22.02 release of the Cray PE. In the second case however the default version of the + ``cce`` module is determined by whatever list of default modules was used when calling the ``module`` + command so may or may not be the one of the 22.02 release. + + - Loading the ``cpe`` module after loading the other Cray PE modules also does not always have the + desired effect in many versions of the Cray PE. This is because of a bug in the ``cpe `` module that + reloads the modules in the wrong order which may trigger the reload of a module with whatever version + was the default when the ``module`` command was called rather than the version the the ``cpe`` module + intends to (re-)load. + + +### The compiler wrapper module craype + +The ``craype`` module loads the compiler wrappers. There is only one set of compiler wrappers for all compilers. +Which compiler will be called, which libraries will be included, but also processor and GPU target options will +be used, is all determined by other modules. Hence it is in principle possible to use a single Makefile for +a project and still reconfigure the build by loading certain modules. + + +### Target modules + +The targets for CPU and GPU optimization, the network library for MPI bt also some other compiler options, +can be set through target modules: + +- ``craype-x86-*`` (and similar options can be expected on ARM-based systems) set the target for CPU + optimisations. For LUMI, the ``crape-x86-rome``, ``craype-x86-milan`` and ``craype-x86-trento`` modules + are relevant. + + This can also be used to cross-compile to a different target architecture unless the compiler target + gets overwritten by a compiler flag added to the command line through the Makefile or other means, + something that unfortunately happens more and more often in faulty software installation procedures. + +- ``creype-accel-*`` sets the target for OpenMP offload (and likely other technologies in the future). E.g., + loading ``craype-accel-amd-gfx90a`` tells the compilers to target AMD MI200 family GPUs, while loading + ``craype-accel-host`` tells the compiler to use the CPU instead (according to the documentation, the latter + is for ``PrgEnv-cray`` only). + +- ``craype-network-*`` selects the communication library to be used by Cray MPICH. On Slingshot 11 EX systems, + only ``craype-network-ofi`` is supported, but Slingshot 10 EX systems also offer support for UCX through the + ``craype-network-ucx`` module. + +- The ``craype-hugepages*`` modules enable Cray Huge Pages support. To fully enable this support they have to + be used at link-time and at run-time. At link time, support is compiled into the binary wile at run-time they + are used to set the actual size of the huge pages. + + The ``craype-hugepages*`` modules are not supported by all compilers. E.g., the AOCC compiler does not support + huge pages at the moment, and loading the module at link time will cause an error message from the linker. + + +### The compiler modules + +The compiler modules have already been discussed with the ``PrgEnv-*`` modules above. The different regular +compiler modules also all belong to the same family so no two different compilers can be loaded simultaneously +and Lmod will automatically unload the other compiler when a new one is loaded. + + +### The MPI modules + +To load the Cray MPI libraries, both one of the ``craype-network-*`` modules and a compiler module has to be +loaded as the MPI libraries are both network- and compiler specific. + +For some unknown reason, the MPI module for the libfabric (``craype-network-ofi``) transport is called ``cray-mpich`` +while the library for the UCX transport (``craype-network-ucx``) is called ``craype-network-ucx``. As a result, +the MPI module fails to reload automatically when switching between both transports, but it does reload automatically +when switching compilers. + +Loading an MPI module will also automatically configure the regular compiler wrappers to compile with support for +that MPI module. However, the libfabric and UCX versions of the MPI library have compatible interfaces, so it is +always possible to swap between those versions at runtime. + + +### The Cray Scientific libraries + +The Cray Scientific Libraries are loaded through the ``cray-libsci`` module (or ``cray-libsci_acc`` for the GPU +versions). Loading this module makes the BLAS, LAPACK, and ScaLAPACK libraries available, and also the Cray IRT +(Iterative Refinement Toolkit), a Cray-specific library. It will also configure the compiler wrappers to link +with these libraries, so no additional include or link options are needed. + + +### The ``fftw`` module + +As FFTW is a third party library, it gets its own module (``cray-fftw``) and is not included in the ``cray-libsci`` module. + + +--- + +## Some unexpected behaviour of the Cray PE + +On Cray EX systems, dynamic linking is the preferred way of linking applications. Whith that comes some +unexpected behaviour of the Cray modules. EasyBuild users expect that at run time the versions of the +libraries that are used, are the ones from the modules that are loaded. This is not always the case +for the runtime libraries of the Cray PE. By default the Cray PE will use a default set of libraries +that is determined by the system default version of the Cray PE (which is set by the sysadmins, not +determined by the ``cpe`` module). This can only be avoided by either using\ +rpath-linking (which is also special in the Cray PE as the wrappers activate rpath linking if the +environment variable ``CRAY_ADD_RPATH`` is defined and set to ``yes``) or by manually adjusting +the library path after loading the modules: +``` bash +export LD_LIBRARY_PATH=${CRAY_LD_LIBRARY_PATH}:$LD_LIBRARY_PATH +``` +The latter cannot be easily automated in modulefiles. Any technique that can be used (without actually +reworking the Cray PE modules) has nasty side effects in some scenarios. + +The net result of this feature is that some reproducibility of the results is lost. Programs will +react differently if the system default version of the Cray PE is changed as that will change the +set of default libraries that will be used at runtime unless rpath-linking is used or users +redfine ``LD_LIBRARY_PATH``. + + --- ## Further reading +- [LUMI documentation: "Developing" section](https://docs.lumi-supercomputer.eu/development/) +- The Cray PE is mostly documented through man pages. There used to be some documentation on the + Cray web site also but the documentation system got reworked after the merger with HPE. + The documentation is now in the [HPE Support Centre](https://support.hpe.com/) where it + is very difficult to find the right version of the documents. +- The [PE-Cray](https://github.com/PE-Cray) GitHub project also provides some additional + documentation, including + - [some whitepapers](https://github.com/PE-Cray/whitepapers), + - [documentation for Cray OpenSMHEMX](https://github.com/PE-Cray/cray-openshmemx) and + - [information about Cray DSMML](https://github.com/PE-Cray/cray-dsmml) + though much of that information ia also not updated regularly anymore. + --- *[[next: Terminology]](1_04_terminology.md)* From 8764aa365f0e87a6161ba0467a202d7c5076f183 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 16 Mar 2022 15:24:08 +0100 Subject: [PATCH 061/159] Adding an example of implementing a hierarchy to the Lmod section of the tutorial. --- docs/2022-CSC_and_LO/1_02_Lmod.md | 191 ++++++++++++++++++++++++++++++ 1 file changed, 191 insertions(+) diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md index 2d1d5121..0caa8a47 100644 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -548,6 +548,197 @@ elements, but that is because the number of arguments to a function is limited i Lua and that limit can actually be met easily by modules for Python, Perl or R packages. +--- + +## Some warnings about writing modulefiles + +***This section is very technical and only useful if you want to manually implement +modules that depend on each other one way or another.*** + +Lmod cannot guarantee that the order of unloading the modules will be the inverse of +the order in which they were loaded. Moreover, unloading a module is not done by reverting +stored actions done when loading the module, but by executing the modulefile again +in a mode that reverts certain actions. This can lead to subtle problems when modulefiles +communicate with each other through environment variables or by detecting which other +modules are loaded. These problems are usually solved by using a proper hierarchy +and basing actions of modulefiles on their position in the hierarchy. + +One case where passing information between modules through environment variables will +go wrong is when that environment variable is subsequently used to compute a directory +name that should be added to a PATH-like variable. Assume we have two versions of +a ``MyPython`` module, e.g., ``MyPython/2.7.18`` and ``MyPython/3.6.10``. That module then +sets an environment variable ``PYTHON_API_VERSION`` to either ``2.7`` or ``3.6``. +Next we have a module ``MyPythonPackage`` that makes a number of Python packages available +for both Python modules. However, as some Python packages have to be installed separately +for each Python version, it does so by adding a directory to the environment variable +``PYTHON_PATH`` that contains the version which it gets by using the Lua function +``os.getenv`` to request the value of ``PYTHON_API_VERSION``. + +One problem becomes clear in the following scenario: +``` bash +module load MyPython/2.7.18 +module load MyPythonPackage/1.0 +module load MyPython/3.6.10 +``` +The ``module load MyPythonPackage`` will find the environment variable ``PYTHON_PACKAGE_API`` +with the value ``2.7`` as set by ``module load MyPython/2.7.18`` and hence add the directory +for the packages for version 2.7 to ``PYTHON_PATH``. The ``module load MyPython/3.6.10`` +command will trigger two operations because of the *"one name rule"*: First it will +automatically unload ``MyPython/2.7.18`` (which will unset ``PYTHON_API_VERSIUON``) and +next it will load ``MyPython/3.6.10`` which will set ``PYTHON_API_VERSION`` to ``3.6``. +However, ``MyPythonPackage`` is not reloaded so the ``PYTHON_PATH`` variable will now point +to the wrong directory. One would be tempted to think that the easy fix for the user would +be to reload ``MyPythonPackage/1.0``: +``` bash +module load MyPythonPackage/1.0 +``` +Because of the *"one name rule"* this will again trigger an unload followed by a load +of the module. The problem is in the unload. One would expect that first unloading +``MyPythonPackage`` would remove the 2.7 directory from the ``PYTHON_PATH`` but it +will not. Lmod does not remeber that last time it loaded ``MyPythonPackage`` it added +the 2.7 directory to ``PythonPath``. Instead it will execute the commands in the +modulefile and reverse certain commands. Since ``PYTHON_API_VERSION`` has now the value +``3.6``, it will try to remove the directory for version ``3.6`` which is not in the +``PYTHON_PATH``. The subsequent load will then add the 3.6 directory to ``PYTHON_PATH`` +so the environment variable now contains both directories. + +In this simple case, a ``module purge`` after the first two ``module load`` commands would +still work as Lmod is able to figure out the right order to unload modules, but in more +complicated examples this may also go wrong. However, a ``module purge`` command after +the load of ``MyPython/3.6.10`` would also fail to clean up the environment as it would +still fail to remove the 2.7 directory from ``PYTHONPATH``. + +??? Note "Running the example" + To test this example for yourself, create a directory and add that directory to + the ``MODULEPATH`` using ``module use``. In that directory, create the following + subdirectories and files: + 1. ``MyPython/2.7.18.lua`` with content: + ``` lua + LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) + setenv( 'PYTHON_API_VERSION', '2.7' ) + ``` + 2. ``MyPython/3.6.10.lua`` with content: + ``` lua + LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) + setenv( 'PYTHON_API_VERSION', '3.6' ) + ``` + 3. ``MyPythonPackage/1.0.lua`` with content: + ``` lua + LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) + LmodMessage( 'PYTHON_API_VERSION = ' .. ( os.getenv( 'PYTHON_API_VERSION' ) or '') ) + prepend_path( 'PYTHON_PATH', 'someroot/python' .. + ( os.getenv( 'PYTHON_API_VERSION' ) or 'TTT' ) .. '/packages' ) + LmodMessage( 'PYTHON_PATH = ' .. ( os.getenv( 'PYTHON_PATH' ) or '') ) + ``` +??? Note "Solution with a hierarchy" + The better way in Lmod to implement the above scenario would be in a module hierarchy. + + Just to show the power of Lmod introspection functions combined with a proper hierarchy + we present a solution using only one version of the code for ``MyPython`` and one version + of the code for ``MyPythonPackages``. + + It is best to start from a clean directory. In that directory, create: + + 1. The files ``level1/MyPython/2.7.18.lua`` and ``level1/MyPython/3.6.10.lua``, + both with the same contents: + ``` lua + LmodMessage( '\nIn ' .. myModuleFullName() .. ' in mode ' .. mode() ) + + local api_version = myModuleVersion():match( '(%d+%.%d+)%..*' ) + + -- Set the variable PYTHON_API_VERSION but not for internal use in the modules. + setenv( 'PYTHON_API_VERSION', api_version ) + + local module_root = myFileName():match( '(.*)/level1/' .. myModuleFullName() ) + prepend_path( 'MODULEPATH', pathJoin( module_root, 'level2/PythonAPI', api_version ) ) + LmodMessage( 'MODULEPATH is now\n ' .. + os.getenv( 'MODULEPATH' ):gsub( ':', '\n ' ) ) + ``` + + 2. The files ``level2/PythonAPI/2.7/MyPythonPackage/1.0.lua`` and + ``level2/PythonAPI/3.6/MyPythonPackage/1.0.lua``, both with the contents: + ``` lua + LmodMessage( '\nIn ' .. myFileName() .. ' in mode ' .. mode() ) + + local python_api_version = myFileName():match( '.*/level2/PythonAPI/([^/]+)/.*' ) + LmodMessage( 'Detected Python API version from hierarchy: ' .. python_api_version ) + LmodMessage( 'Detected Python API version from environment: ' .. + ( os.getenv( 'PYTHON_API_VERSION' ) or '' ) ) + + prepend_path( 'PYTHON_PATH', 'someroot/python' .. python_api_version .. '/packages' ) + + LmodMessage( 'PYTHON_PATH = ' .. (os.getenv( 'PYTHON_PATH' ) or '') ) + ``` + + Now add the ``level1`` subdirectory to ``MODULEPATH``, e.g., if you're in the directory + containing the ``level1`` and ``level2`` subdirectories: + ``` bash + module use $PWD/level1 + ``` + and then try the following commands: + ``` bash + module avail + module load MyPython/2.7.18 + module avail + module load MyPythonPackage/1.0 + module load MyPython/3.6.10 + ``` + and pay attention to the output. + + Initially ``module avail`` will show none of the ``MyPythonPackage`` modules. These are + installed modules but not available modules. ``module load MyPython/2.7.18`` will set the + environment variable ``PYTHON_API_VERSION`` to ``2.7`` and also add a directory to the front + of the ``MODULEPATH`` with the directory name ending on ``level2/PythonAPI/2.7``. Now + ``module avail`` will show the ``MyPythonPackage/1.0`` module. + + The ``MyPythonPackage`` shows two ways to get the version of the Python API to use for + determining the right directory to add to ``PYTHON_PATH``. The fragile way is to enquire + the value of the environment variable ``PYTHON_API_VERSION`` set by loading ``MyPython/2.7.18``. + The more robust way is to use the Lmod introspection function ``myFileName()`` which returns + the full path and file name of the module file that is executing, and extracting the version + from the path with a pattern matching function. In this particular situation both computed + values are the same so both would have worked to correctly add + ``somedir/python2.7/packages`` to the front of ``PYTHON_PATH``. + + The next command, ``module load MyPython/3.6.10`` triggers a chain of events. + + First, Lmod notices that there is already a module loaded with the same name, so it will + unload ``MyPython/2.7.18``. This will unset the environment variable ``PYTHON_API_VERSION`` + (the inverse operation of ``setenv``) and will remove the ``.../level2/PythonAPI/2.7`` + subdirectory from the ``MODULEPATH`` (the inverse action of ``prepend_path``). + + Now due to + the change of the ``MODULEPATH`` the ``MyPythonPackage/1.0`` module which was loaded from + ``.../level2/PythonAPI/2.7`` is no longer available so Lmod will continue with unloading + that module. The interesting bit now is that ``PYTHON_API_VERSION`` is unset. So had we + computed the name of the directory to add to ``PYTHON_PATH`` using the value of that + environment variable, the module would have failed to compute the correct directory name + to remove so ``prepend_path`` would have left the ``PYTHON_PATH`` environment variable + untouched. However, by computing that value from the directory of the modulefile, we get + the right value and can correctly remove ``somedir/python2.7/packages`` from ``PYTHON_PATH``. + Lmod will also remember that the module was only unloaded due to a change in the + ``MODULEPATH`` and not because a user explicitly unloaded the module. I.e., it considers + the module as deactivated but not as unloaded. + + Lmod proceeds with loading the ``MyPython/3.6.10`` module. This will now set + ``PTHON_API_VERSION`` to ``3.6`` and add a directory with name ending on + ``level2/PythonAPI/3.6`` to ``MODULEPATH``. + + Things are not done yet though. As the ``MODULEPATH`` has changed, Lmod looks at its list + of deactivated modules and notices that a different version of ``MyPythonPackage/1.0`` is + now available. Hence it will now automatically load that module from the + ``.../level2/PythonAPI/3.6`` subdirectory so that that module now correctly detects + that ``somedir/python3.6/package`` should be added to ``PYTHON_PATH``. + + Hence at the end of the cycle we have again a correctly configured environment with no + trace of the ``2.7`` version that was loaded initially and with no action required from + the user to ensure that ``MyPythonPackage`` is unloaded and reloaded to ensure the + correct configuration. + + This idea is used on LUMI to implement the various versions of the software stack with + for each software stack also optimised binaries for each of the node types. + + --- ## Further reading From e24aafa8cd2d043f98856e2dda780aba22013877 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 10:08:23 +0100 Subject: [PATCH 062/159] More information about hierarchy in the Cray PE, and definition of toolchain in Terminology. --- docs/2022-CSC_and_LO/1_03_CPE.md | 3 +++ docs/2022-CSC_and_LO/1_04_terminology.md | 21 ++++++++++++++++++--- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md index 8de332a4..42eb197e 100644 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -233,6 +233,9 @@ with these libraries, so no additional include or link options are needed. As FFTW is a third party library, it gets its own module (``cray-fftw``) and is not included in the ``cray-libsci`` module. +The ``cray-fftw`` module can only be loaded if one of the processor target modules (the ``craype-x86-*`` modules) +is loaded first. + --- diff --git a/docs/2022-CSC_and_LO/1_04_terminology.md b/docs/2022-CSC_and_LO/1_04_terminology.md index 3450c8f0..19f1e01f 100644 --- a/docs/2022-CSC_and_LO/1_04_terminology.md +++ b/docs/2022-CSC_and_LO/1_04_terminology.md @@ -28,22 +28,37 @@ can also be included. Additional toolchain components are usually special-purpose libraries: -- an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/)); +- an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/) + or vendor-specific MPI implementations like Intel MPI in a toolchain based on the Intel compilers); - libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), - [LAPACK](http://performance.netlib.org/lapack/)); + [LAPACK](http://performance.netlib.org/lapack/), [ScaLAPACK](http://www.netlib.org/scalapack/)); These libraries are included because they tend to provide a common API, so mixing them in a single executable would cause name conflicts when linking. - a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); +These libraries are included because they are some of the most popular libraries in scientific software, +but also because they provide an API that is available through different implementations. Mixing +implementations in a single build is a recipe for problems as the various implementations might +define the same symbols. + A toolchain that includes all of these libraries is referred to as a **full toolchain**, while a **subtoolchain** is a toolchain that is missing one or more of these libraries. A **compiler-only toolchain** only consists of compilers (no additional libraries). +The toolchain concept also maps nicely on the Programming Environment concept in the Cray +Programming Environment. Each ``PrgEnv-*`` module in fact provides a full toolchain in a +typical Cray PE installation, except for the FFTW library. Note that systems could configure +the contents of a ``PrgEnv-*`` module differently as the MPI library and scientific library are +optional. In the Cray PE, the MPI library is provided by the ``cray-mpic`` module, +the BLAS, LAPACK and ScaLAPACK libraries by the ``cray-libsci`` module (for CPU-only nodes) +and the FFTW library through the ``cray-fftw`` module. + Many EasBuild toolchains are organised in a hierarchy. Each toolchain can use components compiled -with itself or with one of the subtoolchains. +with itself or with one of the subtoolchains. The Cray toolchains however are not currently +organised in a hierarchy. ### System toolchain From 4d47bb740278722e9302387286074057aad5ad4d Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 10:33:37 +0100 Subject: [PATCH 063/159] Add a mermaid.js diagram with the toolchain hierarchy. --- docs/2022-CSC_and_LO/1_04_terminology.md | 42 +++++++++++++++++------- 1 file changed, 30 insertions(+), 12 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_04_terminology.md b/docs/2022-CSC_and_LO/1_04_terminology.md index 19f1e01f..ebb99963 100644 --- a/docs/2022-CSC_and_LO/1_04_terminology.md +++ b/docs/2022-CSC_and_LO/1_04_terminology.md @@ -12,7 +12,7 @@ It is important to be familiar with these terms, so we'll briefly cover them one --- -## *Toolchains* +## Toolchains A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. @@ -82,6 +82,8 @@ because they are widely adopted by the EasyBuild community. The `foss` toolchain consists of all open source components (hence the name: "FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), [ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). +In recent versions (since 2021a), [FlexiBLAS](https://www.mpi-magdeburg.mpg.de/projects/flexiblas) +is used as the BLAS library, with OpenBLAS and LAPACK as the backend. The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. @@ -89,11 +91,27 @@ controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.c Roughly every 6 months, a new version of these common toolchains is agreed upon in the EasyBuild community, after extensive testing. +The common toolchains are also organised in a toolchain hierarchy: + +```mermaid +graph TD; +A[SYSTEM] --> B[GCCcore]; +B --> C[GCC]; +B --> D[iccifort]; +C --> E[gompi: Adds OpenMPI]; +D --> F[iimpi: Adds Intel MPI]; +D --> G[imkl: Adds Intel MKL]; +E --> H[foss: Adds OpenBLAS, LAPACK, ScaLAPACK, FFTW]; +F --> I[intel]; +G --> I; +``` + More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). + --- -## *Modules* +## Modules *Module* is a massively overloaded term in (scientific) software and IT in general (kernel modules, Python modules, and so on). @@ -121,7 +139,7 @@ session such that the corresponding software installation can be used. --- -## *EasyBuild framework* +## EasyBuild framework The EasyBuild *framework* consists of a set of Python modules organised in packages (``easybuild.framework``, ``easybuild.toolchains``, ``easybuild.tools``, etc.) that collectively form **the core of EasyBuild**, @@ -137,7 +155,7 @@ the necessary functionality to facilitate this. --- -## *Easyblocks* +## Easyblocks An *easyblock* is **a Python module that implements a specific software installation procedure**, and can be viewed as a plugin to the EasyBuild framework. Easyblocks can be either *generic* or *software-specific*. @@ -158,7 +176,7 @@ in the [``easybuild-easyblocks`` repository on GitHub](https://github.com/easybu --- -## *Easyconfig parameters* +## Easyconfig parameters An **easyconfig parameter** specifies a particular aspect of a software installation that should be performed by EasyBuild. @@ -188,7 +206,7 @@ only supported by certain easyblocks, and only make sense for particular (types --- -## *Easyconfig files* +## Easyconfig files *Easyconfig files* (or *easyconfigs* for short), are **simple text files written in Python syntax that specify what EasyBuild should install**. @@ -211,7 +229,7 @@ the EasyBuild community maintains a large (and growing) collection of easyconfig --- -## *Easystack files* +## Easystack files [**Easystack files**](https://docs.easybuild.io/en/latest/Easystack-files.html) are a new concept in EasyBuild, providing a way to define a *software stack* @@ -227,7 +245,7 @@ which means it is subject to change in future EasyBuild releases, and may be pro --- -## *Extensions* +## Extensions *Extensions* is the collective term we use for **additional software packages that can be installed on top of another software package**. Common examples are *Python packages*, *R libraries*, and *Perl modules*. @@ -244,7 +262,7 @@ Extensions can be installed in different ways: --- -## *Dependencies* +## Dependencies A *dependency* is a common term in the context of software. It refers to **a software package that is either strictly required by other software, or that can be leveraged to @@ -266,7 +284,7 @@ but we will discriminate build-only dependencies. --- -## *Toolchains* +## Toolchains A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. @@ -319,7 +337,7 @@ More information on these toolchains is available [in the EasyBuild documentatio --- -## *Modules* +## Modules *Module* is a massively overloaded term in (scientific) software and IT in general (kernel modules, Python modules, and so on). @@ -347,7 +365,7 @@ session such that the corresponding software installation can be used. --- -## *Bringing it all together* +## Bringing it all together The EasyBuild **framework** leverages **easyblocks** to automatically build and install (scientific) software, potentially including additional **extensions**, using a particular compiler **toolchain**, From dac3c94281f846ecf1eed5e7518173aa8999cd1a Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 15:02:25 +0100 Subject: [PATCH 064/159] Finished terminology session, extended the installation section with LUMI-specific material. --- docs/2022-CSC_and_LO/1_04_terminology.md | 141 ++---- docs/2022-CSC_and_LO/1_05_installation.md | 543 ++++++++++++++++++++++ 2 files changed, 570 insertions(+), 114 deletions(-) create mode 100644 docs/2022-CSC_and_LO/1_05_installation.md diff --git a/docs/2022-CSC_and_LO/1_04_terminology.md b/docs/2022-CSC_and_LO/1_04_terminology.md index ebb99963..9c643579 100644 --- a/docs/2022-CSC_and_LO/1_04_terminology.md +++ b/docs/2022-CSC_and_LO/1_04_terminology.md @@ -109,38 +109,13 @@ G --> I; More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). ---- - -## Modules - -*Module* is a massively overloaded term in (scientific) software and IT in general -(kernel modules, Python modules, and so on). -In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. - -[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept -on HPC systems: it is a way to specify changes that should be made to one or more -[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a -[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file -is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or -[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, -and specifies which environment variables should be updated, and how (append, -prepend, (re)define, undefine, etc.) upon loading the environment module. -Unloading the environment module will restore the shell environment to its previous state. - -Environment module files are processed via a **modules tool**, of which there -are several conceptually similar yet slightly different implementations. -The Tcl-based [Environment Modules](https://sourceforge.net/projects/modules/) implementation, and -[Lmod](https://lmod.readthedocs.io), a more recent Lua-based implementation (which also supports module -files written in Tcl syntax), are the most commonly used ones. - -**Environment module files are automatically generated for each software installation** by EasyBuild, -and *loading* a module results in changes being made to the environment of the current shell -session such that the corresponding software installation can be used. - --- ## EasyBuild framework +EasyBuild is written in Python and organised in three layers, represented by the three blocks in the logo: +the EasyBuild framework, easyblocks and Easyconfig files. + The EasyBuild *framework* consists of a set of Python modules organised in packages (``easybuild.framework``, ``easybuild.toolchains``, ``easybuild.tools``, etc.) that collectively form **the core of EasyBuild**, and is developed in the [``easybuild-framework`` repository on GitHub](https://github.com/easybuilders/easybuild-framework). @@ -165,12 +140,20 @@ multiple different software packages. Commonly used examples include the ``Confi which implements the ubiquitous ``configure``-``make``-``make install`` procedure, and the ``PythonPackage`` easyblock that can be used to install a Python package. +*In Spack this functionality is provided in the core of the package.* + A **software-specific easyblock** implements an installation procedure that is specific to a particular software packages. Infamous examples include the easyblocks we have for ``GCC``, ``OpenFOAM``, ``TensorFlow``, ``WRF``, ... The installation procedure performed by an easyblock can be controlled by defining **easyconfig parameters** in an easyconfig file. +*In Spack, this functionality is provided by the ``package.py`` files for the various packages. However, EasyBuild +does not need a software-specific easyblock for every package as for many packages the combination of a +generic easyblock with the much simpler description in the easyconfig file is sufficient. +The software-specific easyblocks can be a bit troublesome on Cray PE systems. Some easyblocks contain code that +is specific for certain compilers or toolchains and do not support all compilers or all toolchains.* + A collection of (generic and software-specific) easyblocks is developed by the EasyBuild community in the [``easybuild-easyblocks`` repository on GitHub](https://github.com/easybuilders/easybuild-easyblocks). @@ -227,6 +210,17 @@ In the [``easybuild-easyconfigs`` repository on GitHub](https://github.com/easyb the EasyBuild community maintains a large (and growing) collection of easyconfig files, for a wide range of (scientific) software. +*EasyBuild does not have an equivalent of the concretiser of Spack. EasyBuild always starts from a complete, +precise configuration of a package to install, determined by the easyconfig parameters and possibly a +custom easyblock. Both approaches have their pros and cons. The concretiser in Spack combined with the +configuration of a software package via command line parameters makes it easy to quickly generate a +number of different configurations of a package to test performance or to check which combinations work +well. But one also has to be careful to not cause an explosion of modules and software installations, and +take care of using the appropriate features to ensure that one can generate the same configurations again. +The precise configurations of EayBuild can sometimes slow down things as they may need a lot of file editing, +but when it comes to building a reproducible software stack and sharing precise building instructions, this +approach may be preferred.* + --- ## Easystack files @@ -273,95 +267,14 @@ There are three main types of dependencies for computer software: * a **build dependency** is only required when building/installing a software package; once the software package is installed, it is no longer needed to *use* that software (examples: `CMake`, `pkg-config`); -* a **run-time dependency** (often referred to simply as *dependency*) is a software package that is - required to *use* (or *run*) another software package (example: `Python`); * a **link-time dependency** is somewhere in between a build and runtime dependency: it is only needed when *linking* a software package; it can become either a build or runtime dependency, depending on exactly how the software is installed (example: `OpenBLAS`); +* a **run-time dependency** (often referred to simply as *dependency*) is a software package that is + required to *use* (or *run*) another software package (example: `Python`); -The distinction between link-time and run-time dependencies is mostly irrelevant for this tutorial, -but we will discriminate build-only dependencies. - ---- - -## Toolchains - -A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, -which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. - -We refer to the different parts of a toolchain as **toolchain components**. - -The *compiler component* typically consists of [C](https://en.wikipedia.org/wiki/C_(programming_language)), -[C++](https://en.wikipedia.org/wiki/C%2B%2B), and [Fortran](https://en.wikipedia.org/wiki/Fortran) -compilers in the context of HPC, but additional compilers (for example, -a [CUDA](https://developer.nvidia.com/cuda-zone) compiler for -[GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units) software) -can also be included. - -Additional toolchain components are usually special-purpose libraries: - -* an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/)); -* libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), - [LAPACK](http://performance.netlib.org/lapack/)); -* a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); - -A toolchain that includes all of these libraries is referred to as a **full toolchain**, while -a **subtoolchain** is a toolchain that is missing one or more of these libraries. -A **compiler-only toolchain** only consists of compilers (no additional libraries). - -### System toolchain - -The **`system` toolchain** is a special case which corresponds to using the compilers and libraries -*provided by the operating system*, rather than using toolchain components that were installed using EasyBuild. - -It used sparingly, mostly to install software where no actual compilation is done or -to build a set of toolchain compilers and its dependencies, since the versions of the system compilers -and libraries are beyond the control of EasyBuild, which could affect the reproducibility of the installation. - -### Common toolchains - -The `foss` and `intel` toolchains are also known as the *common toolchains*, -because they are widely adopted by the EasyBuild community. - -The `foss` toolchain consists of all open source components (hence the name: -"FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), -[ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). - -The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version -controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. - -Roughly every 6 months, a new version of these common toolchains is agreed upon -in the EasyBuild community, after extensive testing. - -More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). - ---- - -## Modules - -*Module* is a massively overloaded term in (scientific) software and IT in general -(kernel modules, Python modules, and so on). -In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. - -[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept -on HPC systems: it is a way to specify changes that should be made to one or more -[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a -[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file -is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or -[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, -and specifies which environment variables should be updated, and how (append, -prepend, (re)define, undefine, etc.) upon loading the environment module. -Unloading the environment module will restore the shell environment to its previous state. - -Environment module files are processed via a **modules tool**, of which there -are several conceptually similar yet slightly different implementations. -The Tcl-based [Environment Modules](https://sourceforge.net/projects/modules/) implementation, and -[Lmod](https://lmod.readthedocs.io), a more recent Lua-based implementation (which also supports module -files written in Tcl syntax), are the most commonly used ones. - -**Environment module files are automatically generated for each software installation** by EasyBuild, -and *loading* a module results in changes being made to the environment of the current shell -session such that the corresponding software installation can be used. +EasyBuild currently doesn't treat link-time dependencies differently from run-time dependencies +and hence has no specific easyconfig parameter for them. --- @@ -378,4 +291,4 @@ An **easystack file** can be used to specify a collection of software to install --- -*[[next: Installation]](installation.md)* +*[[next: Installation]](1_05_installation.md)* diff --git a/docs/2022-CSC_and_LO/1_05_installation.md b/docs/2022-CSC_and_LO/1_05_installation.md new file mode 100644 index 00000000..163262ea --- /dev/null +++ b/docs/2022-CSC_and_LO/1_05_installation.md @@ -0,0 +1,543 @@ +# Installing EasyBuild + +*[[back: Terminology]](1_04_terminology.md)* + +--- + +Before deep diving into the wonderful world of EasyBuild and getting your hands dirty with +the hands on exercises coming up in this tutorial, you will need to install EasyBuild. + +In this section we outline a couple of different ways of doing this, and also the things you should pay attention to. +By the end, you will have a functional EasyBuild installation that you can use +for the remainder of this tutorial. + +!!! Remark "EasyBuild on LUMI" + + EasyBuild is already installed for each of the LUMI software stacks. The LUMI software + stacks are based on the Cray PE (using the Cray-specific toolchains). + Moreover, in case you want to install a full shadow stack rather than build on top + of one of the LUMI software stacks, scripts are provided to initialise a new version + of the LUMI software stack based on a specific version of the Cray PE, and those + scripts also take care of the installation of EasyBuild. + + Hence this section of the tutorial is only useful for people who want more background + knowledge or who want to experiment with other EasyBuild toolchains. + + +!!! summary + + * *Requirements*: Linux, Python 2.7 or 3.5+, environment modules tool (Lmod is recommended). + * *Installation methods*: + * ``pip install easybuild`` + * installing EasyBuild with EasyBuild + * development setup + * Verify EasyBuild installation using `eb --version` and `eb --help`. + + +--- + +## Requirements + +### Linux + +The main target platform for EasyBuild is *Linux*, since it is heavily focused on installing software +on HPC system where Linux is dominant operating system (to the point where 100% of the current Top500 list of supercomputers are running Linux). + +EasyBuild is also compatible with *macOS*, but the included easyconfig files are heavily focused +on Linux so most software installations supported by EasyBuild won't work out-of-the-box on macOS. +You can still use the EasyBuild command line interface on macOS for other tasks though, +like development or contributing back to the project. + +### Python + +EasyBuild is implemented in Python, and is **compatible with both Python 2.7 and Python 3.5+** +(that is, Python 3.5 or a newer version of Python 3). + +To check which Python version you have, use: + +```shell +python -V +``` + +*No additional Python packages are required by EasyBuild*, the ones that come with the standard +Python distribution are sufficient. Some additional Python packages *can* be leveraged for specific features. +More on that later. + +### Environment modules tool + +**An *environment modules tool* is required for using EasyBuild.** + +We strongly recommend using [Lmod](https://lmod.readthedocs.io), a Lua-based modern environment +modules implementation and the most commonly used modules tool in the EasyBuild community. +Other implementations, like the original Tcl-based one, are also supported. + +To check if you have a modules tool installed, use: + +``` +module --version +``` + +If this produces output that starts with something like "`Modules based on Lua: Version 8.3.1`" you have Lmod installed, +which is the default modules tool used by EasyBuild, and you are all set for installing and using EasyBuild. +Any sufficiently recent Lmod version (8.x or even 7.x) should be fine. + +If you see output that starts with a line like "`VERSION=3.2.10`" or "`Modules Release 4.5.0`", +you have the original Tcl-based environment modules tool installed, either in its original +C-based imnplementation (``VERSION=3.2.10``) or the new Tcl-based implementation developed at CEA +(``Modules Release 4.5.0``) +and *EasyBuild will need to be configured to use it after installation*. + +If the `module` function is not defined either you do not have a modules tool installed +or your environment is not properly set up to use it. In this case, +please refer to the EasyBuild documentation here for more information. + +HPE Cray supports both the old-style Environments Modules (based on version 3.2.10) and Lmod. On LUMI, +we use Lmod as the default module system for users, and EasyBuild is configured to use that one. + +--- + +## EasyBuild as a Python package + +EasyBuild consists of a number of interdependent Python packages, +and is available via both GitHub at https://github.com/easybuilders, +as well as via the standard Python Package Index (PyPI) at +https://pypi.org/project/easybuild. + +As you may be aware the Python packaging ecosystem is bit convoluted, +which is reflected in the many different ways in which you can install a Python package. +In addition, EasyBuild is packaged in 3 components (framework, easyblocks, easyconfigs) +which slightly complicates the installation. + +
+ +Nevertheless, you don't need to be a rocket scientist to install EasyBuild (and even if you are, +that's OK too), so don't worry. + +You can install EasyBuild just like you can install any other Python software that is released +via the standard *Python Package Index* (PyPI), through one of the standard Python installation tools +(like `pip`, `virtualenv`, `pipenv`, ...). +And since EasyBuild is a software installation tool in its own right, we actually have a couple +of additional tricks up our sleeve! + +### Python 2 or Python 3? + +For EasyBuild it does not matter much whether you install it on top of Python 2 or Python 3. +Since version 4.5, Python 3 does offer some optional features (requiring additional packages) +that are not available with Python 2. +Since Python 2 is end-of-life, +we strongly recommend using Python 3 if you have the choice. + +By default EasyBuild will use the `python` command to run, +but you can control this if needed via ``$EB_PYTHON``. + + +## Installing EasyBuild + +We present three methods for installing EasyBuild. +It is up to you which one you prefer, both result a fully functional EasyBuild installation. + +Time to get your hands dirty! + +* Method 1: Using ``pip`` +* Method 2: installing EasyBuild with EasyBuild +* Method 3: Development setup + +--- + +### Method 1: Using `pip` + +Since EasyBuild is released as a [Python package on PyPI](https://pypi.org/project/easybuild) +you can install it using `pip`, the most commonly used tool for installing Python packages. + +You may need to take additional steps after the installation, depending on the exact installation command. + +!!! note + There are various other ways of installing Python packages, which we won't cover here. + If you are familiar with other tools like `virtualenv` or `pipenv`, feel free to use those + instead to install EasyBuild. + +#### Running `pip install` + +Installing EasyBuild with `pip` is as simple as running the following command: + +```shell +pip install easybuild +``` + +However, you may need to slightly change this command depending on the context and your personal preferences: + +* To install EasyBuild *system-wide*, you can use `sudo` (if you have admin privileges): + ```shell + sudo pip install easybuild + ``` + +* To install EasyBuild *in your personal home directory*, you can use the `--user` option: + ```shell + pip install --user easybuild + ``` + This will result in an EasyBuild installation in `$HOME/.local/`. + +* To install EasyBuild in a *specific directory* you can use the `--prefix` option: + ```shell + pip install --prefix _PREFIX_ easybuild + ``` + In this command, you should replace '`_PREFIX_`' with the location where you want to have EasyBuild installed + (for example, `$HOME/tools` or `/tmp/$USER`). + +#### `pip` vs `pip3` + +On systems where both Python 2 and Python 3 are installed you may also have different `pip` commands +available. Or maybe `pip` is not available at all, and only "versioned" `pip` commands like `pip3` are +available. + +If you (only) have `pip3` available, you can replace `pip` with `pip3` in any of the `pip install` commands +above. + +If you want to ensure that you are using the ``pip`` installation that corresponds to the Python 3 installation +that you intend to use, you can use ``python3 -m pip`` rather than ``pip3``. + +#### Updating your environment + +If you used the `--user` or `--prefix` option in the `pip install` command, +or if you installed EasyBuild with a `pip` version that does not correspond +to your default Python installation, +you will need to update your environment to make EasyBuild ready for use. +This is not required if you did a system-wide installation in a standard location with the default Python version. + +!!! note + Keep in mind that you will have to make these environment changes again if you start a new shell session. + To avoid this, you can update one of the shell startup scripts in your home directory (`.bashrc` for example). + +#### Updating ``$PATH`` + +Update the `$PATH` environment variable to make sure the `eb` command is available: +```shell + +export PATH=_PREFIX_/bin:$PATH +``` +**Replace '`_PREFIX_`' in this command** with the directory path where EasyBuild was installed into +(use `$HOME/.local` if you used `pip install --user`). + +This is not required if you installing EasyBuild in a standard system location. + +You can check with the `which eb` command to determine whether or not you need to update the ``$PATH`` environment variable. + +#### Updating ``$PYTHONPATH`` + +If you installed EasyBuild to a non-standard location using `pip install --prefix`, +you also need to update the Python search path environment variable +[`$PYTHONPATH`](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH) to instruct Python where +it can find the EasyBuild Python packages. + +This is not required if you used the `--user` option, since Python will automatically consider +`$HOME/.local` when searching for installed Python packages, or if you installed EasyBuild in a standard +system-wide location. + +Update `$PYTHONPATH` by running a command like: + +```shell + +export PYTHONPATH=_PREFIX_/lib/pythonX.Y/site-packages:$PYTHONPATH +``` + +Here, you need to replace the `X` and `Y` with the major and minor version of your Python installation, +which you can determine by running `python -V`. +For example, if you are using Python 3.6, make sure you are using `/python3.6/` in the command to update `$PYTHONPATH`. + +And of course, you again need to **replace '`_PREFIX_`'** with the installation prefix where EasyBuild was installed +into. + +For example: + +```shell +# update $PYTHONPATH if EasyBuild was installed in $HOME/tools with Python 3.6 +export PYTHONPATH=$HOME/tools/lib/python3.6/site-packages:$PYTHONPATH +``` + +#### Setting ``$EB_PYTHON`` + +If you want to control which Python version is used to run EasyBuild, +you can specify the name or the full path to the `python` command that should be used by the `eb` command +via the `$EB_PYTHON` environment variable. + +This may be required when you installing EasyBuild with a version of `pip` that does not correspond +with the default Python version. + +For example, to ensure that `eb` uses `python3.6`: + +```shell +export EB_PYTHON=python3.6 +``` + +#### Setting ``$EB_VERBOSE`` + +To determine which `python` commands are being considered by the `eb` command, +you can define the `$EB_VERBOSE` environment variable. For example: + +```shell +$ EB_VERBOSE=1 eb --version +>> Considering 'python3.6'... +>> 'python3' version: 3.6.8, which matches Python 3 version requirement (>= 3.5) +>> Selected Python command: python3 (/usr/bin/python3.6) +>> python3.6 -m easybuild.main --version +This is EasyBuild 4.3.3 (framework: 4.3.3, easyblocks: 4.3.3) on host example +``` + +!!! Tip "Debugging startup problems" + + Using `ÈB_VERBOSE` is useful if EasyBuild fails to start up and complains it + cannot find a suitable Python executable. Rather often the error message is + caused by a failure elsewhere in EasyBuild. + + +--- + +### Method 2: Installing EasyBuild with EasyBuild + +!!! note + This section covers an alternative method for installing EasyBuild. + + If you already have EasyBuild installed through ``pip``, + you can skip ahead to the next section. + +If you prefer having EasyBuild available through an environment module file, +you can consider installing EasyBuild with EasyBuild. This can be done in 3 steps: + +* Step 1: Installing EasyBuild with ``pip`` into a temporary location (only needed if EasyBuild is not installed yet) +* Step 2: Using EasyBuild to install EasyBuild as a module +* Step 3: Loading the EasyBuild module + +!!! note + A [bootstrap script](https://docs.easybuild.io/en/latest/Installation.html#bootstrapping-easybuild) + is available that automates this procedure, + but is known to be problematic in some contexts, and is not being actively + maintained anymore. + + As a result, we do not recommend using the bootstrap script anymore. + + +#### Step 1: Installing EasyBuild into a temporary location + +If you don't have EasyBuild installed yet, you need to install it in a temporary location first. +The recommended way of doing this is using ``pip``. + +For example, to install EasyBuild into a subdirectory `/tmp/$USER` using the default Python 3 version: + +```shell +# pick installation prefix, and install EasyBuild into it +export EB_TMPDIR=/tmp/$USER/eb_tmp +python3 -m pip install --ignore-installed --prefix $EB_TMPDIR easybuild +``` + +```shell +# update environment to use this temporary EasyBuild installation +export PATH=$EB_TMPDIR/bin:$PATH +export PYTHONPATH=$(/bin/ls -rtd -1 $EB_TMPDIR/lib*/python*/site-packages | tail -1):$PYTHONPATH +export EB_PYTHON=python3 +``` + +#### Step 2: Using EasyBuild to install EasyBuild + +Once you have a working (recent) temporary EasyBuild installation, you can use it to +install EasyBuild as a module. Usually this is done in the location where you would +like to install other software too. + +You can use the ``eb --install-latest-eb-release`` command for this, +combined with the ``--prefix`` option to control which directories are used by EasyBuild for the installation. + +For example, to install the latest version of EasyBuild as a module into ``$HOME/easybuild``: + +```shell +eb --install-latest-eb-release --prefix $HOME/easybuild +``` + +!!! note + You may see a harmless deprecation warning popping up when performing this installation, just ignore it. + +#### Step 3: Loading the EasyBuild module + +Once step 2 is completed, you should be able to load the module that was generated alongside +the EasyBuild installation. You will need to do this every time you start a new shell session. + +First, make the module available by running the following command (which will update the module search path +environment variable `$MODULEPATH`): + +```shell + +module use _PREFIX_/modules/all +``` + +**Replace '`_PREFIX_`'** with the path to the directory that you used when running step 2 +(for example, ``$HOME/easybuild``). + +Then, load the `EasyBuild` module to update your environment and make EasyBuild available for use: + +```shell + +module load EasyBuild +``` + +Note that in this case, we don't need to make any changes to our environment for EasyBuild to work correctly. +The environment module file that was generated by EasyBuild specifies all changes that need to be made. + +### Method 3: Development setup + +If you are planning to make changes to EasyBuild, or if you prefer using the latest *bleeding edge* +version of EasyBuild that is being developed, you can consider *cloning* the 3 main EasyBuild repositories +from GitHub, and updating your environment to run EasyBuild from there. + +This can be done as follows (into ``$HOME/easybuild``): + +```shell +mkdir -p $HOME/easybuild +cd $HOME/easybuild +``` + +```shell +# clone EasyBuild repositories from GitHub +git clone https://github.com/easybuilders/easybuild-framework.git +git clone https://github.com/easybuilders/easybuild-easyblocks.git +git clone https://github.com/easybuilders/easybuild-easyconfigs.git +``` +```shell +# update environment for running EasyBuild from there +export PATH=$HOME/easybuild/easybuild-framework:$PATH +export PYTHONPATH=$HOME/easybuild/easybuild-framework:$HOME/easybuild/easybuild-easyblocks:$HOME/easybuild/easybuild-easyconfigs:$PYTHONPATH + +# control which Python command is used to run EasyBuild +export EB_PYTHON=python3 +``` + + +### Approach on LUMI + +*Documentation on the inner workings of the LUMI software stack can be found in the +[LUMI-SoftwareStack GitHub, docs subdirectory](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/docs).* + +To keep the different versions of the LUMI software stack as independent from one another as possible, +EasyBuild is bootstrapped for each software stack by the ``prepare_LUMI_stack.sh`` script. We use the +system Python 3 for running EasyBuild. That Python version currently doesn't have ``pip`` installed. +EasyBuild is first installed in a temporary work directory from files downloaded from PyPi. These +are each untarred and then installed into their location by running +``python3 setup.py install --prefix=...`` in their unpacked directory. +We do so only for the framework and easyblocks files as the easconfig files are not used to +install EasyBuild. Instead we create our own EasyConfig file for EasyBuild which contains +some additional packages that enable extra features in EasyBuild and also provide more information +to Lmod. Next the configuration module for EasyBuild (see the next section, +["Configuring EasyBuild"](1_06_configuring.md), for more information) and use the temporary +installation of EasyBuild with our own EasyConfig file to do a proper installation of EasyBuild +with module in the final location. + + +TODO: follow development setup but with downloading rather than cloning, then use that one with the +EasyBuild configuration module to install EasyBuild with itself. + + + +## Verifying the installation + +Regardless of how EasyBuild was installed, you can now run a couple of basic commands to verify the installation: + +#### Checking the version + +To check which EasyBuild version you have installed, run: + +```shell + +eb --version +``` + +The output should match with the latest EasyBuild version. + + +#### Consulting the help output + +You can consult the help output of the `eb` command, which produces a long list of available options +along with a short informative message. + +```shell +eb --help +``` + +#### Showing the default EasyBuild configuration + +To inspect the current EasyBuild configuration, you can use this command: + +```shell +eb --show-config +``` + +This should tell you that EasyBuild (ab)uses `$HOME/.local/easybuild` as a default location. +More on configuring EasyBuild in the [next part of the tutorial](configuration.md). + +#### System information + +You ask EasyBuild to collect and print some information about the +system you are using it on (OS, CPU, Python, etc.) using this command: + +```shell +eb --show-system-info +``` + +## Updating EasyBuild + +Before we wrap up here, a brief word about updating EasyBuild. + +Once you have EasyBuild installed, the easiest way to update to a newer version is by instructing EasyBuild +to install the latest available version as a module: + +``` + +eb --install-latest-eb-release +``` + +This will result in a *new* EasyBuild installation, which is entirely separate from the EasyBuild installation +you are currently using (so it is *not* an in-place update). +The location where this new EasyBuild version will be installed is determined by the active +EasyBuild configuration. + +If you have installed EasyBuild through ``pip``, and you prefer updating that installation, +you can use ``pip install --upgrade easybuild`` (perhaps with additional options like ``--user`` or ``--prefix``). + +!!! Note "EasyBuild upgrade policy on LUMI" + + Even minor version or patch level updates of EasyBuild may break things (and this has happned + in the past on other systems managed previously by LUMI User Support Team members). Hence a very + conservative upgrade policy is used on LUMI. + + In general we fix the version of EasyBuild within a particular version of the LUMI software stack + and only transition to the latest version when starting a new software stack. The reason is that we + want to be able to rebuild an existing software stack with as little unexpected problems as possible. + A move to a newer version of EasyBuild is acceptable should we run into major problems with an existing + version that cannot be solved in an easyconfig file or LUMI-specific custom easyblock, but in those + cases we will first rebuild the whole software stack on a test setup to ensure that no problems are + introduced by upgrading EasyBuild. + + +--- + +## Exercise (optional) + +*This exercise is not needed to proceed the course when working on LUMI as we will +use the installation that is already present.* + +Install EasyBuild in your home directory. + +Make sure that the EasyBuild installation uses the `python3` command to run, +rather than the standard `python` command. + +Choose your own adventure, or try all these installation methods! + +* install EasyBuild with `pip` (or another very similar command...) using either the `--user` or `--prefix` option; +* perform a (manual) "bootstrap" installation into `$HOME/easybuild`, as outlined in installation method 2; +* set up a development installation; + +Check that the installation works by running the verification commands outlined +[above](#verifying-the-installation). + +--- + +**Make sure you have a working EasyBuild installation before proceeding +with the rest of the tutorial!** + +*[[next: Configuring EasyBuild]](1_06_configuration.md)* From 99a46fe8d0f0a8c38f166e49b5b0641c6ae04d4d Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 17:31:43 +0100 Subject: [PATCH 065/159] Tutorial on EasyBuild configuration added, and some restructuring. --- .../1_04_LUMI_software_stack.md | 17 + ..._04_terminology.md => 1_05_terminology.md} | 0 ...5_installation.md => 1_06_installation.md} | 0 docs/2022-CSC_and_LO/1_07_configuration.md | 610 +++++++++++ docs/2022-CSC_and_LO/1_08_basic_usage.md | 976 ++++++++++++++++++ 5 files changed, 1603 insertions(+) create mode 100644 docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md rename docs/2022-CSC_and_LO/{1_04_terminology.md => 1_05_terminology.md} (100%) rename docs/2022-CSC_and_LO/{1_05_installation.md => 1_06_installation.md} (100%) create mode 100644 docs/2022-CSC_and_LO/1_07_configuration.md create mode 100644 docs/2022-CSC_and_LO/1_08_basic_usage.md diff --git a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md new file mode 100644 index 00000000..0e578a59 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md @@ -0,0 +1,17 @@ +# LUMI software stacks + +*[[back: The Cray Programming Environment]](1_03_CPE.md)* + +--- + + + + + + + + +--- + +*[[next: Terminology]](1_05_terminology.md)* + diff --git a/docs/2022-CSC_and_LO/1_04_terminology.md b/docs/2022-CSC_and_LO/1_05_terminology.md similarity index 100% rename from docs/2022-CSC_and_LO/1_04_terminology.md rename to docs/2022-CSC_and_LO/1_05_terminology.md diff --git a/docs/2022-CSC_and_LO/1_05_installation.md b/docs/2022-CSC_and_LO/1_06_installation.md similarity index 100% rename from docs/2022-CSC_and_LO/1_05_installation.md rename to docs/2022-CSC_and_LO/1_06_installation.md diff --git a/docs/2022-CSC_and_LO/1_07_configuration.md b/docs/2022-CSC_and_LO/1_07_configuration.md new file mode 100644 index 00000000..17e60afe --- /dev/null +++ b/docs/2022-CSC_and_LO/1_07_configuration.md @@ -0,0 +1,610 @@ +# Configuring EasyBuild + +*[[back: Installation]](1_06_installation.md)* + +--- + +After installing EasyBuild, you should **configure** it. + +EasyBuild should work fine out-of-the-box if you use Lmod as your modules tool. +If you are not using Lmod, please see here for more information. + +Nevertheless, we strongly recommend you to inspect the default configuration, +and to configure EasyBuild according to your preferences and the system on which you will use it. + +!!! Note "EasyBuild configuration on LUMI" + + On LUMI serveral configurations of EasyBuild are already available. + + The ``EasyBuild-user`` module is the most important one. It will configure EasyBuild + to install software in either a default location in the user's home directory + (``$HOME/EasyBuild``) or the location pointed to by the environment variable + ``EBU_USER_PREFIX``. The latter can be used to install software in the project directory + so that it is available to all users of the project. It enables the creation of a custom + software environment build on top of the LUMI software stack. Software in this environment is + build in exactly the same way as it would if it were installed in the central repository, but + one can now also easily create custom configurations without confusing other projects with + different or non-standard configurations of software. + + There are two more admin-only configuration modules for EasyBuild. The main one is + ``EasyBuild-production`` which is used for software installations in the central software stack. + The other one is ``EasyBuild-infrastructure`` which is used to install infrastructure modules + that need to be installed in a pure Lmod hierarchy, e.g., the modules that load the toolchains. + + +--- + +## Available configuration settings + +One of the central policies in the EasyBuild project is to avoid hardcoded settings in the codebase. +While this significantly increases the ability to configure EasyBuild to your liking, it also results +in a large amount of available configuration settings. + +The full list of configuration settings can be consulted via `eb --help`, +which shows the corresponding command line option accompanied by a short description. +At the time of writing, *over 240 different configuration settings* are supported by EasyBuild. + +For the sake of this tutorial we will focus on a specific subset of configuration settings, +and cover only the most prominent and important ones. + +We will refer to EasyBuild configuration settings using the names as they appears in the output of `eb --help`, +and omit the leading dashes (`--`) for the sake of clarity. + +Keep in mind that *every* configuration setting can be defined in 3 different ways, +see below for more details. + +--- + +### Overall prefix + +*(default: `HOME/.local/easybuild`)* + +The `prefix` configuration setting specifies the overall prefix that EasyBuild should use, +which **determines the default value for various other configuration settings**: + +* `installpath`: `` +* `buildpath`: `/build` +* `sourcepath`: `/sources` +* `repositorypath` (easyconfigs archive): `/ebfiles_repo` +* `containerpath`: `/containers` + +Here, `` represents the value of the `prefix` configuration setting. + +If one of the configuration settings affected by `prefix` is defined specifically, +the `prefix` value becomes irrelevant for that specific configuration setting. + +--- + +### Install path + +*(default: ``)* + +The **location for both the software installation directories and generated module files** +can be controlled via the `installpath` configuration setting. +Software installation directories will be placed in `/software`, while +`/modules/all` will be used for generated module files. + +The `installpath` location is usually set to a directory on a *shared filesystem* when installing +software for an HPC cluster. Of course, software can also be installed on a local filesystem, +which is particularly useful to test and evaluate software installations. + +Separate configuration settings are available for both software and modules locations, +as well as for controlling the name of the `software` and `modules/all` subdirectories. + +*The EasyBuild community recommends to only change the `installpath` configuration setting to control the location +of software installations and accompanying module files, +such that the software and modules directories are located in the same parent directory, +and the default `software` and `modules/all` names for the subdirectories are used.* + +!!! Note "Path for software and for modules on LUMI" + + On LUMI we do not follow that recommendation. There is a three for the software installations + themselves with subdirectories based on the version of the software stack and LUMI hardware + partition, and a separate tree for the modules organised in a similar way. + + This makes it slightly easier to organise the module tree with user-friendly labeling, but above + all also makes the synchronisation process of the 4 instances of the software directory more robust + as it is now easy to synchonise all modules in the last step, which is a much quicker process than + syncrhonising the software installations. + + We also use short paths for software installations (to avoid overrunning the maximum length of a + shebang line in scripts) while we use longer, more descriptive names for subdirectories in the + module tree. + + +--- + +### Build path + +*(default: `/build`)* + +For each installation it performs, EasyBuild creates a **separate build directory** where software will be compiled +before installing it. This directory is cleaned up automatically when the installation is successfully completed. +To control the location where these build directories are created, you can use the `buildpath` configuration setting. + +Keep in mind that build directories may grow out to several GBs in size during an installation, +and that the commands that run in there can be fairly I/O-intensive since they may involve +manipulating lots of small files. In addition, a build directory that corresponds to a failing installation +is *not* cleaned up automatically, but it will be cleaned up and recycled when the same installation is re-attempted. +Running out of disk space in the location where build directories are created will result in failing +installations. + +It is strongly recommend to use the path to a directory on a *local filesystem* for the value of the +`buildpath` configuration setting, since using a shared filesystem like Lustre or GPFS is known to cause +problems when building certain software packages. Using an in-memory location (like `/dev/shm/$USER`) can +significantly speed up the build process, but may also lead to problems (due to space limitations, +or specific mount options like `noexec`). + +!!! Note "buildpath on LUMI" + + The configuration modules on LUMI will use a RAM disk for the build path. On the login nodes, + ``$XDG_RUNTIME_DIR`` is used as that space is automatically cleared when the last session of a user + ends. However, on the compute nodes a job- or user-specific subdirectory of ``/dev/shm`` is currently used + as ``$XDG_RUNTIME_DIR`` does not exist. + + +--- + +### Source path + +*(default: `/sources`)* + +For most supported software, EasyBuild can **automatically download the source files** required for the installation. +Before trying to download a source file, EasyBuild will first check if it is already present in the source path. + +The locations considered by EasyBuild when checking for available source files, as well as the location to +store downloaded source files, can be controlled via the ``sourcepath`` configuration setting. + +The `sourcepath` value is a colon (`:`) separated list of directory paths. +Each of these paths will be considered in turn when checking for available source files, +until one of them provides the desired source file. Searching for source files is done +based on filename, and a couple of subdirectories are considered. +For example, for a software package named '`Example'`, EasyBuild will consider locations +like `/e/Example/`, `/Example/`, and so on. + +The first path listed in `sourcepath` is the location where EasyBuild will store downloaded source files, +organised by software name through subdirectories, so EasyBuild expects to have write permissions to this path. +For the other paths listed in `sourcepath` only read permissions are required. + +*Make sure you have write permissions to the first path listed in `sourcepath`, so EasyBuild is able +to store downloaded files there. Feel free to list additional paths if you already have a cache of downloaded +files available somewhere.* + +*Storing the downloaded files not only reduces the amount of downloads while developing new easyconfig files +but also greatly helps when recompiling a software stack, as it is not that uncommon that download sites change +of files become unavailable.* + + +--- + +### Easyconfigs archive + +*(default: `/ebfiles_repo`)* + +EasyBuild keeps track of the easyconfig files that were used for installations in the easyconfigs +archive, the location of which is specified by the `repositorypath` configuration setting. + +By default the specified path is assumed to be a regular directory, but using a Git repository +as easyconfigs archive is also supported (for more details, see +[the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Configuration.html#easyconfigs-repository-repository-repositorypath)). + + +--- + +### Modules tool & module syntax + +*(default: `Lmod` as modules tool, and `Lua` as module syntax)* + +By default, EasyBuild assumes you are using **[Lmod](https://lmod.readthedocs.io) as modules tool**. +In addition, it will generate module files in [Lua](https://www.lua.org/) syntax, as supported by Lmod +(next to `Tcl` syntax). + +To diverge from this, you can define the `modules-tool` configuration setting to indicate you +are using a different modules tool; see the output of `eb --avail-modules-tools` for a list of supported +modules tools. +Note that for anything other than Lmod, you *must* make +sure that the actual modules tool binary command is available through `$PATH` (more information +on this [in the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Configuration.html#modules-tool-modules-tool)). + +If you prefer that EasyBuild generates module files in [Tcl](https://tcl.tk/) syntax, you can instruct +it to do so via the `module-syntax` configuration setting. Note that this is *required* if you are using +a modules tool other than Lmod, since only Lmod supports module files in `Lua` syntax. + +*We recommend using EasyBuild with the defaults settings: Lmod as modules tool, and Lua as module syntax.* + +--- + +### Robot search path + +*(default: `robot-paths` specifies the location to the easyconfig files included with EasyBuild installation; +dependency resolution is* ***not*** *enabled)* + +When EasyBuild needs to locate one or more easyconfig files, it will do so via the **robot search path**. +This applies to both easyconfig files that were specified using only their filename as an argument to the +`eb` command, as well as to easyconfigs required to resolve dependencies (more on that later). + +To control the robot search path, two configuration settings are available: `robot` and `robot-paths`. +Both accept a colon-separated list of locations to consider when looking for easyconfig files, +with `robot` having a higher priority than `robot-paths` (i.e., the directories specified with `robot` +appear before those specified by `robot-paths` in the robot search path). + +The key difference between these two configuration settings is that defining `robot` also enables dependency resolution, +and hence makes EasyBuild install missing dependencies, alongside +specifying a list of paths to consider when searching for easyconfig files. +On the other hand, defining `robot-paths` does not have the side effect of enabling dependency resolution. + +In addition, you can use the `--robot` command line option without specifying any paths to it, +to only enable dependency resolution. ***This option is needed on LUMI if you want automatic +dependency resolution as it is disabled on LUMI to prevent accidental mistakes when installing +software.*** + +!!! note "Easyconfig files included with EasyBuild and the robot search path" + Keep in mind that when either of the `robot` or `robot-paths` configuration settings are + defined, the default value corresponding to the location of easyconfigs included with the + EasyBuild is *no longer considered*. + + There are ways around this however, which are outside of the scope of this tutorial. + + For more information, + [see the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Using_the_EasyBuild_command_line.html#controlling-robot-search-path). + + *On LUMI the included easyconfig files cannot be used without changes as they are for the + common toolchains which are currently not supported in the LUMI software stacks.* + + +!!! Note "Easyconfigs archive on LUMI and the robot search path" + + On LUMI we also include the easyconfigs archive at the front of the search path for easyconfig + files. This has the advantage that EasyBuild when checking dependencies will always find the + configuration that is actually installed on the system, even if the easyconfig file in one of + the regular repositories has changed. It also has the disadvantage that one may accidentally + be re-installing with the already installed configuration while one really wants to re-install + the module in a different configuration. + + + +--- + +### Module naming scheme + +*(default: `EasyBuildMNS`)* + +EasyBuild will use a specific naming scheme for the module files it generates for each of the software installations. +This ensures consistency, regardless of who uses EasyBuild to perform the installation. + +Different types of module naming schemes are supported (flat, hierarchical, ...) and you can provide an implementation of +your own custom module naming scheme if desired. A number of different naming schemes are included +with EasyBuild, which you can consult via `eb --avail-module-naming-schemes`. + +The default `EasyBuildMNS` module naming scheme roughly corresponds to the filename of easyconfig files, +and consists of the software name followed by a combination of the software version, toolchain and +an optional label (which corresponds to the value of the `versionsuffix` easyconfig parameter): +`/<-toolchain>`. Just like with [names of easyconfig +files](terminology.md#easyconfig-files), the `<-toolchain>` part is omitted when the `system` toolchain is used, +and the `` value is empty by default. + +!!! Note "Module naming scheme on LUMI" + + On LUMI we use a custom variant of the standard EasyBuild flat naming scheme. The EasyBuild default + naming scheme creates an ``all`` subdirectory in the module directory containing all modules, but also + a directory per category, with the categories determined from the ``moduleclass`` easyconfig parameter. + However, that choice is often rather arbitrary as modules can belong to different categories depending + on who you ask. E.g., GROMACS is used in physics, chemistry and life sciences while EasyBuild can only + assign a single category. + + Hence we do not generate the categories and drop the ``all`` level in the module subdirectory. + + +--- + +## Configuration levels + +Configuring EasyBuild can be done in different ways: + +* through one or more configuration files; +* via `$EASYBUILD_*` environment variables; +* using `eb` command line options; + +Each of the methods corresponds to a *configuration level*. + +*Every* configuration setting can be defined via one of these +mechanisms, without exception! + +--- + +### Configuration level hierarchy + +There is a strict **hierarchy between the different configuration levels** supported by EasyBuild. + +Settings defined via a *configuration file* only override default values. + +A configuration setting that is defined via the corresponding *`$EASYBUILD_*` environment variable* +takes precedence over the value specified in a configuration file (if any). + +Finally, values specified through `eb` command line options **always** win, +regardless of whether the corresponding +configuration setting was already defined some other way, be it via a configuration file or +an environment variable. + +For example, let us consider a fictional configuration setting named `magic`: + +* If a value for `magic` is specified in an EasyBuild configuration file, + then this value will only be used if the corresponding environment variable + (`$EASYBUILD_MAGIC`) is *not* defined **and** if the `--magic` command line option is *not* used; +* If the `$EASYBUILD_MAGIC` environment *is* defined however, then its value + will be used for the `this-is-magic` configuration setting; +* *Unless* the `--magic` command line option is used, + since in that case the value provided as an argument there will be used instead. + +--- + +### Configuration files + +*Configuration files* are the most basic way of configuring EasyBuild. +Two types of are supported by EasyBuild: *user-level* and *system-level* +configuration files. The output of `eb --show-default-configfiles` tells you which locations +are considered, and whether any configuration files were found. + +EasyBuild configuration files are written in the standard [INI format](https://en.wikipedia.org/wiki/INI_file), +and the configuration settings are grouped into different *sections*. + +To create an EasyBuild configuration file, the output of `eb --confighelp` is very useful: +it produces the list of all supported configuration settings which are readily grouped in sections +and with every entry commented out, along with accompanying descriptive comments mentioning the default values, +and in the correct syntax. + +*Configuration files are the recommended mechanism to define configuration settings +that should always be in place, regardless of the software you are installing.* + +#### EasyBuild configuration files vs easyconfig files + +EasyBuild configuration files are sometimes confused with easyconfig files, +due to the similar name. However, these are two entirely different concepts! + +EasyBuild configuration files (usually `*.cfg`) are a way of *configuring the general behaviour of EasyBuild* +across different software installations. They define *configuration settings*, +such as the location where software should be installed, or the syntax that should +be used when generating module files. + +An [easyconfig file](terminology.md#easyconfig-files) (`*.eb`) on the other hand *specifies the details for one particular software installation*. +It does this by defining a set of *easyconfig parameters*, which tell EasyBuild the name and version +of the software to install, which [toolchain](terminology.md#toolchains) and [easyblock](terminology.md#easyblocks) to use, etc. + +For each software installation performed by EasyBuild, there is a corresponding easyconfig file. +There typically are only a handful of configuration files used however, for example a system-level +configuration file, perhaps combined with a user-level one. Or there may be no configuration files involved +at all, since EasyBuild can also be configured through other mechanisms: environment variables and command line +options. + +--- + +### `$EASYBUILD_*` environment variables + +A particularly easy way to configure EasyBuild is through *environment variables*. + +At startup, EasyBuild will pick up any environment variable of which the name starts with '`EASYBUILD_`'. +For each of these, it will determine the corresponding configuration setting (or exit with an error if +none was found). + +Mapping the name of a configuration setting to the name of the corresponding environment variable is straightforward: +use capital letters, replace dashes (`-`) with underscores (`_`), and prefix with `EASYBUILD_`. + +For example: the `module-syntax` configuration setting can be specified by defining +the `$EASYBUILD_MODULE_SYNTAX` environment variable: + +```shell +export EASYBUILD_MODULE_SYNTAX=Tcl +``` + +*Configuring via environment variables is especially practical for controlling the EasyBuild configuration +in a more dynamic way. For example, you can implement a simple shell script that defines `$EASYBUILD_*` +environment variables based on the current context (user, hostname, other environment variables), and +configure EasyBuild through [sourcing](https://bash.cyberciti.biz/guide/Source_command) it.* + +!!! note + Keep in mind that environment variables are only defined for the shell session you are currently working in. + If you want to configure EasyBuild through environment variables in a more persistent way, + you can leverage one of the [shell startup scripts](https://bash.cyberciti.biz/guide/Startup_scripts) (for example `$HOME/.bash_profile` or `$HOME/.bashrc`). + +--- + +### `eb` command line options + +Finally, you can also configure EasyBuild by specifying one or options to the `eb` command. + +As mentioned earlier, the values for configuration settings defined this way override the value that +is specified through any other means. So if you want to be sure that a particular configuration setting +is defined the way you want it to be, you can use the corresponding command line option. + +There are various configuration settings for which it only makes sense to use the command line option. +An example of this is letting the `eb` command print the EasyBuild version (via `eb --version`). +Although you could configure EasyBuild to always print its version and then exit whenever the `eb` command is +run, that would not be very useful... + +*Command line options are typically used to define configuration settings that are only relevant to +that particular EasyBuild session. One example is doing a test installation into a temporary directory:* + +```shell +eb --installpath /tmp/$USER example.eb +``` + +## Inspecting the current configuration (`--show-config`) + +Given the large amount of available configuration settings in EasyBuild and the different configuration levels, +you can easily lose track of exactly how EasyBuild is configured. + +Through the `--show-config` command line option you can +easily inspect the currently active EasyBuild configuration. + +The output of `--show-config` includes a sorted list of all configuration settings that are defined to a +*non-default* value, +along with a couple of important ones that are always shown (like `buildpath`, `installpath`, `sourcepath`, and so on). +In addition, it also indicates at which configuration level each setting was defined, +so you can trace down *where* it was defined if needed. + +This is the output produced by `eb --show-config` for the default EasyBuild configuration, +where EasyBuild was installed via `pip install --user` (which results in the value shown for the +`robot-paths` configuration setting): + +```shell +# +# Current EasyBuild configuration +# (C: command line argument, D: default value, E: environment variable, F: configuration file) +# +buildpath (D) = /home/example/.local/easybuild/build +containerpath (D) = /home/example/.local/easybuild/containers +installpath (D) = /home/example/.local/easybuild +repositorypath (D) = /home/example/.local/easybuild/ebfiles_repo +robot-paths (D) = /home/example/.local/easybuild/easyconfigs +sourcepath (D) = /home/example/.local/easybuild/sources +``` + +As shown here, all configuration settings shown follow the default `prefix` value (`$HOME/.local/easybuild`), +and none of the values diverge from the default value, since all entries are marked with `(D)` for "default value"). + +**Example** + +Now let us do some basic configuring and inspect the resulting output of `--show-config`. + +First, create a user-level EasyBuild configuration file to define the `prefix` configuration setting: + +```shell +mkdir -p $HOME/.config/easybuild +echo '[config]' > $HOME/.config/easybuild/config.cfg +echo 'prefix=/apps' >> $HOME/.config/easybuild/config.cfg +``` + +In addition, define the `buildpath` configuration setting using the corresponding +environment variable: + +```shell +export EASYBUILD_BUILDPATH=/tmp/$USER +``` + +Then run `--show-config` while you specify that the `installpath` configuration +setting should be defined as `/tmp/$USER`: + +```shell +$ eb --installpath=/tmp/$USER --show-config +# +# Current EasyBuild configuration +# (C: command line argument, D: default value, E: environment variable, F: configuration file) +# +buildpath (E) = /tmp/easybuild +containerpath (F) = /apps/containers +installpath (C) = /tmp/easybuild +packagepath (F) = /apps/packages +prefix (F) = /apps +repositorypath (F) = /apps/ebfiles_repo +robot-paths (D) = /home/example/.local/easybuild/easyconfigs +sourcepath (F) = /apps/sources +``` + +The output indicates that the `installpath` setting was specified through a command line option (indicated +with `(C)`), that the `buildpath` setting was defined via an environment +variable (indicated with `(E)`), that the `robot-paths` setting still has the default value (indicated with `(D)`), and that all other configuration +settings were specified via a configuration file, some of which indirectly through the `prefix` value (indicated with +`(F)`). + +--- + +## Exercises (optional) + +*These exercises are not very relevant for LUMI as LUMI already offers a complete configuration +also for user installations of software. However, if you are a very advanced user, you may still +want to make changes to that configuration, and all three options (configuration files, +environment variables and command line parameters) are available to users in the LUMI setup.* + +***Exercise C.1* - Configure EasyBuild** + +Configure EasyBuild to use the `easybuild` subdirectory in your home directory for everything, except for: + +* The location of the build directories: use `/tmp/$USER` for this; +* The locations that should be considered when searching for source files: + include both `$HOME/easybuild/sources` and `/easybuild/sources`, but make + sure that source files that are downloaded by EasyBuild are stored in + `$HOME/easybuild/sources`. + +Leave other configuration settings set to their default value. + +??? success "(click to show solution)" + + This is pretty straightforward. + + Here we just define the corresponding environment variables: + + ```shell + export EASYBUILD_PREFIX=$HOME/easybuild + export EASYBUILD_BUILDPATH=/tmp/$USER + export EASYBUILD_SOURCEPATH=$HOME/easybuild/sources:/easybuild/sources + ``` + + The location where EasyBuild should download source files to + must be listed first in the `sourcepath` configuration setting. + + The output of `--show-config` should look like this: + + ```shell + buildpath (E) = /tmp/example + containerpath (E) = /home/example/easybuild/containers + installpath (E) = /home/example/easybuild + packagepath (E) = /home/example/easybuild/packages + prefix (E) = /home/example/easybuild + repositorypath (E) = /home/example/easybuild/ebfiles_repo + robot-paths (D) = /home/example/easybuild/easyconfigs + sourcepath (E) = /home/example/easybuild/sources:/easybuild/sources + ``` + +--- + +***Exercise C.2* - Install a trivial software package with EasyBuild** + +Try running the following command: + +```shell +eb bzip2-1.0.6.eb +``` + +Where do you expect to find the installation? + +??? success "(click to show solution)" + + The software was installed in `$HOME/easybuild`, + since that's how we configured EasyBuild in *Exercise 3.1*: + + ```shell + $ ls $HOME/easybuild + ebfiles_repo modules software sources + ``` + + The actual installation is in `$HOME/easybuild/software`, + while the module file was generated in `$HOME/easybuild/modules/all`: + + ```shell + $ ls $HOME/easybuild/software + bzip2 + $ ls $HOME/easybuild/software/bzip2 + 1.0.6 + $ ls $HOME/easybuild/software/bzip2/1.0.6 + bin easybuild include lib man + ``` + + ```shell + $ ls $HOME/easybuild/modules/all + bzip2 + $ ls $HOME/easybuild/modules/all/bzip2 + 1.0.6.lua + ``` + + The source file for bzip2 1.0.6 was downloaded to `$HOME/easybuild/sources`: + + ```shell + $ ls $HOME/easybuild/sources/b/bzip2 + bzip2-1.0.6.tar.gz + ``` + + We will discuss this in more detail in the next part of the tutorial. + +--- + +*[[next: Basic usage]](1_08_basic_usage.md)* diff --git a/docs/2022-CSC_and_LO/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_08_basic_usage.md new file mode 100644 index 00000000..5566d376 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_08_basic_usage.md @@ -0,0 +1,976 @@ +# Basic usage + +*[[back: Configuring EasyBuild]](1_07_configuration.md)* + +--- + +Now that we have installed and configured EasyBuild, we can start using it for what it is intended for: +getting scientific software installed without breaking a sweat, or having to resist the urge to +shout out four-letter words. + +We will look at the high-level workflow first, and then cover each aspect in more detail. + +A couple of exercises are available at the end to help to make you more familiar with the +EasyBuild command line interface, so pay attention! + +--- + +## Overall workflow + +Installing software with EasyBuild is as easy as specifying to the **`eb` command** what we +want to install, and then sitting back to enjoy a coffee or tea (or whatever beverage you prefer). + +This is typically done by **specifying the name of one or more easyconfig files**, often combined +with the `--robot` option to let EasyBuild also install missing dependencies. + +It is recommended to first assess the current situation before letting EasyBuild install the software, +and to check which **dependencies** are already installed and which are still missing. In addition, +you may want to inspect the specifics of the **installation procedure** that will be performed by EasyBuild, +and ensure that the active EasyBuild configuration is what it should be. + +--- + +## Specifying easyconfigs + +Letting EasyBuild know what should be installed can be done by specifying one or more easyconfig files, +which is also the most common way. Alternative methods like using the `--software-name` option won't be +covered in this tutorial, since they are not commonly used. + +Arguments passed to the `eb` command, being anything that is *not* an option (which starts with `-` or `--`) or +is a value for a preceding configuration option, are assumed to refer to easyconfig files (with some exceptions). +These could be: + +* the *(absolute or relative) path* to an easyconfig file; +* the *name* of an easyconfig file; +* the path to a *directory* containing easyconfig files; + +Specified paths to files must of course point to existing files; if not, EasyBuild will print an appropriate error message: + +```shell +$ eb /tmp/does_not_exist.eb +ERROR: Can't find path /tmp/does_not_exist.eb +``` + +When only the *name* of an easyconfig file is specified, EasyBuild will automatically try and locate it. +First, it will consider the *current directory*. If no file with the specified name is found there, +EasyBuild will search for the easyconfig file in the [robot search path](../configuration/#robot-search-path). + +If the path to an existing *directory* is provided, EasyBuild will walk through the entire directory +(including all subdirectories), retain all files of which the name ends with `.eb`, and (try to) use these +as easyconfig files. + + +#### Example command + +Suppose we have the current situation in our home directory: + +* two (easyconfig) files named `example1.eb` and `example2.eb`; +* a subdirectory named `some_deps`, which has two easyconfig files `dep1.eb` and `dep2.eb` + alongside a text file named `list.txt`; +* a subdirectory named `more_deps` located *in* the `some_deps` subdirectory, + which contains another easyconfig file `dep3.eb`; + +Or, visually represented: + +```shell +example1.eb +example2.eb +some_deps/ +|-- dep1.eb +|-- dep2.eb +|-- list.txt +|-- more_deps/ + |-- dep3.eb +``` + +In this context, we run the following EasyBuild command from our home directory: + +```shell +eb example1.eb bzip2-1.0.6.eb $HOME/example2.eb some_deps +``` + +EasyBuild will interpret each of these arguments as follows: + +* `example1.eb` is the name of a file in the current directory, so it can be used directly; +* `bzip2-1.0.6.eb` is the name of an easyconfig file to locate via the robot search path + (since it does not exist in the current directory); +* `$HOME/example2.eb` specifies the full path to an existing file, which can be used directly; +* `some_deps` is the relative path to an existing directory, so EasyBuild will scan it and find three + easyconfig files: `some_deps/dep1.eb`, `some_deps/dep2.eb` and `some_deps/more_deps/dep3.eb`, + The `list.txt` file will be ignored since its name does not end with `.eb`. + +--- + +### Easyconfig filenames + +Note that the `eb` command does not care how easyconfig files are named, at least to some extent: the `.eb` +file extension *does* matter when easyconfig files are being picked up in subdirectories. + +File names for easyconfigs being mostly irrelevant is only correct with respect to the arguments passed to +the `eb` command however. As we will learn soon, the name of easyconfig files *does* matter (a lot) +when EasyBuild needs to locate easyconfigs that can be used to resolve a specified dependency +(see [here](#enabling-dependency-resolution)). + +This explains why easyconfig files usually adher to a very specific naming scheme, +corresponding to `--.eb`, where: + +* `` represents the software name; +* `` represents the software version; +* `` represents the toolchain used in the easyconfig file, which consists of the toolchain name + and version separated with a dash (`-`), and which is omitted (including the preceding `-`) when the + [`system` toolchain](../introduction#system-toolchain) is used; +* `` represents the value of the `versionsuffix` easyconfig parameter, + which is sometimes used to distinguish multiple variants of particular software installations + (and is empty by default); + +--- + +### Searching for easyconfigs + +You will frequently need to determine the exact name of an easyconfig file you want to install, +or just check which easyconfigs are available for a given software package. +This can be done by searching for easyconfigs using **`eb --search`** or **`eb -S`**. + +By default all directories listed in the [robot search path](../configuration#robot-search-path) will be +searched. If you want to search in additional directories without changing the robot search path, +you can use the `search-paths` configuration setting, or you can change the robot search path via either +the `robot` or `robot-paths` configuration options. + +Both the `--search` and `-S` options trigger the same search operation, but yield different output: +`eb --search` will print the full path to each easyconfig file that matches the specified search pattern, +while `eb -S` produces a more concise output. + +For example, let's check which easyconfig files are available for OpenFOAM 8 with a `foss` toolchain: + +```shell +$ eb --search openfoam-8-foss + * /home/example/.local/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-8-foss-2020a.eb + * /home/example/.local/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-8-foss-2020b.eb +``` + +The output is a bit more condensed when using `eb -S`: + +```shell +$ eb -S openfoam-8-foss +CFGS1=/home/example/.local/easybuild/easyconfigs/o/OpenFOAM + * $CFGS1/OpenFOAM-8-foss-2020a.eb + * $CFGS1/OpenFOAM-8-foss-2020b.eb +``` + +Note that the search is performed *case-insensitive*. + +The search pattern can include wildcards like `.*` and/or character groups like `[0-9]`, +or other special characters like `^` or `$` to mark the start/end of the filename, +but you need to be careful that `bash` does not expand these before the `eb` command is started, +so it is recommended to wrap the search pattern in single quotes (`'...'`) when using wildcards. + +For example, to check which easyconfigs are available to install TensorFlow 2.4.1 with the `2020b` version of a toolchain: + +```shell +$ eb -S '^tensorflow-2.4.1.*2020b' +CFGS1=/home/example/.local/easybuild/easyconfigs/t/TensorFlow + * $CFGS1/TensorFlow-2.4.1-foss-2020b.eb + * $CFGS1/TensorFlow-2.4.1-fosscuda-2020b.eb +``` + +--- + +#### Search index + +When searching for easyconfig files, you may see a message like this pop up: + +``` +== found valid index for , so using it... +``` + +This indicates that a **search index** was used for this particular directory which significantly speeds +up the search procedure, especially when the easyconfig files are located on a shared filesystem where metadata +operations involving lots of (small) files can be slow. + +For the easyconfig files included with an EasyBuild release, a search index is readily provided. +For other directories, you can create a search index using `eb --create-index `. +[See the EasyBuild documentation for more information](https://easybuild.readthedocs.io/en/latest/Easyconfigs_index.html). + +--- + +## Inspecting easyconfigs + +Once you have determined the name of the easyconfig file that corresponds to the software you want to install, +you may want to take a closer look at its contents before employing it. + +Since easyconfig files are simple text files (in Python syntax), you could use the ubiquitous `cat` command +or your favorite text editor (`vim`, what else). To avoid that you need to locate the easyconfig file first +and copy-paste the full path to it, you can use **`eb --show-ec`**. + +For example, let's inspect the contents of the `bzip2-1.0.6.eb` easyconfig file: + +```shell +$ eb --show-ec bzip2-1.0.6.eb +== temporary log file in case of crash /tmp/eb-jnpzclhl/easybuild-e37cbrj1.log +== Contents of /home/example/.local/easybuild/easyconfigs/b/bzip2/bzip2-1.0.6.eb: +name = 'bzip2' +version = '1.0.6' + +homepage = 'https://sourceware.org/bzip2' +description = """bzip2 is a freely available, patent free, high-quality data compressor. It typically +compresses files to within 10% to 15% of the best available techniques (the PPM family of statistical +compressors), whilst being around twice as fast at compression and six times faster at decompression.""" + +toolchain = SYSTEM +toolchainopts = {'pic': True} + +source_urls = ['https://sourceware.org/pub/bzip2/'] +sources = [SOURCE_TAR_GZ] +patches = ['bzip2-%(version)s-pkgconfig.patch'] +checksums = [ + 'a2848f34fcd5d6cf47def00461fcb528a0484d8edef8208d6d2e2909dc61d9cd', # bzip2-1.0.6.tar.gz + '5a823e820b332eca3684416894f58edc125ac3dace9f46e62f98e45362aa8a6d', # bzip2-1.0.6-pkgconfig.patch +] + +buildopts = "CC=gcc CFLAGS='-Wall -Winline -O3 -fPIC -g $(BIGFILES)'" + +# building of shared libraries doesn't work on OS X (where 'gcc' is actually Clang...) +with_shared_libs = OS_TYPE == 'Linux' + +moduleclass = 'tools' + +== Temporary log file(s) /tmp/eb-jnpzclhl/easybuild-e37cbrj1.log* have been removed. +== Temporary directory /tmp/eb-jnpzclhl has been removed. +``` + +We'll get back to what all of this means later... + +--- + +## Checking dependencies + +!!! note + In some of the examples below, we assume that some software is already installed with EasyBuild. + + If you are following hands-on in a prepared environment, make sure these installations are visible + in the output of "`module avail`". + + +Before kicking off an installation, it is good practice to check which of the required dependencies +are already installed, and which ones are still missing. + +This can be helpful to ensure that your EasyBuild configuration is set up correctly, +and to prevent from accidentally installing an entirely new software stack from scratch +in an unintended location (like `$HOME/.local/easybuild`, the +[default installation path](../configuration/#install-path)). + +--- + +### Dry run + +To get a complete overview of all required dependencies, and see which ones are already installed +and which ones aren't yet, you can use **`eb --dry-run`**. + +Since `--dry-run` produces rather verbose output including the full path to each easyconfig file, +there is a more concise equivalent available as well: `eb --dry-run-short`, which is equivalent with **`eb -D`**. + +For example, to check which of the required dependencies for `SAMtools-1.11-GCC-10.2.0.eb` are already installed: + +```shell +$ eb SAMtools-1.11-GCC-10.2.0.eb -D +== temporary log file in case of crash /tmp/eb-x4qofiph/easybuild-ehhi9fb1.log +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +Dry run: printing build status of easyconfigs and dependencies +CFGS=/home/example/.local/easybuild/easyconfigs + ... + * [x] $CFGS/b/bzip2/bzip2-1.0.8-GCCcore-10.2.0.eb (module: bzip2/1.0.8-GCCcore-10.2.0) + * [x] $CFGS/x/XZ/XZ-5.2.5-GCCcore-10.2.0.eb (module: XZ/5.2.5-GCCcore-10.2.0) + * [x] $CFGS/c/cURL/cURL-7.72.0-GCCcore-10.2.0.eb (module: cURL/7.72.0-GCCcore-10.2.0) + * [x] $CFGS/g/GCC/GCC-10.2.0.eb (module: GCC/10.2.0) + * [x] $CFGS/n/ncurses/ncurses-6.2-GCCcore-10.2.0.eb (module: ncurses/6.2-GCCcore-10.2.0) + * [ ] $CFGS/s/SAMtools/SAMtools-1.11-GCC-10.2.0.eb (module: SAMtools/1.11-GCC-10.2.0) +``` + +(We've trimmed the output a bit here, for the sake of brevity.) + +This output tells us that all dependencies required by `SAMtools-1.11-GCC-10.2.0.eb` are already installed, +since they are all marked with `[x]`, whereas the easyconfig for `SAMtools` itself is not installed yet, +as indicated by lack of an `x` in `[ ]`. + +--- + +### Missing dependencies + +If you are only interested in which dependencies are still *missing*, +you can consult the output of **`eb --missing`**, or the equivalent **`eb -M`**. + +For example, let's see which dependencies are missing in order to get version 3.1.0 of the h5py +Python package installed using the `2020b` version of the `foss` toolchain: + +```shell +$ eb h5py-3.1.0-foss-2020b.eb -M + +2 out of 61 required modules missing: + +* pkgconfig/1.5.1-GCCcore-10.2.0-python (pkgconfig-1.5.1-GCCcore-10.2.0-python.eb) +* h5py/3.1.0-foss-2020b (h5py-3.1.0-foss-2020b.eb) +``` + +That should be pretty self-explanatory: out of the 63 required dependencies (which includes the `foss` toolchain +and everything needed to install it), only 2 dependencies are missing. Great! + +--- + +## Inspecting install procedures + +Finally, before actually installing something you may want to assess *how* exactly EasyBuild is going +to install the software. + +Perhaps you don't trust EasyBuild yet (you will eventually though, hopefully), +or maybe you just want to double check that you have made the right choice before going through with +the actual installation. + +Using **`eb --extended-dry-run`**, or just **`eb -x`** for short, +you can get a **detailed overview of the installation procedure that would be performed by EasyBuild**, +**in a matter of seconds**. + +By means of example, let's inspect some parts of the installation procedure for `Boost-1.74.0-GCC-10.2.0.eb`: + +```shell +$ eb Boost-1.74.0-GCC-10.2.0.eb -x +... + +preparing... [DRY RUN] + +[prepare_step method] +Defining build environment, based on toolchain (options) and specified dependencies... + +Loading toolchain module... + +module load GCC/10.2.0 + +Loading modules for dependencies... + +module load bzip2/1.0.8-GCCcore-10.2.0 +module load zlib/1.2.11-GCCcore-10.2.0 +module load XZ/5.2.5-GCCcore-10.2.0 + +... + +Defining build environment... + + ... + export CXX='mpicxx' + export CXXFLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno -fPIC' + ... + +configuring... [DRY RUN] + +[configure_step method] + running command "./bootstrap.sh --with-toolset=gcc --prefix=/tmp/example/Boost/1.74.0/GCC-10.2.0/obj --without-libraries=python,mpi" + (in /tmp/example/build/Boost/1.74.0/GCC-10.2.0/Boost-1.74.0) + +... + +[sanity_check_step method] +Sanity check paths - file ['files'] + * lib/libboost_system.so + * lib/libboost_thread-mt-x64.so +Sanity check paths - (non-empty) directory ['dirs'] + * include/boost +Sanity check commands + (none) + +... +``` + +We've obviously trimmed the generated output a bit, but it should be sufficient. + +An overview of the installation procedure is shown, following the installation steps as they would be +performed by EasyBuild. The output above shows: + +* how the build environment will be set up during the `prepare` step, by loading the module for both the + toolchains and the dependencies, and defining a set of environment variables like `$CXX`, `$CXXFLAGS`, etc. +* which command will be executed during the configuration step, and in which directory; +* the list of files and directories that will be checked during the sanity check step; + +If you were concerned about EasyBuild being too much of a black box, that is hopefully resolved now. + +!!! note + It is important to highlight here that the reported installation procedure *may* not be 100% correct, + since the [easyblock](../introduction/#easyblocks) can change its mind based on the output of shell commands + that were executed, or based on the contents of a file that was generated during the installation. + Since all "actions" that would be performed during the installation are actually skipped when using `eb -x`, + the reported installation procedure could be partially incorrect. + + In addition, the easyblock may trip over the fact that the installation procedure is not actually being + executed, which sometimes leads to an unexpected error. These situations are handled gracefully by `eb -x` + though, which will report any errors that occurred and then continue anyway with inspecting the remainder of the + installation procedure. Although this obviously limits the value of the generated output, + it doesn't make it completely useless. + +--- + +## Installing software + +You should now be able to make an informed decision on which easyconfig file you should use to +make EasyBuild install the software you require. + +As mentioned before, installing an easyconfig is as simple as passing it to the `eb` command. + +So, let's try to install SAMtools version 1.11: + +```shell +$ eb SAMtools-1.11-GCC-10.2.0.eb +== temporary log file in case of crash /tmp/eb-zh7_fyre/easybuild-4q_lo57b.log +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +== processing EasyBuild easyconfig /home/example/.local/easybuild/easyconfigs/s/SAMtools/SAMtools-1.11-GCC-10.2.0.eb +== building and installing SAMtools/1.11-GCC-10.2.0... +== fetching files... +== creating build dir, resetting environment... +== unpacking... +== patching... +== preparing... +== configuring... +== building... +== testing... +== installing... +== taking care of extensions... +== restore after iterating... +== postprocessing... +== sanity checking... +== cleaning up... +== creating module... +== permissions... +== packaging... +== COMPLETED: Installation ended successfully (took 17 sec) +== Results of the build can be found in the log file(s) /home/example/easybuild/software/SAMtools/1.11-GCC-10.2.0/easybuild/easybuild-SAMtools-1.11-20210309.105601.log +== Build succeeded for 1 out of 1 +== Temporary log file(s) /tmp/eb-zh7_fyre/easybuild-4q_lo57b.log* have been removed. +== Temporary directory /tmp/eb-zh7_fyre has been removed. +``` + +That was... easy. Is that really all there is to it? Well, almost... + +### Enabling dependency resolution + +The SAMtools installation worked like a charm, but remember that all required dependencies were already +available (see [above](#dry-run)). + +If we try this with the `BCFtools-1.11-GCC-10.2.0.eb`, for which the required `GSL` and `HTSlib` dependencies are not available yet, it's less successful: + +```shell +$ eb BCFtools-1.11-GCC-10.2.0.eb -M + +3 out of 23 required modules missing: + +* GSL/2.6-GCC-10.2.0 (GSL-2.6-GCC-10.2.0.eb) +* HTSlib/1.11-GCC-10.2.0 (HTSlib-1.11-GCC-10.2.0.eb) +* BCFtools/1.11-GCC-10.2.0 (BCFtools-1.11-GCC-10.2.0.eb) +``` + +```shell +$ eb BCFtools-1.11-GCC-10.2.0.eb +... +== preparing... +== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/build/BCFtools/1.11/GCC-10.2.0): build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0 (took 2 sec) +== Results of the build can be found in the log file(s) /tmp/eb-3v1dfvnk/easybuild-BCFtools-1.11-20210308.195024.FlxkH.log +ERROR: Build of /home/example/.local/easybuild/easyconfigs/b/BCFtools/BCFtools-1.11-GCC-10.2.0.eb failed (err: 'build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0') +``` + +Oh my, what's this all about? + +If we filter the output a bit and focus on the actual error, the problem is clear: + +``` +Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0 +``` + +The required dependencies `HTSlib/1.11-GCC-10.2.0` and `GSL/2.6-GCC-10.2.0` are not installed yet, +and EasyBuild does not automatically install missing dependencies unless it is told to do so. + +It helpfully suggests to use the `--robot` command line option, so let's try that: + +```shell +$ eb BCFtools-1.11-GCC-10.2.0.eb --robot +... +== resolving dependencies ... +... +== building and installing HTSlib/1.11-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 13 sec) +... +== building and installing GSL/2.6-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 1 min 10 sec) +... +== building and installing BCFtools/1.11-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 8 sec) +... +== Build succeeded for 3 out of 3 +``` + +With dependency resolution enabled the `HTSlib` and `GSL` modules get installed first, +before EasyBuild proceeds with installing `BCFtools`. Great! + +--- + +### Trace output + +As you may have noticed if you tried the previous example hands-on, +some installations take a while. +An installation can be spending quite a bit of time during the build step, but what is actually going on there? + +To provide some more feedback as the installation progresses, you can enable the "`trace`" configuration setting. +Let's do this by defining the `$EASYBUILD_TRACE` environment variable, just to avoid having to type `--trace` +over and over again. + +We will redo the installation of `BCFtools-1.11-GCC-10.2.0.eb` by passing the `--rebuild` +option to the `eb` command (try yourself what happens if you don't use the `--rebuild` option!): + +```shell +$ export EASYBUILD_TRACE=1 +$ eb BCFtools-1.11-GCC-10.2.0.eb --rebuild +... +== configuring... + >> running command: + [started at: 2021-03-08 19:54:53] + [working dir: /tmp/example/build/BCFtools/1.11/GCC-10.2.0/bcftools-1.11] + [output logged in /tmp/eb-9u_ac0nv/easybuild-run_cmd-17m_he2x.log] + ./configure --prefix=/home/example/easybuild/software/BCFtools/1.11-GCC-10.2.0 --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --with-htslib=$EBROOTHTSLIB --enable-libgsl +== building... + >> running command: + [started at: 2021-03-08 19:54:54] + [working dir: /tmp/example/BCFtools/1.11/GCC-10.2.0/bcftools-1.11] + [output logged in /tmp/example/eb-9u_ac0nv/easybuild-run_cmd-bhkgjxi7.log] + make -j 8 + >> command completed: exit 0, ran in 00h00m03s +``` + +That's a bit more comforting to stare at... + +During the *configure* step, the `./configure` command is run with option to +enable support for leveraging `HTSlib` and `GSL`. + +During the *build* step, the software is actually being compiled +by running the `make` command. EasyBuild automatically uses the available cores on the system (in this case 8). + +We even get a pointer to a log file that contains the output of the command being run, +so we can use `tail -f` to see in detail how it progresses. + +Once the `make` command completes, we get a message that the command completed with a exit code 0 +(implying success), and that it took 3 seconds to run. That's good to know. + +Later during the installation, we now also see this output during the sanity check step: + +``` +== sanity checking... + >> file 'bin/bcftools' found: OK + >> file 'bin/plot-vcfstats' found: OK + >> file 'bin/vcfutils.pl' found: OK + >> (non-empty) directory 'libexec/bcftools' found: OK +``` + +Thanks to enabling trace mode, EasyBuild tells us which files & directories it is checking for +in the installation, before declaring it a success. Nice! + +The extra output you get when trace mode is enabled is concise and hence not overwhelming, +while it gives a better insight into what is going on during the installation. +It may also help to spot unexpected actions being taken during the installation early on, +so you can interrupt the installation before it completes, if deemed necessary. + +--- + +## Using installed software + +So far, we have already installed 4 different software packages (SAMtools, HTSlib, GSL, and BCFtools); +we even installed BCFtools twice! + +A lot was going on underneath the covers: locating and unpacking +the source tarballs, setting up the build environment, configuring the build, compiling, +creating and populating the installation directory, performing a quick sanity check on the installation, +cleaning things up, and finally generated the environment module file corresponding to the installation. + +That's great, but how do we now actually *use* these installations? + +This is where the generated module files come into play: they form the access portal to the software +installations, and we'll use the ubiquitous `module` command to digest them. + +First, we need to make sure that the modules tool is aware of where the module files for +our installations are located. If you're unsure where EasyBuild is installing stuff at this point, +check the output of `eb --show-config`; the value of the `installpath` configuration setting is what we are interested in now: + +```shell +$ eb --show-config +... +installpath (E) = /home/example/easybuild +... +repositorypath (E) = /home/example/easybuild/ebfiles_repo +... +sourcepath (E) = /home/example/easybuild/sources +... +``` + +So, what's in this directory? + +```shell +$ ls -l $HOME/easybuild +total 16 +drwxrwxr-x 5 example example 4096 Jun 10 20:11 ebfiles_repo +drwxrwxr-x 5 example example 4096 Jun 10 20:10 modules +drwxrwxr-x 6 example example 4096 Jun 10 20:10 software +drwxrwxr-x 6 example example 4096 Jun 10 20:10 sources +``` + +The `ebfiles_repo` and `sources` directories correspond to the `repositorypath` and `sourcepath` configuration +settings, respectively. The `modules` and `software` directories are what we need now. + +The `modules` subdirectory consists of multiple subdirectories: + +```shell +$ ls $HOME/easybuild/modules +all bio devel numlib tools +``` + +Directories like `bio` and `numlib` correspond to different software categories, +and contain symbolic links to the module files in the `all` directory, +which contains all actual module files for software installed in this EasyBuild installation path. +We'll ignore these separate category directories for now. + +Let's inform the modules tool about the existence of these module files using `"module use"`: + +```shell +module use $HOME/easybuild/modules/all +``` + +This command does little more that updating the `$MODULEPATH` environment variable, +which contains a list of paths that the modules tool should consider when looking for module files. + +Now the modules tool should be aware of our brand new installations: + +```shell +$ module avail + +---------------------- /home/example/easybuild/modules/all ----------------------- + BCFtools/1.11-GCC-10.2.0 GSL/2.6-GCC-10.2.0 SAMtools/1.11-GCC-10.2.0 + EasyBuild/4.3.3 HTSlib/1.11-GCC-10.2.0 bzip2/1.0.6 + +---------------------------- /easybuild/modules/all ----------------------------- + ... +``` + +This output shows both the modules for our own installations as well as the "central" installations in `/easybuild` (which we omitted above for brevity). + +Now we can load these modules and start using these software installations. + +Let's test this for BCFtools. In our current environment, the `bcftools` command is not available yet: + +```shell +$ module list +No modules loaded + +$ bcftools +-bash: bcftools: command not found +``` + +Loading the module for BCFtools changes that: + +```shell +$ module load BCFtools/1.11-GCC-10.2.0 + +$ module list +Currently Loaded Modules: + 1) GCCcore/10.2.0 6) XZ/5.2.5-GCCcore-10.2.0 + 2) zlib/1.2.11-GCCcore-10.2.0 7) cURL/7.72.0-GCCcore-10.2.0 + 3) binutils/2.35-GCCcore-10.2.0 8) HTSlib/1.11-GCC-10.2.0 + 4) GCC/10.2.0 9) GSL/2.6-GCC-10.2.0 + 5) bzip2/1.0.8-GCCcore-10.2.0 10) BCFtools/1.11-GCC-10.2.0 + +$ bcftools --version +bcftools 1.11 +Using htslib 1.11 +... +``` + +Note that the modules for the required dependencies, including the compiler toolchain (which provides runtime libraries +like `libstdc++.so`), are loaded automatically. The "`module load`" command changes the active environment, +by updating environment variables like `$PATH` for example, to make the software available for use. + +##### Resetting your environment + +To restore your environment to a pristine state in which no modules are loaded, you can either +unload the loaded modules one by one using "`module unload`", or you can unload all of them at once using +"`module purge`". + +**If you are using an EasyBuild installation provided by a module, +don't forget to load the `EasyBuild` module again after running "`module purge`".** + +--- + +## Stacking software + +Maybe you have overlooked how the software we are playing around with was not only installed across multiple +different installation directories per software, we are also "stacking" our own installations (in `$HOME/easybuild`) +on top of installations that are provided in a totally different location (`/easybuild`). + +**EasyBuild doesn't care *where* software is installed: as long as the module file that provides access to it +is available, it is happy to pick it up and use it when required.** + +This implies that end users of an HPC system can easily install their +own small software stack on top of what is provided centrally by the HPC support team, +for example. They can even +"replace" a central software installation for their purposes if they need to, since the modules tool will +load the first module file that matches the request being made (there are some caveats with this, but we +won't go into those here). + +--- + +## Exercises + +***Guidelines*** + +Do yourself a favor: don't peek at the solution until you have made an attempt to solve the exercise yourself! + +Please do not spoil solutions to others before they have been discussed by the tutorial organisers. + +The exercises are based on the easyconfig files included with EasyBuild 4.3.3. + +--- + +***Exercise U.0**** - Making installed software available* + +Before working on the exercises for this part of the tutorial, +make sure that the software that is already installed in the prepared environment is available. + +We will assume that you have a small software stack installed using the `2020b` version of the `foss` toolchain. + +**Tip:** execute a "`module use`" command, and verify with "`module avail`" that a bunch of software modules +are available for loading. + +??? success "(click to show solution)" + + Use the following command to make the modules for the software available that is pre-installed + in the prepared environment: + ```shell + module use /easybuild/modules/all + ``` + + If software is installed in a different location than `/easybuild/` in your environment, + you should adjust the command accordingly. + + +--- + +***Exercise U.1**** - Searching easyconfigs* + +See if EasyBuild provides any easyconfig files for installing GROMACS version 2020/5. + +??? success "(click to show solution)" + To check for available easyconfig files, we can use `eb --search` or `eb -S`: + ```shell + $ eb -S gromacs-2020.5 + == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + CFGS1=/home/example/.local/easybuild/easyconfigs/g/GROMACS + * $CFGS1/GROMACS-2020.5-fosscuda-2020a-Python-3.8.2.eb + * $CFGS1/GROMACS-2020.5_fix_threads_gpu_Gmxapitests.patch + ``` + This actually shows one easyconfig file but also a patch file. We can also search specifically + for GROMACS 2020.5 in the `foss` and `fosscuda` toolchains using + ```shell + $ eb -S gromacs-2020.5-foss + == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + CFGS1=/home/example/.local/easybuild/easyconfigs/g/GROMACS + * $CFGS1/GROMACS-2020.5-fosscuda-2020a-Python-3.8.2.eb + ``` + and now we find a single easyconfig file. + +--- + +***Exercise U.2**** - Checking dependencies* + +Check which dependencies are missing to install QuantumESPRESSO version 6.6 with the `2020b` version of the `foss` toolchain. + +??? success "(click to show solution)" + First, we need to determine the name of the easyconfig file for QuantumESPRESSO version 6.6: + ```shell + $ eb -S 'QuantumESPRESSO-6.6.*foss-2020b' + == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + CFGS1=/home/example/.local/easybuild/easyconfigs/q/QuantumESPRESSO + * $CFGS1/QuantumESPRESSO-6.6-foss-2020b.eb + ``` + To determine which dependencies are missing to install this QuantumESPRESSO easyconfig file, we can use `--missing`: + ```shell + $ eb QuantumESPRESSO-6.6-foss-2020b.eb --missing + + 3 out of 58 required modules missing: + + * libxc/4.3.4-GCC-10.2.0 (libxc-4.3.4-GCC-10.2.0.eb) + * ELPA/2020.11.001-foss-2020b (ELPA-2020.11.001-foss-2020b.eb) + * QuantumESPRESSO/6.6-foss-2020b (QuantumESPRESSO-6.6-foss-2020b.eb) + ``` + (some nonessential output removed). + +--- + +***Exercise U.3**** - Performing a dry run* + +Figure out which command EasyBuild would use to compile +the software provided by the `Bowtie2-2.4.2-GCC-9.3.0.eb` easyconfig file, +without actually installing `Bowtie2`. + +Also, which binaries will EasyBuild check for to sanity check the installation? + +??? success "(click to show solution)" + To inspect the installation procedure, we can use `eb -x Bowtie2-2.4.2-GCC-9.3.0.eb`. + + The output for the build step shows the actual compilation command that would be performed (`make ...`): + + ```shell + [build_step method] + >> running command: + [started at: 2021-03-08 20:15:08] + [working dir: /local_scratch/hkenneth/eb-1wodfohg/__ROOT__/local_scratch/hkenneth/Bowtie2/2.4.2/GCC-9.3.0/Bowtie2-2.4.2] + [output logged in /local_scratch/hkenneth/eb-1wodfohg/easybuild-run_cmd-haojzisn.log] + make -j 48 CC="gcc" CPP="g++" CXX="g++" RELEASE_FLAGS="-O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -std=gnu++98" + (in /local_scratch/hkenneth/Bowtie2/2.4.2/GCC-9.3.0/Bowtie2-2.4.2) + ``` + + If the output you get is less detailed, you may not have set `export EASYBUILD_TRACE=1`. + + The output for the sanity check step shows which binaries are expected to be installed: + ``` + [sanity_check_step method] + Sanity check paths - file ['files'] + * bin/bowtie2 + * bin/bowtie2-align-l + * bin/bowtie2-align-s + * bin/bowtie2-build + * bin/bowtie2-build-l + * bin/bowtie2-build-s + * bin/bowtie2-inspect + * bin/bowtie2-inspect-l + * bin/bowtie2-inspect-s + ``` + +--- + +***Exercise U.4**** - Installing software* + +Install version 3.1.0 of the `h5py` Python package and all missing dependencies, +using the `foss/2020b` toolchain, into `/tmp/$USER/easybuild`, +while leveraging the already installed software available from `/easybuild`. + +Enable trace output so you can see which parts of the installation take a while. + +??? success "(click to show solution)" + First, determine the easyconfig file we can use for this: + ```shell + $ eb -S 'h5py-3.1.0.*foss-2020b' + CFGS1=/home/example/.local/easybuild/easyconfigs/h/h5py + * $CFGS1/h5py-3.1.0-foss-2020b.eb + ``` + + Make sure the pre-install software in `/easybuild/` is available: + ```shell + module use /easybuild/modules/all + ``` + + Check which dependencies are missing to install this `h5py` easyconfig: + ```shell + $ eb h5py-3.1.0-foss-2020b.eb --missing + + 2 out of 63 required modules missing: + + * pkgconfig/1.5.1-GCCcore-10.2.0-python (pkgconfig-1.5.1-GCCcore-10.2.0-python.eb) + * h5py/3.1.0-foss-2020b (h5py-3.1.0-foss-2020b.eb) + ``` + + Install `h5py` by specifying the easyconfig file and enabling dependency resolution via `--robot`, + while indicating that we want to install the software into `/tmp/$USER/easybuild` using the `--installpath` + option. Also make sure that trace mode is enabled by defining the `$EASYBUILD_TRACE` environment variable. + ```shell + $ export EASYBUILD_TRACE=1 + $ eb h5py-3.1.0-foss-2020b.eb --robot --installpath /tmp/$USER/easybuild + ... + == building and installing pkgconfig/1.5.1-GCCcore-10.2.0-python... + ... + == building and installing h5py/3.1.0-foss-2020b... + ... + == building... + >> running command: + [started at: 2020-06-10 21:47:32] + [working dir: /tmp/example/h5py/3.1.0/foss-2020b/h5py-3.1.0] + [output logged in /tmp/eb-rjjkbqe1/easybuild-run_cmd-d_dkc4iz.log] + python setup.py configure --mpi --hdf5=$EBROOTHDF5 && /easybuild/software/Python/3.8.6-GCCcore-10.2.0/bin/python setup.py build + >> command completed: exit 0, ran in 00h01m27s + ... + == COMPLETED: Installation ended successfully (took 2 min 46 sec) + ... + == Build succeeded for 2 out of 2 + ``` + + The trace output shows that most time is spent in the build command, + which runs both `python setup.py configure` and `python setup.py build`. + +--- + +***Exercise U.5**** - Using installed software* + +Using the `h5py` installation from the previous exercise to create an empty HDF5 file, +using the following Python statements: + +```python +import h5py +f = h5py.File("empty.hdf5", "w") +f.close() +``` + +Check the resulting file using the `h5stat` command. + +??? success "(click to show solution)" + First, we need to make the modules tool aware of the module files that were installed into `/tmp/$USER/easybuild`: + ```shell + module use /tmp/$USER/easybuild/modules/all + ``` + + Then we can check the `h5py` module is available, and load it: + ```shell + $ module avail h5py + ------------ /tmp/example/easybuild/modules/all ------------ + h5py/3.1.0-foss-2020b + ``` + + ```shell + module load h5py/3.1.0-foss-2020b + ``` + + The Python code snippet can be run directly on the command line using "`python -c '...'`", since it's tiny: + ```shell + python -c 'import h5py; f = h5py.File("empty.hdf5", "w"); f.close()' + ``` + Of course you can also copy the Python code snippet in a file named `test_h5py.py`, + and then run it with `python test_h5py.py`. + + Checking with the `h5stat` command shows that the resulting `empty.hdf5` is indeed a valid HDF5 file: + ```shell + $ ls -l empty.hdf5 + -rw-rw-r-- 1 example example 800 Jun 10 21:54 empty.hdf5 + + $ h5stat empty.hdf5 + Filename: empty.hdf5 + File information + # of unique groups: 1 + # of unique datasets: 0 + ... + ``` + +--- + +If you've made it through the hands-on exercises, congratulations! + +If not, don't worry too much about it. We covered a lot of ground here, +and it's a lot to take in at once, take your time... + +Feel free to ask question in the `#tutorial` channel in the [EasyBuild +Slack](https://docs.easybuild.io/en/latest/#getting-help), +we're happy to help! + +--- + +*[[next: Part 2: Using EasyBuild]](2_00_part2_using.md)* From 6a59f948934e6697bcf08dde9e7a114082717b62 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 17:33:23 +0100 Subject: [PATCH 066/159] Restructuring continued. --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 10 +++++----- docs/2022-CSC_and_LO/1_03_CPE.md | 2 +- docs/2022-CSC_and_LO/1_05_terminology.md | 4 ++-- docs/2022-CSC_and_LO/1_06_installation.md | 10 ++++++---- 4 files changed, 14 insertions(+), 12 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md index 7b138670..0674be92 100644 --- a/docs/2022-CSC_and_LO/1_00_part1_intro.md +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -5,8 +5,8 @@ - [What is EasyBuild?](1_01_what_is_easybuild.md) - [The Lmod module system](1_02_Lmod.md) - [The HPE Cray Programming Environment](1_03_CPE.md) -- [Terminology](1_04_terminology.md) - -- [Installation](installation.md) *(hands-on)* -- [Configuration](configuration.md) *(hands-on)* -- [Basic usage](basic_usage.md) *(hands-on)* +- [LUMI software stacks](1_04_LUMI_software_stack.md) +- [Terminology](1_05_terminology.md) +- [Installation](1_06_installation.md) +- [Configuration](1_07_configuration.md) +- [Basic usage](1_08_basic_usage.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md index 42eb197e..077fbc36 100644 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -281,4 +281,4 @@ redfine ``LD_LIBRARY_PATH``. --- -*[[next: Terminology]](1_04_terminology.md)* +*[[next: LUMI software stacks]](1_04_LUMI_software_stack.md)* diff --git a/docs/2022-CSC_and_LO/1_05_terminology.md b/docs/2022-CSC_and_LO/1_05_terminology.md index 9c643579..8e05012f 100644 --- a/docs/2022-CSC_and_LO/1_05_terminology.md +++ b/docs/2022-CSC_and_LO/1_05_terminology.md @@ -1,6 +1,6 @@ # Terminology -*[[back: The HPE Cray Programming Environment]](1_03_CPE.md)* +*[[back: LUMI software stacks]](1_04_LUMI_software_stack.md)* --- @@ -291,4 +291,4 @@ An **easystack file** can be used to specify a collection of software to install --- -*[[next: Installation]](1_05_installation.md)* +*[[next: Installation]](1_06_installation.md)* diff --git a/docs/2022-CSC_and_LO/1_06_installation.md b/docs/2022-CSC_and_LO/1_06_installation.md index 163262ea..6e68b500 100644 --- a/docs/2022-CSC_and_LO/1_06_installation.md +++ b/docs/2022-CSC_and_LO/1_06_installation.md @@ -1,6 +1,6 @@ # Installing EasyBuild -*[[back: Terminology]](1_04_terminology.md)* +*[[back: Terminology]](1_05_terminology.md)* --- @@ -537,7 +537,9 @@ Check that the installation works by running the verification commands outlined --- -**Make sure you have a working EasyBuild installation before proceeding -with the rest of the tutorial!** +**You'll need a working custom EasyBuild installation for some of the +optional generic EasyBuild exercises, but in general you can use the +installation provided by LUMI.** -*[[next: Configuring EasyBuild]](1_06_configuration.md)* + +*[[next: Configuring EasyBuild]](1_07_configuration.md)* From e277aa543f126ee1298305741c093c0878287cde Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 30 Mar 2022 12:07:24 +0200 Subject: [PATCH 067/159] Finished reworking the basic usage section, except for the exercises. --- docs/2022-CSC_and_LO/1_08_basic_usage.md | 644 ++++++++++++++++------- 1 file changed, 442 insertions(+), 202 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_08_basic_usage.md index 5566d376..f5031e48 100644 --- a/docs/2022-CSC_and_LO/1_08_basic_usage.md +++ b/docs/2022-CSC_and_LO/1_08_basic_usage.md @@ -21,7 +21,7 @@ Installing software with EasyBuild is as easy as specifying to the **`eb` comman want to install, and then sitting back to enjoy a coffee or tea (or whatever beverage you prefer). This is typically done by **specifying the name of one or more easyconfig files**, often combined -with the `--robot` option to let EasyBuild also install missing dependencies. +with the `--robot` (or `-r`) option to let EasyBuild also install missing dependencies. It is recommended to first assess the current situation before letting EasyBuild install the software, and to check which **dependencies** are already installed and which are still missing. In addition, @@ -53,7 +53,7 @@ ERROR: Can't find path /tmp/does_not_exist.eb When only the *name* of an easyconfig file is specified, EasyBuild will automatically try and locate it. First, it will consider the *current directory*. If no file with the specified name is found there, -EasyBuild will search for the easyconfig file in the [robot search path](../configuration/#robot-search-path). +EasyBuild will search for the easyconfig file in the [robot search path](../1_07_configuration/#robot-search-path). If the path to an existing *directory* is provided, EasyBuild will walk through the entire directory (including all subdirectories), retain all files of which the name ends with `.eb`, and (try to) use these @@ -75,12 +75,12 @@ Or, visually represented: ```shell example1.eb example2.eb -some_deps/ -|-- dep1.eb -|-- dep2.eb -|-- list.txt -|-- more_deps/ - |-- dep3.eb +some_deps +├── deb2.eb +├── dep1.eb +├── list.txt +└── more_deps + └── dep3.eb ``` In this context, we run the following EasyBuild command from our home directory: @@ -118,7 +118,7 @@ corresponding to `--.eb`, where: * `` represents the software version; * `` represents the toolchain used in the easyconfig file, which consists of the toolchain name and version separated with a dash (`-`), and which is omitted (including the preceding `-`) when the - [`system` toolchain](../introduction#system-toolchain) is used; + [`system` toolchain](../1_05_terminology#system-toolchain) is used; * `` represents the value of the `versionsuffix` easyconfig parameter, which is sometimes used to distinguish multiple variants of particular software installations (and is empty by default); @@ -131,7 +131,7 @@ You will frequently need to determine the exact name of an easyconfig file you w or just check which easyconfigs are available for a given software package. This can be done by searching for easyconfigs using **`eb --search`** or **`eb -S`**. -By default all directories listed in the [robot search path](../configuration#robot-search-path) will be +By default all directories listed in the [robot search path](../1_07_configuration#robot-search-path) will be searched. If you want to search in additional directories without changing the robot search path, you can use the `search-paths` configuration setting, or you can change the robot search path via either the `robot` or `robot-paths` configuration options. @@ -202,41 +202,91 @@ Since easyconfig files are simple text files (in Python syntax), you could use t or your favorite text editor (`vim`, what else). To avoid that you need to locate the easyconfig file first and copy-paste the full path to it, you can use **`eb --show-ec`**. +!!! Hint + To follow the examples below on LUMI, load ``LUMI/21.12`` and ``EasyBuild-user`` + (though results may differ or the examples not work anymore as the software installation + on LUMI evolves). + For example, let's inspect the contents of the `bzip2-1.0.6.eb` easyconfig file: ```shell -$ eb --show-ec bzip2-1.0.6.eb -== temporary log file in case of crash /tmp/eb-jnpzclhl/easybuild-e37cbrj1.log -== Contents of /home/example/.local/easybuild/easyconfigs/b/bzip2/bzip2-1.0.6.eb: -name = 'bzip2' -version = '1.0.6' - -homepage = 'https://sourceware.org/bzip2' -description = """bzip2 is a freely available, patent free, high-quality data compressor. It typically -compresses files to within 10% to 15% of the best available techniques (the PPM family of statistical -compressors), whilst being around twice as fast at compression and six times faster at decompression.""" - -toolchain = SYSTEM +$ eb --show-ec bzip2-1.0.8-cpeCray-21.12.eb +== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-53o823qb/easybuild-xn6nmt61.log +== Contents of /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeCray-21.12.eb: +# Contributed by Kurt Lust, LUMI project & UAntwerpen + +local_bzip2_version = '1.0.8' # http://www.bzip.org/downloads.html + +name = 'bzip2' +version = local_bzip2_version + +homepage = 'https://www.sourceware.org/bzip2/' + +whatis = [ + 'Description: bzip2 is a freely available, patent free, high-quality data compressor.', + 'The module contains both executables and libraries.' + 'Keywords: BZ2', +] + +description = """ +bzip2 is a freely available, patent free, high-quality data compressor. It +typically compresses files to within 10% to 15% of the best available techniques +(the PPM family of statistical compressors), whilst being around twice as fast +at compression and six times faster at decompression. It is based on the +Burrows-Wheeler block-sorting text compression algorithm and Huffman coding. +""" + +usage = """ +Check the man pages for the available commands or the web-based documentation for the +library functions. +""" + +docurls = [ + 'Web-based documentation: http://www.bzip.org/docs.html', + 'Man pages available for bzcmp, bzdiff, bzegrep, bzfgrep, bzgrep, bzip2, bunzip2, bzless and bzmore', +] + +toolchain = {'name': 'cpeCray', 'version': '21.12'} toolchainopts = {'pic': True} -source_urls = ['https://sourceware.org/pub/bzip2/'] -sources = [SOURCE_TAR_GZ] -patches = ['bzip2-%(version)s-pkgconfig.patch'] +source_urls = ['https://sourceware.org/pub/%(name)s/'] +sources = [SOURCE_TAR_GZ] +patches = ['bzip2-%(version)s-pkgconfig-manpath.patch'] checksums = [ - 'a2848f34fcd5d6cf47def00461fcb528a0484d8edef8208d6d2e2909dc61d9cd', # bzip2-1.0.6.tar.gz - '5a823e820b332eca3684416894f58edc125ac3dace9f46e62f98e45362aa8a6d', # bzip2-1.0.6-pkgconfig.patch + 'ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269', # bzip2-1.0.8.tar.gz + 'de11269dc6e4917023af4cee9ff83b204953ad9cde561dbc9d6fc70d9b9578e3', # bzip2-1.0.8-pkgconfig-manpath.patch ] -buildopts = "CC=gcc CFLAGS='-Wall -Winline -O3 -fPIC -g $(BIGFILES)'" +builddependencies = [ # Create a reproducible build environment. + ('buildtools', '%(toolchain_version)s', '', True), +] -# building of shared libraries doesn't work on OS X (where 'gcc' is actually Clang...) -with_shared_libs = OS_TYPE == 'Linux' +local_bzip2_major_minor = '.'.join(local_bzip2_version.split('.')[:2]) + +sanity_check_paths = { + 'files': [ 'lib/pkgconfig/bzip2.pc', 'lib/libbz2.a', 'lib/libbz2.%s' % SHLIB_EXT, + 'lib/libbz2.%s.%s' % (SHLIB_EXT, local_bzip2_major_minor), + 'lib/libbz2.%s.%s' % (SHLIB_EXT, local_bzip2_version), + 'include/bzlib.h' ] + + [ 'bin/b%s' % x for x in ['unzip2', 'zcat', 'zdiff', 'zgrep', 'zip2', 'zip2recover', 'zmore'] ] + + [ 'share/man/man1/bz%s.1' % x for x in ['cmp', 'diff', 'egrep', 'fgrep', 'grep', 'ip2', 'less', 'more'] ], + 'dirs': [] +} + +sanity_check_commands = [ + 'bzip2 --help', + 'pkg-config --libs bzip2', +] moduleclass = 'tools' -== Temporary log file(s) /tmp/eb-jnpzclhl/easybuild-e37cbrj1.log* have been removed. -== Temporary directory /tmp/eb-jnpzclhl has been removed. +== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-53o823qb/easybuild-xn6nmt61.log* have been removed. +== Temporary directory /run/user/10012026/easybuild/tmp/eb-53o823qb has been removed. ``` +The output may actually be longer for an easyconfig file that is already installed on the system +as a new easyconfig file is generated in the repository with some information about the installation +added to it and as on LUMI these are at the front of the robot search path to ensure that the system +finds the right easyconfig file matching with a module on the system. We'll get back to what all of this means later... @@ -272,25 +322,30 @@ there is a more concise equivalent available as well: `eb --dry-run-short`, whic For example, to check which of the required dependencies for `SAMtools-1.11-GCC-10.2.0.eb` are already installed: ```shell -$ eb SAMtools-1.11-GCC-10.2.0.eb -D -== temporary log file in case of crash /tmp/eb-x4qofiph/easybuild-ehhi9fb1.log -== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +$ eb SAMtools-1.14-cpeGNU-21.12.eb -D +== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-oo0lj9lq/easybuild-2cyomy8v.log Dry run: printing build status of easyconfigs and dependencies -CFGS=/home/example/.local/easybuild/easyconfigs - ... - * [x] $CFGS/b/bzip2/bzip2-1.0.8-GCCcore-10.2.0.eb (module: bzip2/1.0.8-GCCcore-10.2.0) - * [x] $CFGS/x/XZ/XZ-5.2.5-GCCcore-10.2.0.eb (module: XZ/5.2.5-GCCcore-10.2.0) - * [x] $CFGS/c/cURL/cURL-7.72.0-GCCcore-10.2.0.eb (module: cURL/7.72.0-GCCcore-10.2.0) - * [x] $CFGS/g/GCC/GCC-10.2.0.eb (module: GCC/10.2.0) - * [x] $CFGS/n/ncurses/ncurses-6.2-GCCcore-10.2.0.eb (module: ncurses/6.2-GCCcore-10.2.0) - * [ ] $CFGS/s/SAMtools/SAMtools-1.11-GCC-10.2.0.eb (module: SAMtools/1.11-GCC-10.2.0) +CFGS=/appl/lumi + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeGNU/cpeGNU-21.12.eb (module: cpeGNU/21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ncurses/ncurses-6.2-cpeGNU-21.12.eb (module: ncurses/6.2-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/zlib/zlib-1.2.11-cpeGNU-21.12.eb (module: zlib/1.2.11-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeGNU-21.12.eb (module: bzip2/1.0.8-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12-minimal.eb (module: gettext/0.21-cpeGNU-21.12-minimal) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/XZ/XZ-5.2.5-cpeGNU-21.12.eb (module: XZ/5.2.5-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/Brotli/Brotli-1.0.9-cpeGNU-21.12.eb (module: Brotli/1.0.9-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cURL/cURL-7.78.0-cpeGNU-21.12.eb (module: cURL/7.78.0-cpeGNU-21.12) + * [ ] $CFGS/LUMI-EasyBuild-contrib/easybuild/easyconfigs/h/HTSlib/HTSlib-1.14-cpeGNU-21.12.eb (module: HTSlib/1.14-cpeGNU-21.12) + * [ ] $CFGS/LUMI-EasyBuild-contrib/easybuild/easyconfigs/s/SAMtools/SAMtools-1.14-cpeGNU-21.12.eb (module: SAMtools/1.14-cpeGNU-21.12) +== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-oo0lj9lq/easybuild-2cyomy8v.log* have been removed. +== Temporary directory /run/user/10012026/easybuild/tmp/eb-oo0lj9lq has been removed. ``` -(We've trimmed the output a bit here, for the sake of brevity.) +This output tells you that most of the dependencies required by ``SAMtools-1.14-cpeGNU-21.12.eb`` are +already installed, since they are marked with ``[x]``. However, the easyconfig files for +``HTSLib-1.14-cpeGNU-21.12.eb`` and SAMtools itself are not installed yet, denoted by the +lack of an ``x`` in ``[ ]``. -This output tells us that all dependencies required by `SAMtools-1.11-GCC-10.2.0.eb` are already installed, -since they are all marked with `[x]`, whereas the easyconfig for `SAMtools` itself is not installed yet, -as indicated by lack of an `x` in `[ ]`. --- @@ -299,20 +354,19 @@ as indicated by lack of an `x` in `[ ]`. If you are only interested in which dependencies are still *missing*, you can consult the output of **`eb --missing`**, or the equivalent **`eb -M`**. -For example, let's see which dependencies are missing in order to get version 3.1.0 of the h5py -Python package installed using the `2020b` version of the `foss` toolchain: +For example, for the SAMtools easyconfig file used in the previous example we get (with +some lines removed from the output): ```shell -$ eb h5py-3.1.0-foss-2020b.eb -M - -2 out of 61 required modules missing: +$ eb SAMtools-1.14-cpeGNU-21.12.eb -M +2 out of 11 required modules missing: -* pkgconfig/1.5.1-GCCcore-10.2.0-python (pkgconfig-1.5.1-GCCcore-10.2.0-python.eb) -* h5py/3.1.0-foss-2020b (h5py-3.1.0-foss-2020b.eb) +* HTSlib/1.14-cpeGNU-21.12 (HTSlib-1.14-cpeGNU-21.12.eb) +* SAMtools/1.14-cpeGNU-21.12 (SAMtools-1.14-cpeGNU-21.12.eb) ``` -That should be pretty self-explanatory: out of the 63 required dependencies (which includes the `foss` toolchain -and everything needed to install it), only 2 dependencies are missing. Great! +That should be pretty self-explanatory: out of the 113 required dependencies (which includes the `cpeGNU` toolchain +and everything needed to install it), only 2 dependencies (including SAMtools itself) are missing. Great! --- @@ -329,52 +383,78 @@ Using **`eb --extended-dry-run`**, or just **`eb -x`** for short, you can get a **detailed overview of the installation procedure that would be performed by EasyBuild**, **in a matter of seconds**. -By means of example, let's inspect some parts of the installation procedure for `Boost-1.74.0-GCC-10.2.0.eb`: +By means of example, let's inspect some parts of the installation procedure for ``HTSLib-1.14-cpeGNU-21.12.eb``: ```shell -$ eb Boost-1.74.0-GCC-10.2.0.eb -x +$ eb HTSlib-1.14-cpeGNU-21.12.eb -x ... -preparing... [DRY RUN] +*** DRY RUN using 'ConfigureMake' easyblock (easybuild.easyblocks.generic.configuremake @ /appl/lumi/SW/LUMI-21.12/common/EB/EasyBuild/4.5.3/lib/python3.6/site-packages/easybuild/easyblocks/generic/configuremake.py) *** + +== building and installing HTSlib/1.14-cpeGNU-21.12... +fetching files... [DRY RUN] + +... [prepare_step method] Defining build environment, based on toolchain (options) and specified dependencies... Loading toolchain module... -module load GCC/10.2.0 +module load cpeGNU/21.12 Loading modules for dependencies... -module load bzip2/1.0.8-GCCcore-10.2.0 -module load zlib/1.2.11-GCCcore-10.2.0 -module load XZ/5.2.5-GCCcore-10.2.0 +module load buildtools/21.12 +module load zlib/1.2.11-cpeGNU-21.12 +module load bzip2/1.0.8-cpeGNU-21.12 +module load XZ/5.2.5-cpeGNU-21.12 +module load cURL/7.78.0-cpeGNU-21.12 ... Defining build environment... - ... - export CXX='mpicxx' - export CXXFLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno -fPIC' - ... +... + + export CC='cc' + export CFLAGS='-O2 -ftree-vectorize -fno-math-errno' + +... configuring... [DRY RUN] [configure_step method] - running command "./bootstrap.sh --with-toolset=gcc --prefix=/tmp/example/Boost/1.74.0/GCC-10.2.0/obj --without-libraries=python,mpi" - (in /tmp/example/build/Boost/1.74.0/GCC-10.2.0/Boost-1.74.0) + running command "./configure --prefix=/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/HTSlib/1.14-cpeGNU-21.12" + (in /run/user/10012026/easybuild/build/HTSlib/1.14/cpeGNU-21.12/HTSlib-1.14) + +building... [DRY RUN] + +[build_step method] + running command "make -j 256" + (in /run/user/10012026/easybuild/build/HTSlib/1.14/cpeGNU-21.12/HTSlib-1.14) + +testing... [DRY RUN] + +[test_step method] + +installing... [DRY RUN] ... +sanity checking... [DRY RUN] + [sanity_check_step method] Sanity check paths - file ['files'] - * lib/libboost_system.so - * lib/libboost_thread-mt-x64.so + * bin/bgzip + * bin/tabix + * lib/libhts.so Sanity check paths - (non-empty) directory ['dirs'] - * include/boost + * include Sanity check commands - (none) + * bgzip --version + * htsfile --version + * tabix --version ... ``` @@ -385,7 +465,7 @@ An overview of the installation procedure is shown, following the installation s performed by EasyBuild. The output above shows: * how the build environment will be set up during the `prepare` step, by loading the module for both the - toolchains and the dependencies, and defining a set of environment variables like `$CXX`, `$CXXFLAGS`, etc. + toolchains and the dependencies, and defining a set of environment variables like `$CC`, `$CFLAGS`, etc. * which command will be executed during the configuration step, and in which directory; * the list of files and directories that will be checked during the sanity check step; @@ -393,7 +473,7 @@ If you were concerned about EasyBuild being too much of a black box, that is hop !!! note It is important to highlight here that the reported installation procedure *may* not be 100% correct, - since the [easyblock](../introduction/#easyblocks) can change its mind based on the output of shell commands + since the [easyblock](1_05_terminology.md#easyblocks) can change its mind based on the output of shell commands that were executed, or based on the contents of a file that was generated during the installation. Since all "actions" that would be performed during the installation are actually skipped when using `eb -x`, the reported installation procedure could be partially incorrect. @@ -413,64 +493,105 @@ make EasyBuild install the software you require. As mentioned before, installing an easyconfig is as simple as passing it to the `eb` command. -So, let's try to install SAMtools version 1.11: +So, let's try to install libdap version 3.20.9 with the cpeGNU/21.12 toolchain. + +Let's first check if it has any dependencies that still need to be installed: ```shell -$ eb SAMtools-1.11-GCC-10.2.0.eb -== temporary log file in case of crash /tmp/eb-zh7_fyre/easybuild-4q_lo57b.log -== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... -== processing EasyBuild easyconfig /home/example/.local/easybuild/easyconfigs/s/SAMtools/SAMtools-1.11-GCC-10.2.0.eb -== building and installing SAMtools/1.11-GCC-10.2.0... +$ eb libdap-3.20.9-cpeGNU-21.12.eb -D +== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-wm_bk3j6/easybuild-puyu_559.log +Dry run: printing build status of easyconfigs and dependencies + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeGNU/cpeGNU-21.12.eb (module: cpeGNU/21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/zlib/zlib-1.2.11-cpeGNU-21.12.eb (module: zlib/1.2.11-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libtirpc/libtirpc-1.3.2-cpeGNU-21.12.eb (module: libtirpc/1.3.2-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/Brotli/Brotli-1.0.9-cpeGNU-21.12.eb (module: Brotli/1.0.9-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ICU/ICU-69.1-cpeGNU-21.12.eb (module: ICU/69.1-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cURL/cURL-7.78.0-cpeGNU-21.12.eb (module: cURL/7.78.0-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeGNU-21.12.eb (module: bzip2/1.0.8-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ncurses/ncurses-6.2-cpeGNU-21.12.eb (module: ncurses/6.2-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libreadline/libreadline-8.1-cpeGNU-21.12.eb (module: libreadline/8.1-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12-minimal.eb (module: gettext/0.21-cpeGNU-21.12-minimal) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/PCRE2/PCRE2-10.37-cpeGNU-21.12.eb (module: PCRE2/10.37-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/XZ/XZ-5.2.5-cpeGNU-21.12.eb (module: XZ/5.2.5-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libxml2/libxml2-2.9.12-cpeGNU-21.12.eb (module: libxml2/2.9.12-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libxslt/libxslt-1.1.34-cpeGNU-21.12.eb (module: libxslt/1.1.34-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12.eb (module: gettext/0.21-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/file/file-5.41-cpeGNU-21.12.eb (module: file/5.41-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/util-linux/util-linux-2.37.1-cpeGNU-21.12.eb (module: util-linux/2.37.1-cpeGNU-21.12) + * [ ] /pfs/lustrep3/users/kurtlust/LUMI/LUMI-EasyBuild-contrib/easybuild/easyconfigs/l/libdap/libdap-3.20.9-cpeGNU-21.12.eb (module: libdap/3.20.9-cpeGNU-21.12) +== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-wm_bk3j6/easybuild-puyu_559.log* have been removed. +== Temporary directory /run/user/10012026/easybuild/tmp/eb-wm_bk3j6 has been removed. +``` + +and now instal the library: + +```shell +$ eb libdap-3.20.9-cpeGNU-21.12.eb +== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-kfphjoi8/easybuild-kcs00ai5.log +== processing EasyBuild easyconfig +/pfs/lustrep3/users/kurtlust/LUMI/LUMI-EasyBuild-contrib/easybuild/easyconfigs/l/libdap/libdap-3.20.9-cpeGNU-21.12.eb +== building and installing libdap/3.20.9-cpeGNU-21.12... == fetching files... +== ... (took 2 secs) == creating build dir, resetting environment... == unpacking... == patching... == preparing... +== ... (took 6 secs) == configuring... +== ... (took 1 min 6 secs) == building... +== ... (took 53 secs) == testing... == installing... +== ... (took 5 secs) == taking care of extensions... == restore after iterating... == postprocessing... == sanity checking... +== ... (took 3 secs) == cleaning up... == creating module... +== ... (took 2 secs) == permissions... == packaging... -== COMPLETED: Installation ended successfully (took 17 sec) -== Results of the build can be found in the log file(s) /home/example/easybuild/software/SAMtools/1.11-GCC-10.2.0/easybuild/easybuild-SAMtools-1.11-20210309.105601.log +== COMPLETED: Installation ended successfully (took 2 mins 20 secs) +== Results of the build can be found in the log file(s) +/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/libdap/3.20.9-cpeGNU-21.12/easybuild/easybuild-libdap-3.20.9-20220329.154535.log == Build succeeded for 1 out of 1 -== Temporary log file(s) /tmp/eb-zh7_fyre/easybuild-4q_lo57b.log* have been removed. -== Temporary directory /tmp/eb-zh7_fyre has been removed. +== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-kfphjoi8/easybuild-kcs00ai5.log* have been removed. +== Temporary directory /run/user/10012026/easybuild/tmp/eb-kfphjoi8 has been removed. ``` That was... easy. Is that really all there is to it? Well, almost... ### Enabling dependency resolution -The SAMtools installation worked like a charm, but remember that all required dependencies were already +The libdap installation worked like a charm, but remember that all required dependencies were already available (see [above](#dry-run)). -If we try this with the `BCFtools-1.11-GCC-10.2.0.eb`, for which the required `GSL` and `HTSlib` dependencies are not available yet, it's less successful: +If we try this with the `SAMtools-1.14-cpeGNU-21.12.eb`, for which the required `HTSlib` dependencies is not available yet, it's less successful: ```shell -$ eb BCFtools-1.11-GCC-10.2.0.eb -M +$ eb SAMtools-1.14-cpeGNU-21.12.eb -M -3 out of 23 required modules missing: +2 out of 11 required modules missing: -* GSL/2.6-GCC-10.2.0 (GSL-2.6-GCC-10.2.0.eb) -* HTSlib/1.11-GCC-10.2.0 (HTSlib-1.11-GCC-10.2.0.eb) -* BCFtools/1.11-GCC-10.2.0 (BCFtools-1.11-GCC-10.2.0.eb) +* HTSlib/1.14-cpeGNU-21.12 (HTSlib-1.14-cpeGNU-21.12.eb) +* SAMtools/1.14-cpeGNU-21.12 (SAMtools-1.14-cpeGNU-21.12.eb) ``` ```shell -$ eb BCFtools-1.11-GCC-10.2.0.eb +$ eb SAMtools-1.14-cpeGNU-21.12.eb ... == preparing... -== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/build/BCFtools/1.11/GCC-10.2.0): build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0 (took 2 sec) -== Results of the build can be found in the log file(s) /tmp/eb-3v1dfvnk/easybuild-BCFtools-1.11-20210308.195024.FlxkH.log -ERROR: Build of /home/example/.local/easybuild/easyconfigs/b/BCFtools/BCFtools-1.11-GCC-10.2.0.eb failed (err: 'build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0') +== FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12): build failed +(first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12 (took 3 secs) +== Results of the build can be found in the log file(s) +/run/user/10012026/easybuild/tmp/eb-rgj1v43y/easybuild-SAMtools-1.14-20220329.155911.ZtDcX.log + +ERROR: Build of /appl/lumi/LUMI-EasyBuild-contrib/easybuild/easyconfigs/s/SAMtools/SAMtools-1.14-cpeGNU-21.12.eb failed (err: 'build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12') ``` Oh my, what's this all about? @@ -478,36 +599,33 @@ Oh my, what's this all about? If we filter the output a bit and focus on the actual error, the problem is clear: ``` -Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0 +Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12') ``` -The required dependencies `HTSlib/1.11-GCC-10.2.0` and `GSL/2.6-GCC-10.2.0` are not installed yet, -and EasyBuild does not automatically install missing dependencies unless it is told to do so. +The required dependency `HTSlib/1.14-cpeGNU-21.12` is not installed yet, +and EasyBuild does not automatically install missing dependencies unless it is told to do so +(which we didn't do in the configuration for LUMI). -It helpfully suggests to use the `--robot` command line option, so let's try that: +It helpfully suggests to use the `--robot` (or '-r') command line option, so let's try that: ```shell -$ eb BCFtools-1.11-GCC-10.2.0.eb --robot +$ eb SAMtools-1.14-cpeGNU-21.12.eb --robot ... == resolving dependencies ... ... -== building and installing HTSlib/1.11-GCC-10.2.0... +== building and installing HTSlib/1.14-cpeGNU-21.12... ... == COMPLETED: Installation ended successfully (took 13 sec) ... -== building and installing GSL/2.6-GCC-10.2.0... -... -== COMPLETED: Installation ended successfully (took 1 min 10 sec) -... -== building and installing BCFtools/1.11-GCC-10.2.0... +== building and installing SAMtools/1.14-cpeGNU-21.12... ... == COMPLETED: Installation ended successfully (took 8 sec) ... -== Build succeeded for 3 out of 3 +== Build succeeded for 2 out of 2 ``` -With dependency resolution enabled the `HTSlib` and `GSL` modules get installed first, -before EasyBuild proceeds with installing `BCFtools`. Great! +With dependency resolution enabled the `HTSlib` module gets installed first, +before EasyBuild proceeds with installing `SAMtools`. Great! --- @@ -521,35 +639,50 @@ To provide some more feedback as the installation progresses, you can enable the Let's do this by defining the `$EASYBUILD_TRACE` environment variable, just to avoid having to type `--trace` over and over again. -We will redo the installation of `BCFtools-1.11-GCC-10.2.0.eb` by passing the `--rebuild` +We will redo the installation of `SAMtools-1.14-cpeGNU-21.12.eb` by passing the `--rebuild` option to the `eb` command (try yourself what happens if you don't use the `--rebuild` option!): ```shell $ export EASYBUILD_TRACE=1 -$ eb BCFtools-1.11-GCC-10.2.0.eb --rebuild +$ eb SAMtools-1.14-cpeGNU-21.12.eb --rebuild ... == configuring... >> running command: - [started at: 2021-03-08 19:54:53] - [working dir: /tmp/example/build/BCFtools/1.11/GCC-10.2.0/bcftools-1.11] - [output logged in /tmp/eb-9u_ac0nv/easybuild-run_cmd-17m_he2x.log] - ./configure --prefix=/home/example/easybuild/software/BCFtools/1.11-GCC-10.2.0 --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --with-htslib=$EBROOTHTSLIB --enable-libgsl -== building... + [started at: 2022-03-29 18:46:31] + [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] + [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-g7vd83qv.log] + /users/kurtlust/LUMI-user-appl/sources/generic/eb_v4.5.3/ConfigureMake/config.guess + >> command completed: exit 0, ran in < 1s >> running command: - [started at: 2021-03-08 19:54:54] - [working dir: /tmp/example/BCFtools/1.11/GCC-10.2.0/bcftools-1.11] - [output logged in /tmp/example/eb-9u_ac0nv/easybuild-run_cmd-bhkgjxi7.log] - make -j 8 + [started at: 2022-03-29 18:46:31] + [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] + [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-k0etfv8i.log] + ./configure --prefix=/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/SAMtools/1.14-cpeGNU-21.12 --build=x86_64-pc-linux-gnu +--host=x86_64-pc-linux-gnu --with-htslib=$EBROOTHTSLIB >> command completed: exit 0, ran in 00h00m03s +== ... (took 3 secs) +== building... + >> running command: + [started at: 2022-03-29 18:46:34] + [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] + [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-svcps0yj.log] + make -j 256 CC="cc" CXX="CC" CFLAGS="-O2 -ftree-vectorize -fno-math-errno -fPIC" CXXFLAGS="-O2 -ftree-vectorize -fno-math-errno +-fPIC" + >> command completed: exit 0, ran in 00h00m06s +== ... (took 6 secs) ``` That's a bit more comforting to stare at... -During the *configure* step, the `./configure` command is run with option to -enable support for leveraging `HTSlib` and `GSL`. +SAMtools uses a custom easyblock that is derived from the generic `ConfigureMake` easyblock. +During the *configure* step, the `./configure` command is run with `--build` and `--host` +options added by the generic ConfigureMake easyblock (and the other command, `config.guess` plays +a role in determining the value of those flags). The `--with-htslib=$EBROOTHTSLIB` flag is added +via our easyconfig file to tell SAMtools to use an already available version of HTSlib rather +than the built-in one. During the *build* step, the software is actually being compiled -by running the `make` command. EasyBuild automatically uses the available cores on the system (in this case 8). +by running the `make` command. EasyBuild automatically uses the available cores on the system (in this case 256). We even get a pointer to a log file that contains the output of the command being run, so we can use `tail -f` to see in detail how it progresses. @@ -561,14 +694,19 @@ Later during the installation, we now also see this output during the sanity che ``` == sanity checking... - >> file 'bin/bcftools' found: OK - >> file 'bin/plot-vcfstats' found: OK - >> file 'bin/vcfutils.pl' found: OK - >> (non-empty) directory 'libexec/bcftools' found: OK + >> file 'bin/blast2sam.pl' found: OK + >> file 'bin/bowtie2sam.pl' found: OK + >> file 'bin/export2sam.pl' found: OK + >> file 'bin/interpolate_sam.pl' found: OK +... + >> loading modules: SAMtools/1.14-cpeGNU-21.12... + >> running command 'samtools version' ... + >> result for command 'samtools version': OK ``` -Thanks to enabling trace mode, EasyBuild tells us which files & directories it is checking for -in the installation, before declaring it a success. Nice! +Thanks to enabling trace mode, EasyBuild tells us which files (& directories, but there are non in this case) +it is checking for +in the installation, and which command it is trying to run before declaring it a success. Nice! The extra output you get when trace mode is enabled is concise and hence not overwhelming, while it gives a better insight into what is going on during the installation. @@ -579,13 +717,13 @@ so you can interrupt the installation before it completes, if deemed necessary. ## Using installed software -So far, we have already installed 4 different software packages (SAMtools, HTSlib, GSL, and BCFtools); -we even installed BCFtools twice! +So far, we have already installed 4 different software packages (SAMtools, HTSlib, libdap); +we even installed SAMtools twice! A lot was going on underneath the covers: locating and unpacking the source tarballs, setting up the build environment, configuring the build, compiling, creating and populating the installation directory, performing a quick sanity check on the installation, -cleaning things up, and finally generated the environment module file corresponding to the installation. +cleaning things up, and finally generating the environment module file corresponding to the installation. That's great, but how do we now actually *use* these installations? @@ -593,98 +731,165 @@ This is where the generated module files come into play: they form the access po installations, and we'll use the ubiquitous `module` command to digest them. First, we need to make sure that the modules tool is aware of where the module files for -our installations are located. If you're unsure where EasyBuild is installing stuff at this point, -check the output of `eb --show-config`; the value of the `installpath` configuration setting is what we are interested in now: +our installations are located. On LUMI, when using the EasyBuild-user module to configure EasyBuild, +everything is taken care of for you and the LUMI modules will also automatically add the +suitable module directories for user-installed software to the search path for modules. +By default, EasyBuild-config will install in `$HOME/EasyBuild`, but it is possible to +build the installation in a different directory by pointing to it with the environment +variable `EBU_USER_PREFIX`. Of course this variable needs to be set before loading the `LUMI` +module. (Note that one reason why we don't load a software stack by default is that in +the current setup of LUMI this module would be loaded before the user gets the chance to set +that environment variable in `.bash_profile` or `.bashrc`.) + +When loading the `EasyBuild-user` module, the module command will show you were EasyBuild +will install the software and put the modules, and also put its repository of +processed easyconfig file. ```shell -$ eb --show-config -... -installpath (E) = /home/example/easybuild -... -repositorypath (E) = /home/example/easybuild/ebfiles_repo -... -sourcepath (E) = /home/example/easybuild/sources -... +ml EasyBuild-user + +EasyBuild configured to install software from the LUMI/21.12 software stack for the LUMI/L +partition in the user tree at /users/kurtlust/LUMI-user-appl. + * Software installation directory: /users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L + * Modules installation directory: /users/kurtlust/LUMI-user-appl/modules/LUMI/21.12/partition/L + * Repository: /users/kurtlust/LUMI-user-appl/ebrepo_files/LUMI-21.12/LUMI-L + * Work directory for builds and logs: /run/user/10012026/easybuild + Clear work directory with clear-eb ``` -So, what's in this directory? - -```shell -$ ls -l $HOME/easybuild -total 16 -drwxrwxr-x 5 example example 4096 Jun 10 20:11 ebfiles_repo -drwxrwxr-x 5 example example 4096 Jun 10 20:10 modules -drwxrwxr-x 6 example example 4096 Jun 10 20:10 software -drwxrwxr-x 6 example example 4096 Jun 10 20:10 sources -``` +EasyBuild will copy each easyconfig file it installs to +the repository and add some lines to it with information about the installation. It +also has some options that may edit the source easyconfig, e.g., when asking EasyBuild +to try to build with another toolchain. -The `ebfiles_repo` and `sources` directories correspond to the `repositorypath` and `sourcepath` configuration -settings, respectively. The `modules` and `software` directories are what we need now. +You can always check where EasyBuild is installing stuff by checking the output of +`eb --show-config`. -The `modules` subdirectory consists of multiple subdirectories: +If you're unsure where EasyBuild is installing stuff at this point, +check the output of `eb --show-config`; the value of the `installpath` configuration setting is what we are interested in now: ```shell -$ ls $HOME/easybuild/modules -all bio devel numlib tools +$ eb --show-config +... +buildpath (E) = /run/user/XXXXXXXX/easybuild/build +... +installpath-modules (E) = /users/XXXXXXXX/LUMI-user/modules/LUMI/21.12/partition/L +installpath-software (E) = /users/XXXXXXXX/LUMI-user/SW/LUMI-21.12/L... +... +repositorypath (E) = /users/XXXXXXXX/LUMI-user/ebrepo_files/LUMI-21.12/LUMI-L +... +sourcepath (E) = /users/XXXXXXXX/LUMI-user/sources:/appl/lumi/sources/easybuild +... ``` -Directories like `bio` and `numlib` correspond to different software categories, -and contain symbolic links to the module files in the `all` directory, -which contains all actual module files for software installed in this EasyBuild installation path. -We'll ignore these separate category directories for now. +This is sligthly different from the default EasyBuild setup, where the modules, software, +repository and sources would be installed in respectively the subdirectories `modules`, +`software`, `ebfiles_repo` and `sources` of the directory pointed to by the `installpath` +line. -Let's inform the modules tool about the existence of these module files using `"module use"`: +The modules directory is also a simplified one from the standard EasyBuild one as that also +provides a module categorisation besides a directory containing all modules. As this categorisation +is largely arbitrary and hard to use in the module system, we decided simply not to use it in +our installation and use a custom naming scheme. -```shell -module use $HOME/easybuild/modules/all -``` - -This command does little more that updating the `$MODULEPATH` environment variable, -which contains a list of paths that the modules tool should consider when looking for module files. +However, if you would be using EasyBuild on another system with its default configuration, the +above setup would be used. For more information, we refer to the generic EasyBuild tutorials on the +[EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). Now the modules tool should be aware of our brand new installations: ```shell $ module avail +... +-- EasyBuild managed user software for software stack LUMI/21.12 on LUMI-L --- + HTSlib/1.14-cpeGNU-21.12 libdap/3.20.9-cpeGNU-21.12 + SAMtools/1.14-cpeGNU-21.12 ----------------------- /home/example/easybuild/modules/all ----------------------- - BCFtools/1.11-GCC-10.2.0 GSL/2.6-GCC-10.2.0 SAMtools/1.11-GCC-10.2.0 - EasyBuild/4.3.3 HTSlib/1.11-GCC-10.2.0 bzip2/1.0.6 - ----------------------------- /easybuild/modules/all ----------------------------- - ... +----- EasyBuild managed software for software stack LUMI/21.12 on LUMI-L ----- +... ``` -This output shows both the modules for our own installations as well as the "central" installations in `/easybuild` (which we omitted above for brevity). +This output shows both the modules for our own installations as well as the "central" installations +(which we omitted for brevity). Now we can load these modules and start using these software installations. -Let's test this for BCFtools. In our current environment, the `bcftools` command is not available yet: +Let's test this for SAMtools. In our current environment, the `samtools` command is not available yet: ```shell $ module list -No modules loaded -$ bcftools --bash: bcftools: command not found +Currently Loaded Modules: + 1) perftools-base/21.12.0 + 2) cce/13.0.0 + 3) craype/2.7.13 + 4) cray-dsmml/0.2.2 + 5) cray-mpich/8.1.12 + 6) cray-libsci/21.08.1.2 + 7) PrgEnv-cray/8.2.0 + 8) ModuleLabel/label (S) + 9) init-lumi/0.1 (S) + 10) craype-x86-rome + 11) craype-accel-host + 12) libfabric/1.11.0.4.106 + 13) craype-network-ofi + 14) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta + 15) partition/L (S) + 16) LUMI/21.12 (S) + + Where: + S: Module is Sticky, requires --force to unload or purge +$ samtools +-bash: samtools: command not found ``` -Loading the module for BCFtools changes that: +Loading the module for SAMtools changes that: ```shell -$ module load BCFtools/1.11-GCC-10.2.0 +$ module load SAMtools/1.14-cpeGNU-21.12 + +Lmod is automatically replacing "cce/13.0.0" with "gcc/11.2.0". +Lmod is automatically replacing "PrgEnv-cray/8.2.0" with "cpeGNU/21.12". + +Due to MODULEPATH changes, the following have been reloaded: + 1) cray-mpich/8.1.12 $ module list + Currently Loaded Modules: - 1) GCCcore/10.2.0 6) XZ/5.2.5-GCCcore-10.2.0 - 2) zlib/1.2.11-GCCcore-10.2.0 7) cURL/7.72.0-GCCcore-10.2.0 - 3) binutils/2.35-GCCcore-10.2.0 8) HTSlib/1.11-GCC-10.2.0 - 4) GCC/10.2.0 9) GSL/2.6-GCC-10.2.0 - 5) bzip2/1.0.8-GCCcore-10.2.0 10) BCFtools/1.11-GCC-10.2.0 - -$ bcftools --version -bcftools 1.11 -Using htslib 1.11 + 1) perftools-base/21.12.0 + 2) ModuleLabel/label (S) + 3) init-lumi/0.1 (S) + 4) craype-x86-rome + 5) craype-accel-host + 6) libfabric/1.11.0.4.106 + 7) craype-network-ofi + 8) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta + 9) partition/L (S) + 10) LUMI/21.12 (S) + 11) gcc/11.2.0 + 12) craype/2.7.13 + 13) cray-mpich/8.1.12 + 14) cray-libsci/21.08.1.2 + 15) cray-dsmml/0.2.2 + 16) cpeGNU/21.12 + 17) ncurses/6.2-cpeGNU-21.12 + 18) zlib/1.2.11-cpeGNU-21.12 + 19) bzip2/1.0.8-cpeGNU-21.12 + 20) gettext/0.21-cpeGNU-21.12-minimal + 21) XZ/5.2.5-cpeGNU-21.12 + 22) Brotli/1.0.9-cpeGNU-21.12 + 23) cURL/7.78.0-cpeGNU-21.12 + 24) HTSlib/1.14-cpeGNU-21.12 + 25) SAMtools/1.14-cpeGNU-21.12 + + Where: + S: Module is Sticky, requires --force to unload or purge + +$ samtools --version +samtools 1.14 +Using htslib 1.14 +Copyright (C) 2021 Genome Research Ltd.... ... ``` @@ -696,24 +901,61 @@ by updating environment variables like `$PATH` for example, to make the software To restore your environment to a pristine state in which no modules are loaded, you can either unload the loaded modules one by one using "`module unload`", or you can unload all of them at once using -"`module purge`". +"`module purge`". On LUMI, `module purge` will unload all application modules but will not undo the selection +of the software stack. It will reset the software stack though to use those modules that fit best +with the hardware of the current node (i.e., you may find a different `partition` module). -**If you are using an EasyBuild installation provided by a module, -don't forget to load the `EasyBuild` module again after running "`module purge`".** +```shell +$ module purge +The following modules were not unloaded: + (Use "module --force purge" to unload all): + + 1) LUMI/21.12 6) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta + 2) craype-x86-rome 7) partition/L + 3) craype-accel-host 8) init-lumi/0.1 + 4) libfabric/1.11.0.4.106 9) ModuleLabel/label + 5) craype-network-ofi +$ module list + +Currently Loaded Modules: + 1) LUMI/21.12 (S) + 2) craype-x86-rome + 3) craype-accel-host + 4) libfabric/1.11.0.4.106 + 5) craype-network-ofi + 6) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta + 7) partition/L (S) + 8) init-lumi/0.1 (S) + 9) ModuleLabel/label (S) + + Where: + S: Module is Sticky, requires --force to unload or purge +``` + +Runnin `module --force purge` instead will remove all modules, including the `init-lumi` +module which does part of the initialisation. You will not be able to use the software +stacks completely as before without first loading `init-lumi` in its most recent (or default) +version again! + +```shell +$ module --force purge +$ module list +No modules loaded +``` --- ## Stacking software Maybe you have overlooked how the software we are playing around with was not only installed across multiple -different installation directories per software, we are also "stacking" our own installations (in `$HOME/easybuild`) -on top of installations that are provided in a totally different location (`/easybuild`). +different installation directories per software, we are also "stacking" our own installations (in `$HOME/EasyBuild` +or `$EBU_USER_PREFIX`) on top of installations that are provided in a totally different location (`/appl/lumi`). **EasyBuild doesn't care *where* software is installed: as long as the module file that provides access to it is available, it is happy to pick it up and use it when required.** -This implies that end users of an HPC system can easily install their -own small software stack on top of what is provided centrally by the HPC support team, +This implies that end users of LUMI can easily install their +own small software stack on top of what is provided centrally by the LUMI User Support, for example. They can even "replace" a central software installation for their purposes if they need to, since the modules tool will load the first module file that matches the request being made (there are some caveats with this, but we @@ -723,6 +965,8 @@ won't go into those here). ## Exercises +TODO: Rework for LUMI and provide the necessary EasyConfig files. + ***Guidelines*** Do yourself a favor: don't peek at the solution until you have made an attempt to solve the exercise yourself! @@ -967,10 +1211,6 @@ If you've made it through the hands-on exercises, congratulations! If not, don't worry too much about it. We covered a lot of ground here, and it's a lot to take in at once, take your time... -Feel free to ask question in the `#tutorial` channel in the [EasyBuild -Slack](https://docs.easybuild.io/en/latest/#getting-help), -we're happy to help! - --- *[[next: Part 2: Using EasyBuild]](2_00_part2_using.md)* From a8c9b044439d0379228dc1aa55c53d9addfcd804 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 30 Mar 2022 12:27:23 +0200 Subject: [PATCH 068/159] Additions to the LUMI software stack page of the tutorial. --- .../1_04_LUMI_software_stack.md | 86 ++++++++++++++++++- 1 file changed, 85 insertions(+), 1 deletion(-) diff --git a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md index 0e578a59..3bfae70c 100644 --- a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md +++ b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md @@ -1,9 +1,93 @@ -# LUMI software stacks +# LUMI software stacks (technical) *[[back: The Cray Programming Environment]](1_03_CPE.md)* --- +The user-facing documentation on how to use the LUMI software stacks is +avialable in [the LUMI documentation](https://docs.lumi-supercomputer.eu/computing/softwarestacks/). +On this page we focus more on the technical implementation behind it. + +--- + +# An overview of LUMI + +LUMI has different node types providing compute resources: + +- LUMI has 16 login nodes, though many of those are reserved for special purposes and not + available to all users. TODO + + +--- + +## CrayEnv and LUMI modules + +On LUMI, two types of software stacks are currently offered: + + - ``CrayEnv`` (module name) offers the Cray PE and enables one to use + it completely in the way intended by HPE-Cray. The environment also offers a + limited selection of additional tools, often in updated versions compared to + what SUSE Linux, the basis of the Cray Linux environment, offers. Those tools + are installed and managed via EasyBuild. However, EasyBuild is not available + in that partition. + + It also rectifies a problem caused by the fact that there is only one + configuration file for the Cray PE on LUMI, so that starting a login shell + will not produce an optimal set of target modules for all node types. + The ``CrayEnv`` module recognizes on which node type it is running and + (re-)loading it will trigger a reload of the recommended set of target + modules for that node. + + - ``LUMI`` is an extensible software stack that is mostly managed through + [EasyBuild][easybuild]. Each version of the LUMI software stack is based on + the version of the Cray Programming Environment with the same version + number. + + A deliberate choice was made to only offer a limited number of software + packages in the globally installed stack as the setup of redundancy on LUMI + makes it difficult to update the stack in a way that is guaranteed to not + affect running jobs and as a large central stack is also hard to manage. + However, the EasyBuild setup is such that users can easily install + additional software in their home or project directory using EasyBuild build + recipes that we provide or they develop, and that software will fully + integrate in the central stack (even the corresponding modules will be made + available automatically). + + Each ``LUMI`` module will also automatically activate a set of application + modules tuned to the architecture on which the module load is executed. To + that purpose, the ``LUMI`` module will automatically load the ``partition`` + module that is the best fit for the node. After loading a version of the + ``LUMI`` module, users can always load a different version of the ``partition`` + module. + +Note that the ``partition`` modules are only used by the ``LUMI`` module. In the +``CrayEnv`` environment, users should overwrite the configuration by loading their +set of target modules after loading the ``CrayEnv`` module. + + +--- + +## The ``partition`` module + +The ``LUMI`` module currently supports four partition modules, but that number may +be reduced in the future: + +| Partition | CPU target | Accelerator | +|:----------------|-----------------------|:----------------------------| +| ``partition/L`` | ``craype-x86-rome`` | ``craype-accel-host`` | +| ``partition/C`` | ``craype-x86-milan`` | ``craype-accel-host`` | +| ``partition/G`` | ``craype-x86-trento`` | ``craype-accel-amd-gfx90a`` | +| ``partition/D`` | ``craype-x86-rome`` | ``craype-accel-nvidia80`` | + +All ``partition`` modules also load `craype-network-ofi``. + +``pattition/D`` may be dropped in the future as it seems we have no working CUDA setup +and can only use the GPU nodes in the LUMI-D partition for visualisation and not with CUDA. + +Furthermore if it would turn out that there is no advantage in optimizing for Milan +specifically, or that there are no problems at all in running Milan binaries on Rome +generation CPUs, ``partition/L`` and ``partition/C`` might also be united in a single +partition. From 75706448f0014f0402e53d6d13764e9f800850fe Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 30 Mar 2022 12:28:17 +0200 Subject: [PATCH 069/159] Integrating parts of the old tutorial into the new one, texts still need to be rewritten. --- docs/2022-CSC_and_LO/2_00_part2_using.md | 7 + docs/2022-CSC_and_LO/2_01_troubleshooting.md | 488 +++++++++ .../2_02_creating_easyconfig_files.md | 998 ++++++++++++++++++ .../2_03_implementing_easyblocks.md | 665 ++++++++++++ docs/2022-CSC_and_LO/3_00_part3_advanced.md | 9 + .../2022-CSC_and_LO/3_01_easybuild_library.md | 293 +++++ docs/2022-CSC_and_LO/3_02_hooks.md | 190 ++++ docs/2022-CSC_and_LO/3_03_slurm_jobs.md | 304 ++++++ .../3_04_module_naming_schemes.md | 617 +++++++++++ .../3_05_github_integration.md | 442 ++++++++ 10 files changed, 4013 insertions(+) create mode 100644 docs/2022-CSC_and_LO/2_00_part2_using.md create mode 100644 docs/2022-CSC_and_LO/2_01_troubleshooting.md create mode 100644 docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md create mode 100644 docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md create mode 100644 docs/2022-CSC_and_LO/3_00_part3_advanced.md create mode 100644 docs/2022-CSC_and_LO/3_01_easybuild_library.md create mode 100644 docs/2022-CSC_and_LO/3_02_hooks.md create mode 100644 docs/2022-CSC_and_LO/3_03_slurm_jobs.md create mode 100644 docs/2022-CSC_and_LO/3_04_module_naming_schemes.md create mode 100644 docs/2022-CSC_and_LO/3_05_github_integration.md diff --git a/docs/2022-CSC_and_LO/2_00_part2_using.md b/docs/2022-CSC_and_LO/2_00_part2_using.md new file mode 100644 index 00000000..e033499c --- /dev/null +++ b/docs/2022-CSC_and_LO/2_00_part2_using.md @@ -0,0 +1,7 @@ +# Part II: Using EasyBuild + +*[[back to start page]](index.md)* + +* [Troubleshooting](2_01_troubleshooting.md) *(hands-on)* +* [Creating easyconfig files](2_02_creating_easyconfig_files.md) *(hands-on)* +* [Implementing easyblocks](2_03_implementing_easyblocks.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/2_01_troubleshooting.md b/docs/2022-CSC_and_LO/2_01_troubleshooting.md new file mode 100644 index 00000000..86a2f1d3 --- /dev/null +++ b/docs/2022-CSC_and_LO/2_01_troubleshooting.md @@ -0,0 +1,488 @@ +# Troubleshooting + +*[[back: Overview]](index.md)* + +--- + +When installing scientific software you are bound to run into problems +that make the installation fail sooner or later, even when using EasyBuild. + +In this part we take a look at how you can **troubleshoot a failing installation**, +and focus on a couple of EasyBuild aspects that can be helpful in that context, +like the **error messages produced by EasyBuild**, the detailed **log file** that is +available for the installation and the **build directory** where most software is +being compiled before it actually gets installed. + +At the end of this part an exercise is available in which you get +to fix a failing installation, so pay attention! + +## EasyBuild error messages + +When EasyBuild detects that something went wrong, it will usually produce a +short (and hopefully helpful) error message. + +Things that could go wrong during an installation include: + +* missing source or patch files; +* a checksum error on a downloaded source or patch file; +* required dependencies that are not specified in the easyconfig file; +* failing shell commands; +* running out of available memory or disk space; +* a segmentation fault caused by a flipped bit triggered by a cosmic ray ([really, it happens!](https://blogs.oracle.com/linux/attack-of-the-cosmic-rays-v2)); + +Unfortunately this is not an exhaustive list, there are plenty of other +potential problems that could result in a failing installation... + +For each of the shell commands that EasyBuild executes during an +installation, it will check the exit status. +If the exit status is zero, EasyBuild will usually assume that the shell command +ran correctly, and it will continue with the rest of the installation procedure. +If the exit status is anything but zero, EasyBuild will assume that a problem +has occurred, and the installation will be interrupted. + +### Example + +Here is an example of an EasyBuild error message (slightly reformatted for clarity): + +``` +$ eb example.eb +... +== building... +== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): +build failed (first 300 chars): cmd "make" exited with exit code 2 and output: +/usr/bin/g++ -O2 -ftree-vectorize -march=native -fno-math-errno -std=c++14 -c -o core.o core.cpp +g++: error: unrecognized command line option '-std=c++14' (took 1 sec) +== Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log +ERROR: Build of /home/easybuild/subread.eb failed (err: ...) +``` + +Let's break this down a bit: during the `build` step of the installation +procedure EasyBuild was running `make` as a shell command, which +failed (exit code 2, so not zero). +The `make` command tripped over the compilation of `core.cpp` that failed because +`-std=c++14` is not a known option to the `g++` command. + +OK fine, but now what? Can you spot something suspicious here? +Wait a minute... Why is `make` using `/usr/bin/g++` for the compilation?! +That's not where our toolchain compiler is installed, +that's somewhere under `/easybuild/software`. + +Let's see what `/usr/bin/g++` is: + +```shell +$ /usr/bin/g++ --version +g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) +``` + +Oh my that's pretty ancient (GCC 4.8.5 was released in June 2015). +That could definitely explain why it doesn't know about the C++14 standard yet... + +Your next step in this case should probably be figuring +out why `/usr/bin/g++` is being used rather than just `g++`, which would +result in using the right compiler version because EasyBuild sets up the build +environment carefully. + +This is a fictitious example of course, but hopefully it gives you a feeling +of how errors that occur during the installation are handled. + +## EasyBuild log files + +Finding the cause of a problem that made the installation fail is, unfortunately, not always that straightforward... + +EasyBuild includes the first 300 characters of the output produced by a failing +shell command in the error message, which is a simple way to try include +actionable information in the error message while not flooding your terminal +with the full output of the command that failed. + +In some cases there won't be any useful information in there however, +since the actual error message(s) could only appear way later, perhaps even after +the command was already running for several minutes. + +In that case, you will have the dive into the log file that is created by EasyBuild for +every installation, which is located in the unique temporary directory for the EasyBuild session. + +See for example this output line from our earlier example error message: + +``` +== Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log +``` + +You can open this file with your favorite text editor or a tool like `less` +to take a look at the information collected in the log file, which includes +things like: + +* informative messages produced by both the EasyBuild framework and the easyblock + describing how the installation is progressing; +* how the build environment was set up: which modules were loaded, which environment variables were set; +* the exact shell commands that were executed, and in which directory they were run; +* the full output produced by these commands, and their exit code; + +Note that the installation log is also copied into each software installation +directory for successful installation, into the `easybuild` subdirectory. +For example: + +``` +/easybuild/software/HDF5/1.10.7-gompi-2020b/easybuild/easybuild-HDF5-1.10.7-20210308.214453.log +``` + +### Last log + +The `eb` command supports a handy little option that prints the location +to the most recently updated build log. You can leverage this to quickly +open the build log of the last failed EasyBuild session in an editor: + +``` +vim $(eb --last-log) +``` + +### Navigating log files + +Usually you want to go to the end of the log file and then work your way up, +either by scrolling or by searching for specific patterns. Here are a couple +of suggestions of patterns you can use to locate errors: + +* `ERROR` +* `Error 1` +* `error: ` +* `failure` +* `not found` +* `No such file or directory` +* `bazel` +* `Segmentation fault` + +Using "`error`" as a search pattern is not very useful: you will hit a lot of log lines +that are not actually errors at all (like the compilation of an `error.c` file), +and you'll miss others that do include errors but mention `ERROR` or `Error` +rather than `error`. + +When using `less` to view a log file, you can navigate it by: + +* hitting '`$`' followed by '`G`' to go to the end of the log file; +* using your arrow keys to scroll up/down; +* typing '`?`' followed by some text and Enter to search backwards for a particular +pattern ('`/`' to search forwards, '`n`' for next match); +* hitting '`q`' to exit; + +It can also be helpful to zoom in on a specific step of the installation procedure, +which you can do by looking for step markers like these: + +``` +== 2020-06-13 01:34:48,816 example INFO configuring... +== 2020-06-13 01:34:48,817 example INFO Starting configure step +... +== 2020-06-13 01:34:48,823 main.EB_HPL INFO Running method configure_step part of step configure +``` + +If you want to look at the start of the output produced by a particular command, +you can look for the log message that looks like this (this is from the installation +log for `HDF5`): + +``` +== 2020-06-09 13:11:19,968 run.py:222 INFO running cmd: make install +== 2020-06-09 13:11:25,186 run.py:538 INFO cmd " make install " exited with exit code 0 and output: +Making install in src +make[1]: Entering directory `/tmp/example/HDF5/1.10.7/gompi-2020b/hdf5-1.10.7/src' +... +``` + +It can be useful to look for the *first* error that occurred in the output of a command, since subsequent errors are +often fallout from earlier errors. You can do this by first navigating +to the start of the output for a command using "`INFO running cmd`" as a search pattern, and then looking for patterns +like "`error:`" from there. + +## Inspecting the build directory + +When an installation fails the corresponding build directory is *not* cleaned up +automatically, that is only done for successful installations. +This allows you to dive in and check for clues in the files that are stored there. + +The location of the build directory is mentioned in the EasyBuild error message: + +``` +== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): ... +``` + +For software using a classic `configure` script, you may have to locate +and inspect the `config.log` file in the build directory to determine the underlying cause of an error. For software using CMake as a configuration tool you often have to check in +`CMakeOutput.log` or `CMakeError.log` for clues, which are sneakily hidden by CMake in a `CMakeFiles` subdirectory of the build directory. + +As a side note here: as EasyBuild does not clean out old and failed builds you will need to eventually manually remove these build directories from the `buildpath` directory. + +## Exercise + +Let's work our way through a less smooth software installation, +using the easyconfig file that is provided below. + +Don't worry if most of this is still unclear to you, we'll get +to writing easyconfig files from scratch [later in this tutorial](../creating_easyconfig_files). + +```python +easyblock = 'MakeCp' + +name = 'Subread' +version = '2.0.1' + +homepage = 'http://subread.sourceforge.net' +description = "High performance read alignment, quantification and mutation discovery" + +toolchain = {'name': 'GCC', 'version': '8.5.0'} + +# download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz +sources = ['subread-%(version)s-source.tar.gz'] +checksums = ['d808eb5b1823c572cb45a97c95a3c5acb3d8e29aa47ec74e3ca1eb345787c17b'] + +start_dir = 'src' + +# -fcommon is required to compile Subread 2.0.1 with GCC 10, +# which uses -fno-common by default (see https://www.gnu.org/software/gcc/gcc-10/porting_to.html) +buildopts = '-f Makefile.Linux CFLAGS="-fast -fcommon"' + +files_to_copy = ['bin'] + +sanity_check_paths = { + 'files': ['bin/featureCounts', 'bin/subread-align'], + 'dirs': ['bin/utilities'], +} + +sanity_check_commands = ["featureCounts --version"] + +moduleclass = 'bio' +``` + +Do you spot any potential problems yet with this easyconfig file? + +***Preparation*** + +Start by copying the text above in a file named `subread.eb`, +so you can gradually fix the problem you'll encounter. + +Also make sure that the pre-installed software stack is available, +and that the EasyBuild module is loaded (unless you installed EasyBuild +yourself): + +``` +module use /easybuild/modules/all +module load EasyBuild +``` + +For this exercise, make sure EasyBuild is configured to +use `$HOME/easybuild` as `prefix`, and to use `/tmp/$USER` as `buildpath`: + +```shell +export EASYBUILD_PREFIX=$HOME/easybuild +export EASYBUILD_BUILDPATH=/tmp/$USER +``` + +Check your configuration via `eb --show-config`. + +Strictly speaking the configuration doesn't matter much for the sake of this +exercise, but it may help with the step-wise approach we'll take and +grasping the solutions. + +Remember though: *no peeking* before you tried to solve each step yourself! + +--- + +***Exercise T.1**** - Sources* + +Try to install the `subread.eb` easyconfig file, see what happens. + +Can you fix the problem you run into, perhaps without even changing +the easyconfig file? + +??? success "(click to show solution)" + + The installation fails because the source file `subread-2.0.1-source.tar.gz` + is not found: + ``` + $ eb subread.eb + ... + == FAILED: Installation ended unsuccessfully (build directory: /tmp/example/Subread/2.0.1/GCC-8.5.0): build failed (first 300 chars): + Couldn't find file subread-2.0.1-source.tar.gz anywhere, and downloading it didn't work either... + Paths attempted (in order): ... + ``` + + In this case, the problem is that the easyconfig file does not specify + where the sources can be downloaded from. Not automatically at least, + but there is a helpful comment included: + ```python + # download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz + sources = ['subread-%(version)s-source.tar.gz'] + ``` + + We can download the source tarball ourselves, + and move it to the location where EasyBuild expects to find it + (in the `sourcepath` directory): + ``` + curl -OL https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz + mv subread-2.0.1-source.tar.gz $HOME/easybuild/sources/s/Subread/ + ``` + + If downloading is problematic for some reason, the source tarball is also available + in `/easybuild/sources/s/Subread`. + + Or, we can change the easyconfig file to specify the location where + the easyconfig file can be downloaded from: + ```python + source_urls = ['https://download.sourceforge.net/subread/'] + sources = ['subread-%(version)s-source.tar.gz'] + ``` + Note that the `source_urls` value is a *list* of candidate download URLs, + *without* the filename of the source file itself. + + This way, EasyBuild will download the source file when running `eb subread.eb`. + + The source tarball is fairly large (23MB), so don't be alarmed if the download takes a little while. + + ```shell + $ ls -lh $HOME/easybuild/sources/s/Subread + total 23M + -rw-rw-r-- 1 easybuild easybuild 23M Jun 13 17:42 subread-2.0.1-source.tar.gz + ``` + +--- + +***Exercise T.2**** - Toolchain* + +After fixing the problem with missing source file, try the installation again. + +What's wrong now? How can you fix it quickly? + +Take into account that we just want to get this software package installed, +we don't care too much about details like the version of the dependencies or +the *toolchain* here... + + +??? success "(click to show solution)" + + The installation fails because the easyconfig specifies that GCC 8.5.0 + should be used as toolchain: + ``` + $ eb subread.eb + ... + == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-8.5.0): build failed (first 300 chars): + No module found for toolchain: GCC/8.5.0 (took 1 sec) + ``` + + We don't have this GCC version installed, but we do have GCC 10.2.0: + + ```shell + $ module avail GCC/ + ----------------- /easybuild/modules/all ------------------ + GCC/10.2.0 + ``` + + So let's try using that instead. + + Edit the easyconfig file so it contains this: + + ```python + toolchain = {'name': 'GCC', 'version': '10.2.0'} + ``` +--- + +***Exercise T.3**** - Build step* + +With the first two problems fixed, now we can actually try to build the software. + +Can you fix the next problem you run into? + +??? success "(click to show solution)" + + The compilation fails, but the error message we see is incomplete due to + EasyBuild truncating the command output (only the 300 first characters of the output are shown): + ``` + == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-10.2.0): build failed (first 300 chars): + cmd " make -j 1 -f Makefile.Linux CFLAGS="-fast -fcommon"" exited with exit code 2 and output: + gcc -mtune=core2 -O3 -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\""2.0.1"\" -D_FILE_OFFSET_BITS=64 -fmessage-length=0 -ggdb -fast -fcommon -c -o core.o core.c + gcc: error: unrecognized command line opti (took 1 sec) + ``` + + If you open the log file and scroll to the end, + the error is pretty obvious: + ``` + gcc: error: unrecognized command line option -fast; did you mean -Ofast? + make: *** [core.o] Error 1 + ``` + + The easyconfig file hard specifies the `-fast` compiler flag via the `CFLAGS` argument to the build command: + ```python + # -fcommon is required to compile Subread 2.0.1 with GCC 10, + # which uses -fno-common by default (see https://www.gnu.org/software/gcc/gcc-10/porting_to.html) + buildopts = '-f Makefile.Linux CFLAGS="-fast -fcommon"' + ``` + + EasyBuild sets up the build environment, so there should be no need + to hard specify compiler flags (certainly not incorrect ones). + The comment above the `buildopts` definition makes it clear that the `-fcommon` + flag *is* required though, because GCC 10 became a bit stricter by + using `-fno-common` by default. Note that we are using `-fcommon` + as an escape mechanism here: it would be better to fix the source code + and create a patch file instead. + + An easy way to fix this problem is to replace the `-fast` with `-Ofast`, + as the compiler error suggests. + + In this case it is advised to change the `CFLAGS` argument that is added + to be build command to replace the `-fast` with `$CFLAGS`, + which is defined in the build environment by EasyBuild. + + ```python + buildopts = '-f Makefile.Linux CFLAGS="$CFLAGS -fcommon"' + ``` + + Note that we need to be careful with quotes here: we use inner double quotes + to ensure that `$CFLAGS` will be expanded to its value when the build command is run. + +--- + +***Exercise T.4**** - Sanity check* + +After fixing the compilation issue, you're really close to getting the installation working, we promise! + +Don't give up now, try one last time and fix the last problem that occurs... + +??? success "(click to show solution)" + + Now the installation itself works but the sanity check fails, + and hence the module file does not get generated: + ``` + $ eb subread.eb + ... + == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-10.2.0): build failed (first 300 chars): + Sanity check failed: sanity check command featureCounts --version exited with code 255 + (output: featureCounts: unrecognized option '--version' + ... + ``` + + If you look at the full output in the log file you can see + that the correct option to check the version of the `featureCounts` command + is "`-v`" rather than "`--version`", so we need to fix this in the easyconfig file. + + Make the following change in the easyconfig file: + ```python + sanity_check_commands = ["featureCounts -v"] + ``` + + After doing so, **you don't have to redo the installation + from scratch**, you can use the `--module-only` option to only run the + sanity check and generate the module file again: + ``` + eb subread.eb --module-only + ``` + +--- + +In the end, you should be able to install Subread 2.0.1 with the GCC 10.2.0 toolchain by fixing the problems with the `subread.eb` easyconfig file. + +Check your work by manually loading the module and checking the version +via the `featureCounts` command, which should look like this: + +```shell +$ featureCounts -v +featureCounts v2.0.1 +``` + +--- + +*[[next: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md new file mode 100644 index 00000000..d46e003a --- /dev/null +++ b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md @@ -0,0 +1,998 @@ +# Creating easyconfig files + +*[[back: Troubleshooting]](2_02_troubleshooting.md)* + +--- + +We already know how to install easyconfig files that are included with EasyBuild, +but what about installing software for which no easyconfig is available yet? + +To do this we will need to **create additional easyconfig files**, +since every software installation performed by EasyBuild is done based on an easyconfig file. + +In this part of the tutorial we will look at the guts of easyconfig files and even create some ourselves! + +## Easyconfigs vs easyblocks + +Before we dive into writing [easyconfig files](../introduction/#easyconfig-files), +let us take a brief look at how they relate to [easyblocks](../introduction/#easyblocks). + +As we discussed [earlier](../introduction/#terminology), an easyconfig file (`*.eb`) is required +for each installation +performed by EasyBuild which specifies the details of the installation (which software +version, toolchain, etc.), while the installation procedure is implemented +in an easyblock (a Python module). + +When can we leverage a *generic easyblock*, perhaps via a "fat" easyconfig file that includes +a lot of carefully defined easyconfig parameters, and when should we use a minimal easyconfig file +together with a custom *software-specific* easyblock? + +This is not an easy question to answer in a general sense, since it depends on several factors: +the complexity of the software you want to get installed, how much flexibility you want, +how "intelligent" the installation procedure should be with respect to the compiler toolchain and dependencies +that are used for the installation, etc. + +In a nutshell, custom software-specific easyblocks are "do once and forget": they are central solution to peculiarities in the installation procedure of a particular software package. + +Reasons to consider implementing a software-specific easyblock rather than using a generic easyblock include: + +* 'critical' values for easyconfig parameters required to make installation succeed; +* toolchain-specific aspects of the build and installation procedure (e.g., configure options); +* interactive commands that need to be run; +* custom (configure) options for dependencies; +* having to create or adjust specific (configuration) files; +* 'hackish' usage of a generic easyblock; +* complex or very non-standard installation procedure; + +Implementing easyblocks is out of scope for this basic tutorial, for more information please consult +the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). + +## Writing easyconfig files + +Writing an easyconfig file boils down to defining a set of easyconfig parameters in a text file, +which we give a '`.eb`' extension by convention. The name of an easyconfig file doesn't matter +when using it directly to install software, but it does matter when EasyBuild needs to find it +to resolve a dependency for example (as we [discussed earlier](../basic_usage/#easyconfig-filenames)). + +The syntax for easyconfig files is *Python syntax*: you are basically defining a bunch of Python variables +that correspond to easyconfig parameters. + +The order in which the easyconfig parameters are defined doesn't matter, but we generally try to strick to a particular +order which roughly corresponds to the order in which the easyconfig parameters are used during the installation. +That is mostly helpful for humans staring at easyconfig files or comparing them. + +### Mandatory parameters + +A limited number of easyconfig parameters are *mandatory*, they must be defined in every easyconfig file: + +* `name`: the name of the software to install; +* `version`: the version of the software to install; +* `homepage`: a URL to the website of the software; +* `description`: a short description of the software; +* `toolchain`: the compiler toolchain to use for the installation; + +**`name`, `version`** + +It should be no surprise that specifying the name and version of the software you want to install is mandatory. +This information may influence the value of several other easyconfig parameters (like the name of the source file), and is also used to the determine the name of the module file to install. + +```python +name = 'example' +version = '1.0' +``` + +**`homepage`, `description`** + +The homepage and description are included in the generated module file for the installation. +That way the "`module show`" command provides some useful high-level information about the installation. + +```python +homepage = 'https://example.org' +description = "This is just an example." +``` + +Usually it does not matter whether you use single or double quotes to specify string values, +but you will often see that single quotes are used for values that don't have spaces (words) +and double quotes for values that do have spaces (sentences). There is no technical reason for +this, it just feels more natural to some people. There are cases where it is important to use +the right type of quotes however, we will get back to that later (keep it in mind for the exercises!). + +For multi-line descriptions, you will need to use "triple quoting" (which is standard Python syntax): + +```python +description = """This is an example + of a multi-line description. + It is spread across multiple lines.""" +``` + +**`toolchain`** + +EasyBuild also requires that the [compiler toolchain](../introduction/#toolchains) is specified, via the `toolchain` +easyconfig parameter. + +This can either be the [`system` toolchain](../introduction/#system-toolchain), for which a constant named `SYSTEM` is available: + +```python +toolchain = SYSTEM +``` + +Usually we specify a 'proper' toolchain like the compiler-only toolchain GCC 10.2.0 which we used before, +or the full toolchain `foss` 2020b. The name and version of the toolchain can be specified using a small Python dictionary, +for example: + +```python +toolchain = {'name': 'GCC', 'version': '10.2.0'} +``` + +### Commonly used parameters + +You will often need to specify additional easyconfig parameters to get something useful done. +We will cover the most commonly used ones here, but keep in mind that these are *not* mandatory. + +A full overview of all known easyconfig parameters can be obtained via "`eb --avail-easyconfig-params`" +or just "`eb -a`" for short, or can be consulted in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_parameters.html). + +#### Sources, patches, and checksums + +In most easyconfig files you will see that a list of source files is specified via the `sources` +easyconfig parameter, usually combined +with one or more URLs where these sources can be downloaded specified via `source_urls`. +There also may be patch files listed (specified via `patches`), +and checksums for both the source files and patches (specified via `checksums`). + +The `sources` easyconfig parameter is commonly defined but it is *not* mandatory, +because some easyconfig files only specify bundles of software packages and hence only +serve to generate a module file. + +Here is an example of how these easyconfig parameters can be specified: + +```python +source_urls = [ + 'https://example.org/download/', + 'https://example.org/download/archive/', +] +sources = ['example-1.0-src.tar.gz'] +patches = ['example-fix.patch'] +checksums = [ + '9febae18533d035ac688d977cb2ca050e6ca8379311d7a14490ad1ef948d45fa', + '864395d648ad9a5b75d1a745c8ef82b78421d571584037560a22a581ed7a261c', +] +``` + +Each of these require a *list* of values, so even if there is only a single source file or download URL +you must use square brackets as shown in the example. The default value for each of these is an empty list (`[]`). + +Some things worth pointing out here: + +* The download URLs specified via `source_urls` do *not* include the name of the file, that is added + automatically by EasyBuild when it tries to download the file (only if it's not available already.) +* If multiple download URLs are specified, they are each tried once in order until the download of the source file was + successful. This can be useful to include backup locations where source files can be downloaded from. +* Names of source files and patches should not include hardcoded software versions, they usually use a + template value like `%(version)s` instead: + ```python + sources = ['example-%(version)s-src.tar.gz'] + ``` + EasyBuild will use the value of the `version` easyconfig parameter to determine the actual name of the source + file. This way the software version is only specified in one place and the easyconfig file is easier to + update to other software versions. A list of template values can be consulted via the EasyBuild command + line via the `--avail-easyconfig-templates` option, or in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_templates.html). +* Source files can also be specified in ways other than just using a filename, see the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-param-sources-alt) for more information. +* Specified checksums are usually SHA256 checksum values, but [other types are also supported](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html?highlight=checksums#checksums). + + +#### Easyblock + +The easyblock that should be used for the installation can be specified via the `easyblock` easyconfig parameter. + +This is not mandatory however, because by default EasyBuild will determine the easyblock to use based on the +name of the software. If '`example`' is specified as software name, EasyBuild will try to locate a +software-specific easyblock named `EB_example` (in a Python module named `example.py`). Software-specific +easyblocks follow the convention that the class name starts with `'EB_`', followed by the software name +(where some characters are replaced, like '`-`' with '`_minus_`'). + +**Generic easyblocks** + +Usually the `easyblock` value is the name of a *generic* easyblock, if it is specified. The name of +a generic easyblock does *not* start with '`EB_`', so you can easily distinguish it from a software-specific +easyblock. + +Here are a couple of commonly used generic easyblocks: + +* `ConfigureMake`: implements the standard `./configure`, `make`, `make install` installation procedure; +* `CMakeMake`: same as `ConfigureMake`, but with `./configure` replaced with `cmake` for the configuration step; +* `PythonPackage`: implements the installation procedure for a single Python package, by default using + "`python setup.py install`" but other methods like using "`pip install`" are also supported; +* `Bundle`: a simple generic easyblock to bundle a set of software packages together in a single installation directory; +* `PythonBundle`: a customized version of the `Bundle` generic easyblock to install a bundle of Python packages + in a single installation directory; + +A full overview of the available generic easyblocks is available in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/generic_easyblocks.html). You can also consult the output of +`eb --list-easyblocks`, which gives an overview of *all* known easyblocks, and how they relate to each other. + +**Custom easyconfig parameters** + +Most generic easyblocks provide additional easyconfig parameters to steer their behaviour. +You can consult these via "`eb -a --easyblock`" or just "`eb -a -e`", which results in an +additional "`EASYBLOCK-SPECIFIC`" section to be added. See the (partial) output of this command for example: + +```shell +$ eb -a -e ConfigureMake +Available easyconfig parameters (* indicates specific to the ConfigureMake easyblock): +... +EASYBLOCK-SPECIFIC +------------------ +build_cmd* Build command to use [default: "make"] +build_type* Value to provide to --build option of configure script, e.g., x86_64-pc-linux-gnu (determined by config.guess shipped with EasyBuild if None, False implies to leave it up to the configure script) [default: None] +configure_cmd* Configure command to use [default: "./configure"] +configure_cmd_prefix* Prefix to be glued before ./configure [default: ""] +host_type* Value to provide to --host option of configure script, e.g., x86_64-pc-linux-gnu (determined by config.guess shipped with EasyBuild if None, False implies to leave it up to the configure script) [default: None] +install_cmd* Build command to use [default: "make install"] +prefix_opt* Prefix command line option for configure script ('--prefix=' if None) [default: None] +tar_config_opts* Override tar settings as determined by configure. [default: False] +``` + +#### Dependencies + +You will often need to list one or more [dependencies](../introduction/#dependencies) that are required +to install or run the software. +We distinguish between two main different types of dependencies: runtime dependencies and build dependencies. + +*Runtime dependencies* are required for using the installed software, and may also have to be available +during the installation. These dependencies can be specified via the `dependencies` easyconfig parameter. +EasyBuild will load the modules for these dependencies when setting up the build environment, +and will include load statements for them in the generated module file. + +*Build dependencies* are only required during the installation of the software, not for using the +software once it is installed. The modules for these dependencies will be loaded in the build environment +set up by EasyBuild during the installation, but they will *not* be loaded by the generated module file. +You can specify build dependencies via the `builddependencies` easyconfig parameter. +One typical example of a build dependency is `CMake`, which is only needed for configuring +the build. + +Here is a simple example of specifying dependencies: + +```python +builddependencies = [('CMake', '3.18.4')] + +dependencies = [ + ('Python', '3.8.2'), + ('HDF5', '1.10.6'), + ('SciPy-bundle', '2020.03', '-Python-%(pyver)s'), +] +``` + +Both `builddependencies` and `dependencies` require a list of tuples, +each of which specifying one dependency. +The name and version of a dependency is specified with a 2-tuple (a tuple with two string values). + +In some cases additional information may have to be provided, as is shown in the example above for the `SciPy-bundle` +dependency where a 3rd value is specified corresponding to the `versionsuffix` value of this dependency. +If this is not specified, it is assumed to be the empty string (`''`). + +Note how we use the '`%(pyver)s'` template value in the `SciPy-bundle` dependency +specification, to avoid hardcoding the Python version in different places. + +See also the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#dependencies) for additional options on specifying dependencies. + + +#### Version suffix + +In some cases you may want to build a particular software package in different configurations, or include a label in the module name to highlight a particular aspect +of the installation. + +The `versionsuffix` easyconfig parameter can be used for this purpose. The name of this parameter implies that this label will be added after the +software version (and toolchain label) in the standard module naming scheme. + +If you are configuring the software to build with a particular non-default value, +you can indicate this as follows for example: + +```python +versionsuffix = '-example-label' +``` + +This mechanism is used frequently to indicate that a software installation depends +on a particular version of Python, where the `%(pyver)s` template comes in useful again: + +```python +versionsuffix = '-Python-%(pyver)s' +... +dependencies = [('Python', '2.7.18')] +``` + +Even though Python 2 is officially dead and +buried some scientific software still requires it, and mixing modules where +some use Python 2 and other use Python 3 doesn't work well. +The `versionsuffix` label is helpful to inform the user that a particular Python version is required by the installation. + +#### Customizing configure, build, test, and install commands + +When using a generic easyblock like `ConfigureMake` or `CMakeMake`, you will often +find yourself having to specify options to the configure, build, test, or install commands, or to inject additional commands right before them. + +For this the following standard easyconfig parameters are available: + +* `preconfigopts`: string value that is glued *before* the configure command; +* `configopts`: string value that is added *after* the configure command, which can be used to specify configuration options; + +Equivalent easyconfig parameters are available for the `build`, `test` and `install` steps: `prebuildopts`, `buildopts`, `pretestopts`, `testopts`, `preinstallopts`, and `installopts`. + +Here is a fictitious example of how they can be used: + +```python +easyblock = 'ConfigureMake' +... +dependencies = [('HDF5', '1.10.6')] +... +configopts = '--enable-hdf5-support' + +prebuildopts = 'export HDF5_PREFIX="$EBROOTHDF5" && ' + +installopts = "PREFIX='%(installdir)s'" +``` + +Here we are: + +* Adding the `--enable-hdf5-support` configure option, to convince the `ConfigureMake` easyblock to run the following + command during the configure step: + ```shell + ./configure --prefix ... --enable-hdf5-support + ``` + (where the '`...`' represents the path to installation directory where the software should be installed). + +* Specifying that an additional command has to be run before running `make` in the build step. We use '`&&`' to glue the + command to the `make` command, so `make` will only be run if the command we specified ran correctly. So, the build step will run something like: + ```shell + export HDF5_PREFIX="$EBROOTHDF5" && make -j 4 + ``` + The '`4`' value passed to the `-j` option shown here, which specifies how many commands `make` can run in parallel, is automatically determined by EasyBuild based on the number of available cores (taking into account `ulimit` settings, and cpuset and cgroup restrictions). + +* Passing the location where the software should be installed via the `PREFIX` argument to the `make install` command during the installation step. This results in the + following command being run: + ```shell + make install PREFIX=... + ``` + (where the '`...`' again represents the path to installation directory). + Even though the + installation directory is already specified in the configure command, it is + apparently blatantly ignored by the software we are installing here, and we are expected to specify it + this way instead. How rude! + +The `$EBROOTHDF5` environment variable that we are using in `prebuildopts` corresponds to the path of +the installation directory of the HDF5 dependency. EasyBuild includes a statement +to define an `$EBROOT*` environment variable +like this in every environment module file it generates (see the output of "`module show HDF5`"). + +#### Sanity check + +One seemingly trivial yet important aspect of the installation procedure that EasyBuild performs +is the sanity check step. + +By default EasyBuild does a simple sanity check that verifies whether there is a non-empty `bin` subdirectory +in the installation, next to a non-empty `lib` or `lib64` directory (either is sufficient). + +It is recommended to customize the sanity check and check for something more specific, like a particular +binary or directory, or making sure that a trivial command (like `example -V` or `example --help`) +runs correctly. + +To specify a custom set of files and/or directories to check, +you can use the `sanity_check_paths` easyconfig parameter. The expected value is Python dictionary +with two keys: `files` and `dirs`. For example: + +```python +sanity_check_paths = { + 'files': ['bin/example'], + 'dirs': ['examples/one', 'examples/two'], +} +``` + +In addition, you can specify one or more commands that should be working without a problem (that is, have a zero exit status) via the `sanity_check_commands` easyconfig parameter. +These commands will be run just like a user would: after loading the module that was generated for this installation. +Here is an example: + +```python +sanity_check_commands = [ + "example --version", + "example --help", +] +``` + + +#### Module class + +Finally, you will usually see the `moduleclass` easyconfig parameter to be defined as well, for example: + +```python +moduleclass = 'lib' +``` + +This is done to categorize software, and it is used to group the generated module files into smaller sets ([remember what we saw when installing software earlier](../basic_usage/#using-installed-software)). + +## Generating tweaked easyconfigs + +Sometimes you may want to install software that differs only slightly from an +existing easyconfig file, like a newer software version or using a different +compiler toolchain. Do we need to create an easyconfig file for this too? + +We do, but EasyBuild does provide some help so you don't need to *manually* +create the easyconfig file. You can use one of the `--try-*` options provided +by the `eb` command to make EasyBuild *generate* a new easyconfig file based on +an existing one. + +For example, to try installing a different software version you can use the `--try-software-version` option: + +```shell +eb example-1.2.3.eb --try-software-version 1.2.4 +``` + +Or, to try using a different compiler toolchain you can use `--try-toolchain`: + +```shell +eb example-1.2.3-foss-2020b.eb --try-toolchain intel,2020b +``` + +It is important to keep in mind the *"try"* aspect here: while easyconfigs that +are generated by EasyBuild via a `--try-*` option often do work fine, there is +no strong guarantee they will. Newer software versions may come with changes to +the installation procedure, additional dependencies that are required, etc. +Using a different compiler toolchain may be as simple as just switching one for +another, but it may require additional changes to be made to configure options, for example. + +## Copying easyconfigs + +One additional handy command line option we want to highlight is `--copy-ec`, which can be used to +copy easyconfig files to a specific location. That may sound trivial, but +keep in mind that you can specify easyconfigs to the `eb` command using only +the filename, and letting the robot search mechanism locate them. + +So to copy an easyconfig file, we would have to use `eb --search` first to +get the full location to it, copy-paste that, and then use the `cp` command. + +It is a lot easier with `--copy-ec`: + +```shell +$ eb --copy-ec SAMtools-1.11-GCC-10.2.0.eb SAMtools.eb +... +SAMtools-1.10-GCC-10.2.0.eb copied to SAMtools.eb +``` + +If you omit the target location, the easyconfig file will simply be copied +to the current working directory, retaining the original filename. + +You can copy multiple easyconfig files at once, as long as the target location +is an existing directory. + +## Example + +By means of example, we are going to puzzle together an easyconfig file to install the +example software package `eb-tutorial`. + +The sources for `eb-tutorial` version 1.0.1 are available at: + +``` +https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.0.1.tar.gz +``` + +You can consult the unpacked sources at [https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1). + +### Preparation + +Make sure EasyBuild is properly configured before you start: + +```shell +export EASYBUILD_PREFIX=$HOME/easybuild +export EASYBUILD_BUILDPATH=/tmp/$USER +``` + +and that the installed software in `/easybuild` is available: + +```shell +module use /easybuild/modules/all +``` + +### Mandatory parameters + +Let's start by getting the mandatory easyconfig parameters defined in the easyconfig file: + +```python +name = 'eb-tutorial' +version = '1.0.1' + +homepage = 'https://easybuilders.github.io/easybuild-tutorial' +description = "EasyBuild tutorial example" +``` + +If we try using this (very) minimal easyconfig file, EasyBuild will inform us that we failed to specify one of the mandatory easyconfig parameters: `toolchain`: + +``` +$ eb example.eb +== Temporary log file in case of crash /tmp/eb-90j723rl/easybuild-q21plqvx.log +== found valid index for /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs, so using it... +ERROR: Failed to process easyconfig /home/example/example.eb: mandatory parameters not provided in pyheader: toolchain +``` + +We will use `GCC/10.2.0` as toolchain, since we know it is already installed in `/easybuild` in the prepared environment, so we also define the `toolchain` easyconfig parameter: + +```python +toolchain = {'name': 'GCC', 'version': '10.2.0'} +``` + +In addition, we'll also specify the `moduleclass`. +This is not required, but it is usually set to a sensible value: + +```python +moduleclass = 'tools' +``` + +The default value is '`base`', at least '`tools`' has *some* meaning. + +### Easyblock + +Let us see what happens if we take our current easyconfig file for a spin: + +```shell +$ eb example.eb +== temporary log file in case of crash /tmp/eb-8_vxjfn7/easybuild-k3aaoan2.log +ERROR: Failed to process easyconfig /home/example/example.eb: +No software-specific easyblock 'EB_eb_minus_tutorial' found for eb-tutorial +``` + +That didn't get us very far... + +The error shows that there is no software-specific easyblock available for installing the software with the name '`eb-tutorial`'. +Does that mean we have to implement an easyblock? + +In this simple case it doesn't, since we can leverage one of the available *generic easyblocks*. +But, which one? + +Build instructions are usually included in a `README` file, or in the documentation. +In this case, there's indeed a minimal [`README` +file](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1/README) available, +which tells us that we should use the `cmake` command to configure the installation, followed by `make` and `make install`. + +[We briefly discussed](#easyblock) a generic easyblock that does exactly this: `CMakeMake`. + +```python +easyblock = 'CMakeMake' +``` + +The "`easyblock =`" line is usually at the top of the easyconfig file, but strictly speaking +the order of the parameter definitions doesn't matter (unless one is defined in terms of another one). + +### CMake build dependency + +Does using the `CMakeMake` generic easyblock help at all? + +``` +$ eb example.eb +== temporary log file in case of crash /tmp/eb-yutbor1p/easybuild-4jc9v1u9.log +== found valid index for /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs, so using it... +== processing EasyBuild easyconfig /home/example/example.eb +== building and installing eb-tutorial/1.0.1-GCC-10.2.0... +== fetching files... +== creating build dir, resetting environment... +== unpacking... +== patching... +== preparing... +== configuring... +== FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/ebtutorial/1.0.1/GCC-10.2.0): +build failed (first 300 chars): cmd " cmake -DCMAKE_INSTALL_PREFIX=/home/example/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER='gcc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno' -DCMAKE_CXX_COMPILER='g++' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=native (took 0 sec) +``` + +It did help: EasyBuild made an attempt to configure the build using the `cmake` command, but that failed almost +instantly. We need to dive into the log file to see the actual reason. By starting at the end of the log file and +scrolling up, you should be able to locate the following error message: + +``` +/bin/bash: cmake: command not found +``` + +Ah, that explains it, `cmake` isn't even installed on this system. Or is it? + +```shell +$ module avail CMake + +--------------------------- /easybuild/modules/all ---------------------------- + CMake/3.18.4-GCCcore-10.2.0 +``` + +Since a module is available for `CMake` that is compatible with the toolchain we are using (GCC 10.2.0), +we can use it as a dependency for the installation. +It is only needed for building the software, not for running it, so it's only a *build* dependency: + +```python +builddependencies = [('CMake', '3.18.4')] +``` + +There is usually no need to specify toolchain for (build) dependencies, EasyBuild will automatically consider +[subtoolchains](../terminology/#toolchains) compatible with the specified toolchain to locate module for the dependencies. + +You can verify this via `eb -D` (equivalent with `eb --dry-run`): + +``` +$ eb example.eb -D + ... + * [x] /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs/g/GCC/GCC-10.2.0.eb (module: GCC/10.2.0) + * [x] /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs/c/CMake/CMake-3.18.4-GCCcore-10.2.0.eb (module: CMake/3.18.4-GCCcore-10.2.0) + * [ ] /home/example/example.eb (module: eb-tutorial/1.0.1-GCC-10.2.0) +``` + +### Sources + +If you try again after adding `CMake` as a build dependency, you will see the installation fail again in the +configuration step. Inspecting the log file reveals this: + +``` +CMake Error: The source directory "/tmp/example/ebtutorial/1.0.1/GCC-10.2.0" does not appear to contain CMakeLists.txt. +``` + +Wait, but there *is* a `CMakeLists.txt`, we can see it in the [unpacked sources](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1)! + +Let's inspect the build directory: + +``` +$ ls /tmp/$USER/ebtutorial/1.0.1/GCC-10.2.0 +easybuild_obj +$ ls /tmp/$USER/ebtutorial/1.0.1/GCC-10.2.0/easybuild_obj +$ +``` + +There's nothing there at all! And that's not strange because we didn't actually +specify any sources in our easyconfig file... + +The `sources` easyconfig parameter is commonly defined but it is *not* mandatory, +because some easyconfig files only specify bundles of software packages and hence only +serve to generate a module file. + +So we need to specify one or more source files that should be used, +via the `sources` easyconfig parameter which specifies a *list* of +names of source files: + +```python +sources = ['eb-tutorial-1.0.1.tar.gz'] +``` + +We can avoid hardcoding the version number here by using a *template value*: + +```python +sources = ['eb-tutorial-%(version)s.tar.gz'] +``` + +And since this is a standard way of naming software files, there's +even a constant available that we can use: + +```python +sources = [SOURCE_TAR_GZ] +``` + +That way, we only have the software version specified *once* in the easyconfig file, +via the `version` easyconfig parameter. That will come in useful later (see [Exercise 7.2](#exercises))... + +If now we try installing the easyconfig file again, EasyBuild complains +that it can't find the specified source file anywhere: + +``` +Couldn't find file eb-tutorial-1.0.1.tar.gz anywhere, and downloading it didn't work either... +``` + +To let EasyBuild automatically download the source file if it is not available yet, +we have to specify *where* it can be downloaded. This is done via `source_urls`: + +```python +source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] +sources = [SOURCE_TAR_GZ] +``` + +### Required configure option + +With `sources` and `source_urls` defined, we can try again. Yet again we see the configure step fail. +Is this a ruse to make you hate CMake with a passion? Maybe... + +Here's what we find in the log file: + +``` +CMake Error at CMakeLists.txt:7 (message): + EBTUTORIAL_MSG is not set! +``` + +Apparently the `eb-tutorial` software has a required configure option. It's almost as if that +was done on purpose, how silly! + +Options to the configure command can be specified by the `configopts` easyconfig parameter. +To define the value of a CMake option, we need to use `-DNAME_OF_OPTION`, so: + +```python +configopts = "-DEBTUTORIAL_MSG='Hello from the EasyBuild tutorial!' " +``` + +We need to be a little bit careful with quotes here. If we use outer double quotes, +we have to use single quotes to specify the actual value for the `EBTUTORIAL_MSG` configure option. +That works fine here, but that's not always the case! +In some cases we will have to use inner doubles quotes, for example to get environment variables +expanded when the configure command is run (see [Exercise 7.1](#exercises)). + +### Sanity check + +Hopefully that brings us closer to getting the installation to work... + +``` +$ eb example.eb +.... +== sanity checking... +== FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/ebtutorial/1.0.1/GCC-10.2.0): build failed (first 300 chars): +Sanity check failed: no (non-empty) directory found at 'lib' or 'lib64' in /home/easybuild/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 (took 2 sec) +``` + +It got all the way to the sanity check step, that's great! + +The sanity check failed because no '`lib`' or `'lib64'` directory was found. +Indeed: + +``` +$ ls $HOME/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 +bin +$ ls $HOME/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0/bin +eb-tutorial +``` + +There is only a binary named `eb-tutorial` in the `bin` subdirectory. +So we need to customize the standard sanity check: + +```python +sanity_check_paths = { + 'files': ['bin/eb-tutorial'], + 'dirs': [], +} +``` + +Since we want to obtain a *working* installation, we might as well try to run this `eb-tutorial` command as well: + +```python +sanity_check_commands = ['eb-tutorial'] +``` + +Let us now retry, but use `--module-only` rather than redoing the whole installation. +`--module-only` still sanity checks the installation, so if it creates +a module, we know it will work as expected. +By enabling trace mode via `--trace` we can get some more information too: + +```shell +$ eb example.eb --module-only --trace +... +== sanity checking... + >> file 'bin/eb-tutorial' found: OK + >> running command 'eb-tutorial' ... + >> result for command 'eb-tutorial': OK +... +== COMPLETED: Installation ended successfully (took 4 sec) +``` + +Yes, great success! + +To convince yourself that the installation works as intended, try to load the `eb-tutorial` module and +run the `eb-tutorial` command yourself: + +``` +$ module use $HOME/easybuild/modules/all +$ module load eb-tutorial +$ eb-tutorial +Hello from the EasyBuild tutorial! +``` + +### Complete easyconfig + +Here is the complete easyconfig we puzzled together for this example: + +```python +easyblock = 'CMakeMake' + +name = 'eb-tutorial' +version = '1.0.1' + +homepage = 'https://easybuilders.github.io/easybuild-tutorial' +description = "EasyBuild tutorial example" + +toolchain = {'name': 'GCC', 'version': '10.2.0'} + +source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] +sources = [SOURCE_TAR_GZ] +checksums = ['d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9'] + +builddependencies = [('CMake', '3.18.4')] + +configopts = "-DEBTUTORIAL_MSG='Hello from the EasyBuild tutorial!' " + +sanity_check_paths = { + 'files': ['bin/eb-tutorial'], + 'dirs': [], +} + +sanity_check_commands = ['eb-tutorial'] + +moduleclass = 'tools' +``` + +One additional easyconfig parameter we defined here is `checksums`, +which specifies SHA256 checksums for source (and patch) files, +so EasyBuild can verify them before performing an installation. + +You can let EasyBuild determine *and* inject these SHA256 checksums +automatically via `eb --inject-checksums`: + +``` +$ eb example.eb --inject-checksums +... +== injecting sha256 checksums for sources & patches in example.eb... +== * eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 +``` + +--- + +## Exercises + +--- + +***Exercise E.1**** - Making `eb-tutorial` a bit more personal* + +Change the easyconfig file for `eb-tutorial` to make the message printed by the `eb-tutorial` command +a bit more personal: include the username of the account that was used to install the software in it +(using the `$USER` environment variable). + +??? success "(click to show solution)" + For this we need to change the value that is passed to the `EBTUTORIAL_MSG` configure option: + ```python + configopts = '-DEBTUTORIAL_MSG="Hello from the EasyBuild tutorial! I was installed by $USER." ' + ``` + Here we have to use inner double quotes, to ensure that the `$USER` environment variable is expanded + by the shell when running the `cmake` configure command. + + When you run the `eb-tutorial` command yourself, you should get output like this (not a message that + includes a literal '`$USER`' string): + + ```shell + Hello from the EasyBuild tutorial! I was installed by example. + ``` + + To re-install the `example.eb` easyconfig, you will need to use `eb --rebuild` or `eb --force`. + +--- + +***Exercise E.2**** - Installing eb-tutorial version 1.1.0* + +Install version 1.1.0 of the `eb-tutorial` example software, +which is a trivial version bump compared to version 1.0.1. + +The sources are available via: + +``` +https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.1.0.tar.gz +``` + +You can leverage the `eb-tutorial` easyconfig file we have composed in the example above, +**but you should not make any manual changes to it!** + +??? success "(click to show solution)" + You can use the `--try-software-version` option for this: + ```shell + $ eb example.eb --try-software-version 1.1.0 + ... + == building and installing eb-tutorial/1.1.0-GCC-10.2.0... + ... + == COMPLETED: Installation ended successfully (took 4 sec) + ``` + To test: + ``` + $ module use $HOME/easybuild/modules/all + $ module load eb-tutorial/1.1.0-GCC-10.2.0 + $ eb-tutorial + I have a message for you: + Hello from the EasyBuild tutorial! + ``` + (`eb-tutorial` version 1.0.1 doesn't print "`I have a message for you:`") + +--- + +***Exercise E.3**** - Installing py-eb-tutorial 1.0.0* + +Try composing an easyconfig file for the `py-eb-tutorial` example software, which is a tiny Python package. +The source tarball can be downloaded from this link: [py-eb-tutorial-1.0.0.tar.gz](https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/py-eb-tutorial-1.0.0.tar.gz). + +A couple of tips: + +* There is a [generic easyblock](#easyblock) available for installing Python packages, which will come in useful here. + +* By default EasyBuild performs an `import` check when install Python packages, using a Python module name that is derived from the software name by default, which will be incorrect in this case. You can specify the correct name to use in the import check by specifying it via the `options` +easyconfig parameter in your easyconfig file: + ```python + options = {'modulename': 'example'} + ``` + (you will need to change '`example`' here, of course) + +* Leverage the software that is already pre-installed in `/easybuild` in the prepared environment. + Remember that some already installed modules may be a *bundle* of a couple of other software packages. + +Please also take this into account: + +* Unfortunately this software doesn't come with documentation. That is done to make it an example that + is representative for software that you may run into in the wild (it's *not* because + we were lazy when preparing the exercises, really!). + You can inspect the sources of this software [here](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/py-eb-tutorial-1.0.0). Definitely take a look at the `setup.py` file, it includes some clues + about the requirements to get this software installed. + +* Make sure the installation actually *works*, by checking that the `py-eb-tutorial` command runs correctly. + Maybe you will need to make sure other required software is available as well, for it to work correctly... + + +??? success "(click to show solution)" + + Here is a complete working easyconfig file for `py-eb-tutorial`: + ```python + + easyblock = 'PythonPackage' + + name = 'py-eb-tutorial' + version = '1.0.0' + versionsuffix = '-Python-%(pyver)s' + + homepage = 'https://easybuilders.github.io/easybuild-tutorial' + description = "EasyBuild tutorial Python example" + + source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] + sources = [SOURCE_TAR_GZ] + checksums = ['fcf73a9efc65527a210b993e8889d41ebf05977eef1f6a65ebac3188152cd496'] + + toolchain = {'name': 'foss', 'version': '2020b'} + + dependencies = [ + ('Python', '3.8.6'), + ('SciPy-bundle', '2020.11'), + ('eb-tutorial', '1.0.1'), + ] + + use_pip = True + + options = {'modulename': 'eb_tutorial'} + + sanity_check_paths = { + 'files': ['bin/py-eb-tutorial'], + 'dirs': ['lib/python%(pyshortver)s/site-packages'], + } + + sanity_check_commands = ["py-eb-tutorial"] + + moduleclass = 'tools' + ``` + + Some remarks: + + * We used the `PythonPackage` generic easyblock. There is also a `PythonBundle` easyblock for installing + bundles of Python packages, which is used for `SciPy-bundle` for example. But we don't need that here, + since we are only dealing with a single Python package. + + * The `versionsuffix` is not strictly needed, but it's common to tag Python packages with the Python version + for which they were installed. + * The SHA256 checksum for the source tarball was added automatically via `eb py-eb-tutorial.eb --inject-checksums`. + + * `py-eb-tutorial` only wants to be installed with `pip install`, so we had to set `use_pip = True`. + You can consult the custom easyconfig parameters supported by the `PythonPackage` easyblock via + "`eb -a -e PythonPackage`", see the `EASYBLOCK-SPECIFIC` part of the output. + Even when the default installation mechanism used by `PythonPackage` + (which consists of running `python setup.py install`) works fine, + it is recommended to instruct EasyBuild to use `pip install` instead. + + * By default EasyBuild will try to import `py_eb_tutorial`, while the actual name of the Python package + provided by `py-eb-tutorial` is just `eb_tutorial`. We fixed this by specifying the correct Python module name to + use via `options`. + + * Strictly speaking we don't need to specify a custom `sanity_check_paths`, since the default used + by Python package is already pretty decent (it will check for a non-empty `lib/python3.8/site-packages` + directory in the installation). We also want to make sure the `py-eb-tutorial` command is available in + the `bin` subdirectory however. Hardcoding to `python3.8` can be avoided using the `%(pyshortver)s` + template value. + + * A good way to check whether the `py-eb-tutorial` command works correctly is by running it as a sanity check + command. If the `eb-tutorial` command is not available the `py-eb-tutorial` command will fail, + since it basically just runs the `eb-tutorial` command. So we need to include `eb-tutorial` as a (runtime) + dependency in the `py-eb-tutorial` easyconfig file. + +*[[next: Implementing easyblocks]](2_03_implementing_easyblocks.md)* diff --git a/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md new file mode 100644 index 00000000..5355548e --- /dev/null +++ b/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md @@ -0,0 +1,665 @@ +# Implementing easyblocks + +*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* + +--- + +## The basics + +An *easyblock* is a Python module that implements a software build and installation procedure. + +This concept is essentially implemented as a Python script that plugs into the EasyBuild framework. + +EasyBuild will leverage easyblocks as needed, depending on which software packages it needs to install. Which easyblock is required is determined by the ``easyblock`` easyconfig parameter, if it is present, or by the software name. + + +## Generic vs software-specific easyblocks + +Easyblocks can either be *generic* or *software-specific*. + +*Generic* easyblocks implement a "standard" software build and installation procedure that is used by multiple different +software packages. +A commonly used example is the +[``ConfigureMake``](https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/generic/configuremake.py) +generic easyblock, which implements the standard ``configure`` - ``make`` - ``make install`` installation procedure used +by most GNU software packages. + +*Software-specific* easyblocks implement the build and installation procedure for a particular software package. +Typically this involves highly customised steps, for example specifying dedicated configuration options, creating +or adjusting specific files, executing non-standard shell commands, etc. Usually a custom implementation of the +sanity check is also included. + +Using a generic easyblock requires specifying the ``easyblock`` parameter in the easyconfig file. +If it is not specified, EasyBuild will try and find the software-specific easyblock derived from the software name. + +The distinction between generic and software-specific easyblocks can be made based on the naming scheme that is used +for an easyblock (see below). + + +## Naming + +Easyblocks need to follow a strict naming scheme, to ensure that EasyBuild can pick them up as needed. This involves two aspects: + +* the name of the Python class; +* the name and location of the Python module file. + +### Python class name + +The name of the Python class is determined by the *software name* for software-specific easyblocks. +It consists of a prefix '``EB_``', followed by the (encoded) software name. + +Because of limitations in Python on characters allowed in names of Python classes, +only alphanumeric characters and underscores (``_``) are allowed. Any other characters are replaced following an encoding scheme: + +* spaces are replaced by underscores (``_``); +* dashes ``-`` are replaced by ``_minus_``; +* underscores are replaced by ``_underscore_``; + +The ``encode_class_name`` function provided in ``easybuild.tools.filetools`` returns the expected class name +for a given software name; for example: + +```shell +$ python3 -c "from easybuild.tools.filetools import encode_class_name; print(encode_class_name('netCDF-Fortran'))" +EB_netCDF_minus_Fortran +``` + +**Python class name for *generic* easyblocks** + +For *generic* easyblocks, the class name does *not* include an ``EB_`` prefix (since there is no need for an escaping +mechanism) and hence the name is fully free to choose, taking into account the restriction to alphanumeric characters +and underscores. + +For code style reasons, the class name should start with a capital letter and use CamelCasing. + +Examples include ``Bundle``, ``ConfigureMake``, ``CMakePythonPackage``. + +### Python module name and location + +The *filename* of the Python module is directly related to the name of Python class it provides. + +It should: + +* *not* include the ``EB_`` prefix of the class name for software-specific easyblocks; +* consists only of lower-case alphanumeric characters (``[a-z0-9]``) and underscores (``_``); + * dashes (``-``) are replaced by underscores (``_``); + * any other non-alphanumeric characters (incl. spaces) are simply dropped; + +Examples include: + +* ``gcc.py`` (for *GCC*) +* ``netcdf_fortran.py`` (for *netCDF-Fortran*) +* ``gamess_us.py`` (for *GAMESS (US)*) + +The ``get_module_path`` function provided by the EasyBuild framework in the +``easybuild.framework.easyconfig.easyconfig`` module returns the (full) +module location for a particular software name or easyblock class name. For example: + +```python +>>> from easybuild.framework.easyconfig.easyconfig import get_module_path +>>> get_module_path('netCDF-Fortran') +'easybuild.easyblocks.netcdf_fortran' +>>> get_module_path('EB_netCDF_minus_Fortran') +'easybuild.easyblocks.netcdf_fortran' +``` + +The location of the Python module is determined by whether the easyblock is generic or software-specific. +Generic easyblocks are located in the ``easybuid.easyblocks.generic`` namespace, while software-specific easyblocks +live in the ``easybuild.easyblocks`` namespace directly. + +To keep things organised, the actual Python module files +for software-specific easyblocks are kept in 'letter' subdirectories, +rather than in one large '``easyblocks``' directory +(see +[https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/](https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/)), +but this namespace is collapsed transparently by EasyBuild (you don't need to import from letter subpackages). + +To let EasyBuild pick up one or more new or customized easyblocks, you can use the [``--include-easyblocks``](https://docs.easybuild.io/en/latest/Including_additional_Python_modules.html#including-additional-easyblocks-include-easyblocks) +configuration option. As long as both the filename of the Python module and the name of the Python class +are correct, EasyBuild will use these easyblocks when needed. + +## Structure of an easyblock + +The example below shows the overal structure of an easyblock: + +```python +from easybuild.framework.easyblock import EasyBlock +from easybuild.tools.run import run_cmd + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def configure_step(self): + """Custom implementation of configure step for Example""" + + # run configure.sh to configure the build + run_cmd("./configure.sh --install-prefix=%s" % self.installdir) +``` + + +Each easyblock includes an implementation of a ``class`` that (directly or indirectly) derives from the abstract +``EasyBlock`` class. + +Typically some useful functions provided by the EasyBuild framework are imported at the top of the Python module. + +In the class definition, one or more '``*_step``' methods (and perhaps a couple of others) are redefined, +to implement the corresponding step in the build and installation procedure. + +Each easyblock *must* implement the ``configure``, ``build`` and ``install`` steps, since these are not implemented +in the abstract ``EasyBlock`` class. This could be done explicitly by redefining the corresponding ``*_step`` methods, +or implicitly by deriving from existing (generic) easyblocks. + +The full list of methods that can be redefined in an easyblock can be consulted in +the [API documentation](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyblock.html). + +## Deriving from existing easyblocks + +When implementing an easyblock, it is common to derive from an existing (usually generic) easyblock, +and to leverage the functionality provided by it. This approach is typically used when only a specific part +of the build and installation procedure needs to be customised. + +In the (fictitious) example below, we derive from the generic ``ConfigureMake`` easyblock to redefine the ``configure`` +step. In this case, we are *extending* the ``configure`` step as implemented by ``ConfigureMake`` rather than +redefining it entirely, since we call out to the original ``configure_step`` method at the end. + +```python +from easybuild.easyblocks.generic.configuremake import ConfigureMake +from easybuild.tools.filetools import copy_file + + +class EB_Example(ConfigureMake): + """Custom easyblock for Example""" + + def configure_step(self): + """Custom implementation of configure step for Example""" + + # use example make.cfg for x86-64 + copy_file('make.cfg.x86', 'make.cfg') + + # call out to original configure_step implementation of ConfigureMake easyblock + super(EB_Example, self).configure_step() +``` + +## Easyconfig parameters + +All of the easyconfig parameters that are defined in an easyconfig file +are available via the ``EasyConfig`` instance that can be accessed through ``self.cfg`` in an easyblock. +For instance, if the easyconfig file specifies + +```python +name = 'example' +version = '2.5.3' +versionsuffix = '-Python-3.7.4' +``` + +then these three parameters are accessible within an easyblock via ``self.cfg['name']``, ``self.cfg['version']`` +and ``self.cfg['versionsuffix']``. + +A few of the most commonly used parameters can be referenced directly: + +* **``self.name``** is equivalent with ``self.cfg['name']``; +* **``self.version``** is equivalent with ``self.cfg['version']``; +* **``self.toolchain``** is equivalent with ``self.cfg['toolchain']``; + + +### Updating parameters + +You will often find that you need to *update* some easyconfig parameters in an easyblock, +for example ``configopts`` which specifies options for the configure command. + +Because of implementation details (related to +how template values like ``%(version)s`` are handled), you need to be a bit careful here... + +To completely redefine the value of an easyconfig parameter, you can use simple assignment. For example: + +```python +self.cfg['example'] = "A new value for the example easyconfig parameter." +``` + +If want to *add* to the existing value however, you *must* use the ``self.cfg.update`` method. For example: + +```python +self.cfg.update('some_list', 'example') +``` + +This will ***not*** work (because ``self.cfg['some_list']`` does not return a reference to the original value, +but to a *temporary copy* thereof): + +```python +# anti-pattern, this does NOT work as expected! +self.cfg['some_list'].append('example') +``` + +### Custom parameters + +Additional custom easyconfig parameters can be defined in an easyblock to allowing steering its behaviour. +This is done via the ``extra_options`` *static* method. Custom parameters can be specified to be mandatory. + +The example below shows how this can be implemented: + +```python +from easybuild.easyblocks.generic.configuremake import ConfigureMake +from easybuild.framework.easyconfig import CUSTOM, MANDATORY + + +class EB_Example(ConfigureMake): + """Custom easyblock for Example""" + + @staticmethod + def extra_options(): + """Custom easyconfig parameters for Example""" + extra_vars = { + 'required_example_param': [None, "Example required custom parameter", MANDATORY], + 'optional_example_param': [None, "Example optional custom parameter", CUSTOM], + } + return ConfigureMake.extra_options(extra_vars) +``` + +The first element in the list of a defined custom parameter corresponds to the default value for that parameter +(both ``None`` in the example above). The second element provides some informative help text, and the last element +indicates whether the parameter is mandatory (``MANDATORY``) or just an optional custom parameter (``CUSTOM``). + +## Easyblock constructor + +In the ``class`` constructor of the easyblock, i.e. the ``__init__`` method, one or more class variables +can be initialised. These can be used for sharing information between different ``*_step`` methods in the easyblock. + +For example: + +```python +from easybuild.framework.easyblock import EasyBlock + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def __init__(self, *args, **kwargs): + """Constructor for Example easyblock, initialises class variables.""" + + # call out to original constructor first, so 'self' (i.e. the class instance) is initialised + super(EB_Example, self).__init__(*args, **kwargs) + + # initialise class variables + self.example_value = None + self.example_list = [] +``` + +## File operations + +File operations is a common use case for implementing easyblocks, hence the EasyBuild framework provides a +number of useful functions related to this, including: + +* ``read_file()``: read file at a specified location and returns its contents; + +* ``write_file(, )`` at a specified location with provided contents; + to append to an existing file, use ``append=True`` as an extra argument; + +* ``copy_file(, )`` to copy an existing file; + +* ``apply_regex_substitutions(, )`` to patch an existing file; + +All of these functions are provided by the [``easybuild.tools.filetools``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html) module. + +## Executing shell commands + +For executing shell commands two functions are provided by the +[``easybuild.tools.run``](https://docs.easybuild.io/en/latest/api/easybuild.tools.run.html) module: + +* ``run_cmd()`` to run a non-interactive shell command; + +* ``run_cmd_qa(, )`` to run an interactive shell command; + +Both of these accept a number of optional arguments: + +* ``simple=True`` to just return ``True`` or ``False`` to indicate a successful execution, + rather than the default return value, i.e., a tuple that provides the command output and the exit code (in that + order); + +* ``path=`` to run the command in a specific subdirectory; + +The ``run_cmd_qa`` function takes two additional specific arguments: + +* ``no_qa=`` to specify a list of patterns to recognize non-questions; + +* ``std_qa=`` to specify regular expression patterns for common questions, and the matching answer; + +## Manipulating environment variables + +To (re)define environment variables, the ``setvar`` function provided by the +[``easybuild.tools.environment``](https://docs.easybuild.io/en/latest/api/easybuild.tools.environment.html) +module should be used. + +This makes sure that the changes being made to the specified environment variable are kept track of, +and that they are handled correctly under ``--extended-dry-run``. + +## Logging and errors + +It is good practice to include meaningful log messages in the ``*_step`` methods being customised in the easyblock, +to enrich the EasyBuild log with useful information for later debugging or diagnostics. + +For logging, the provided ``self.log`` logger class should be used. +You can use the ``self.log.info`` method to log an informative message. +Similar methods are available for logging debug messages (``self.log.debug``), which are +only emitted when ``eb`` is run with debugging mode enabled (``--debug`` or ``-d``), +and for logging warning messages (``self.log.warning``). + +If something goes wrong, you can raise an ``EasyBuildError`` instance to report the error. + +For example: + +```python +from easybuild.framework.easyblock import EasyBlock +from easybuild.tools.build_log import EasyBuildError +from easybuild.tools.run import run_cmd + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def configure_step(self): + """Custom implementation of configure step for Example""" + + cmd = "./configure --prefix %s" % self.installdir) + out, ec = run_cmd(cmd) + + success = 'SUCCESS' + if success in out: + self.log.info("Configuration command '%s' completed with success." % cmd) + else: + raise EasyBuildError("Pattern '%s' was not found in output of '%s'." % (success, cmd)) +``` + +## Custom sanity check + +For software-specific easyblocks, a custom sanity check is usually included to verify that the installation was +successful or not. + +This is done by redefining the ``sanity_check_step`` method in the easyblock. For example: + +```python +from easybuild.framework.easyblock import EasyBlock + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def sanity_check_step(self): + """Custom sanity check for Example.""" + + custom_paths = { + 'files': ['bin/example'], + 'dirs': ['lib/examples/'], + } + custom_commands = ['example --version'] + + # call out to parent to do the actual sanity checking, pass through custom paths and commands + super(EB_Example, self).sanity_check_step(custom_paths=custom_paths, custom_commands=custom_commands) +``` + +You can both specify file paths and subdirectories to check for, which are specified relative to the installation directory, +as well as simple commands that should execute successfully after completing the installation and loading the generated module file. + +It is up to you how extensive you make the sanity check, but it is recommended to make the check as complete +as possible to catch any potential build or installation problems that may occur, while ensuring that it can +run relatively quickly (in seconds, or at most a couple of minutes). + +## Version-specific parts + +In some cases version-specific actions or checks need to be included in an easyblock. +For this, it is recommended to use ``LooseVersion`` rather than directly comparing version numbers using string values. + +For example: + +```python +from distutils.version import LooseVersion +from easybuild.framework.easyblock import EasyBlock + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def sanity_check_step(self): + """Custom sanity check for Example.""" + + custom_paths = { + 'files': [], + 'dirs': [], + } + + # in older versions, the binary used to be named 'EXAMPLE' rather than 'example' + if LooseVersion(self.version) < LooseVersion('1.0'): + custom_paths['files'].append('bin/EXAMPLE') + else: + custom_paths['files'].append('bin/example') + + super(EB_Example, self).sanity_check_step(custom_paths=custom_paths) +``` + +## Compatibility with ``--extended-dry-run`` and ``--module-only`` + +Some special care must be taken to ensure that an easyblock is fully compatible with ``--extended-dry-run`` / ``-x`` +(see [Inspecting install procedures](../basic_usage/#inspecting-install-procedures)) and ``--module-only``. + +For compatibility with ``--extended-dry-run``, you need to take into account that specified operations +like manipulating files or running shell commands will not actually be executed. You can check +whether an easyblock is being run in dry run mode via ``self.dry_run``. + +For example: + +```python +from easybuild.framework.easyblock import EasyBlock +from easybuild.tools.build_log import EasyBuildError +from easybuild.tools.run import run_cmd + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def configure_step(self): + """Custom implementation of configure step for Example""" + + cmd = "./configure --prefix %s" % self.installdir) + out, ec = run_cmd(cmd) + + success = 'SUCCESS' + if success in out: + self.log.info("Configuration command '%s' completed with success." % cmd) + + # take into account that in dry run mode we won't get any output at all + elif self.dry_run: + self.log.info("Ignoring missing '%s' pattern since we're running in dry run mode." % success) + + else: + raise EasyBuildError("Pattern '%s' was not found in output of '%s'." % (success, cmd)) +``` + +For ``--module-only``, you should make sure that both the ``make_module_step``, including the ``make_module_*`` submethods, +and the ``sanity_check_step`` methods do not make any assumptions about the presence of certain environment variables, or that class variables have been defined already. + +This is required because under ``--module-only`` the large majority of the ``*_step`` functions are +simply *skipped* entirely. So, if the ``configure_step`` method is responsible for defining class variables that are +picked up in ``sanity_check_step``, the latter may run into unexpected initial values like ``None``. +A possible workaround is to define a separate custom method to define the class variables, and to call out to this +method from ``configure_step`` and ``sanity_check_step`` (for the latter, conditionally, i.e., only if the class +variables still have the initial values). + +For example: + +```python +from easybuild.framework.easyblock import EasyBlock + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def __init__(self, *args, **kwargs): + """Easyblock constructor.""" + super(EB_Example, self).__init__(*args, **kwargs) + + self.command = None + + def set_command(self): + """Initialize 'command' class variable.""" + # $CC environment variable set by 'prepare' step determines exact command + self.command = self.name + '-' + os.getenv('CC') + + def configure_step(self): + """Custom configure step for Example.""" + + self.set_command() + self.cfg.update('configopts', "COMMAND=%s" % self.command) + + super(EB_Example, self).configure_step() + + def sanity_check_step(self): + """Custom implementation of configure step for Example""" + + if self.command is None: + self.set_command() + + super(EB_Example, self).sanity_check_step(custom_commands=[self.command]) +``` + +## Exercise + +### Exercise I.1 + +Try implementing a new custom easyblock for ``eb-tutorial``, which derives directly +from the base ``EasyBlock`` class. + +Your easyblock should: + +* define a custom mandatory easyconfig parameter named ``message``; +* run `cmake` to configure the installation, which includes at least: + * specifying the correct installation prefix (using the `-DCMAKE_INSTALL_PREFIX=...` option); + * passing down the value of ``message`` easyconfig parameter via `-DEBTUTORIAL_MSG=...` +* run `make` to build `eb-tutorial`; +* run `make install` to install the generated binary; +* perform a custom sanity check to ensure the installation is correct; +* pick up on commonly used easyconfig parameters like `configopts` and `preinstallopts` where appropriate; + +??? success "(click to show solution)" + + Here's a complete custom easyblock for ``eb-tutorial`` that derives from the base ``EasyBlock`` class, + which should be included in a file named ``eb_tutorial.py``. + + We need to implement the ``configure_step``, ``build_step``, and ``install_step`` methods in + order to have a fully functional easyblock. + + The configure, build, and install steps take into account the corresponding easyconfig + parameters that allow customizing these commands from an easyconfig file. + + ```python + from easybuild.framework.easyblock import EasyBlock + from easybuild.framework.easyconfig import MANDATORY + from easybuild.tools.run import run_cmd + + + class EB_eb_minus_tutorial(EasyBlock): + """Custom easyblock for eb-tutorial.""" + + @staticmethod + def extra_options(): + extra = EasyBlock.extra_options() + extra.update({ + 'message': [None, "Message that eb-tutorial command should print", MANDATORY], + }) + return extra + + def configure_step(self): + """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" + + cmd = ' '.join([ + self.cfg['preconfigopts'], + "cmake", + "-DCMAKE_INSTALL_PREFIX='%s'" % self.installdir, + "-DEBTUTORIAL_MSG='%s'" % self.cfg['message'], + self.cfg['configopts'], + ]) + run_cmd(cmd) + + def build_step(self): + """Build step for eb-tutorial""" + + cmd = ' '.join([ + self.cfg['prebuildopts'], + "make", + self.cfg['buildopts'], + ]) + run_cmd(cmd) + + def install_step(self): + """Install step for eb-tutorial""" + + cmd = ' '.join([ + self.cfg['preinstallopts'], + "make install", + self.cfg['installopts'], + ]) + run_cmd(cmd) + + def sanity_check_step(self): + custom_paths = { + 'files': ['bin/eb-tutorial'], + 'dirs': [], + } + custom_commands = ['eb-tutorial'] + return super(EB_eb_minus_tutorial, self).sanity_check_step(custom_paths=custom_paths, + custom_commands=custom_commands) + ``` + +### Exercise I.2 + +Try implementing another new custom easyblock for ``eb-tutorial``, +which derives from the generic ``CMakeMake`` easyblock. + +Your easyblock should only: + +* define a custom mandatory easyconfig parameter named ``message``; +* pass down the value of ``message`` easyconfig parameter via `-DEBTUTORIAL_MSG=...` +* perform a custom sanity check to ensure the installation is correct; + +??? success "(click to show solution)" + + When deriving from the ``CMakeMake`` generic easyblock, there is a lot less to worry about. + + We only need to customize the ``configure_step`` method to ensure that the ``-DEBTUTORIAL_MSG`` configuration + option is specified; the ``CMakeMake`` easyblock already takes care of specifying the location of + the installation directory (and a bunch of other configuration options, like compiler commands and flags, etc.). + + Implementing the ``build_step`` and ``install_step`` methods is no longer needed, + the standard procedure that is run by the ``CMakeMake`` generic easyblock is fine, + and even goes beyond what we did in the previous exercise (like building in parallel with ``make -j``). + + ```python + from easybuild.easyblocks.generic.cmakemake import CMakeMake + from easybuild.framework.easyconfig import MANDATORY + from easybuild.tools.run import run_cmd + + class EB_eb_minus_tutorial(CMakeMake): + """Custom easyblock for eb-tutorial.""" + + @staticmethod + def extra_options(): + extra = CMakeMake.extra_options() + extra.update({ + 'message': [None, "Message that eb-tutorial command should print", MANDATORY], + }) + return extra + + def configure_step(self): + """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" + self.cfg.update('configopts', "-DEBTUTORIAL_MSG='%s'" % self.cfg['message']) + + super(EB_eb_minus_tutorial, self).configure_step() + + def sanity_check_step(self): + custom_paths = { + 'files': ['bin/eb-tutorial'], + 'dirs': [], + } + custom_commands = ['eb-tutorial'] + return super(EB_eb_minus_tutorial, self).sanity_check_step(custom_paths=custom_paths, + custom_commands=custom_commands) + ``` + +*[[next: PArt 3: OVerview]](3_00_part3_advanced.md)* diff --git a/docs/2022-CSC_and_LO/3_00_part3_advanced.md b/docs/2022-CSC_and_LO/3_00_part3_advanced.md new file mode 100644 index 00000000..708c57f8 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_00_part3_advanced.md @@ -0,0 +1,9 @@ +# Part III: Advanced topics + +*[[back to start page]](index.md)* + +* [Using EasyBuild as a library](3_01_easybuild_library.md) *(hands-on)* +* [Using hooks to customise EasyBuild](3_02_hooks.md) *(hands-on)* +* [Submitting installations as Slurm jobs](3_03_slurm_jobs.md) *(hands-on)* +* [Module naming schemes (incl. hierarchical)](3_04_module_naming_schemes.md) *(hands-on)* +* [GitHub integration to facilitate contributing to EasyBuild](3_05_github_integration.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/3_01_easybuild_library.md b/docs/2022-CSC_and_LO/3_01_easybuild_library.md new file mode 100644 index 00000000..c037f49f --- /dev/null +++ b/docs/2022-CSC_and_LO/3_01_easybuild_library.md @@ -0,0 +1,293 @@ +# Using EasyBuild as a library + +*[[back: Configuring EasyBuild]](1_07_configuration.md)* + +--- + +You can use EasyBuild as a Python library and implement a script to automate a particular task. + +All the functionality provided by the EasyBuild framework can be leveraged, +so you only have to glue things together and add the missing pieces. + + +## Setting up the EasyBuild configuration + +Before you call any of the functions provided by the EasyBuild framework, +you should initialize EasyBuild by **setting up the configuration**. +Some of the EasyBuild framework functions assume that EasyBuild has been initialized properly, +and calling them without doing so will result in errors. + +To set up the EasyBuild configuration, you should call the ``set_up_configuration`` function +that is provided by the ``easybuild.tools.options`` module. + +This function takes a couple of optional arguments that are useful in the context of +a Python script that uses EasyBuild as a library: + +* Via ``args`` you can provide a list of command line arguments that should be passed to the option parser. + By default, the standard command line arguments are picked up via ``sys.args``, which may not be what you want. +* By passing ``silent=True`` you can specify that the ``set_up_configuration`` function should not print + anything when it is called. By default, it will print the location to the temporary EasyBuild log file. + +### Cleaning up the temporary directory + +When EasyBuild is configured, a temporary directory specific to that EasyBuild session will be created automatically. +You should clean up that directory, especially if you will be creating temporary files, or if the script will be run +often. + +Note that cleaning up the temporary directory implies removes the temporary log files, +so you probably only want to do this if no errors occurred. + +### Minimal example script + +Here is a minimal (and pretty useless) example Python script, which sets up the EasyBuild configuration +and cleans up the temporary directory: + +```python +#!/usr/bin/env python3 +from easybuild.tools.filetools import remove_dir +from easybuild.tools.options import set_up_configuration + +opts, _ = set_up_configuration(args=[], silent=True) + +remove_dir(opts.tmpdir) +``` + + +## Example use cases + +Once the EasyBuild configuration has been set up, the functions provided by the EasyBuild framework +can be called from a Python script (or directly from the Python interpreter). + +A full overview of all functions is available via the [EasyBuild API documentation](https://docs.easybuild.io/en/latest/api/easybuild.html). + +We highlight a couple commonly used functions in the sections below. + +### File operations + +The [``easybuild.tools.filetools``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html) +module provides a bunch of functions for performing file operations. + +That includes straightforward things like reading, writing, and copying files +(see [``read_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.read_file), [``write_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.write_file), [``copy_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.copy_file)), but also more specific functionality like applying a patch file ([``apply_patch``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.apply_patch)), +computing different types of checksums for a file ([``compute_checksum``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.compute_checksum)), and downloading files +([``download_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.download_file)). + +Here is a trivial example where we download a file from a specified URL to a temporary directory, +compute the SHA256 checksum, and then copy the downloaded file to the current working directory: + +```python +#!/usr/bin/env python3 +import os +import sys +import tempfile + +from easybuild.tools.filetools import download_file, compute_checksum, copy_file, remove_dir +from easybuild.tools.options import set_up_configuration + +url = sys.argv[1] + +opts, _ = set_up_configuration() + +fn = os.path.basename(url) +tmp_path = os.path.join(opts.tmpdir, fn) +download_file(fn, url, tmp_path) + +sha256 = compute_checksum(tmp_path, checksum_type='sha256') +print("SHA256 checksum for %s: %s" % (fn, sha256)) + +target_dir = os.getcwd() +copy_file(tmp_path, target_dir) +print("%s copied to %s" % (fn, target_dir)) + +# cleanup temp dir +remove_dir(opts.tmpdir) +``` + +Example output: +```shell +$ export TEST_URL='https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.0.1.tar.gz' +$ python3 download_and_copy.py $TEST_URL +== Temporary log file in case of crash /tmp/eb-or_xhaw8/easybuild-ewooa00c.log +SHA256 checksum for eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 +eb-tutorial-1.0.1.tar.gz copied to /home/example +``` +We are deliberately *not* specifying the `args` optional argument to the `set_up_configuration` call here, +so we can pass EasyBuild configuration options as command line arguments to this script. +Of course, only those configuration options that are taken into account by the EasyBuild +framework functions being called have any impact. For example: + +```shell +# wait for max. 1h for the download to complete +python3 download_and_copy.py $TEST_URL --download-timeout 3600 +``` + +### Running shell commands + +To run shell commands, we can leverage the ``run_cmd`` functions from the ``easybuild.tools.run`` module. + +Here is a simple example for running the '`make`' command via a Python script: + +```python +#!/usr/bin/env python3 +import sys + +from easybuild.tools.filetools import remove_dir, which +from easybuild.tools.run import run_cmd +from easybuild.tools.options import set_up_configuration + +opts, _ = set_up_configuration(args=[]) + +cmd = 'make' +cmd_path = which(cmd) +if cmd_path: + print(">>> '%s' command found at %s" % (cmd, cmd_path)) +else: + sys.stderr.write("ERROR: '%s' command not found!\n" % cmd) + sys.exit(1) + +cmd = ' '.join(["make"] + sys.argv[1:]) +out, ec = run_cmd(cmd) + +print("\n>>> Output of '%s' (exit code %s):\n\n%s" % (cmd, ec, out)) + +remove_dir(opts.tmpdir) +``` + +Note that in this case it is important to use `args=[]` when calling `set_up_configuration`, +to avoid that the additional script arguments which are options for the '`make`' command +are picked up by the EasyBuild option parser. + +Example usage: + +```shell +$ python3 run_make.py --version +== Temporary log file in case of crash /tmp/eb-zfvbceg6/easybuild-7tynetaj.log + +>> Output of 'make --version' (exit code 0): + +GNU Make 3.81 +``` + +### Interacting with the modules tool + +You can interact with the environment modules tool in a Python script using the +[``easybuild.tools.modules``](https://docs.easybuild.io/en/latest/api/easybuild.tools.modules.html) +Python module that is a part of the EasyBuild framework. +The modules tool that will be used is determined by the active EasyBuild configuration. + +Here is an example script that checks which modules are available and currently loaded, +loads the default module file for ``bzip2``, and inspects the resulting changes to the environment. + +```python +#!/usr/bin/env python3 +import os +import sys + +from easybuild.tools.filetools import remove_dir +from easybuild.tools.modules import get_software_root_env_var_name, modules_tool +from easybuild.tools.options import set_up_configuration + +opts, _ = set_up_configuration() + +# obtain ModulesTool instance for preferred modules tool (determined by active EasyBuild configuration) +mod_tool = modules_tool() +print("Active modules tool: %s version %s" % (mod_tool.NAME, mod_tool.version)) + +avail_modules = mod_tool.available() +print("Found %d available modules in total" % len(avail_modules)) + +avail_eb_modules = mod_tool.available('EasyBuild') +print("Found %d available modules for EasyBuild: %s" % (len(avail_eb_modules), ', '.join(avail_eb_modules))) + +loaded_modules = mod_tool.loaded_modules() +print("%d modules are currently loaded: %s" % (len(loaded_modules), ', '.join(loaded_modules))) + +# load default module for bzip2, check changes to environment +name = 'bzip2' +env_var_name = get_software_root_env_var_name(name) +if any(m.startswith(name + '/') for m in avail_modules): + + print("Current $%s value: %s" % (env_var_name, os.getenv(env_var_name, '(no set)'))) + print("Loading (default) '%s' module..." % name) + + mod_tool.load([name]) + print("Loaded modules: %s" % ', '.join(mod_tool.loaded_modules())) + + # inspect $_LMFILES_ environment variable to determine path to loaded bzip2 module file + for mod_file_path in os.getenv('_LMFILES_').split(':'): + if name in mod_file_path: + print("Path to loaded %s module: %s" % (name, mod_file_path)) + break + + # $EBROOTBZIP2 should be set now (if the bzip2 module was installed with EasyBuild) + print("Current $%s value: %s" % (env_var_name, os.getenv(env_var_name, '(no set)'))) +else: + sys.stderr.write("No modules available for %s\n" % name) + sys.exit(1) + +remove_dir(opts.tmpdir) +``` + +### Parsing easyconfig files + +Here is another small example Python script, which uses the EasyBuild framework functionality +to locate and parse an easyconfig file, and inspect the value of specific easyconfig parameters. + +We define a small helper function named ``parse_easyconfig``, because the EasyBuild framework API +is a bit awkward to use for this simple use case. + +```python +#!/usr/bin/env python3 +import sys + +from easybuild.framework.easyconfig.tools import det_easyconfig_paths, parse_easyconfigs +from easybuild.tools.options import set_up_configuration + + +def parse_easyconfig(ec_fn): + """ + Helper function: find and parse easyconfig with specified filename, + and return parsed easyconfig file (an EasyConfig instance). + """ + # determine path to easyconfig file + ec_path = det_easyconfig_paths([ec_fn])[0] + + # parse easyconfig file; + # the 'parse_easyconfigs' function expects a list of tuples, + # where the second item indicates whether or not the easyconfig file was automatically generated or not + ec_dicts, _ = parse_easyconfigs([(ec_path, False)]) + + # only retain first parsed easyconfig, ignore any others (which are unlikely anyway) + return ec_path, ec_dicts[0]['ec'] + + +# check whether required arguments are provided +if len(sys.argv) < 3: + sys.stderr.write("ERROR: Usage: %s ") + sys.exit(1) + +ec_fn = sys.argv[1] +keys = sys.argv[2:] + +set_up_configuration(args=[], silent=True) + +ec_path, ec = parse_easyconfig(ec_fn) + +print("Inspecting %s ..." % ec_path) +for key in keys: + print("%s: %s" % (key, ec[key])) +``` + +Example usage: + +```shell +$ ./inspect_easyconfig.py Subread-2.0.0-GCC-8.3.0.eb name version sources sanity_check_paths +name: Subread +version: 2.0.0 +sources: ['subread-2.0.0-source.tar.gz'] +sanity_check_paths: {'files': ['bin/exactSNP', 'bin/featureCounts', 'bin/subindel', 'bin/subjunc', 'bin/sublong', 'bin/subread-align', 'bin/subread-buildindex'], 'dirs': ['bin/utilities']} +``` + +--- + +*[[next: Using hooks to customise EasyBuild]](3_02_hooks.md)* diff --git a/docs/2022-CSC_and_LO/3_02_hooks.md b/docs/2022-CSC_and_LO/3_02_hooks.md new file mode 100644 index 00000000..a2663ff0 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_02_hooks.md @@ -0,0 +1,190 @@ +# Using hooks to customise EasyBuild + +*[[back: Using EasyBuild as a library]](3_01_easybuild_library.md)* + +--- + +Sometimes you may have the need to customise the behaviour of EasyBuild, +while you want to avoid copying easyconfig files in order to make small changes +(like adding an additional configuration option), having to maintain a set +of modified easyblocks, or using a patched version of the EasyBuild framework. + +EasyBuild provides support for implementing so-called *hooks*, +which are small functions that are triggered automatically at specific times. + +Through these hook functions you can take additional actions, or even tweak +internal data structures, to change the software installation procedure that EasyBuild +would normally execute. + +This is a very powerful feature, but it is important to aware of some details in order +to make good use of it. + +Support for hooks is also covered in detail in the EasyBuild documentation, see +[here](https://docs.easybuild.io/en/latest/Hooks.html). + + +## Implementing and using hooks + +Using hooks is done by implementing functions with specific names in a Python module, +and configuring EasyBuild to use them by specifying the path to the file that contains them +via the ``hooks`` configuration option. + +For example, if the file named ``my_eb_hooks.py`` in your home directory contains the implementation +of your hooks, you can configure EasyBuild to use them by defining the ``$EASYBUILD_HOOKS`` environment +variable: + +```shell +export EASYBUILD_HOOKS=$HOME/my_eb_hooks.py +``` + +## Available hooks + +EasyBuild supports various hooks, the full list can be consulted via ``eb --avail-hooks``. + +There are 3 different types of hooks: + +- start/end hooks +- parse hook +- pre/post-step hooks + +These are discussed in more details in the sections below. + +### Start/end hooks + +The first type of hooks are the ``start`` and ``end`` hooks, which are respectively triggered +at the very start of the EasyBuild session (right after setting up the EasyBuild configuration), +and at the very end of the session (right before stopping the logger and doing final cleanup). + +These hooks are only called *once* for every EasyBuild session, no matter how many installations +are performed during that session. + +The ``end`` hook is *not* triggered if an error occurred during one of the installations, +only on successful completion of the EasyBuild session. + +These hooks can be implementing by defining a function named ``start_hook`` and ``end_hook`` +in the Python module file that is provided via the ``hooks`` configuration option. +No arguments are provided when calling these hooks. + +### Parse hook + +The ``parse`` hook is triggered right after an easyconfig file is being parsed, +*before* EasyBuild sets up its internal data structures based on the parsed easyconfig. + +If you want to dynamically change one or more easyconfig parameters without changing the corresponding +easyconfig files, using this hook may be appropriate. + +Note that parsing an easyconfig file can happen for a variety of reasons, +not only when the easyconfig file will actually be installed. EasyBuild will also +parse easyconfig files to check whether they resolve required dependencies, +to check whether the corresponding module file is already installed, etc. + +This hook can be implemented via a function named ``parse_hook``, and exactly one +argument is provided when it is called: the [``EasyConfig``](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyconfig.easyconfig.html#easybuild.framework.easyconfig.easyconfig.EasyConfig) +instance that represents the parsed easyconfig file. + +### Pre/post-step hooks + +The third type of hooks are *pre/post-step* hooks, which are triggered right before or +after a particular installation step is executed. + +EasyBuild performs each software installation by stepping through over a dozen different methods, +and for each of these steps there a pre- and post-hook is triggered, which results in over 30 +additional hooks. + +To use any of these hooks, you need to implement a function that follow a strict naming scheme: +``__hook``. For example, the hook that is triggered right before the ``configure`` +step is run is a function named ``pre_configure_hook``. + +Every time these hooks are called, a single argument is provided: an [``EasyBlock``](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyblock.html#easybuild.framework.easyblock.EasyBlock) +instance that represents the easyblock that is being used to perform the installation. +The parsed easyconfig file can be accessed via the ``cfg`` class variable of the ``EasyBlock`` instance. + +These hooks are useful for influencing the installation procedure at a particular stage. + +## Caveats + +There are a couple of important caveats to take into account when implementing hooks. + +### Breaking EasyBuild with hooks + +Since hooks allow you to inject custom code into EasyBuild at runtime, +it is also easy to break EasyBuild by using hooks... + +Make sure to carefully test your hook implementations, and constrain the actions +you take a much as possible, for example by adding conditions to control for which +software names you will actually modify the installation procedure, etc. + +Any errors that are triggered or raised while a hook function is running +will interrupt the EasyBuild session. + +So don't forget: with great power comes great responsibility! + +### Template values + +Depending on the type of hook, you may observe "raw" values of easyconfig parameters where +template values have not been resolved yet, or values in which template values have been resolved already. + +In the ``parse`` hook, you will always see unresolved template values. + +In the pre/post-step hooks you will see resolved template values, +unless you explicitly disable templating. + +To obtain easyconfig parameter values with unresolved template values in step hooks, +you can use the ``disable_templating`` [context manager](https://docs.python.org/3/reference/compound_stmts.html#with). +For example: + +```python +from easybuild.framework.easyconfig.easyconfig import disable_templating +from easybuild.tools.build_log import print_warning + +def pre_source_hook(eb): + """Print warning when software version was found in 'raw' name of source file.""" + with disable_templating(eb.cfg): + for src in eb.cfg['sources']: + if eb.version in src: + msg = "Software version '%s' found in name of source file (%s), " % (eb.version, src) + msg += "please use %(version)s template value instead!" + print_warning(msg) +``` + + +### Manipulating easyconfig parameters + +If you want update a particular easyconfig parameter without overwriting the existing value, +a bit of care has to be taken: you should use the ``update`` method of the ``EasyConfig`` instance +for this, unless you disable template resolution. This is particularly important when +updating easyconfig parameters that have *mutable* value (like a ``list`` or ``dict``). + +Here's a correct example of a pre-install hook: + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # always copy the README directory too when installing pigz + eb.cfg.update('files_to_copy', 'README') +``` + +This seemingly equivalent implementation will ***not*** work (the value of the `files_to_copy` +easyconfig parameter will *not* be updated): + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # incorrect way of adding 'README' to 'files_to_copy' (DON'T USE THIS!) + eb.cfg['files_to_copy'].append('README') +``` + +To use this coding style successfully, you have to disable the templating mechanism +when updating the easyconfig parameter: + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # this works, but it is better to use the 'update' method instead... + with disable_templating(eb.cfg): + eb.cfg['files_to_copy'].append('README') +``` + +--- + +*[[next: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* diff --git a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md new file mode 100644 index 00000000..0ffa12a9 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md @@ -0,0 +1,304 @@ +# Submitting installations as Slurm jobs + +*[[back: Using hooks to customise EasyBuild]](3_02_hooks.md)* + +--- + +EasyBuild can submit jobs to different backends including Slurm to install software, +to *distribute* the often time-consuming installation of a set of software applications and +the dependencies they require to a cluster. + +This is done via the ``--job`` command line option. + +It is important to be aware of some details before you start using this, which we'll cover here. + +## Configuration + +The EasyBuild configuration that is active at the time that ``eb --job`` is used +will be *passed down* into the submitted job automatically, via command line options to the ``eb`` +command that is run in the job script. + +This includes not only command line options used next to ``--job``, but also configuration settings +that are specified via an [EasyBuild configuration file](configuration.md#configuration-files) or through +[``$EASYBUILD_*`` environment variables](configuration.md#easybuild_-environment-variables). + +This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables +that are in place in the job environment are most likely *irrelevant*, since configuration settings +they specify they will most likely be overruled by the corresponding command line options. + + +## Using ``eb --job`` + +### Job backend + +The default job backend in EasyBuild v4.x is [``GC3Pie``](https://gc3pie.readthedocs.io). +To let EasyBuild submit jobs to Slurm instead, you should set the ``job-backend`` configuration setting +to ``Slurm``, for example by setting the corresponding environment variable: + +```shell +export EASYBUILD_JOB_BACKEND='Slurm' +``` + +### Job resources + +To submit an installation as a job, simply use ``eb --job``: + +```shell +eb example.eb --job +``` + +By default, EasyBuild will submit single-core jobs requesting for 24 hours of walltime. +You can tweak the requested resources via the ``job-cores`` and ``job-max-walltime`` configuration options. +For example: + +```shell +# submit job to install example, using 5 cores and 2 hours of max. walltime +eb example.eb --job --job-cores 5 --job-max-walltime 2 +``` + +Note that not all ``job-*`` configuration settings apply to all job backends, +see the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Submitting_jobs.html) for more details. + +### Controlling Slurm submission options + +When using Slurm as a job backend, EasyBuild will automatically generate job scripts which +use the ``eb`` command to perform a single installation. These scripts will be submitted +using the ``sbatch`` command. + +EasyBuild currently doesn't provide away to customize the Slurm submission options, +for example to submit to a particular partition, or to use a particular account, +build you can set the corresponding ``$SBATCH_*`` environment variables prior to running ``eb --job``. + +For example, to specify a particular account that should be used for the jobs submitted by EasyBuild +(equivalent with using the ``-A`` or ``--account`` command line option for ``sbatch``): + +```shell +export SBATCH_ACCOUNT='example_project' +``` + +Or to submit to a particular Slurm partition (equivalent with the ``-p`` or ``--partition`` option for ``sbatch``): + +```shell +export SBATCH_PARTITION='example_partition' +``` + +For more information about supported ``$SBATCH_*`` environment variables, +see the [Slurm documentation](https://slurm.schedmd.com/sbatch.html#lbAJ). + +## Combining ``--job`` and ``--robot`` + +If one or more dependencies are still missing for the software you want to install, +you can combine ``--job`` and ``--robot`` to get EasyBuild to submit a *separate* job +for each of the installations. These jobs will *not* ``--robot``, they will each only +perform a single installation. + +Dependencies between jobs will be "registered" at submission time, so Slurm will put jobs +on hold until the jobs that install the required (build) dependencies have completed successfully, +and cancel jobs if the job to install a dependency failed for some reason. + +## Attention points + +There are a couple of important things to keep an eye on when submitting installations as jobs... + +### Differences on cluster workernodes + +Sometimes the resources available on the login nodes and cluster workernodes are slightly different, +and you may need to take this into account in your EasyBuild configuration. + +For example, plenty of disk space may be available in the `/tmp` temporary filesystem on a login node, +while the workernodes require you to use a different location for temporary files and directories. +As a result, you may need to slightly change your EasyBuild configuration when submitting installations +as jobs, to avoid that they fail almost instantly due to a lack of disk space. + +Keep in mind that the active EasyBuild configuration is passed down into the submitted jobs, +so any configuration that is present on the workernodes may not have any effect. + +For example, if you commonly use `/tmp/$USER` for build directories on a login node, +you may need to tweak that when submitting jobs to use a different location: + +```shell +# EasByuild is configured to use /tmp/$USER on the login node +login01 $ eb --show-config | grep buildpath +buildpath (E) = /tmp/example + +# use /localdisk/$USER for build directories when submitting installations as jobs +login01 $ eb --job --buildpath /localdisk/$USER example.eb --robot +``` + +### Temporary log files and build directories + +The temporary log file that EasyBuild creates is most likely going to end up on the local disk +of the workernode on which the job was started (by default in `$TMPDIR` or `/tmp`). +If an installation fails, the job will finish and temporary files will likely be cleaned up instantly, +which may leave you wondering about the actual cause of the failing installation... + +To remedy this, there are a couple of EasyBuild configuration options you can use: + +* You can use ``--tmp-logdir`` to specify a different location where EasyBuild should store temporary log files, + for example: + ```shell + $ eb --job example.eb --tmp-logdir $HOME/eb_tmplogs + ``` + +* If you prefer having the entire log file stored in the Slurm job output files, + you can use ``--logtostdout`` when submitting the jobs. This will result in extensive logging + to your terminal window when submitting the jobs, but it will also make EasyBuild + log to ``stdout`` when the installation is running in the job, and hence the log messages will be + captured in the job output files. + +The same remark applies to build directories: they should be on a local filesystem (to avoid problems +that often occur when building software on a parallel filesystem like GPFS or Lustre), +which will probably be cleaned up automatically when a job fails. Here it is less easy to provide +general advice on how to deal with this, but one thing you can consider is retrying the installation +in an interactive job, so you can inspect the build directory after the installation fails. + +### Lock files + +EasyBuild creates [locks](https://docs.easybuild.io/en/latest/Locks.html) +to prevent that the same installation is started multiple times on +different system to the same installation directory. + +If an installation fails or gets interrupted, EasyBuild cleans up those locks automatically. + +However, if a Slurm job that is using EasyBuild to install software gets cancelled (because it +ran out of walltime, tried to consume too much memory, through an ``scancel`` command, etc.), +EasyBuild will not get the chance to clean up the lock file. + +If this occurs you will need to either clean up the lock file (which is located in the `software/.locks` +subdirectory of ``installpath``) manually, or re-submit the job with ``eb --job --ignore-locks``. + +## Example + +As an example, we will let EasyBuild submit jobs to install ``AUGUSTUS`` with the ``foss/2020b`` toolchain. + +### Configuration + +Before using ``--job``, let's make sure that EasyBuild is properly configured: + +```shell +# use $HOME/easybuild for software, modules, sources, etc. +export EASYBUILD_PREFIX=$HOME/easybuild + +# use ramdisk for build directories +export EASYBUILD_BUILDPATH=/dev/shm/$USER + +# use Slurm as job backend +export EASYBUILD_JOB_BACKEND=Slurm +``` + +In addition, add the path to the centrally installed software to ``$MODULEPATH`` via ``module use``: + +```shell +module use /easybuild/modules/all +``` + +Load the EasyBuild module: + +```shell +module load EasyBuild +``` + +Let's assume that we also need to inform Slurm that jobs should be submitted into a particular account: + +```shell +export SBATCH_ACCOUNT=example_project +``` + +This will be picked up by the ``sbatch`` commands that EasyBuild will run to submit the software installation jobs. + + +### Submitting jobs to install AUGUSTUS + +Now we can let EasyBuild submit jobs for AUGUSTUS. + +Let's first check what is still missing: + +```shell +$ eb AUGUSTUS-3.4.0-foss-2020b.eb --missing +... +11 out of 61 required modules missing: + +* HTSlib/1.11-GCC-10.2.0 (HTSlib-1.11-GCC-10.2.0.eb) +* lpsolve/5.5.2.11-GCC-10.2.0 (lpsolve-5.5.2.11-GCC-10.2.0.eb) +* Boost/1.74.0-GCC-10.2.0 (Boost-1.74.0-GCC-10.2.0.eb) +* GSL/2.6-GCC-10.2.0 (GSL-2.6-GCC-10.2.0.eb) +* SAMtools/1.11-GCC-10.2.0 (SAMtools-1.11-GCC-10.2.0.eb) +* BCFtools/1.11-GCC-10.2.0 (BCFtools-1.11-GCC-10.2.0.eb) +* METIS/5.1.0-GCCcore-10.2.0 (METIS-5.1.0-GCCcore-10.2.0.eb) +* BamTools/2.5.1-GCC-10.2.0 (BamTools-2.5.1-GCC-10.2.0.eb) +* MPFR/4.1.0-GCCcore-10.2.0 (MPFR-4.1.0-GCCcore-10.2.0.eb) +* SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0.eb) +* AUGUSTUS/3.4.0-foss-2020b (AUGUSTUS-3.4.0-foss-2020b.eb) +``` + +Several dependencies are not installed yet, so we will need to use ``--robot`` to ensure that +EasyBuild also submits jobs to install these first. + +To speed up the installations a bit, we will request 10 cores for each submitted job (via ``--job-cores``). +That should be sufficient to let each installation finish in (well) under 1 hour, +so we only request 1 hour of walltime per job (via ``--job-max-walltime``). + +In order to have some meaningful job output files, we also enable trace mode (via ``--trace``). + +``` +$ eb AUGUSTUS-3.4.0-foss-2020b.eb --job --job-cores 10 --job-max-walltime 1 --robot --trace +... +== resolving dependencies ... +... +== List of submitted jobs (11): Boost-1.74.0-GCC-10.2.0 (Boost/1.74.0-GCC-10.2.0): 1000011; GSL-2.6-GCC-10.2.0 (GSL/2.6-GCC-10.2.0): 1000004; SAMtools-1.11-GCC-10.2.0 (SAMtools/1.11-GCC-10.2.0): 1000005; HTSlib-1.11-GCC-10.2.0 (HTSlib/1.11-GCC-10.2.0): 1000006; BCFtools-1.11-GCC-10.2.0 (BCFtools/1.11-GCC-10.2.0): 1000001; lpsolve-5.5.2.11-GCC-10.2.0 (lpsolve/5.5.2.11-GCC-10.2.0): 1000007; BamTools-2.5.1-GCC-10.2.0 (BamTools/2.5.1-GCC-10.2.0): 1000008; METIS-5.1.0-GCCcore-10.2.0 (METIS/5.1.0-GCCcore-10.2.0): 1000009; MPFR-4.1.0-GCCcore-10.2.0 (MPFR/4.1.0-GCCcore-10.2.0): 1000010; SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0): 1000002; AUGUSTUS-3.4.0-foss-2020b (AUGUSTUS/3.4.0-foss-2020b): 1000003 +== Submitted parallel build jobs, exiting now +``` + +### Inspecting the submitted jobs + +Once EasyBuild has submitted the jobs, we can inspect them via Slurm's ``squeue`` command: + +``` +$ squeue -u $USER -la + JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON) +1000001 small BCFtools user123 PENDING 0:00 2:00:00 1 (Dependency) +1000002 small SuiteSpa user123 PENDING 0:00 2:00:00 1 (Dependency) +1000003 small AUGUSTUS user123 PENDING 0:00 2:00:00 1 (Dependency) +1000004 small GSL-2.6- user123 RUNNING 0:21 2:00:00 1 node003 +1000005 small SAMtools user123 RUNNING 0:21 2:00:00 1 node007 +1000006 small HTSlib-1 user123 RUNNING 0:21 2:00:00 1 node007 +1000007 small lpsolve- user123 RUNNING 0:21 2:00:00 1 node011 +1000008 small BamTools user123 RUNNING 0:21 2:00:00 1 node011 +1000009 small METIS-5. user123 RUNNING 0:21 2:00:00 1 node013 +1000010 small MPFR-4.1 user123 RUNNING 0:21 2:00:00 1 node029 +1000011 small Boost-1. user123 RUNNING 0:24 2:00:00 1 node029 +``` + +Note that 3 jobs can not be started yet, because those installations require on one or more +missing dependencies. As soon as the jobs for those dependencies (successfully) complete, +these jobs will be able to start. + +### Final result + +After about 20 minutes, AUGUSTUS and all missing dependencies should be installed: + +``` +$ ls -lrt $HOME/easybuild/modules/all/*/*.lua | tail -11 +-rw-rw----. 1 example example 1634 Mar 29 10:13 /users/example/easybuild/modules/all/HTSlib/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1792 Mar 29 10:13 /users/example/easybuild/modules/all/SAMtools/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1147 Mar 29 10:13 /users/example/easybuild/modules/all/BamTools/2.5.1-GCC-10.2.0.lua +-rw-rw----. 1 example example 957 Mar 29 10:13 /users/example/easybuild/modules/all/lpsolve/5.5.2.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1549 Mar 29 10:13 /users/example/easybuild/modules/all/METIS/5.1.0-GCCcore-10.2.0.lua +-rw-rw----. 1 example example 1525 Mar 29 10:14 /users/example/easybuild/modules/all/GSL/2.6-GCC-10.2.0.lua +-rw-rw----. 1 example example 1221 Mar 29 10:15 /users/example/easybuild/modules/all/MPFR/4.1.0-GCCcore-10.2.0.lua +-rw-rw----. 1 example example 1678 Mar 29 10:15 /users/example/easybuild/modules/all/BCFtools/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1292 Mar 29 10:21 /users/example/easybuild/modules/all/Boost/1.74.0-GCC-10.2.0.lua +-rw-rw----. 1 example example 1365 Mar 29 10:28 /users/example/easybuild/modules/all/SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0.lua +-rw-rw----. 1 example example 2233 Mar 29 10:30 /users/example/easybuild/modules/all/AUGUSTUS/3.4.0-foss-2020b.lua + +$ module use $HOME/easybuild/modules/all + +$ module avail AUGUSTUS + +-------- /users/hkenneth/easybuild/modules/all -------- + AUGUSTUS/3.4.0-foss-2020b +``` + +--- + +*[[next: Module naming schemes]](3_04_module_naming_scheme.md)* diff --git a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md new file mode 100644 index 00000000..14229d97 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md @@ -0,0 +1,617 @@ +# Module naming schemes + +*[[back: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* + +--- + +Up until now we have used EasyBuild's default module naming scheme (`EasyBuildMNS`), +which produces module files with names that closely resemble to the names of the +corresponding easyconfig files. +For example, when installing `h5py-3.1.0-foss-2020b.eb` the generated module was named `h5py/3.1.0-foss-2020b`. + +EasyBuild supports several different module naming schemes: + +```shell +$ eb --avail-module-naming-schemes +List of supported module naming schemes: + CategorizedHMNS + CategorizedModuleNamingScheme + EasyBuildMNS + HierarchicalMNS + MigrateFromEBToHMNS +``` + +In this part of the tutorial we will take a closer look at `HierarchicalMNS`, +which is the standard **hierarchical** module naming scheme included with EasyBuild. + +We will also take a quick look at implementing our own custom module naming scheme. + +## Flat vs hierarchical + +The default module naming scheme `EasyBuildMNS` is an example of regular *"flat"* module naming scheme, which is characterized by: + +* all module files are directly available for loading; +* each module name uniquely identifies a particular installation; + +In contrast, a *hierarchical* module naming scheme +consists of a *hierarchy* of module files. + +The typical module hierarchy has 3 levels: + +* a **core** level, where module files for software that was installed using the + [`system` toolchain](../introduction/#system-toolchain) are kept; +* a **compiler** level, where module files for software that was installed using a +*compiler-only toolchain* are stored; +* and an **MPI** level, which houses module files for software that was installed using a toolchain that includes (at least) a compiler and MPI component; + +Here is a simple example of such a 3-level module hierarchy: + +
+ +In this example the core level only includes a single module `GCC/9.3.0`, +while the compiler level includes two modules: `OpenMPI/4.0.3` and `MPICH/3.3.2`. +In the MPI level, three modules are available: one for `FFTW`, one for `ScaLAPACK`, and one for `HDF5`. + +Initially only the modules on the top level of a module hierarchy are available for loading. +If you run "`module avail`" with the example module hierarchy, you will only see the `GCC/9.3.0` module. + +Some modules in the top level of the hierarchy act as a "gateway" to modules in the +next level below. +To make additional modules available for loading one of these gateway modules has to be loaded. In our exampe, loading the `GCC/9.3.0` module results in two additional modules coming into view from the compiler level, as indicated by the arrows: the modules for `OpenMPI` and `MPICH`. These correspond to installations of `OpenMPI` +and `MPICH` that were built using `GCC/9.3.0` as a (compiler-only) toolchain. + +Similarly, the `OpenMPI/4.0.3` module serves as a gateway to the three modules in the MPI level. Only by loading the `OpenMPI` module will these additional three modules become +available for loading. They correspond to software installations built using the `gompi/2020a` toolchain that +consists of the `GCC/9.3.0` compiler module and the `OpenMPI/4.0.3` MPI module. Software installed using +`foss/2020a` (which is a full toolchain that also includes OpenBLAS, FFTW and ScaLAPACK) would also be stored +in this level of the module hierarchy. + +The characteristics of a module hierarchy are: + +* not all module files are directly available for loading; +* some modules serve as a gateway to more modules; +* to access some software installations you will first need to load one or more gateway modules in order + to use them; + +You can probably think of other ways to organize module files in a hierarchical module tree, but here we will stick to the standard core / compiler / MPI hierarchy. + +### Pros & cons + +So why go through all this trouble of organizing modules hierarchically? + +There are a couple of advantages to this approach: + +* shorter module names; +* less overwhelming list of available modules; +* only compatible modules can be loaded together; + +However, the are some minor disadvantages too: + +* not all existing modules are directly visible; +* gateway modules may have little meaning to end users; + +#### Length of module names + +When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For our `HDF5` installation for example, +we have `HDF5/1.10.6-gompi-2020a` as module name. The `-gompi-2020a` part of the name refers to the toolchain that was +used for this installation, but it may be confusing to some people (what kind of Pokémon is a "gompi"?!). + +In the example module hierarchy shown above, the module for `HDF5` is named `HDF5/1.10.6` which is basically the bare +essentials: software name and version. That's way better, nice and clean! + +#### Amount of available modules + +The output of "`module avail`" can be quite overwhelming if lots of module files +are installed and a flat module naming scheme is used, since *all* modules are +*always* available. +EasyBuild makes it very easy to install lots of software, +so the number of installed modules can easily grow into the hundreds or even thousands... + +This often explosive growth of modules is less of an issue when using a hierarchical module naming scheme, since +initially only a small set of modules are available, and relatively limited +groups of additional modules become available as gateway modules are loaded. + +#### Loading compatible modules + +Since all modules are available at once when using a flat module naming scheme, you can easily load +modules together that are not compatible with each other. + +Imagine loading two modules that were built with a different compiler toolchain (different compiler, +different MPI library). That's likely to end in tears, unless you have the necessary technical expertise +to fully understand what is going on, *and* you are being very careful... + +In a module hierarchy this can be prevented, since modules for software that were installed with a +different compiler and/or a different MPI library are located in a different part of the module hierarchy, +and so these modules can not be loaded together easily. + + +#### Visibility of existing modules + +One downside of a module hierarchy is that not all existing modules are directly available for loading +or are even *visible* to the user, since the output of "`module avail`" only shows a subset of all modules. + +[Lmod](https://lmod.readthedocs.io) has a solution for this though: it provides a separate "`module spider`" +command to search for module files throughout the entire module hierarchy. So as long as the end users are +aware of this additional command, it should not be difficult to discover which software installations exist +and how they can be accessed. The "`module spider`" command will inform the user which of the gateway modules +need to be loaded in order to load a specific module file. + +#### Semantics of gateway modules + +An additional potential problem of a module hierarchy is that the semantics of the gateway modules may not be clear +to end users. They may wonder why they need to pick a specific compiler and MPI library, or which of the +available options is the best one. Maybe there are not even be aware what exactly a "compiler" is, or how it is +relevant to the software they need in their bioinformatics pipeline... + +This can be partially resolved by loading a default compiler and MPI module so a particular set of modules +is available right after login, which could be the ones used in the most recent toolchain, or the +recommended versions. More experienced users could then leverage the "`module spider`" command to navigate +the module hierarchy. + +## Using a custom module naming scheme + +Next to the module naming schemes that are included with EasyBuild, +you can also define your own module naming scheme (MNS), and configure EasyBuild to use it. + +### Implementation + +To use a custom module naming scheme, you need to implement a Python module file, +where you define a Python class that derives from the general [``ModuleNamingScheme``](https://docs.easybuild.io/en/latest/api/easybuild.tools.module_naming_scheme.mns.html#easybuild.tools.module_naming_scheme.mns.ModuleNamingScheme) class. + +For a flat module naming scheme, it is sufficient to implement the ``det_full_module_name`` method, +which should return a string value (the full module name). +You may also need to customize the ``is_short_modname_for`` method, which verifies whether +a given (short) module name is for the software with a particular given name (or not). + +The argument provided to ``det_full_module_name`` can be a small Python dictionary which only specifies +the value of a handful of easyconfig parameters (`name`, `version`, `toolchain`, and `versionsuffix`), +or an ``EasyConfig`` instance which represents a parsed easyconfig file and contains values for *all* +known easyconfig parameters. + +For simple module naming schemes, just have ``name``, ``version``, ``toolchain``, and ``versionsuffix`` +available is sufficient. If it is not, you can list which additional easyconfig parameters are required for +your module naming scheme via a class constant ``REQUIRED_KEYS``. + +A fairly straightforward example of a *flat* module name scheme is the default EasyBuild module naming scheme +[``EasyBuildMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/easybuild_mns.py). + +For a *hierarchical* module naming scheme, various additional methods have to be implemented. + +Some of these, like ``det_module_subdir`` and ``det_short_module_name``, determine which part of +the (full) module name determines the location of the module in the module hierarchy, and which part +is the user-facing (or "short") module name. Others, like ``det_modpath_extensions``, +determine which modules are *gateway* modules that open up an additional level of the hierarchy +(by *extending* the ``$MODULEPATH``). + +A typical example of a hierarchical module naming scheme is [``HierarchicalMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/hierarchical_mns.py) that comes with EasyBuild (which is quite complex because it includes quite a bit of special handling for particular compiler +toolchains), which implements the traditional ``Core``-``Compiler``-``MPI`` module hierarchy we discussed +above. + +### Configuring EasyBuild + +To let EasyBuild use a custom module naming scheme, you need to: + +* specify the path to the Python module file implementing it via the ``include-module-naming-schemes`` + EasyBuild configuration option; + +* indicate that you also want to *use* this custom module naming scheme via the ``module-naming-scheme`` + EasyBuild configuration option. + +For example: + +```shell +export EASYBUILD_INCLUDE_MODULE_NAMING_SCHEMES=$HOME/easybuild/example_mns.py +export EASYBUILD_MODULE_NAMING_SCHEME=ExampleMNS +``` + +#### Example custom module naming scheme + +Here is an example of a custom module naming scheme, where: + +* the ``versionsuffix`` goes directly after the ``version``; +* all dashes are replaced by underscores; +* all module names are lowercase; + +Note that we also need to customise the ``is_short_modname_for`` method, +to make sure it returns ``True`` when the EasyBuild framework checks whether +``scipy_bundle/2020.11_foss_2020b`` is a module name for ``SciPy-bundle``. + +```python +import os + +from easybuild.tools.module_naming_scheme.mns import ModuleNamingScheme + + +class ExampleMNS(ModuleNamingScheme): + + REQUIRED_KEYS = ['name', 'version', 'versionsuffix', 'toolchain'] + + def det_full_module_name(self, ec): + """ + Determine full module name: + - all lowercase + - replace all dashes with underscores + """ + parts = [ec['version']] + + # versionsuffix directly after version (but only if it's not empty) + if ec['versionsuffix']: + parts.append(ec['versionsuffix']) + + # only add toolchain name/version for non-system toolchain + tc = ec['toolchain'] + if tc['name'].lower() != 'system': + parts.extend([tc['name'], tc['version']]) + + modname = ec['name'] + '/' + '_'.join(parts) + + modname = modname.replace('-', '_').replace('__', '_') + + return modname.lower() + + def is_short_modname_for(self, short_modname, name): + """Determine whether short module name is a module for the software with specified name.""" + return short_modname.startswith(name.lower().replace('-', '_') + '/') +``` + +We can see what the module names with this module naming scheme would like like via ``eb -D``: + +``` +$ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D + ... + * [ ] $CFGS/g/GCC/GCC-10.2.0.eb (module: gcc/10.2.0) + ... + * [ ] $CFGS/p/Python/Python-2.7.18-GCCcore-10.2.0.eb (module: python/2.7.18_gcccore_10.2.0) + ... + * [ ] $CFGS/o/OpenMPI/OpenMPI-4.0.5-GCC-10.2.0.eb (module: openmpi/4.0.5_gcc_10.2.0) + ... + * [ ] $CFGS/s/SciPy-bundle/SciPy-bundle-2020.03-foss-2020a-Python-2.7.18.eb (module: scipy_bundle/2020.03_python_2.7.18_foss_2020a) +``` + +## Example module hierarchy: HDF5 + +!!! Warning + **This exercise will only work if you have write access to the software installation prefix**, + because EasyBuild will try to copy the installation log file to each installlation directory. + +Now that we know more about hierarchical module naming schemes, +let us see how EasyBuild can help us with generating a hierarchical module tree. + +In this example we will use EasyBuild to generate modules organised in a hierarchy +for some of the software that is already installed in the prepared environment. + +The good news is that the existing installations can be reused. There is absolutely no need +to reinstall the software, we are just creating a different "view" on these software installations. + +### Preparing the environment + +Before running EasyBuild to generate a hierarchical module tree, we have to be a bit careful +with preparing our environment. + +**We must absolutely avoid mixing modules from a flat and hierarchical module naming scheme!** + +Some module files will have the same name in both module trees (like `GCC/10.2.0` for example), +but their contents will be different. +Mixing modules from a flat and hierarchical module tree *will* trigger problems... + +So we have to make sure that the module files we already have in `/easybuild` are *not* visible. +The easiest way to do this is to unload all modules (using "`module purge`") +and resetting the module search path to be empty, which we can do with "`module unuse $MODULEPATH`". + +```shell +module purge +module unuse $MODULEPATH +``` + +In this part of the tutorial, we are assuming you are *not* using an EasyBuild installation provided through +a module. We have just made all modules unavailable, so we would have to first +install EasyBuild again in our hierarchical module tree before we can continue. + +**We strongly recommend using an EasyBuild installation that was [installed via "`pip install`" +or "`pip3 install`"](../installation/#method-1-using-pip-recommended) in this part of the tutorial.** + +An easy way to do this is in the prepared environment is to run: + +```shell +pip3 install --user easybuild +export PATH=$HOME/.local/bin:$PATH +export EB_PYTHON=python3 +``` + +### Configuring EasyBuild + +First of all, we need to make sure that EasyBuild is properly configured. +We can do this by defining this set of environment variables: + +```shell +export EASYBUILD_PREFIX=$HOME/easybuild +export EASYBUILD_BUILDPATH=/tmp/$USER +export EASYBUILD_INSTALLPATH_SOFTWARE=/easybuild/software +export EASYBUILD_MODULE_NAMING_SCHEME=HierarchicalMNS +export EASYBUILD_INSTALLPATH_MODULES=$HOME/hmns/modules +``` + +To make sure we didn't make any silly mistakes, we double check using `eb --show-config`: + + +```shell +$ eb --show-config +# +# Current EasyBuild configuration +# (C: command line argument, D: default value, E: environment variable, F: configuration file) +# +buildpath (E) = /tmp/example +containerpath (E) = /home/example/easybuild/containers +installpath (E) = /home/example/easybuild +installpath-modules (E) = /home/example/hmns/modules +installpath-software (E) = /easybuild/software +module-naming-scheme (E) = HierarchicalMNS +packagepath (E) = /home/example/easybuild/packages +prefix (E) = /home/example/easybuild +repositorypath (E) = /home/example/easybuild/ebfiles_repo +robot-paths (D) = /home/example/.local/easybuild/easyconfigs +sourcepath (E) = /home/example/easybuild/sources +``` + +There are a couple of things worth pointing out here: + +* We have defined the `module-naming-scheme` configuration setting to `HierarchicalMNS`, + which makes EasyBuild use the included standard hierarchical module naming scheme (the classic + core / compiler / MPI one we discussed above). +* We have specified *different* locations for the software (via `installpath-software`) + and the module files (via `installpath-modules`). This is important because we want to + reuse the software that is already installed in `/easybuild/software` while we want to + generate an entirely new module tree for it (in `$HOME/hmns/modules`). + +The other configuration settings are the same as before, and mostly irrelevant for this example. + +### Generating modules for HDF5 + +Let us now generate a hierarchical module tree for `HDF5` and all of its dependencies, +including the toolchain. That sounds complicated, and it sort of is since there are +a lot of details you have to get right for the module hierarchy to works as intended, +but EasyBuild can do all the hard work for us. + +The steps we will have to go through are: + +* Tell EasyBuild we want to "install" the `HDF5-1.10.7-gompi-2020b.eb` easyconfig file; +* Enable dependency resolution via `--robot`; +* Instruct EasyBuild to only generate the module files, not to install the software (since it is + there already in `/easybuild/software`), via the `--module-only` option. + +These steps translate to this single `eb` command: + +``` +$ eb HDF5-1.10.7-gompi-2020b.eb --robot --module-only +... +== building and installing MPI/GCC/10.2.0/OpenMPI/4.0.5/HDF5/1.10.7... +... +== sanity checking... +== cleaning up [skipped] +== creating module... +... +== COMPLETED: Installation ended successfully (took 9 sec) +... +== Build succeeded for 41 out of 41 +``` + +This should take a couple of minutes in total, for generating 41 modules. +Remember that this also includes generating module files for the toolchain and +all of its components. + +In addition, there is a bit more going on one that just generating module files, +since the sanity check step is still being run for each of the installations +when using `--module-only` to ensure the installation is actually functional. +After all, there is no point in generating a module for an obviously broken +installation... + +### Loading the HDF5 module + +After generating the hierarchical module tree for HDF5, how do we access the HDF5 installation through it? + +Here's what the module tree looks like on disk: + +``` +$ ls $HOME/hmns/modules/all +Compiler Core MPI +``` + +Those are basically the 3 levels in the module hierarchy we showed in our example earlier. + +The starting point is the top level of the module hierarchy named `Core`: + +``` +module use $HOME/hmns/modules/all/Core +``` + +Let us see what that gives us in terms of available modules: + +``` +$ module avail + +--------------------- /home/easybuild/hmns/modules/all/Core --------------------- + Bison/3.5.3 GCCcore/10.2.0 flex/2.6.4 help2man/1.47.4 + Bison/3.7.1 (D) M4/1.4.18 gettext/0.21 ncurses/6.2 + GCC/10.2.0 binutils/2.35 gompi/2020b zlib/1.2.11 +``` + + +Nice and short module names, but only a limited set of them. + +We know a module file exists for `HDF5`, but we can't see it yet (and hence +we can't load it either). + +``` +$ module avail HDF5 +No module(s) or extension(s) found! +Use "module spider" to find all possible modules and extensions. +``` + +Let us see if `module spider` is of any help, as "`module avail`" so kindly suggests: + +``` +$ module spider HDF5 +... + + You will need to load all module(s) on any one of the lines below + before the "HDF5/1.10.7" module is available to load. + + GCC/10.2.0 OpenMPI/4.0.5 +``` + +This tells us we need to load two gateway modules before we can load the module +for HDF5. + +Let us start with loading the `GCC` compiler module: + +``` +module load GCC/10.2.0 +``` + +And then check again which modules are available: + +``` +$ module avail + +-------------- /home/easybuild/hmns/modules/all/Compiler/GCC/10.2.0 -------------- + OpenMPI/4.0.5 + +------------ /home/easybuild/hmns/modules/all/Compiler/GCCcore/10.2.0 ------------ + Autoconf/2.69 UCX/1.9.0 libpciaccess/0.16 + ... + Szip/2.1.1 libfabric/1.11.0 zlib/1.2.11 (L,D) + Autoconf/2.69 XZ/5.2.5 libtool/2.4.6 + +--------------------- /home/easybuild/hmns/modules/all/Core --------------------- + Bison/3.5.3 GCCcore/10.2.0 (L) flex/2.6.4 help2man/1.47.4 + Bison/3.7.1 (D) M4/1.4.18 gettext/0.21 ncurses/6.2 + GCC/10.2.0 (L) binutils/2.35 gompi/2020b zlib/1.2.11 +``` + +Good news, we now have additional modules available! + +The compiler level of our hierarchy actually consists of two directories here: `Compiler/GCCcore/10.2.0` +and `Compiler/GCC/10.2.0`. The modules in the `GCCcore` directory are ones we can use in other compiler +toolchains that use GCC 10.2.0 as a base compiler (the details of that are out of scope here). + +The module we are interested in is `OpenMPI/4.0.5`, which is another gateway module. + +Remember that the "`module spider`" output told us that there does indeed exist a module for `HDF5`, but that +we need to load *both* the `GCC/10.2.0` and `OpenMPI/4.0.5` modules first. + +So, let us do exactly that (remember that `GCC/10.2.0` is already loaded): + +``` +module load OpenMPI/4.0.5 +``` + +If you now check the output of "`module avail`" again, you should see the `HDF5/1.10.7` module: + +``` +$ module avail + +-------- /home/easybuild/hmns/modules/all/MPI/GCC/10.2.0/OpenMPI/4.0.5 ------- + HDF5/1.10.7 + +------------ /home/easybuild/hmns/modules/all/Compiler/GCC/10.2.0 ------------ + OpenMPI/4.0.5 (L) + +... +``` + +To use HDF5, we need to load this `HDF5/1.10.7` module. We can verify that the installation works +using one of the commands provided by HDF5, `h5dump` for example: + +``` +module load HDF5/1.10.7 +``` + +``` +$ h5dump --version +h5dump: Version 1.10.7 +``` + +If you now check which modules are loaded via "`module list`", you will notice that all module names +and nice and short now, which is one of the advantages of using a hierarchical module tree: + +```shell +$ module list + +Currently Loaded Modules: + 1) GCCcore/10.2.0 5) numactl/2.0.13 9) hwloc/2.2.0 13) PMIx/3.1.5 + 2) zlib/1.2.11 6) XZ/5.2.5 10) libevent/2.1.12 14) OpenMPI/4.0.5 + 3) binutils/2.35 7) libxml2/2.9.10 11) UCX/1.9.0 15) Szip/2.1.1 + 4) GCC/10.2.0 8) libpciaccess/0.16 12) libfabric/1.11.0 16) HDF5/1.10.7 +``` + +## Exercise + +Now it is your turn! + +Try to get a feeling for how a hierarchical module tree works by: + +* installing the missing modules for the `SciPy-bundle-2020.11-foss-2020b.eb` in the module hierarchy we + generated for HDF5; +* figure out where the `SciPy-bundle` module is located in the hierarchy, and then also load it; + +You can verify your work by running this command (since `pandas` is one of the Python packages included +in the `SciPy-bundle` installation): + +```shell +python -c 'import pandas; print(pandas.__version__)' +``` + +Start from a clean slate, by first running: + +```shell +module purge +module unuse $MODULEPATH +``` + +??? success "(click to show solution)" + + * Step 0: check which modules are still missing, using `--missing` or `-M`: + ```shell + eb SciPy-bundle-2020.11-foss-2020b.eb -M + ``` + The output should tell you that 15 out of 50 required modules are still missing. + + * Install the missing modules in the module hierarchy we have generated in `$HOME/hmns/modules`: + ```shell + eb SciPy-bundle-2020.11-foss-2020b.eb --robot --module-only + ``` + Don't forget to use both `--robot` (to enable dependency resolution) and `--module-only` + (to only run the sanity check and generate module files, not install the software again). + + * Start at the top of the module hierarchy (the `Core` level), + and run module spider to check which gateway modules to load to make `SciPy-bundle` available: + ```shell + module use $HOME/hmns/modules/all/Core + module spider SciPy-bundle/2020.11 + ``` + * Load the gateway modules: + ```shell + module load GCC/10.2.0 OpenMPI/4.0.5 + ``` + * Check that the `SciPy-bundle` module is available, and load it: + ```shell + $ module avail SciPy-bundle + ----- /home/example/hmns/modules/all/MPI/GCC/10.2.0/OpenMPI/4.0.5 ------ + SciPy-bundle/2020.11 + ``` + ``` + module load SciPy-bundle/2020.11 + ``` + * Run the test command: + ``` + $ python -c 'import pandas; print(pandas.__version__)' + 1.1.4 + ``` + +!!! Warning + **This exercise will only work if you have write access to the software installation prefix**, + because EasyBuild will try to copy the installation log file to each installlation directory. + +--- + +*[[next: GitHub integration]](3_05_github_integration.md)* diff --git a/docs/2022-CSC_and_LO/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_05_github_integration.md new file mode 100644 index 00000000..59dcbb67 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_05_github_integration.md @@ -0,0 +1,442 @@ +# GitHub integration to facilitate contributing to EasyBuild + +*[[back: Module naming schemes]](3_04_module_naming_schemes.md)* + +--- + +To contribute changes to the EasyBuild code (framework or easyblocks) or easyconfigs, +you will need to be a bit familiar with Git and GitHub. Or maybe not? + +## Manual contribution procedure + +0) Create and setup a [GitHub account](https://github.com/join) (and register your SSH public key); + +1) Clone and fork the appropriate GitHub repository, for example when contributing an easyconfig file: + +```shell +git clone git@github.com:easybuilders/easybuild-easyconfigs.git +cd easybuild-easyconfigs +git remote add my_fork git@github.com:your_github_account/easybuild-easyconfigs.git +``` + +**You should change '`your_github_account`' in the last line to your own GitHub user name!** + +2) Create and check out a new branch, starting from the (up-to-date) ``develop`` branch: + +``` +git checkout develop +git pull origin develop +git checkout -b example +``` + +3) Stage the changes you want to contribute, after you make sure that your easyconfig file has the +[correct filename](../basic_usage/#easyconfig-filenames), and that it's located in the appropriate directory. + +```shell +mkdir -p easybuild/easyconfigs/e/example/ +mv example.eb easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb +git add easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb +``` + +4) Commit those changes with a sensible commit message: + +```shell +git commit -m "This is just an example" +``` + +5) Push your branch to your fork of the repository on GitHub: + +```shell +git push my_fork example +``` + +6) Open the pull request through the GitHub web interface, making sure that: + +* the target branch is correct (should be `develop`); +* an appropriate title is used; +* a short description of the changes is provided; +* the changes are indeed the ones you want to propose; +* clicking the (correct) green button; + +
Pfft
+ +That didn't exactly motivate you to contribute, did it... + +## Github integration features + +Over the years we noticed that some people were keen on contributing to EasyBuild, +but they were not very familiar with Git or GitHub. That meant they had to overcome a +relatively steep learning curve before they could contribute... + +
Gandalf vs Git
+ +In addition, the contribution workflow can be a bit daunting and time consuming, +even if you're already familiar with the procedure. You will have dozens of +branches flying around in no time, and if you get stuck in a weird corner +with `git` you may quickly end up demotivated. + +This is frustrating not only for the people who wanted to contribute but +also for the EasyBuild maintainers, and it doesn't agree with the philosophy of +a project that aims to *automate* tedious software installation procedures. + +At the end of 2015 efforts were made to tackle this issue by implementing +GitHub integration features in EasyBuild, which automate the contribution +workflow by running `git` commands and interacting with the [GitHub API](https://developer.github.com/v3/). + +We will briefly go over some of these features here, but they are also covered in detail [in the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html). + +### Requirements & configuration + +First of all, the GitHub integration features impose a couple of additional [requirements](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) +and configuration. + + +**Additional dependencies** + +Both the `GitPython` and `keyring` Python packages as well as the `keyrings.cryptfile` add-on package must be installed. +In the prepared environment, you can do this via: + +```shell +pip3 install --user GitPython keyring keyrings.cryptfile +``` + +!!! Note + You may experiences problems installing the ``cryptography`` Python packages, + which is a dependency of keyring. The underlying cause is that you need to have + the [``Rust``](https://www.rust-lang.org/) compiler installed to install the latest version + of ``cryptography`` (see [here](https://github.com/pyca/cryptography/issues/5771)). + + You can work around this issue using: + + ```shell + pip3 install --user 'cryptography<3.4' + ``` + +**SSH public key in GitHub account** + +You need to have a GitHub account that has your SSH public key registered in it +(via [https://github.com/settings/keys](https://github.com/settings/keys)). + +If you need to generate an SSH key pair, you can run the following command: + +```shell +ssh-keygen -t rsa -b 4096 +``` + +You can copy the SSH public key from the output of this command: + +```shell +cat .ssh/id_rsa.pub +``` + + +**Forked repository in GitHub** + +In addition, you must have *forked* the EasyBuild repository you want to contribute to +(for example [https://github.com/easybuilders/easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs)). + +**EasyBuild configuration, incl. GitHub token** + +You also have to configure EasyBuild a bit more, so it knows about your +GitHub user name *and* has a GitHub token available in order to perform actions +in GitHub with your credentials. + +To do this, you should define the `github-user` configuration option and +run the "`eb --install-github-token`" command: + +```shell +# replace 'ebtutorial' with your own GitHub username! +$ export EASYBUILD_GITHUB_USER=ebtutorial +$ eb --install-github-token +``` + +To create a GitHub token: + +* Visit [https://github.com/settings/tokens](https://github.com/settings/tokens). +* Click *"Personal access tokens"*. +* Click followed by *"Generate new token"*. +* Give the token a name (for example *"Token for EasyBuild"*). +* Select both the '`repo`' and '`gist`' scopes. +* Click the green *"Generate token"* button. +* Copy the generated token. +* Paste the token when asked by `--install-github-token` (and hit *Enter*). +* Enter a password to encrypt your GitHub token. + +The output should look something like this: + +```shell +$ eb --install-github-token +== temporary log file in case of crash /tmp/eb-9z0bdve9/easybuild-hfpti62w.log +Token: +Validating token... +Token seems to be valid, installing it. +Please set a password for your new keyring: +Please confirm the password: +Token 'fed..987' installed! +``` + + +**Checking status of GitHub integration** + +You can check the status of the GitHub integration using "`eb --check-github`": + +```shell +$ eb --check-github +== temporary log file in case of crash /tmp/eb-4ckdlyfy/easybuild-gp69ev2w.log + +Checking status of GitHub integration... + +Making sure we're online...OK + +* GitHub user...ebtutorial => OK +Please enter password for encrypted keyring: +* GitHub token...fed..987 (len: 40) => OK (validated) +* git command...OK ("git version 1.8.3.1; ") +* GitPython module...OK (GitPython version 3.1.3) +* push access to ebtutorial/easybuild-easyconfigs repo @ GitHub...OK +* creating gists...OK +* location to Git working dirs... not found (suboptimal) + +All checks PASSed! + +Status of GitHub integration: +* --from-pr: OK +* --new-pr: OK +* --review-pr: OK +* --update-pr: OK +* --upload-test-report: OK +``` + +If you see '`OK`' for each of the status checks, you're all set +to try out the GitHub integration features! + +!!! Note + If your SSH private key is protected with a password, you may need + to enter your password a couple of times when running "`eb --check-github`". + + You can avoid this by [using an SSH agent](https://help.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent). + +!!! Note + If you see the ``push access`` check fail with ``Failed to fetch branch 'main'``, + you will need to rename the ``master`` branch in your fork of the ``easybuild-easyconfigs`` + repository from ``master`` to ``main`` (this is required since EasyBuild v4.3.3). + + This can be done via the *pencil* icon at + [https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches](https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches) + (replace ``YOUR_GITHUB_ACCOUNT`` with the name of your GitHub account in this URL). + +### Creating pull requests + +The GitHub integration in EasyBuild allows you to **create pull requests +using the `eb` command**, without even leaving your shell environment. +How cool is that‽ + +To create a pull request to the `easybuild-easyconfigs` repository, +you can either do it in a single go by +running "`eb --new-pr`" and passing it one or more easyconfig files to add +into the pull request. + +The more detailed option is to first create a branch in your repository fork +in GitHub via "`eb --new-branch-github`" and then later open the pull request +via "`eb --new-pr-from-branch`". This method can be useful when preparing multiple +interdependent pull requests, or to check whether your changes pass the unit tests +(which are run automatically in the GitHub Actions CI environment for +all branches pushed to your fork). + +The `--new-pr` option can also be used to open pull requests to the easyblocks +and framework repositories, and it will even automatically determine the target +repository based on the contents of the files you provide. Whoa! + +You can control the target repository for your pull request using +`--pr-target-account` (default is `easybuilders`) and `--pr-target-repo`. + +If you want you can carefully double check your work before actually +opening the pull request by doing a dry run via "`eb --dry-run --new-pr`" +or "`eb -D --new-pr`". + +Finally, you can use "`eb --preview-pr`" to see how the easyconfig files +for which you plan to create a pull request differ from existing easyconfig +files. + +### Updating pull requests + +To update an existing pull request with additional changes +you can use "`eb --update-pr`" and pass the pull request ID, +alongside the paths to the updated files. + +If you have only created a branch (for example via `eb --new-branch-github`) +you can update it via `--update-branch-github` in the same way, +passing the branch name instead of a pull request ID. + +### Using a pull request + +Next to creating and updating branches and pull requests +you can also *use* easyconfig files and easyblocks from a pull request, +regardless of its status (open, merged, or closed). This is particularly +useful when testing contributions, or to install software for which +support is not yet included in the latest EasyBuild release. + +Using the `--from-pr` option you can install easyconfig files from the +pull request with specified ID. By default all easyconfig files that are +touched by the pull request will be installed, but you can specify +particular ones to use as well. It is generally advised to also use the +`--robot` option to ensure that the easyconfig files are installed in the +correct order with respect to dependencies. + +Similarly, using a new or updated easyblock from a pull request is as simple +as using the `--include-easyblocks-from-pr` option. And of course you can +combine it with `--from-pr`! + +Via `--upload-test-report` you can let EasyBuild submit a comment into the +easyconfig pull request to show that the installation worked on your system. This is +useful for others to know, in particular EasyBuild maintainers, since the comment +will include information about your system (OS, processor, etc.) and your EasyBuild configuration. + +## Demo + +That is a lot to digest, so let us make this a bit more concrete with an example: +we will open a pull request for the [`eb-tutorial` example software](../adding_support_software/#example) to *a fork* of the [`easybuild-easyconfigs` repository](https://github.com/easybuilders/easybuild-easyconfigs) using the `eb` command, +and submit a test report in it. + +!!! Note + Make sure that you have correctly configured the GitHub integration, + [see above](#requirements-configuration). + +### Creating pull request + +We first configure EasyBuild to target the `ebtutorial` GitHub account rather +than the default `easybuilders` GitHub organisation, +by defining the `pr-target-account` configuration setting: + +```shell +export EASYBUILD_PR_TARGET_ACCOUNT=ebtutorial +``` + +In the output of "`eb --show-config`" you should see a line like this: + +``` +pr-target-account (E) = ebtutorial +``` + +We only do this to avoid that lots of pull requests for the `eb-tutorial` +example software are opened in the [central easyconfigs repository](https://github.com/easybuilders/easybuild-easyconfigs). + +Opening a pull request is as simple as running "`eb --new-pr`" and passing +the easyconfig file: + +```shell +$ eb --new-pr example.eb +== temporary log file in case of crash /tmp/eb-ggr6scbq/easybuild-hnk271xj.log +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +== fetching branch 'develop' from https://github.com/ebtutorial/easybuild-easyconfigs.git... +== copying files to /tmp/eb-ggr6scbq/git-working-dirxwk1fzaw/easybuild-easyconfigs... +== pushing branch '20200622095415_new_pr_eb-tutorial100' to remote 'github_ebtutorial_qgtfU' (git@github.com:ebtutorial/easybuild-easyconfigs.git) +Enter passphrase for key '/home/example/.ssh/id_rsa': +Please enter password for encrypted keyring: + +Opening pull request +* target: ebtutorial/easybuild-easyconfigs:develop +* from: ebtutorial/easybuild-easyconfigs:20200622095415_new_pr_eb-tutorial100 +* title: "{tools}[GCC/10.2.0] eb-tutorial v1.0.1" +* labels: new +* description: +""" +(created using `eb --new-pr`) + +""" +* overview of changes: + easybuild/easyconfigs/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb | 26 ++++++++++++++++++++++++++ + 1 file changed, 26 insertions(+) + +Opened pull request: https://github.com/ebtutorial/easybuild-easyconfigs/pull/ +== Temporary log file(s) /tmp/eb-ggr6scbq/easybuild-hnk271xj.log* have been removed. +== Temporary directory /tmp/eb-ggr6scbq has been removed. +``` + +Take a moment to grasp what we did here: we ran **a single `eb` command** which +took care of the **[whole contribution procedure](#contribution-procedure)** for us, including: + +* Cloning the `easybuilders/easybuild-easyconfigs` repository and checking out the `develop` branch (in a temporary + directory); +* Picking a sensible name for a branch and creating it; +* Adding the `eb-tutorial` easyconfig file to the branch, in the correct location + (`easybuild/easyconfigs/e/eb-tutorial/`) and with the correct filename (`eb-tutorial-1.0.1-GCC-10.2.0.eb`); +* Pushing the branch to our fork (`example/easybuild-easyconfigs`); +* Actually opening the pull request, using an informative title. + +That is so... easy! + +This feature not only *significantly* lowers the bar for contributing, +it also saves quite a bit of time since you don't need to double check +various details (like targeting the `develop` branch) or spend time on +coming up with a nice looking title or funny branch name (although you +still can if you really want to). + +There are a couple of nice side effects too, like not having any local branches +to tidy up on once the pull request gets merged (since `--new-pr` created the +branch only in a temporary directory). + +If many contributions are made via `--new-pr` it also simplifies the task +of EasyBuild maintainers, since pull requests opened this way have a particular +structure to them and thus are easier to digest because they look familiar. + +### Uploading test report + +After opening the pull request, we should also upload a test report to show that the installation is working. +This is just as easy as creating the pull request. + +First make sure that the pre-installed software in the prepared environment +is available, since the required dependencies for `eb-tutorial` are already +installed there: + +```shell +module use /easybuild/modules/all +``` + +You can verify which dependencies are still missing using `--from-pr` combined with `--missing`: + +```shell +# change '1' to the ID of your own pull request (see output of --new-pr) +$ eb --from-pr 1 --missing +== temporary log file in case of crash /tmp/eb-ioi9ywm1/easybuild-e3v0xa1b.log +Please enter password for encrypted keyring: +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + +1 out of 20 required modules missing: + +* eb-tutorial/1.0.1-GCC-10.2.0 (eb-tutorial-1.0.1-GCC-10.2.0.eb) +``` + +Uploading a test report boils down to combining `--from-pr` with `--upload-test-report`: + +```shell +# change '1' to the ID of your own pull request (see output of --new-pr) +$ eb --rebuild --from-pr 1 --upload-test-report +Please enter password for encrypted keyring: +... +== processing EasyBuild easyconfig /tmp/eb-bnb1pv3n/files_pr65/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb +== building and installing eb-tutorial/1.0.1-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 2 sec) +... +Adding comment to easybuild-easyconfigs issue #65: 'Test report by @ebtutorial +**SUCCESS** +Build succeeded for 1 out of 1 (1 easyconfigs in this PR) +example - Linux centos linux 7.8.2003, x86_64, Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz (haswell), Python 3.6.8 +See https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 for a full test report.' +== Test report uploaded to https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 and mentioned in a comment in easyconfigs PR#1 +``` + +Note that we may need to use `--rebuild` here since `eb-tutorial` may already be installed. + +This results in a comment being added to the pull request: + +
Test report comment
+ +The gist linked from this comment provides more detailed information: + +
Test report gist
+ +--- + +*[[next: Overview]](index.md)* From abfbf61873895d38113d3b8a7970fe53c8b625fe Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 30 Mar 2022 18:58:28 +0200 Subject: [PATCH 070/159] Troubleshooting section reworked for LUMI. --- docs/2022-CSC_and_LO/2_01_troubleshooting.md | 159 +++++++++++++------ 1 file changed, 108 insertions(+), 51 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_01_troubleshooting.md b/docs/2022-CSC_and_LO/2_01_troubleshooting.md index 86a2f1d3..30f2f5cc 100644 --- a/docs/2022-CSC_and_LO/2_01_troubleshooting.md +++ b/docs/2022-CSC_and_LO/2_01_troubleshooting.md @@ -4,8 +4,8 @@ --- -When installing scientific software you are bound to run into problems -that make the installation fail sooner or later, even when using EasyBuild. +Whatever tool you use, when installing scientific software you'll +be running into problems rather sooner than later. In this part we take a look at how you can **troubleshoot a failing installation**, and focus on a couple of EasyBuild aspects that can be helpful in that context, @@ -50,8 +50,8 @@ $ eb example.eb == building... == FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): build failed (first 300 chars): cmd "make" exited with exit code 2 and output: -/usr/bin/g++ -O2 -ftree-vectorize -march=native -fno-math-errno -std=c++14 -c -o core.o core.cpp -g++: error: unrecognized command line option '-std=c++14' (took 1 sec) +/usr/bin/g++ -O2 -ftree-vectorize -march=znver2 -fno-math-errno -c -o core.o core.cpp +cc1plus: error: bad value (‘znver2’) for ‘-march=’ switch (took 1 sec) == Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log ERROR: Build of /home/easybuild/subread.eb failed (err: ...) ``` @@ -60,22 +60,24 @@ Let's break this down a bit: during the `build` step of the installation procedure EasyBuild was running `make` as a shell command, which failed (exit code 2, so not zero). The `make` command tripped over the compilation of `core.cpp` that failed because -`-std=c++14` is not a known option to the `g++` command. +`-march=znver2` is not a known option to the `g++` command. OK fine, but now what? Can you spot something suspicious here? Wait a minute... Why is `make` using `/usr/bin/g++` for the compilation?! That's not where our toolchain compiler is installed, -that's somewhere under `/easybuild/software`. +that's somewhere under `/opt/cray/pe/gcc`. Let's see what `/usr/bin/g++` is: -```shell +``` $ /usr/bin/g++ --version -g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) +g++ (SUSE Linux) 7.5.0 ``` -Oh my that's pretty ancient (GCC 4.8.5 was released in June 2015). -That could definitely explain why it doesn't know about the C++14 standard yet... +Oh my that's an ancient compiler (7.5 was released on November 14, 2019, +a few months after the release of the Zen2 architecture, but +the base version, 7.1, is really from May 2, 2017, long before the Zen2 architecture was around) +That could definitely explain why it doesn't know about the Zen2 architecture yet... Your next step in this case should probably be figuring out why `/usr/bin/g++` is being used rather than just `g++`, which would @@ -98,7 +100,7 @@ In some cases there won't be any useful information in there however, since the actual error message(s) could only appear way later, perhaps even after the command was already running for several minutes. -In that case, you will have the dive into the log file that is created by EasyBuild for +In that case, you will have to dive into the log file that is created by EasyBuild for every installation, which is located in the unique temporary directory for the EasyBuild session. See for example this output line from our earlier example error message: @@ -122,14 +124,14 @@ directory for successful installation, into the `easybuild` subdirectory. For example: ``` -/easybuild/software/HDF5/1.10.7-gompi-2020b/easybuild/easybuild-HDF5-1.10.7-20210308.214453.log +/appl/lumi/SW/LUMI-21.12/L/EB/ncurses/6.2-cpeGNU-21.12/easybuild/easybuild-ncurses-6.2-20220302.110244.log ``` ### Last log The `eb` command supports a handy little option that prints the location to the most recently updated build log. You can leverage this to quickly -open the build log of the last failed EasyBuild session in an editor: +open the build log of the last ***failed*** EasyBuild session in an editor: ``` vim $(eb --last-log) @@ -206,7 +208,10 @@ For software using a classic `configure` script, you may have to locate and inspect the `config.log` file in the build directory to determine the underlying cause of an error. For software using CMake as a configuration tool you often have to check in `CMakeOutput.log` or `CMakeError.log` for clues, which are sneakily hidden by CMake in a `CMakeFiles` subdirectory of the build directory. -As a side note here: as EasyBuild does not clean out old and failed builds you will need to eventually manually remove these build directories from the `buildpath` directory. +As a side note here: as EasyBuild does not clean out old and failed builds you will need to eventually manually remove these build directories from the `buildpath` directory. On the login nodes of LUMI this is currently not much of an issue as `$XDG_RUNTIME_DIR` is (ab)used for build and +temporary files and that directory is cleaned automatically. However, when building on compute nodes, where `$XDG_RUNTIME_DIR` is +not available, it is essential to manually clean that directory as the space is not automatically cleaned when your session +ends. The `EasyBuild-user` module does define the bash function `clear-eb` that can be used to clear that space. ## Exercise @@ -214,7 +219,7 @@ Let's work our way through a less smooth software installation, using the easyconfig file that is provided below. Don't worry if most of this is still unclear to you, we'll get -to writing easyconfig files from scratch [later in this tutorial](../creating_easyconfig_files). +to writing easyconfig files from scratch [later in this tutorial](2_02_creating_easyconfig_files). ```python easyblock = 'MakeCp' @@ -225,7 +230,7 @@ version = '2.0.1' homepage = 'http://subread.sourceforge.net' description = "High performance read alignment, quantification and mutation discovery" -toolchain = {'name': 'GCC', 'version': '8.5.0'} +toolchain = {'name': 'PrgEnv-gnu', 'version': '21.10'} # download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz sources = ['subread-%(version)s-source.tar.gz'] @@ -253,25 +258,23 @@ Do you spot any potential problems yet with this easyconfig file? ***Preparation*** -Start by copying the text above in a file named `subread.eb`, +Start by copying the text above in a file named `subread.eb` +(which does not follow the EasyBuild conventions but that is not a problem for this exercise), so you can gradually fix the problem you'll encounter. Also make sure that the pre-installed software stack is available, -and that the EasyBuild module is loaded (unless you installed EasyBuild +and that the EasyBuild-user module is loaded (unless you installed EasyBuild yourself): ``` -module use /easybuild/modules/all -module load EasyBuild +module load LUMI/21.12 +module load EasyBuild-user ``` -For this exercise, make sure EasyBuild is configured to -use `$HOME/easybuild` as `prefix`, and to use `/tmp/$USER` as `buildpath`: - -```shell -export EASYBUILD_PREFIX=$HOME/easybuild -export EASYBUILD_BUILDPATH=/tmp/$USER -``` +This will configure EasyBuild correctly for this exercise, though if you already have +an existing EasyBuild user installation you may want to work in a different one +by pointing `$EBU_USER_PREFIX` to the desired work directory before loading +`LUMI/21.12`. Check your configuration via `eb --show-config`. @@ -297,7 +300,7 @@ the easyconfig file? ``` $ eb subread.eb ... - == FAILED: Installation ended unsuccessfully (build directory: /tmp/example/Subread/2.0.1/GCC-8.5.0): build failed (first 300 chars): + == FAILED: Installation ended unsuccessfully (build directory: /run/user/XXXXXXXX/easybuild/build/Subread/2.0.1/cpeGNU-21.12): build failed (first 300 chars): Couldn't find file subread-2.0.1-source.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order): ... ``` @@ -315,12 +318,10 @@ the easyconfig file? (in the `sourcepath` directory): ``` curl -OL https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz - mv subread-2.0.1-source.tar.gz $HOME/easybuild/sources/s/Subread/ + mkdir -p $EBU_USER_PREFIX/sources/s/Subread + mv subread-2.0.1-source.tar.gz $EBU_USER_PREFIX/sources/s/Subread/ ``` - If downloading is problematic for some reason, the source tarball is also available - in `/easybuild/sources/s/Subread`. - Or, we can change the easyconfig file to specify the location where the easyconfig file can be downloaded from: ```python @@ -335,9 +336,9 @@ the easyconfig file? The source tarball is fairly large (23MB), so don't be alarmed if the download takes a little while. ```shell - $ ls -lh $HOME/easybuild/sources/s/Subread + $ ls -lh $EBU_USER_PREFIX/sources/s/Subread total 23M - -rw-rw-r-- 1 easybuild easybuild 23M Jun 13 17:42 subread-2.0.1-source.tar.gz + -rw-rw-r-- 1 XXXXXXXX XXXXXXXX 23M Mar 30 16:08 subread-2.0.1-source.tar.gz ``` --- @@ -350,26 +351,52 @@ What's wrong now? How can you fix it quickly? Take into account that we just want to get this software package installed, we don't care too much about details like the version of the dependencies or -the *toolchain* here... +the toolchain here... ??? success "(click to show solution)" The installation fails because the easyconfig specifies that GCC 8.5.0 should be used as toolchain: + ``` $ eb subread.eb ... - == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-8.5.0): build failed (first 300 chars): - No module found for toolchain: GCC/8.5.0 (took 1 sec) + ERROR: Failed to process easyconfig /pfs/lustrep3/users/kurtlust/easybuild-tutorial/Troubleshooting/subread.eb: Toolchain PrgEnv-gnu not found, + available toolchains: ... + ... + ``` + + `PrgEnv-gnu` is an HPE Cray PE module that may look like a toolchain - it certainly has + the same function: provide compiler, MPI and basic math libraries - but it is not + recognised as a toolchain by EasyBuild. EasyBuild prefers to manage its own modules so that it knows + well what is in it which is not the case with the `PrgEnv-*` modules from the Cray PE + as the content may differe between systems and as the versions of the compilers etc. that + are loaded differ on other modules that are loaded. Hence we created Cray-specific toolchains. + You'll actually find two series of Cray toolchains in the list of available toolchains. The + `CrayGNU`, `CrayIntel`, `CrayPGI` and `CrayCCE` are included with the EasyBuild distribution + and where developed by CSCS for their systems using Environment Modules. These were not compatible + with the initial releases of the Cray PE with Lmod modules so new ones were developed on which we + also built for the LUMI toolchains. Those are called `cpeCray`, `cpeGNU`, `cpeAOCC` and `cpeAMD` + and are maintained by LUST and available via the LUMI repositories. + + Changing the toolchain name to `cpeGNU` is not enough to solve all problems though: + + ``` + $ eb subread.eb + ... + ERROR: Build of /pfs/lustrep3/users/kurtlust/easybuild-tutorial/Troubleshooting/subread.eb failed (err: 'build failed (first 300 chars): + No module found for toolchain: cpeGNU/21.10') + ... ``` - We don't have this GCC version installed, but we do have GCC 10.2.0: + We don't have this `cpeGNU` version installed, but we do have GCC 21.12: ```shell - $ module avail GCC/ - ----------------- /easybuild/modules/all ------------------ - GCC/10.2.0 + $ module avail cpeGNU/ + ----- Infrastructure modules for the software stack LUMI/21.12 on LUMI-L ----- + cpeGNU/21.12 + ... ``` So let's try using that instead. @@ -377,7 +404,7 @@ the *toolchain* here... Edit the easyconfig file so it contains this: ```python - toolchain = {'name': 'GCC', 'version': '10.2.0'} + toolchain = {'name': 'cpeGNU', 'version': '21.12'} ``` --- @@ -392,13 +419,13 @@ Can you fix the next problem you run into? The compilation fails, but the error message we see is incomplete due to EasyBuild truncating the command output (only the 300 first characters of the output are shown): ``` - == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-10.2.0): build failed (first 300 chars): - cmd " make -j 1 -f Makefile.Linux CFLAGS="-fast -fcommon"" exited with exit code 2 and output: - gcc -mtune=core2 -O3 -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\""2.0.1"\" -D_FILE_OFFSET_BITS=64 -fmessage-length=0 -ggdb -fast -fcommon -c -o core.o core.c - gcc: error: unrecognized command line opti (took 1 sec) + == FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12): build failed + (first 300 chars): cmd " make -j 256 -f Makefile.Linux CFLAGS="-fast -fcommon"" exited with exit code 2 and output: + gcc -mtune=core2 -O3 -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\""2.0.1"\" -D_FILE_OFFSET_BITS=64 -fmessage-length=0 + -ggdb -fast -fcommon -I/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86 (took 4 secs) ``` - If you open the log file and scroll to the end, + If you open the log file (e.g., with `view $(eb --last-log)`) and scroll to the end, the error is pretty obvious: ``` gcc: error: unrecognized command line option -fast; did you mean -Ofast? @@ -416,7 +443,8 @@ Can you fix the next problem you run into? to hard specify compiler flags (certainly not incorrect ones). The comment above the `buildopts` definition makes it clear that the `-fcommon` flag *is* required though, because GCC 10 became a bit stricter by - using `-fno-common` by default. Note that we are using `-fcommon` + using `-fno-common` by default (and we're using GCC 11 in `cpeGNU/21.12`). + Note that we are using `-fcommon` as an escape mechanism here: it would be better to fix the source code and create a patch file instead. @@ -433,6 +461,32 @@ Can you fix the next problem you run into? Note that we need to be careful with quotes here: we use inner double quotes to ensure that `$CFLAGS` will be expanded to its value when the build command is run. + As you can see after the change by doing a dry-run: + + ``` + $ eb subread.eb -x + ... + + Defining build environment... + + export BLAS_INC_DIR='/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/include' + ... + export CC='cc' + export CFLAGS='-O2 -ftree-vectorize -fno-math-errno' + ... + [build_step method] + running command "make -j 256 -f Makefile.Linux CFLAGS="$CFLAGS -fcommon"" + (in /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12/Subread-2.0.1/src) + ... + ``` + + EasyBuild will launch the command + ``` + make -j 256 -f Makefile.Linux CFLAGS="$CFLAGS -fcommon" + ``` + in a shell where `CFLAGS` is defined and set to an appropriate value (determined by + defaults in EasyBuild, settings in the EasyBuild configuration and settings in the + easyconfig file that we shall discuss later). --- @@ -449,10 +503,11 @@ Don't give up now, try one last time and fix the last problem that occurs... ``` $ eb subread.eb ... - == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-10.2.0): build failed (first 300 chars): - Sanity check failed: sanity check command featureCounts --version exited with code 255 + == FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12): + build failed (first 300 chars): Sanity check failed: sanity check command featureCounts --version exited with code 255 (output: featureCounts: unrecognized option '--version' ... + ... ``` If you look at the full output in the log file you can see @@ -473,12 +528,14 @@ Don't give up now, try one last time and fix the last problem that occurs... --- -In the end, you should be able to install Subread 2.0.1 with the GCC 10.2.0 toolchain by fixing the problems with the `subread.eb` easyconfig file. +In the end, you should be able to install Subread 2.0.1 with the cpeGNU 21.12 toolchain by fixing the problems with the `subread.eb` easyconfig file. Check your work by manually loading the module and checking the version via the `featureCounts` command, which should look like this: ```shell +$ module load Subread/2.0.1-cpeGNU-21.12 +... $ featureCounts -v featureCounts v2.0.1 ``` From 5881892df8e7cb2e5c909d01e30a66cd7ea97193 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 1 Apr 2022 18:15:00 +0200 Subject: [PATCH 071/159] Section on creating easyconfig files. --- .../2_02_creating_easyconfig_files.md | 527 +++++++++--------- 1 file changed, 278 insertions(+), 249 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md index d46e003a..99ac7d88 100644 --- a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md @@ -4,8 +4,9 @@ --- -We already know how to install easyconfig files that are included with EasyBuild, -but what about installing software for which no easyconfig is available yet? +We already know how to install easyconfig files that are provided by LUMI, +but what about installing software for which no easyconfig is available yet, +or for which we need to adapt an existing easyconfig? To do this we will need to **create additional easyconfig files**, since every software installation performed by EasyBuild is done based on an easyconfig file. @@ -14,10 +15,10 @@ In this part of the tutorial we will look at the guts of easyconfig files and ev ## Easyconfigs vs easyblocks -Before we dive into writing [easyconfig files](../introduction/#easyconfig-files), -let us take a brief look at how they relate to [easyblocks](../introduction/#easyblocks). +Before we dive into writing [easyconfig files](../1_05_terminology/#easyconfig-files), +let us take a brief look at how they relate to [easyblocks](../1_05_terminology/#easyblocks). -As we discussed [earlier](../introduction/#terminology), an easyconfig file (`*.eb`) is required +As we discussed [earlier](../1_05_terminology/#terminology), an easyconfig file (`*.eb`) is required for each installation performed by EasyBuild which specifies the details of the installation (which software version, toolchain, etc.), while the installation procedure is implemented @@ -30,34 +31,47 @@ together with a custom *software-specific* easyblock? This is not an easy question to answer in a general sense, since it depends on several factors: the complexity of the software you want to get installed, how much flexibility you want, how "intelligent" the installation procedure should be with respect to the compiler toolchain and dependencies -that are used for the installation, etc. +that are used for the installation, whether you want an easyconfig that is more user-focused (with easy-to-understand +parameters) or one that is more focused towards people who understand installation commands, etc. -In a nutshell, custom software-specific easyblocks are "do once and forget": they are central solution to peculiarities in the installation procedure of a particular software package. +In a nutshell, custom software-specific easyblocks are "do once and forget": they are central solution to peculiarities in the installation procedure of a particular software package. However, they also hide a lot from direct view, so if things go +wrong, it is often harder to debug the exact problem. And since a single easyblock has to cover +multiple software versions, multiple toolchains and multiple possible configurations of the package, +they are often harder to develop and certainly harder to test. As a result many of the easyblocks +included with EasyBuild work poorly on HPE Cray systems, e.g., because they want to add compiler flags +specific for a certain compiler and don't recognise the Cray compilers. Reasons to consider implementing a software-specific easyblock rather than using a generic easyblock include: -* 'critical' values for easyconfig parameters required to make installation succeed; -* toolchain-specific aspects of the build and installation procedure (e.g., configure options); -* interactive commands that need to be run; -* custom (configure) options for dependencies; -* having to create or adjust specific (configuration) files; -* 'hackish' usage of a generic easyblock; -* complex or very non-standard installation procedure; - -Implementing easyblocks is out of scope for this basic tutorial, for more information please consult -the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). +- 'critical' values for easyconfig parameters required to make installation succeed; +- toolchain-specific aspects of the build and installation procedure (e.g., configure options); +- interactive commands that need to be run; + *For example: The easyblock to [install maple](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/m/maple.py) + responds to a number of interactive questions.* +- custom (configure) options for dependencies; + *For example, the [easyblock for VMD](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/v/vmd.py) + will add configure options depending on the dependency list of the package.* +- having to create or adjust specific (configuration) files; +- 'hackish' usage of a generic easyblock; +- complex or very non-standard installation procedure; + *For example, the [easyblock to install the gcc compilers from source](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/g/gcc.py), + bootstrapping with the system compiler and then re-installing with itself.* + +For implementing easyblocks we refer to the +["Implementing easyblocks" section of this tutorial](2_03_implementing_easyblocks.md) +and the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). ## Writing easyconfig files Writing an easyconfig file boils down to defining a set of easyconfig parameters in a text file, which we give a '`.eb`' extension by convention. The name of an easyconfig file doesn't matter when using it directly to install software, but it does matter when EasyBuild needs to find it -to resolve a dependency for example (as we [discussed earlier](../basic_usage/#easyconfig-filenames)). +to resolve a dependency for example (as we [discussed earlier](../1_08_basic_usage/#easyconfig-filenames)). The syntax for easyconfig files is *Python syntax*: you are basically defining a bunch of Python variables that correspond to easyconfig parameters. -The order in which the easyconfig parameters are defined doesn't matter, but we generally try to strick to a particular +The order in which the easyconfig parameters are defined doesn't matter, but we generally try to stick to a particular order which roughly corresponds to the order in which the easyconfig parameters are used during the installation. That is mostly helpful for humans staring at easyconfig files or comparing them. @@ -65,13 +79,13 @@ That is mostly helpful for humans staring at easyconfig files or comparing them. A limited number of easyconfig parameters are *mandatory*, they must be defined in every easyconfig file: -* `name`: the name of the software to install; -* `version`: the version of the software to install; -* `homepage`: a URL to the website of the software; -* `description`: a short description of the software; -* `toolchain`: the compiler toolchain to use for the installation; +- `name`: the name of the software to install; +- `version`: the version of the software to install; +- `homepage`: a URL to the website of the software; +- `description`: a description of the software; +- `toolchain`: the compiler toolchain to use for the installation; -**`name`, `version`** +#### `name`, `version` It should be no surprise that specifying the name and version of the software you want to install is mandatory. This information may influence the value of several other easyconfig parameters (like the name of the source file), and is also used to the determine the name of the module file to install. @@ -81,7 +95,7 @@ name = 'example' version = '1.0' ``` -**`homepage`, `description`** +#### `homepage`, `description` The homepage and description are included in the generated module file for the installation. That way the "`module show`" command provides some useful high-level information about the installation. @@ -91,10 +105,8 @@ homepage = 'https://example.org' description = "This is just an example." ``` -Usually it does not matter whether you use single or double quotes to specify string values, -but you will often see that single quotes are used for values that don't have spaces (words) -and double quotes for values that do have spaces (sentences). There is no technical reason for -this, it just feels more natural to some people. There are cases where it is important to use +Usually it does not matter whether you use single or double quotes to specify string values. +There are cases where it is important to use the right type of quotes however, we will get back to that later (keep it in mind for the exercises!). For multi-line descriptions, you will need to use "triple quoting" (which is standard Python syntax): @@ -105,23 +117,43 @@ description = """This is an example It is spread across multiple lines.""" ``` -**`toolchain`** +If no homepage is known for the package, the convention in the EasyBuild community is to use +`(none)` as the value. + +The `description` field is used in two different places in the LUA module file generated by +EasyBuild: +- In the help block, for the first section of the help information +- If there is no `whatis` parameter in the easyconfig file, it is also used to generate + a whatis-command with the description in the module file. + +*Note:* We ask people who contribute easyconfigs to LUMI to instead use a decent description +that is useful in the help information, and to specify a short description via the `whatis` +parameter: + +```python +whatis = ['Description: Blosc is an extremely fast, multi-threaded, meta-compressor library'] +``` + +Note that in this case the word "Description:" has to be explicitly specified. -EasyBuild also requires that the [compiler toolchain](../introduction/#toolchains) is specified, via the `toolchain` + +#### `toolchain` + +EasyBuild also requires that the [compiler toolchain](../1_05_terminology/#toolchains) is specified, via the `toolchain` easyconfig parameter. -This can either be the [`system` toolchain](../introduction/#system-toolchain), for which a constant named `SYSTEM` is available: +This can either be the [`system` toolchain](../1_05_terminology/#system-toolchain), for which a constant named `SYSTEM` is available: ```python toolchain = SYSTEM ``` -Usually we specify a 'proper' toolchain like the compiler-only toolchain GCC 10.2.0 which we used before, -or the full toolchain `foss` 2020b. The name and version of the toolchain can be specified using a small Python dictionary, +Usually we specify a 'proper' toolchain like the `cpeGNU/21.12` toolchain we have used bvefore. +The name and version of the toolchain can be specified using a small Python dictionary, for example: ```python -toolchain = {'name': 'GCC', 'version': '10.2.0'} +toolchain = {'name': 'cpeGNU', 'version': '21.12'} ``` ### Commonly used parameters @@ -177,8 +209,11 @@ Some things worth pointing out here: file. This way the software version is only specified in one place and the easyconfig file is easier to update to other software versions. A list of template values can be consulted via the EasyBuild command line via the `--avail-easyconfig-templates` option, or in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_templates.html). -* Source files can also be specified in ways other than just using a filename, see the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-param-sources-alt) for more information. -* Specified checksums are usually SHA256 checksum values, but [other types are also supported](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html?highlight=checksums#checksums). +* Source files can also be specified in ways other than just using a filename, see the + [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-param-sources-alt) for more information. + It is also possible to download a given commit from a GitHub repository. +* Specified checksums are usually SHA256 checksum values, but + [other types are also supported](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html?highlight=checksums#checksums). #### Easyblock @@ -189,7 +224,9 @@ This is not mandatory however, because by default EasyBuild will determine the e name of the software. If '`example`' is specified as software name, EasyBuild will try to locate a software-specific easyblock named `EB_example` (in a Python module named `example.py`). Software-specific easyblocks follow the convention that the class name starts with `'EB_`', followed by the software name -(where some characters are replaced, like '`-`' with '`_minus_`'). +(where some characters are replaced, like '`-`' with '`_minus_`'). It is possible to use different +naming conventions for software-specific easyblocks, but then EasyBuild will not automatically detect +that there is one for the package and it will also need to be specified via the `easyblock` parameter. **Generic easyblocks** @@ -234,7 +271,7 @@ tar_config_opts* Override tar settings as determined by configure. [defau #### Dependencies -You will often need to list one or more [dependencies](../introduction/#dependencies) that are required +You will often need to list one or more [dependencies](../1_05_terminology/#dependencies) that are required to install or run the software. We distinguish between two main different types of dependencies: runtime dependencies and build dependencies. @@ -248,12 +285,17 @@ software once it is installed. The modules for these dependencies will be loaded set up by EasyBuild during the installation, but they will *not* be loaded by the generated module file. You can specify build dependencies via the `builddependencies` easyconfig parameter. One typical example of a build dependency is `CMake`, which is only needed for configuring -the build. +the build. On LUMI we define a `buildtools` module for each version of the `LUMI` software stack +which contains most popular build tools. This ensures that we use a consistent set of build tools +and that they do not change over the life time of a stack, also in case of OS upgrades, to +increase the reproducibility of the build process Here is a simple example of specifying dependencies: ```python -builddependencies = [('CMake', '3.18.4')] +builddependencies = [ + ('buildtools', '%(toolchain_version)s', '', True) +] dependencies = [ ('Python', '3.8.2'), @@ -268,20 +310,38 @@ The name and version of a dependency is specified with a 2-tuple (a tuple with t In some cases additional information may have to be provided, as is shown in the example above for the `SciPy-bundle` dependency where a 3rd value is specified corresponding to the `versionsuffix` value of this dependency. -If this is not specified, it is assumed to be the empty string (`''`). +If this is not specified, it is assumed to be the empty string (`''`). Note how we use the '`%(pyver)s'` template value in the `SciPy-bundle` dependency -specification, to avoid hardcoding the Python version in different places. +specification, to avoid hardcoding the Python version in different places. (Though this +specific parameter is less useful on LUMI as we currently try to build on top of `cray-python`.) + +The `buildtools` build dependency shows that there is a fourth parameter specifying the toolchain +used for that dependency and is needed if that toolchain is different from the one used in the example. +As it is not possible to load several Cray toolchains together (they are not in a hierachical relation) +the only useful value on LUMI is `True` which tells that `buildtools` is build with the `SYSTEM` +toolchain. Here also we use a template, `%(toolchain_version)s` which - as its name suggests - expands +to the version of the toolchain, as we version our `buildtools` modules after the version of the Cray +toolchains for which they are intended. + +See also the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#dependencies) +for additional options on specifying dependencies. That page specifies two more dependency types: -See also the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#dependencies) for additional options on specifying dependencies. +* `hiddendependencies` are currently not used on LUMI, and if we would use them in the future, it will likely + be through a way that does not require this parameter. +* `osdependencies` can be used to let EasyBuild check if certain needed OS packages are installed. + See, e.g., the [easyconfigs for the `buildtools` package](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/easybuild/easyconfigs/b/buildtools) + on LUMI. #### Version suffix -In some cases you may want to build a particular software package in different configurations, or include a label in the module name to highlight a particular aspect +In some cases you may want to build a particular software package in different configurations, +or include a label in the module name to highlight a particular aspect of the installation. -The `versionsuffix` easyconfig parameter can be used for this purpose. The name of this parameter implies that this label will be added after the +The `versionsuffix` easyconfig parameter can be used for this purpose. +The name of this parameter implies that this label will be added after the software version (and toolchain label) in the standard module naming scheme. If you are configuring the software to build with a particular non-default value, @@ -303,7 +363,8 @@ dependencies = [('Python', '2.7.18')] Even though Python 2 is officially dead and buried some scientific software still requires it, and mixing modules where some use Python 2 and other use Python 3 doesn't work well. -The `versionsuffix` label is helpful to inform the user that a particular Python version is required by the installation. +The `versionsuffix` label is helpful to inform the user that a particular Python version is +required by the installation. #### Customizing configure, build, test, and install commands @@ -330,6 +391,7 @@ prebuildopts = 'export HDF5_PREFIX="$EBROOTHDF5" && ' installopts = "PREFIX='%(installdir)s'" ``` +(again an imaginary example as on LUMI we advise to use the Cray-provided HDF5 modules.) Here we are: @@ -406,7 +468,13 @@ Finally, you will usually see the `moduleclass` easyconfig parameter to be defin moduleclass = 'lib' ``` -This is done to categorize software, and it is used to group the generated module files into smaller sets ([remember what we saw when installing software earlier](../basic_usage/#using-installed-software)). +This is done to categorize software, and it is used to group the generated module files into smaller sets +([remember what we saw when installing software earlier](../1_08_basic_usage/#using-installed-software)). + +This is currently not used on LUMI since we feel that (a) it is not easy to explain to users how they can +then only make certain classes that are useful to them visible and more importantly (b) since it is not +always intuitive to decide which moduleclass should be used for a package, or from a user 's perspective, +in which category to look for a package. ## Generating tweaked easyconfigs @@ -428,7 +496,7 @@ eb example-1.2.3.eb --try-software-version 1.2.4 Or, to try using a different compiler toolchain you can use `--try-toolchain`: ```shell -eb example-1.2.3-foss-2020b.eb --try-toolchain intel,2020b +eb example-1.2.3-foss-2020b.eb --try-toolchain cpeCray,21.12 ``` It is important to keep in mind the *"try"* aspect here: while easyconfigs that @@ -477,20 +545,18 @@ You can consult the unpacked sources at [https://github.com/easybuilders/easybui ### Preparation -Make sure EasyBuild is properly configured before you start: +Make sure that a proper version of the `LUMI` software stack is loaded +(we recommend `LUMI/21.12` for the session for which this tutorial was designed) +and also `EasyBuild-user` is loaded to configure EasyBuild to install in +`$HOME/EasyBuild` or `$EBU_USER_PREFIX`. -```shell -export EASYBUILD_PREFIX=$HOME/easybuild -export EASYBUILD_BUILDPATH=/tmp/$USER +``` shell +module load LUMI/21.12 +module load EasyBuild-user` ``` -and that the installed software in `/easybuild` is available: -```shell -module use /easybuild/modules/all -``` - -### Mandatory parameters +### Easyblock Let's start by getting the mandatory easyconfig parameters defined in the easyconfig file: @@ -499,48 +565,26 @@ name = 'eb-tutorial' version = '1.0.1' homepage = 'https://easybuilders.github.io/easybuild-tutorial' -description = "EasyBuild tutorial example" -``` -If we try using this (very) minimal easyconfig file, EasyBuild will inform us that we failed to specify one of the mandatory easyconfig parameters: `toolchain`: +whatis = [ 'Description: EasyBuild tutorial example'] -``` -$ eb example.eb -== Temporary log file in case of crash /tmp/eb-90j723rl/easybuild-q21plqvx.log -== found valid index for /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs, so using it... -ERROR: Failed to process easyconfig /home/example/example.eb: mandatory parameters not provided in pyheader: toolchain +description = """ +This is a short C++ example program that can be buid using CMake. +""" ``` -We will use `GCC/10.2.0` as toolchain, since we know it is already installed in `/easybuild` in the prepared environment, so we also define the `toolchain` easyconfig parameter: - -```python -toolchain = {'name': 'GCC', 'version': '10.2.0'} -``` - -In addition, we'll also specify the `moduleclass`. -This is not required, but it is usually set to a sensible value: - -```python -moduleclass = 'tools' -``` - -The default value is '`base`', at least '`tools`' has *some* meaning. - -### Easyblock - -Let us see what happens if we take our current easyconfig file for a spin: +Let's see what EasyBuild does with this very minimal easyconfig file: ```shell -$ eb example.eb -== temporary log file in case of crash /tmp/eb-8_vxjfn7/easybuild-k3aaoan2.log -ERROR: Failed to process easyconfig /home/example/example.eb: +$ eb eb-tutorial.eb +== Temporary log file in case of crash /run/user/XXXXXXXXX/easybuild/tmp/eb-k_82wotb/easybuild-wg0k_reb.log +ERROR: Failed to process easyconfig /pfs/lustrep4/users/XXXXXXXX/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v1.eb: No software-specific easyblock 'EB_eb_minus_tutorial' found for eb-tutorial ``` -That didn't get us very far... - -The error shows that there is no software-specific easyblock available for installing the software with the name '`eb-tutorial`'. -Does that mean we have to implement an easyblock? +It is not mandatory to specify an easyblock in the easyconfig. However, in the absense of that +specification, EasyBuild goes looking for an application-specific easyblock with the standard name, +in this case `EB_eb_minus_tutorial`, which it does not have. Does that mean we have to implement an easyblock? In this simple case it doesn't, since we can leverage one of the available *generic easyblocks*. But, which one? @@ -559,81 +603,90 @@ easyblock = 'CMakeMake' The "`easyblock =`" line is usually at the top of the easyconfig file, but strictly speaking the order of the parameter definitions doesn't matter (unless one is defined in terms of another one). -### CMake build dependency -Does using the `CMakeMake` generic easyblock help at all? +### Mandatory parameters + +When trying this improved easyconfig file, EasyBuild will inform us that we failed to specify +one of the mandatory easyconfig parameters: `toolchain`: ``` $ eb example.eb -== temporary log file in case of crash /tmp/eb-yutbor1p/easybuild-4jc9v1u9.log -== found valid index for /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs, so using it... -== processing EasyBuild easyconfig /home/example/example.eb -== building and installing eb-tutorial/1.0.1-GCC-10.2.0... -== fetching files... -== creating build dir, resetting environment... -== unpacking... -== patching... -== preparing... -== configuring... -== FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/ebtutorial/1.0.1/GCC-10.2.0): -build failed (first 300 chars): cmd " cmake -DCMAKE_INSTALL_PREFIX=/home/example/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER='gcc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno' -DCMAKE_CXX_COMPILER='g++' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=native (took 0 sec) +== Temporary log file in case of crash /run/user/XXXXXXXXX/easybuild/tmp/eb-22om7hut/easybuild-ipozjipf.log +ERROR: Failed to process easyconfig /pfs/lustrep4/users/XXXXXXXX/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v2.eb: +mandatory parameters not provided in pyheader: toolchain ``` -It did help: EasyBuild made an attempt to configure the build using the `cmake` command, but that failed almost -instantly. We need to dive into the log file to see the actual reason. By starting at the end of the log file and -scrolling up, you should be able to locate the following error message: +We will use `cpeCray/21.12` as toolchain, so we also define the `toolchain` easyconfig parameter: +```python +toolchain = {'name': 'cpeCray', 'version': '21.12'} ``` -/bin/bash: cmake: command not found + +In addition, we'll also specify the `moduleclass`. +This is not required, but it is usually set to a sensible value: + +```python +moduleclass = 'tools' ``` -Ah, that explains it, `cmake` isn't even installed on this system. Or is it? +The default value is '`base`', at least '`tools`' has *some* meaning. -```shell -$ module avail CMake +### CMake build dependency ---------------------------- /easybuild/modules/all ---------------------------- - CMake/3.18.4-GCCcore-10.2.0 -``` +The `CMakeMake` easyblock needs the `cmake` command. On LUMI we are lucky as +`cmake` is already installed in the OS. It may be a somewhat older version, but for this +program is is probably enough. Yet in general it is better to use sufficiently recent +build tools, and `cmake` is one of those tools that is typically entered as a build +dependency. After all, the less you rely on the OS, the more likely it becomes that +your easyconfig is useful for other sites also. -Since a module is available for `CMake` that is compatible with the toolchain we are using (GCC 10.2.0), -we can use it as a dependency for the installation. -It is only needed for building the software, not for running it, so it's only a *build* dependency: +In the [section on Lmod](../1_02_Lmod#module-extensions) we've already seen that on LUMI +the `cmake` command is available through the `buildtools` modules, and as discussed in +other examples on this page, LUMI has one for every `LUMI` software stack with its version +number the same as the stack and corresponding toolchains. It is a good practice to +add this module as a build dependency: ```python -builddependencies = [('CMake', '3.18.4')] +builddependencies = [ + ('buildtools', '%(toolchain_version)s', '', True) +] ``` -There is usually no need to specify toolchain for (build) dependencies, EasyBuild will automatically consider -[subtoolchains](../terminology/#toolchains) compatible with the specified toolchain to locate module for the dependencies. +In a more traditional EasyBuild setup with the common toolchains, there is usually no need to specify +the toolchain for (build) dependencies. EasyBuild will automatically consider +[subtoolchains](../1_05_terminology/#toolchains) compatible with the specified toolchain to locate +modules for the dependencies. However, the Cray PE toolchains on LUMI are currently not part of +such a hierarchy and the `SYSTEM` toolchain we used for `buildtools` is not automatically considered +which is why we need the 4-element version of the dependency specification. -You can verify this via `eb -D` (equivalent with `eb --dry-run`): +You can verify that EasyBuild now locates the dependency via `eb -D` (equivalent with `eb --dry-run`): ``` -$ eb example.eb -D +$ eb eb-tutorial.eb -D ... - * [x] /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs/g/GCC/GCC-10.2.0.eb (module: GCC/10.2.0) - * [x] /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs/c/CMake/CMake-3.18.4-GCCcore-10.2.0.eb (module: CMake/3.18.4-GCCcore-10.2.0) - * [ ] /home/example/example.eb (module: eb-tutorial/1.0.1-GCC-10.2.0) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeCray/cpeCray-21.12.eb (module: cpeCray/21.12) + * [ ] /pfs/lustrep4/users/kulust/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v4.eb (module: eb-tutorial/1.0.-cpeCray-21.12) ``` ### Sources -If you try again after adding `CMake` as a build dependency, you will see the installation fail again in the +If you try again after adding `buildtools` as a build dependency, you will see the installation fail again in the configuration step. Inspecting the log file reveals this: ``` -CMake Error: The source directory "/tmp/example/ebtutorial/1.0.1/GCC-10.2.0" does not appear to contain CMakeLists.txt. +CMake Error: The source directory "/run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12" does not appear to contain CMakeLists.txt. ``` -Wait, but there *is* a `CMakeLists.txt`, we can see it in the [unpacked sources](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1)! +Wait, but there *is* a `CMakeLists.txt`, we can see it in the +[unpacked sources](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1)! Let's inspect the build directory: ``` -$ ls /tmp/$USER/ebtutorial/1.0.1/GCC-10.2.0 +$ ls /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12 easybuild_obj -$ ls /tmp/$USER/ebtutorial/1.0.1/GCC-10.2.0/easybuild_obj +$ ls /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12/easybuild_obj $ ``` @@ -672,7 +725,8 @@ If now we try installing the easyconfig file again, EasyBuild complains that it can't find the specified source file anywhere: ``` -Couldn't find file eb-tutorial-1.0.1.tar.gz anywhere, and downloading it didn't work either... +Couldn't find file eb-tutorial-1.0.1.tar.gz anywhere, and downloading it didn't work either... +Paths attempted (in order):... ``` To let EasyBuild automatically download the source file if it is not available yet, @@ -716,11 +770,13 @@ expanded when the configure command is run (see [Exercise 7.1](#exercises)). Hopefully that brings us closer to getting the installation to work... ``` -$ eb example.eb +$ eb eb-tutorial.eb .... == sanity checking... -== FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/ebtutorial/1.0.1/GCC-10.2.0): build failed (first 300 chars): -Sanity check failed: no (non-empty) directory found at 'lib' or 'lib64' in /home/easybuild/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 (took 2 sec) +== ... (took 2 secs) +== FAILED: Installation ended unsuccessfully (build directory: /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12): +build failed (first 300 chars): Sanity check failed: no (non-empty) directory found at 'lib' or 'lib64' in +/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 (took 7 secs) ``` It got all the way to the sanity check step, that's great! @@ -729,9 +785,9 @@ The sanity check failed because no '`lib`' or `'lib64'` directory was found. Indeed: ``` -$ ls $HOME/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 +$ ls /users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 bin -$ ls $HOME/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0/bin +$ ls /users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 eb-tutorial ``` @@ -757,10 +813,11 @@ a module, we know it will work as expected. By enabling trace mode via `--trace` we can get some more information too: ```shell -$ eb example.eb --module-only --trace +$ eb eb-tutorial.eb --module-only --trace ... == sanity checking... >> file 'bin/eb-tutorial' found: OK + >> loading modules: eb-tutorial/1.0.1-cpeCray-21.12... >> running command 'eb-tutorial' ... >> result for command 'eb-tutorial': OK ... @@ -773,7 +830,6 @@ To convince yourself that the installation works as intended, try to load the `e run the `eb-tutorial` command yourself: ``` -$ module use $HOME/easybuild/modules/all $ module load eb-tutorial $ eb-tutorial Hello from the EasyBuild tutorial! @@ -820,12 +876,73 @@ You can let EasyBuild determine *and* inject these SHA256 checksums automatically via `eb --inject-checksums`: ``` -$ eb example.eb --inject-checksums +$ eb eb-tutorial.eb --inject-checksums ... == injecting sha256 checksums for sources & patches in example.eb... == * eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 ``` +Finally, we should consider changing the name of the easyconfig that we just developed +to align with the EasyBuild conventions as otherwise it would not be found when used as +a dependency of another package. In this case, the name should be +`eb-tutorial-1.0.0-cpeCray-21.12.eb`. In fact, EasyBuild stored a processed version +of our easyconfig with that name in the repository: + +``` +$ ls $EASYBUILD_REPOSITORYPATH +eb-tutorial +$ ls $EASYBUILD_REPOSITORYPATH/eb-tutorial +eb-tutorial-1.0.1-cpeCray-21.12.eb +$ cat $EASYBUILD_REPOSITORYPATH/eb-tutorial/eb-tutorial-1.0.1-cpeCray-21.12.eb +... +``` + +Note that EasyBuild has added an additional parameter to the easyconfig file, `buildstats`, +with a lot of information about how easybuild was called and properties of the node on which +it was run. This file is still a valid easyconfig file though from which we can build the +program again. + +Let's also inspect the installation directory when the whole build process has finished successfully. +This is very easy after loading the module, as EasyBuild-generated modules define a number of environment +variables for each module: + +``` +$ module load eb-tutorial +$ env | grep TUTORIAL +EBVERSIONEBMINTUTORIAL=1.0.1 +EBDEVELEBMINTUTORIAL=/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12/easybuild/eb-tutorial-1.0.1-cpeCray-21.12-easybuild-devel +EBROOTEBMINTUTORIAL=/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 +``` +The most interesting one of those variables is the `EBROOT` variable which points to the installation directory. +As variable names cannot contain minus signs, the minus in the module name is replaced with `MIN` in the name +of the variable (which is not the most consistent thing however as in the name of an easyblock it is replaced +with `minus`). + +Let's have a look in that directory: + +``` +$ ls $EBROOTEBMINTUTORIAL +bin easybuild +$ ls EBROOTEBMINTUTORIAL/easybuild +easybuild-eb-tutorial-1.0.1-20220401.184518.log +easybuild-eb-tutorial-1.0.1-20220401.184518_test_report.md +eb-tutorial-1.0.1-cpeCray-21.12-easybuild-devel +eb-tutorial-1.0.1-cpeCray-21.12.eb +reprod +$ ls EBROOTEBMINTUTORIAL/easybuild/reprod +easyblocks +eb-tutorial-1.0.1-cpeCray-21.12.eb +eb-tutorial-1.0.1-cpeCray-21.12.env +hooks +ls $EBROOTEBMINTUTORIAL/easybuild/reprod/easyblocks +cmakemake.py configuremake.py +``` + +As you can see, EasyBuild has also created the `easybuild` subdirectory (and it actually told us about that +at the end of the installation) which contains a lot of information about the build, also to make it easier +to reproduce a build process afterwards. + + --- ## Exercises @@ -854,6 +971,9 @@ a bit more personal: include the username of the account that was used to instal ``` To re-install the `example.eb` easyconfig, you will need to use `eb --rebuild` or `eb --force`. + EasyBuild will also complain about modules that are loaded already if you tested the program above. + A good way to get rid of all those modules on LUMI is to simply use `module purge`. You don't need + to reload the software stack, but you will need to load `EasyBuild-user` again. --- @@ -874,125 +994,34 @@ You can leverage the `eb-tutorial` easyconfig file we have composed in the examp ??? success "(click to show solution)" You can use the `--try-software-version` option for this: ```shell - $ eb example.eb --try-software-version 1.1.0 + $ eb eb-tutorial.eb --try-software-version 1.1.0 ... - == building and installing eb-tutorial/1.1.0-GCC-10.2.0... + == building and installing eb-tutorial/1.1.0-cpeCray-21.12... ... == COMPLETED: Installation ended successfully (took 4 sec) ``` To test: ``` - $ module use $HOME/easybuild/modules/all - $ module load eb-tutorial/1.1.0-GCC-10.2.0 + $ module load eb-tutorial/1.1.0-cpeCray-21.12 $ eb-tutorial I have a message for you: - Hello from the EasyBuild tutorial! + Hello from the EasyBuild tutorial! I was installed by XXXXXXXX. ``` (`eb-tutorial` version 1.0.1 doesn't print "`I have a message for you:`") ---- - -***Exercise E.3**** - Installing py-eb-tutorial 1.0.0* - -Try composing an easyconfig file for the `py-eb-tutorial` example software, which is a tiny Python package. -The source tarball can be downloaded from this link: [py-eb-tutorial-1.0.0.tar.gz](https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/py-eb-tutorial-1.0.0.tar.gz). - -A couple of tips: - -* There is a [generic easyblock](#easyblock) available for installing Python packages, which will come in useful here. - -* By default EasyBuild performs an `import` check when install Python packages, using a Python module name that is derived from the software name by default, which will be incorrect in this case. You can specify the correct name to use in the import check by specifying it via the `options` -easyconfig parameter in your easyconfig file: - ```python - options = {'modulename': 'example'} - ``` - (you will need to change '`example`' here, of course) - -* Leverage the software that is already pre-installed in `/easybuild` in the prepared environment. - Remember that some already installed modules may be a *bundle* of a couple of other software packages. - -Please also take this into account: - -* Unfortunately this software doesn't come with documentation. That is done to make it an example that - is representative for software that you may run into in the wild (it's *not* because - we were lazy when preparing the exercises, really!). - You can inspect the sources of this software [here](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/py-eb-tutorial-1.0.0). Definitely take a look at the `setup.py` file, it includes some clues - about the requirements to get this software installed. - -* Make sure the installation actually *works*, by checking that the `py-eb-tutorial` command runs correctly. - Maybe you will need to make sure other required software is available as well, for it to work correctly... - - -??? success "(click to show solution)" - - Here is a complete working easyconfig file for `py-eb-tutorial`: - ```python - - easyblock = 'PythonPackage' - - name = 'py-eb-tutorial' - version = '1.0.0' - versionsuffix = '-Python-%(pyver)s' - - homepage = 'https://easybuilders.github.io/easybuild-tutorial' - description = "EasyBuild tutorial Python example" - - source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] - sources = [SOURCE_TAR_GZ] - checksums = ['fcf73a9efc65527a210b993e8889d41ebf05977eef1f6a65ebac3188152cd496'] - - toolchain = {'name': 'foss', 'version': '2020b'} - - dependencies = [ - ('Python', '3.8.6'), - ('SciPy-bundle', '2020.11'), - ('eb-tutorial', '1.0.1'), - ] - - use_pip = True - - options = {'modulename': 'eb_tutorial'} - - sanity_check_paths = { - 'files': ['bin/py-eb-tutorial'], - 'dirs': ['lib/python%(pyshortver)s/site-packages'], - } - - sanity_check_commands = ["py-eb-tutorial"] - - moduleclass = 'tools' + EasyBuild has also created a new easyconfig for this configuration and stored + in the repository and the `easybuild` subdirectory from the installation + directory. As on LUMI the repository is in the search path we can actually copy + the file back to the current directory: + ``` + eb --copy-ec eb-tutorial-1.1.0-cpeCray-21.12. ``` + Some of the formatting is lost though and the checksum is still missing, so you may want + to do some cleaning up. ``` + eb eb-tutorial-1.1.0-cpeCray-21.12.eb --inject-checksum + ``` + - Some remarks: - - * We used the `PythonPackage` generic easyblock. There is also a `PythonBundle` easyblock for installing - bundles of Python packages, which is used for `SciPy-bundle` for example. But we don't need that here, - since we are only dealing with a single Python package. - - * The `versionsuffix` is not strictly needed, but it's common to tag Python packages with the Python version - for which they were installed. - * The SHA256 checksum for the source tarball was added automatically via `eb py-eb-tutorial.eb --inject-checksums`. - - * `py-eb-tutorial` only wants to be installed with `pip install`, so we had to set `use_pip = True`. - You can consult the custom easyconfig parameters supported by the `PythonPackage` easyblock via - "`eb -a -e PythonPackage`", see the `EASYBLOCK-SPECIFIC` part of the output. - Even when the default installation mechanism used by `PythonPackage` - (which consists of running `python setup.py install`) works fine, - it is recommended to instruct EasyBuild to use `pip install` instead. - - * By default EasyBuild will try to import `py_eb_tutorial`, while the actual name of the Python package - provided by `py-eb-tutorial` is just `eb_tutorial`. We fixed this by specifying the correct Python module name to - use via `options`. - - * Strictly speaking we don't need to specify a custom `sanity_check_paths`, since the default used - by Python package is already pretty decent (it will check for a non-empty `lib/python3.8/site-packages` - directory in the installation). We also want to make sure the `py-eb-tutorial` command is available in - the `bin` subdirectory however. Hardcoding to `python3.8` can be avoided using the `%(pyshortver)s` - template value. - - * A good way to check whether the `py-eb-tutorial` command works correctly is by running it as a sanity check - command. If the `eb-tutorial` command is not available the `py-eb-tutorial` command will fail, - since it basically just runs the `eb-tutorial` command. So we need to include `eb-tutorial` as a (runtime) - dependency in the `py-eb-tutorial` easyconfig file. +--- *[[next: Implementing easyblocks]](2_03_implementing_easyblocks.md)* From 0ba449e2ddcd121db5500a5cd9d04967acf61f2f Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 1 Apr 2022 19:12:05 +0200 Subject: [PATCH 072/159] Adding in an additional section about external modules. --- docs/2022-CSC_and_LO/2_00_part2_using.md | 3 ++- docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md | 2 +- ...lementing_easyblocks.md => 2_04_implementing_easyblocks.md} | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-) rename docs/2022-CSC_and_LO/{2_03_implementing_easyblocks.md => 2_04_implementing_easyblocks.md} (99%) diff --git a/docs/2022-CSC_and_LO/2_00_part2_using.md b/docs/2022-CSC_and_LO/2_00_part2_using.md index e033499c..c50036be 100644 --- a/docs/2022-CSC_and_LO/2_00_part2_using.md +++ b/docs/2022-CSC_and_LO/2_00_part2_using.md @@ -4,4 +4,5 @@ * [Troubleshooting](2_01_troubleshooting.md) *(hands-on)* * [Creating easyconfig files](2_02_creating_easyconfig_files.md) *(hands-on)* -* [Implementing easyblocks](2_03_implementing_easyblocks.md) *(hands-on)* +* [Using external modules from the Cray PE](2_03_external_modules.md) +* [Implementing easyblocks](2_04_implementing_easyblocks.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md index 99ac7d88..f7d6a646 100644 --- a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md @@ -1024,4 +1024,4 @@ You can leverage the `eb-tutorial` easyconfig file we have composed in the examp --- -*[[next: Implementing easyblocks]](2_03_implementing_easyblocks.md)* +*[[next: Using external modules from the Cray PE]](2_03_external_modules.md)* diff --git a/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md similarity index 99% rename from docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md rename to docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md index 5355548e..e0c07510 100644 --- a/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md +++ b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md @@ -1,6 +1,6 @@ # Implementing easyblocks -*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* +*[[back: Using external modules from the Cray PE]](2_03_external_modules.md)* --- From a778c98bffef48d79c6d0e8be647b5304bdd6991 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 1 Apr 2022 19:12:42 +0200 Subject: [PATCH 073/159] Adding in a section taken from a previous tutorial and integrating some of the material in an earlier section on the Cray PE. --- docs/2022-CSC_and_LO/1_03_CPE.md | 222 ++++++++++++++---- docs/2022-CSC_and_LO/2_03_external_modules.md | 81 +++++++ 2 files changed, 259 insertions(+), 44 deletions(-) create mode 100644 docs/2022-CSC_and_LO/2_03_external_modules.md diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md index 077fbc36..d3a2ff84 100644 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -15,76 +15,207 @@ tools and to configure the environment in a flexible way. ## Cray PE components -- **Cray Compiling Environment (CCE)**: +### Cray compiler environments - CCE consists of Cray compilers performing code analysis during compilation to generate highly optimized code. - Supported languages include Fortran, C and C++, and UPC (Unified Parallel C). - The Cray C/C++ compiler is basically the Clang/LLVM compiler with a back-end configured by HPE Cray that also - adds some Cray-specific options while the Cray Fortran compiler uses an HPE-Cray specific front-end - with a back-end based on LLVM. The Fortran compiler supports most of Fortran 2018 and tends to be - considerably stricter than the GNU or Intel Fortran compilers. +Cray PE supports multiple compilers, Cray and third party compilers as well: AOCC, Cray, Intel, GNU. - The CCE compilers will also support OpenMP offload to NVIDIA and AMD GPUs but that is still work in progress - at the time of the development of this tutorial. The Fortran compiler (but not the C/C++ compiler) also supports - OpenACC for offloading to GPU. +Users can access the compilers loading a programming environment module (see below). - The classic pre-LLVM Cray C/C++ compiler is not available on Cray EX systems. +Compilers in the Cray Programming Environment are almost always used through the Cray compiler wrappers. In fact, +until recently, some compilers could not be used without the wrappers as they failed to even detect their own +include files and some components. The behaviour of the wrapper will depend on the other modules that are +loaded in the environment. E.g., there is no separate wrapper for MPI as the MPI header files and libraries are +automatically activated through the regular wrappers as soon as a Cray MPI module is loaded. -- **Cray Scientific and Mathematical Libraries (CSML)**: +The commands to invoke compiler wrappers are `ftn` (Fortran), `cc` (C), `CC` (C++). They wrap automatically +to the right compilers based on the programming environment module that is loaded and the compiler module. - A set of high performance libraries providing portability for scientific applications, sparse and dense linear - algebra (BLAS, LAPACK, ScaLAPACK) and fast Fourier transforms (FFTW). +The online help can be accessed with the `-help` option. E.g.: `cc -help`, `CC -help`. +One of the most frequently used options of the compiler wrappers is `-craype-verbose`: +``` + ftn -help | grep verbose -A 1 + -craype-verbose Print the command which is forwarded + to compiler invocation +``` +More information is available with the `info` or `man` commands. +E.g.: both `info cc` or `man cc` will display the man page of the C compiler wrapper. - There is a special (more limited) version of the library with GPU-optimized routines. +The compiler wrappers call the correct compiler in the currently loaded programming environment, +with appropriate options to build and link applications with relevant libraries, as required by the modules loaded. +Besides a number of generic options, they also will pass all other options to the underlying compile, +so you can still use the regular compiler options also. - CSML also includes the HDF5 and netCDF libraries, in versions with and without MPI. +The compiler wrappers should replace direct calls to compiler drivers in Makefiles and build scripts +to ensure that the proper libraries are used. -- **Cray Message Passing Toolkit**: +!!! Note + For system libraries, only dynamic linking is supported by compiler wrappers on the Cray EX system - A collection of software libraries used to perform data transfers between nodes running in parallel applications. - It includes the Message Passing Interface (MPI) and OpenSHMEM parallel programming models. +--- - The HPE Cray MPI implementation is based on the MPICH3 sources from ANL. However, some algorithms are tweaked - specifically for HPE Cray hardware and the Cray interconnects. +#### Cray Compiling Environment (CCE) -- **Cray Environment Setup and Compiling Support**: - - Libraries supporting code compilation and setting up the development environment, including compiler drivers. +The Cray Compiling Environment is set by the module `PrgEnv-cray`, which is loaded by default at login. +The compiler itself is contained in the `cce` module. -- **Cray Performance and Analysis Tools**: +CCE consists of Cray compilers performing code analysis during compilation to generate highly optimized code. +Supported languages include Fortran, C and C++, and UPC (Unified Parallel C). +The Cray C/C++ compiler is basically the Clang/LLVM compiler with a back-end configured by HPE Cray that also +adds some Cray-specific options while the Cray Fortran compiler uses an HPE-Cray specific front-end +with a back-end based on LLVM. The Fortran compiler supports most of Fortran 2018 and tends to be +considerably stricter than the GNU or Intel Fortran compilers. - Tools to analyze the performance and behavior of programs that are run on Cray systems, and a Performance API (PAPI). +The CCE compilers will also support OpenMP offload to NVIDIA and AMD GPUs but that is still very immature and +very much work in progress at the time of the development of this tutorial. +The Fortran compiler (but not the C/C++ compiler) also supports +OpenACC for offloading to GPU and is being updated to the newest versions of this standard. -- **Cray Debugging Support Tools**: +The classic pre-LLVM Cray C/C++ compiler is not available on Cray EX systems. - Debugging tools, including `gdb4hpc` and `Valgrind4hpc`. +Compiler-specific manpages can be accessed on the system with `man crayftn`, `man craycc` or `man crayCC`. + +More details are given in the +[Cray Fortran Reference Manual](https://support.hpe.com/connect/s/search?language=en_US#q=Cray%20Fortran%20Reference%20manual&t=All&sort=relevancy&numberOfResults=25) + and the + [Cray Compiling Environment Release](https://support.hpe.com/connect/s/search?language=en_US#q=Cray%20Compiling%20Environment%20Release&t=All&sort=relevancy&numberOfResults=25) +that unfortunately are hidden deep in the new [support pages of HPE](https://support.hpe.com/). +The [Clang Compiler User’s Manual](https://clang.llvm.org/docs/UsersManual.html) +is another source of information for the Cray C and C++ Clang compilers. + +For more information about compiler pragmas and directives, see `man intro_directives` on the system. + +--- + +### Third-Party Compilers + +#### GNU + +The GNU C/C++ and Fortran compilers are probably the best supported third party +compiler in the Cray PE. + +Compiler-specific manpages can be accessed on the system with `man gfortran`, `man gcc` or `man g++`. + +More details are provided by the [GCC online documentation](https://gcc.gnu.org/onlinedocs). + + +#### AOCC + +There AOCC compilers, the AMD Optimizing C/C++ Compiler and matching fortran compilers, +AMD's compiler offering for CPU-only systems, have a matching programming environment module +and are a full citizen of the Cray PE. + +Compiler-specific documentation is available in the [AOCC User Guide](https://developer.amd.com/amd-aocc/#userguide). + +Cray provides a bundled package of support libraries to install into the PE environment to enable AOCC, +and Cray PE utilities such as debuggers and performance tools work with AOCC. + + +#### AMD ROCm compilers + +The AMD ROCm compilers (in the AMD world sometimes known as AOMP) are supported on systems +with AMD CPUs. However, at the time of writing of this tutorial they are not yet available in the +LUMI environment and the integration with the Cray environment still seems to be work in progress. + + +#### Intel (not on LUMI) + +The Cray PE also provides a programming environment module to enable +the Intel® oneAPI compiler and tools. + +The documentation is available in the [Intel® oneAPI Programming Guide](https://software.intel.com/content/www/us/en/develop/documentation/oneapi-programming-guide/top/oneapi-development-environment-setup.html) + +Cray provides a bundled package of support libraries to install into the Cray PE to enable the Intel compiler, allowing +utilities such as debuggers and performance tools to work with it. + + +#### NVIDIA HPC toolkit (not on LUMI) -- **Third-party compilers**: +The NVIDIA HPC Toolkit compilers (formerly PGI) are supported on systems with NVIDIA GPUs. - The Cray PE also supports several third-party compilers. Not all compilers are supported on all systems: - - The GNU C/C++ and Fortran compilers will be present on most systems with the Cray PE. +### Cray Scientific and Math Library - - The AMD AOCC C/C++ and Fortran compilers are supported on systems with AMD CPUs. +* Manpages: `intro_libsci`, `intro_fftw3` - - The AMD ROCm compilers (sometimes known as AOMP) are supported on systems with AMD GPU technology. +The Cray Scientific and Math Libraries (CSML, also known as LibSci) are a collection of numerical routines +optimized for best performance on Cray systems. - - The Intel C/C++ and Fortran compilers (also the oneAPI versions) are supported on systems with Intel - processors. HPE Cray does not include them in licenses for AMD CPUs. +These libraries satisfy dependencies for many commonly used applications on Cray systems for a wide variety of domains. - - The NVIDIA HPC Toolkit compilers are supported on systems with NVIDIA GPUs. +When the module for a CSML package (such as `cray-libsci` or `cray-fftw`) is loaded, +all relevant headers and libraries for these packages are added to the compile +and link lines of the `cc`, `ftn`, and `CC` compiler wrappers, so linking with them is +completely transparant (to the extent that users wonder where the libraries are). + +The CSML collection contains the following Scientific Libraries: + +* BLAS (Basic Linear Algebra Subroutines) +* BLACS (Basic Linear Algebra Communication Subprograms) +* CBLAS (Collection of wrappers providing a C interface to the Fortran BLAS library) +* IRT (Iterative Refinement Toolkit) +* LAPACK (Linear Algebra Routines) +* LAPACKE (C interfaces to LAPACK Routines) +* ScaLAPACK (Scalable LAPACK) +* `libsci_acc` (library of Cray-optimized BLAS, LAPACK, and ScaLAPACK routines) +* HDF5 +* NetCDF (Network Common Data Format) +* FFTW3 (the Fastest Fourier Transforms in the West, release 3) + + +### Cray Message Passing Toolkit + +* Manpages: `intro_mpi` + +* Website: [http://www.mpi-forum.org](http://www.mpi-forum.org) + +MPI is a widely used parallel programming model that establishes a practical, portable, efficient, +and flexible standard for passing messages between ranks in parallel processes. + +Cray MPI is derived from Argonne National Laboratory MPICH and implements the MPI-3.1 standard +as documented by the MPI Forum in MPI: A Message Passing Interface Standard, Version 3.1. + +Support for MPI varies depending on system hardware. To see which functions and environment variables the +system supports, please have a look at the corresponding man pages with `man intro_mpi` on the system. + +Note that though on LUMI at the time of the tutorial both the OpenFabric Interface (OFI) based and UCX-based +versions of the library are supported, the finalised LUMI system will only support OFI. + + +### DSMML + +* Modules: `cray-dsmml` +* +* Manpages: `intro_dsmml` +* +* Website: [https://pe-cray.github.io/cray-dsmml](https://pe-cray.github.io/cray-dsmml) + +Distributed Symmetric Memory Management Library (DSMML) is a HPE Cray proprietary memory management library. + +DSMML is a standalone memory management library for maintaining distributed shared symmetric memory +heaps for top-level PGAS languages and libraries like Coarray Fortran, UPC, and OpenSHMEM. + +DSMML allows user libraries to create multiple symmetric heaps and share information with other libraries. + +Through DSMML, interoperability can be extracted between PGAS programming models. + +Further details are available in the man page on the system with `man intro_dsmml`. + + +### Other components + +- **Cray Performance and Analysis Tools**: + + Tools to analyze the performance and behavior of programs that are run on Cray systems, and a Performance API (PAPI). + +- **Cray Debugging Support Tools**: + + Debugging tools, including `gdb4hpc` and `Valgrind4hpc`. --- ## Configuring the Cray PE through modules -Compilers in the Cray Programming Environment are almost always used through the Cray compiler wrappers. In fact, -until recently, some compilers could not be used without the wrappers as they failed to even detect their own -include files and some components. The behaviour of the wrapper will depend on the other modules that are -loaded in the environment. E.g., there is no separate wrapper for MPI as the MPI header files and libraries are -automatically activated through the regular wrappers as soon as a Cray MPI module is loaded. - Multiple releases of the HPE Cray PE can be installed simultaneously on the system and users can mix-and-match components. However, it is also possible to load only components for a specific release of the PE. Cray PE versions have version numbers of the form ``yy.mm``, e.g., ``22.02`` for the version released in @@ -95,7 +226,7 @@ Below we only discuss those modules that are important when building software wi profilers, etc., are not included in the list. -### The PrgEnv-* modules +### The programming environment modules The usual way to initialise the Cray PE is by loading one of the ``PrgEnv-*`` modules. This module will then load the appropriate compiler, compiler wrappers and other libraries. Some of the components that will be @@ -117,7 +248,10 @@ All ``PrgEnv-*`` modules belong to the same module family. Hence Lmod will autom already loaded ``PrgEnv-*`` module when you load a different one. -### Selecting the version through the cpe module +### Selecting the version through the cpe meta-module + +The Cray PE on the EX system provides the meta-module `cpe`: the purpose of the meta-module is +similar to the scope of the `cdt` and `cdt-cuda` meta-modules available on the XC systems. Loading one of the ``cpe/yy.mm`` modules (e.g., ``cpe/22.02``) has the following effects: diff --git a/docs/2022-CSC_and_LO/2_03_external_modules.md b/docs/2022-CSC_and_LO/2_03_external_modules.md new file mode 100644 index 00000000..5d6632c0 --- /dev/null +++ b/docs/2022-CSC_and_LO/2_03_external_modules.md @@ -0,0 +1,81 @@ +# Using external modules from the Cray PE + +*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* + +--- + +## What are external modules? + +EasyBuild supports the use of modules that were not installed via EasyBuild. +We refer to such modules as [external modules](https://docs.easybuild.io/en/latest/Using_external_modules.html). +These modules do not define the `EBROOT*` and `EBVERSION*` environment variables that EasyBuild would define +and uses internally in several easyblocks and some easyconfig files. + +This feature is used extensively on Cray systems to interface with the Cray PE (which comes with its own +modules and cannot be installed via EasyBuild): +[external modules can be used as dependencies](https://docs.easybuild.io/en/latest/Using_external_modules.html#using-external-modules-as-dependencies), +by including the module name in the dependencies list, +along with the `EXTERNAL_MODULE` constant marker. + +For example, to specify the module `cray-fftw` as a dependency, you should write the following in your easyconfig file: +``` python +dependencies = [('cray-fftw', EXTERNAL_MODULE)] +``` + +For such dependencies, EasyBuild will: + +* load the module before initiating the software build and install procedure + +* include a `module load` statement in the generated module file (for runtime dependencies) + +!!! Note + The default version of the external module will be loaded unless a specific version is given as dependency, + and here that version needs to be given as part of the name of the module and not as the second element in the + tuple. + + ```python + dependencies = [('cray-fftw/3.3.8.12', EXTERNAL_MODULE)] + ``` + +If the specified module is not available, EasyBuild will exit with an error message stating that the dependency +can not be resolved because the module could not be found, without searching for a matching easyconfig file +from which it could generate the module. + + +--- + +## EasyBuild Metadata for external modules + +[Metadata](https://docs.easybuild.io/en/latest/Using_external_modules.html#metadata-for-external-modules) + can be supplied to EasyBuild for external modules: using the `--external-modules-metadata` +configuration option, the location of one or more metadata files can be specified. + +The files are expected to be in INI format, with a section per module name +and key-value assignments specific to that module. + +The external modules metadata file can be also defined with the corresponding environment variable: +``` +echo $EASYBUILD_EXTERNAL_MODULES_METADATA +/apps/common/UES/jenkins/production/easybuild/cpe_external_modules_metadata-21.04.cfg +``` + +The following keys are +[supported by EasyBuild](https://docs.easybuild.io/en/latest/Using_external_modules.html#supported-metadata-values): + +* name: software name(s) provided by the module +* version: software version(s) provided by the module +* prefix: installation prefix of the software provided by the module + +For instance, the external module version loaded by the dependency `cray-fftw` can be specified as follows: +```ini +[cray-fftw] +name = FFTW +prefix = FFTW_DIR/.. +version = 3.3.8.10 +``` + +The environment variable `$EBROOTFFTW` will also be defined according to the `prefix` specified in the metadata file. + +--- + +*[[next: Implementing easyblocks]](2_04_implementing_easyblocks.md)* From ca73ca7a053a18c34dc5a182bb2a5c4aa517accb Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 5 Apr 2022 15:54:42 +0200 Subject: [PATCH 074/159] Implementing EasyBlocks adapted for LUMI. --- .../2_04_implementing_easyblocks.md | 113 ++++++++++++++++-- 1 file changed, 101 insertions(+), 12 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md index e0c07510..a91b0dc8 100644 --- a/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md +++ b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md @@ -27,7 +27,10 @@ by most GNU software packages. *Software-specific* easyblocks implement the build and installation procedure for a particular software package. Typically this involves highly customised steps, for example specifying dedicated configuration options, creating or adjusting specific files, executing non-standard shell commands, etc. Usually a custom implementation of the -sanity check is also included. +sanity check is also included. Much of the work done in software-specific easyblocks can often also be done +in generic easyblocks using parameters such as ``confdigopts`` etc., but a software-specific easyblock can +hide some of that complexity from the user. Other software-specific easyblocks implement very specific +installation procedures that do not fit in one of the generic ones. Using a generic easyblock requires specifying the ``easyblock`` parameter in the easyconfig file. If it is not specified, EasyBuild will try and find the software-specific easyblock derived from the software name. @@ -38,7 +41,8 @@ for an easyblock (see below). ## Naming -Easyblocks need to follow a strict naming scheme, to ensure that EasyBuild can pick them up as needed. This involves two aspects: +Easyblocks need to follow a strict naming scheme, to ensure that EasyBuild can pick them up automatically as needed. +This involves two aspects: * the name of the Python class; * the name and location of the Python module file. @@ -52,7 +56,7 @@ Because of limitations in Python on characters allowed in names of Python classe only alphanumeric characters and underscores (``_``) are allowed. Any other characters are replaced following an encoding scheme: * spaces are replaced by underscores (``_``); -* dashes ``-`` are replaced by ``_minus_``; +* dashes ``-`` are replaced by ``_minus_`` (note the inconsistency with the naming of ``EBROOT`` and ``EBVERSION`` variables); * underscores are replaced by ``_underscore_``; The ``encode_class_name`` function provided in ``easybuild.tools.filetools`` returns the expected class name @@ -117,6 +121,11 @@ To let EasyBuild pick up one or more new or customized easyblocks, you can use t configuration option. As long as both the filename of the Python module and the name of the Python class are correct, EasyBuild will use these easyblocks when needed. +On LUMI, the EasyBuild configuration modules take care of setting this parameter (using the corresponding environment +variable), pointing to custom easyblocks in the LUMI software stack itself and a repo (with a fixed name) that users +can create themselves. At this moment it does not yet include possible other easyblock repositories in other repositories. + + ## Structure of an easyblock The example below shows the overal structure of an easyblock: @@ -232,7 +241,7 @@ self.cfg['some_list'].append('example') ### Custom parameters -Additional custom easyconfig parameters can be defined in an easyblock to allowing steering its behaviour. +Additional custom easyconfig parameters can be defined in an easyblock to steer its behaviour. This is done via the ``extra_options`` *static* method. Custom parameters can be specified to be mandatory. The example below shows how this can be implemented: @@ -256,7 +265,9 @@ class EB_Example(ConfigureMake): ``` The first element in the list of a defined custom parameter corresponds to the default value for that parameter -(both ``None`` in the example above). The second element provides some informative help text, and the last element +(both ``None`` in the example above). The second element provides some informative help text +(which can then be displayed with ``eb -a -e ``, eg, ``eb -a -e EB_GCC``), +and the last element indicates whether the parameter is mandatory (``MANDATORY``) or just an optional custom parameter (``CUSTOM``). ## Easyblock constructor @@ -519,6 +530,29 @@ class EB_Example(EasyBlock): super(EB_Example, self).sanity_check_step(custom_commands=[self.command]) ``` + +## Easyblocks in the Cray ecosystem + +The generic easyblocks are usually rather independent of compilers etc. and tend to work well with all toolchains. +However, software-specific easyblocks may contain code that is specific for certain toolchains and are often only +tested with the common toolchains (foss and intel and their subtoolchains). Many of those easyblocks will fail +on Cray systems (or any system that uses other toolchains) as they don't recognise the compiler and rather than +implementing some generic behaviour that may or may not work, produce an error message instead that the compiler +toolchain is not supported. + +Several packages on LUMI therefore use generic easyblocks rather than the software-specific easyblocks that may +exist for those applications. Adapting those software-specific easyblocks for LUMI poses an interesting maintenance +problem. Either one could decide to not contribute back to the community, but this implies then that all modifications +made to the corresponding easyblocks in the EasyBuild distribution should be monitored and implemented in the custom +easyblocks for Cray also. On the other hand, contributing back to the community also poses two problems. First it +would also require to implement the Cray toolchains as used on LUMI in the core of EasyBuild (which already contains +a different set of toolchains targeted more at how the Cray PE works with the regular environment modules), and that +only makes sense if these toolchains are first extended to not only cover the programming environments supported on +LUMI but also the Intel and NVIDIA programming environments. Second, the EasyBuild community has no easy way of testing +any modification made to such an easyblock on a Cray PE system. Hence every update made in the community may break +the Cray PE support again. + + ## Exercise ### Exercise I.1 @@ -570,9 +604,9 @@ Your easyblock should: cmd = ' '.join([ self.cfg['preconfigopts'], - "cmake", - "-DCMAKE_INSTALL_PREFIX='%s'" % self.installdir, - "-DEBTUTORIAL_MSG='%s'" % self.cfg['message'], + 'cmake', + '-DCMAKE_INSTALL_PREFIX=\'%s\'' % self.installdir, + '-DEBTUTORIAL_MSG="%s"' % self.cfg['message'], self.cfg['configopts'], ]) run_cmd(cmd) @@ -582,7 +616,7 @@ Your easyblock should: cmd = ' '.join([ self.cfg['prebuildopts'], - "make", + 'make', self.cfg['buildopts'], ]) run_cmd(cmd) @@ -592,7 +626,7 @@ Your easyblock should: cmd = ' '.join([ self.cfg['preinstallopts'], - "make install", + 'make install', self.cfg['installopts'], ]) run_cmd(cmd) @@ -607,6 +641,58 @@ Your easyblock should: custom_commands=custom_commands) ``` + We also need to adapt our easyconfig file for ``eb-tutorial``: + + - The ``easyblock`` line is no longer needed as we will rely on the automatic selection of the + software-specific easyblock. + - We don't need to define the message through ``configopts`` but via the easyblock-specific + configuration parameter ``message``. In fact, we were so careful when implementing the ``configure_step`` + that even variable expansion will still work so we can still include ``$USER`` in the message. + - The sanity check is also no longer needed as it is done by the software-specific easyblock. + + So the easyconfig file simplifies to: + + ```python + name = 'eb-tutorial' + version = "1.1.0" + + homepage = 'https://easybuilders.github.io/easybuild-tutorial' + + whatis = [ 'Description: EasyBuild tutorial example'] + + description = """ + This is a short C++ example program that can be buid using CMake. + """ + + toolchain = {'name': 'cpeCray', 'version': '21.12'} + + builddependencies = [ + ('buildtools', '%(toolchain_version)s', '', True) + ] + + source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] + sources = [SOURCE_TAR_GZ] + checksums = ['def18b69b11a3ec34ef2a81752603b2118cf1a57e350aee41de9ea13c2e6a7ef'] + + message = 'Hello from the EasyBuild tutorial! I was installed by $USER.' + + moduleclass = 'tools' + + ``` + + Running this example on LUMI is a little tricky as using ``--include-easyblocks`` to point EasyBuild to + our new easyblock interfers with settings already made by the EasyBuild configuration modules (``EasyBuild-user``) + and causes error messages about the toolchains. So either the easyblock needs to be copied to the user location + that can be found by looking at the output of ``eb --show-config`` or we simply need to extend the list of + easyblocks that EasyBuild searches with the easyblocks in the current directory: + + ``` shell + EASYBUILD_INCLUDE_EASYBLOCKS="$EASYBUILD_INCLUDE_EASYBLOCKS,./*.py" + ``` + + + + ### Exercise I.2 Try implementing another new custom easyblock for ``eb-tutorial``, @@ -648,7 +734,7 @@ Your easyblock should only: def configure_step(self): """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" - self.cfg.update('configopts', "-DEBTUTORIAL_MSG='%s'" % self.cfg['message']) + self.cfg.update('configopts', '-DEBTUTORIAL_MSG="%s"'% self.cfg['message']) super(EB_eb_minus_tutorial, self).configure_step() @@ -662,4 +748,7 @@ Your easyblock should only: custom_commands=custom_commands) ``` -*[[next: PArt 3: OVerview]](3_00_part3_advanced.md)* + This is a much simpler easyblock as we already use all the logic that has been written for us to build + with CMake. + +*[[next: Part 3: Overview]](3_00_part3_advanced.md)* From bdc33246228121403fb49f18bb5fcbe0a49630ac Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 5 Apr 2022 15:54:55 +0200 Subject: [PATCH 075/159] Correction of typos. --- docs/2022-CSC_and_LO/2_03_external_modules.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_03_external_modules.md b/docs/2022-CSC_and_LO/2_03_external_modules.md index 5d6632c0..b19313ff 100644 --- a/docs/2022-CSC_and_LO/2_03_external_modules.md +++ b/docs/2022-CSC_and_LO/2_03_external_modules.md @@ -8,10 +8,10 @@ EasyBuild supports the use of modules that were not installed via EasyBuild. We refer to such modules as [external modules](https://docs.easybuild.io/en/latest/Using_external_modules.html). -These modules do not define the `EBROOT*` and `EBVERSION*` environment variables that EasyBuild would define +External modules do not define the `EBROOT*` and `EBVERSION*` environment variables that EasyBuild would define and uses internally in several easyblocks and some easyconfig files. -This feature is used extensively on Cray systems to interface with the Cray PE (which comes with its own +External modules are used extensively on Cray systems to interface with the Cray PE (which comes with its own modules and cannot be installed via EasyBuild): [external modules can be used as dependencies](https://docs.easybuild.io/en/latest/Using_external_modules.html#using-external-modules-as-dependencies), by including the module name in the dependencies list, @@ -47,7 +47,7 @@ from which it could generate the module. ## EasyBuild Metadata for external modules [Metadata](https://docs.easybuild.io/en/latest/Using_external_modules.html#metadata-for-external-modules) - can be supplied to EasyBuild for external modules: using the `--external-modules-metadata` +can be supplied to EasyBuild for external modules: using the `--external-modules-metadata` configuration option, the location of one or more metadata files can be specified. The files are expected to be in INI format, with a section per module name @@ -76,6 +76,10 @@ version = 3.3.8.10 The environment variable `$EBROOTFFTW` will also be defined according to the `prefix` specified in the metadata file. +On LUMI, users in generally don't need to be too concerned about the metadata file as the EasyBuild-user (and other hidden +EasyBuild configuration modules) take care of pointing to the right metadata file, which is specific for each version of the +Cray PE and hence each version of the LUMI software stack. + --- *[[next: Implementing easyblocks]](2_04_implementing_easyblocks.md)* From 87138ea2ef72af1945bf7fcade8c574df8952287 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 5 Apr 2022 15:55:13 +0200 Subject: [PATCH 076/159] Using EasyBuild as a library corrected for LUMI. --- .../2022-CSC_and_LO/3_01_easybuild_library.md | 26 ++++++++++++------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_01_easybuild_library.md b/docs/2022-CSC_and_LO/3_01_easybuild_library.md index c037f49f..f0c255eb 100644 --- a/docs/2022-CSC_and_LO/3_01_easybuild_library.md +++ b/docs/2022-CSC_and_LO/3_01_easybuild_library.md @@ -254,10 +254,12 @@ def parse_easyconfig(ec_fn): # parse easyconfig file; # the 'parse_easyconfigs' function expects a list of tuples, - # where the second item indicates whether or not the easyconfig file was automatically generated or not + # where the second item indicates whether or not the easyconfig file was + # automatically generated or not ec_dicts, _ = parse_easyconfigs([(ec_path, False)]) - # only retain first parsed easyconfig, ignore any others (which are unlikely anyway) + # only retain first parsed easyconfig, ignore any others (which are unlikely + # anyway) return ec_path, ec_dicts[0]['ec'] @@ -278,15 +280,21 @@ for key in keys: print("%s: %s" % (key, ec[key])) ``` -Example usage: +Example usage (ensure that ``EasyBuild-user`` is loaded as EasyBuild should be +configured properly!): -```shell -$ ./inspect_easyconfig.py Subread-2.0.0-GCC-8.3.0.eb name version sources sanity_check_paths -name: Subread -version: 2.0.0 -sources: ['subread-2.0.0-source.tar.gz'] -sanity_check_paths: {'files': ['bin/exactSNP', 'bin/featureCounts', 'bin/subindel', 'bin/subjunc', 'bin/sublong', 'bin/subread-align', 'bin/subread-buildindex'], 'dirs': ['bin/utilities']} ``` +$ ./inspect_easyconfig.py GMP-6.2.1-cpeCray-21.12.eb name version sources sanity_check_paths +Inspecting /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/GMP/GMP-6.2.1-cpeCray-21.12.eb ... +name: GMP +version: 6.2.1 +sources: ['gmp-6.2.1.tar.bz2'] +sanity_check_paths: {'files': ['include/gmp.h', 'include/gmpxx.h', 'lib/libgmp.a', 'lib/libgmp.la', 'lib/libgmp.so', 'lib/libgmpxx.a', 'lib/libgmpxx.la', 'lib/libgmpxx.so', 'lib/pkgconfig/gmp.pc', 'lib/pkgconfig/gmpxx.pc'], 'dirs': ['share/info']} +``` + +This feature can be used, e.g., to extract information from easyconfig files to generate documentation +for the library of easyconfig files. + --- From 492683e0c7ef3a39a83a3c6be91700e29ec7893d Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 5 Apr 2022 15:55:33 +0200 Subject: [PATCH 077/159] Section about hooks extended with references to additional examples. --- docs/2022-CSC_and_LO/3_02_hooks.md | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_02_hooks.md b/docs/2022-CSC_and_LO/3_02_hooks.md index a2663ff0..f91f7d7c 100644 --- a/docs/2022-CSC_and_LO/3_02_hooks.md +++ b/docs/2022-CSC_and_LO/3_02_hooks.md @@ -19,9 +19,32 @@ would normally execute. This is a very powerful feature, but it is important to aware of some details in order to make good use of it. -Support for hooks is also covered in detail in the EasyBuild documentation, see -[here](https://docs.easybuild.io/en/latest/Hooks.html). - +Support for hooks is also +[covered in detail in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Hooks.html). + +## Why hooks? + +Hooks can be implemented for various reasons. Some examples used by some sites in the EasyBuild +community: + +- Enforce some site policies on easyconfig files. A practical example for LUMI could be imposing + the use of the `whatis` parameter and ensuring that it contains a `Description` line to have a + more consistent format for module files. +- Ensuring that some information is always in the module file. E.g., on LUMI the `site_contacts` + parameter is added automatically when installing in the central stack if it is not already in + the easyconfig file to refer to the LUST support pages. +- Modify the behaviour of standard easyconfig files that come with EasyBuild to adapt them to the system + while allowing users to simply use the standard easyconfig files. E.g., we could consider this option + to offer the common EasyBuild FOSS toolchain and its subtoolchains to users on LUMI-C, while using hooks to + adapt, e.g., the easyconfig files for Open MPI to work on LUMI. + +Some site have really large hook files to implement policies and modify standard EasyBuild build recipes, e.g., +- [The hooks file from Jülich Supercomputing Centre](https://github.com/easybuilders/JSC/blob/2022/Custom_Hooks/eb_hooks.py) +- ComputeCanada [cc_hooks_common.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_common.py) + and [cc_hooks_gento.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_gentoo.py) +- [The hooks file from the EESSI software stack](https://github.com/EESSI/software-layer/blob/main/eb_hooks.py). + [EESSI](https://www.eessi-hpc.org/) is an effort to build a software stack distributed via + [CernVM-FS](https://cernvm.cern.ch/fs/) using EasyBuild to build all software. ## Implementing and using hooks From 8e734c5005002d90851d6048aae2107ed1e38ed8 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 13:57:15 +0200 Subject: [PATCH 078/159] Slurm job submission from EasyBuild a bit reworked. --- docs/2022-CSC_and_LO/3_03_slurm_jobs.md | 109 +++++++++++++++--------- 1 file changed, 69 insertions(+), 40 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md index 0ffa12a9..567f8abd 100644 --- a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md +++ b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md @@ -6,12 +6,30 @@ EasyBuild can submit jobs to different backends including Slurm to install software, to *distribute* the often time-consuming installation of a set of software applications and -the dependencies they require to a cluster. +the dependencies they require to a cluster. Each individual package is installed in a separate +job and job dependencies are used to manage the dependencies between package so that no build +is started before the dependencies are in place. This is done via the ``--job`` command line option. It is important to be aware of some details before you start using this, which we'll cover here. +!!! Warning "This section is not supported on LUMI, use at your own risk" + + EasyBuild on LUMI is currently not fully configured to support job submission via Slurm. Several + changes would be needed to the configuration of EasyBuild, including the location of the + temporary files and build directory. Those have to be made by hand. + + Due to the setup of the central software stack, this feature is currently useless to install + the central stack. For user installations, there are also limitations as the enviornment + on the compute nodes is different from the login nodes so, e.g., different locations for + temporary files are being used. These would only be refreshed if the EasyBuild configuration + modules are reloaded on the compute nodes which cannot be done currently in the way Slurm + job submission is set up in EasyBuild. + + Use material in this section with care; it has not been completely tested. + + ## Configuration The EasyBuild configuration that is active at the time that ``eb --job`` is used @@ -25,6 +43,8 @@ that are specified via an [EasyBuild configuration file](configuration.md#config This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables that are in place in the job environment are most likely *irrelevant*, since configuration settings they specify they will most likely be overruled by the corresponding command line options. +It does also imply however that the EasyBuild configuration that is in place when ``eb --job`` is used +does also work on the compute nodes to which the job is submitted. ## Using ``eb --job`` @@ -39,6 +59,9 @@ to ``Slurm``, for example by setting the corresponding environment variable: export EASYBUILD_JOB_BACKEND='Slurm' ``` +On LUMI this is taken care of in the EasyBuild configuration modules such as ``EasyBuild-user``. + + ### Job resources To submit an installation as a job, simply use ``eb --job``: @@ -73,13 +96,13 @@ For example, to specify a particular account that should be used for the jobs su (equivalent with using the ``-A`` or ``--account`` command line option for ``sbatch``): ```shell -export SBATCH_ACCOUNT='example_project' +export SBATCH_ACCOUNT='project_XXXXXXXXX' ``` Or to submit to a particular Slurm partition (equivalent with the ``-p`` or ``--partition`` option for ``sbatch``): ```shell -export SBATCH_PARTITION='example_partition' +export SBATCH_PARTITION='small' ``` For more information about supported ``$SBATCH_*`` environment variables, @@ -113,24 +136,29 @@ as jobs, to avoid that they fail almost instantly due to a lack of disk space. Keep in mind that the active EasyBuild configuration is passed down into the submitted jobs, so any configuration that is present on the workernodes may not have any effect. -For example, if you commonly use `/tmp/$USER` for build directories on a login node, -you may need to tweak that when submitting jobs to use a different location: +For example, on LUMI it is possible to use ``$XDG_RUNTIME_DIR`` on the login nodes which has +the advantage that any leftovers of failed builds will be cleaned up when the user ends their last +login session on that node, but it is not possible to do so on the compute nodes. ```shell # EasByuild is configured to use /tmp/$USER on the login node -login01 $ eb --show-config | grep buildpath -buildpath (E) = /tmp/example +uan01 $ eb --show-config | grep buildpath +buildpath (E) = /run/user/XXXXXXXX/easybuild/build -# use /localdisk/$USER for build directories when submitting installations as jobs -login01 $ eb --job --buildpath /localdisk/$USER example.eb --robot +# use /dev/shm/$USER for build directories when submitting installations as jobs +login01 $ eb --job --buildpath /dev/shm/$USER/easybuild example.eb --robot ``` + ### Temporary log files and build directories -The temporary log file that EasyBuild creates is most likely going to end up on the local disk -of the workernode on which the job was started (by default in `$TMPDIR` or `/tmp`). -If an installation fails, the job will finish and temporary files will likely be cleaned up instantly, -which may leave you wondering about the actual cause of the failing installation... +The problems for the temporary log files are twofold. First, they may end up in a place +that is not available on the compute nodes. E.g., for the same reasons as for the build +path, the LUMI EasyBuild configuration will place the temporary files in a subdirectory of +``$XDG_RUNTIME_DIR`` on the loginnodes but a subdirectory of ``/dev/shm/$USER`` on the +compute nodes. The second problem however is that if an installation fails, those log files are +not even accessible anymore which may leave you wondering about the actual cause of the failing +installation... To remedy this, there are a couple of EasyBuild configuration options you can use: @@ -139,6 +167,7 @@ To remedy this, there are a couple of EasyBuild configuration options you can us ```shell $ eb --job example.eb --tmp-logdir $HOME/eb_tmplogs ``` + This will move at least the log file to a suitable place. * If you prefer having the entire log file stored in the Slurm job output files, you can use ``--logtostdout`` when submitting the jobs. This will result in extensive logging @@ -146,11 +175,13 @@ To remedy this, there are a couple of EasyBuild configuration options you can us log to ``stdout`` when the installation is running in the job, and hence the log messages will be captured in the job output files. -The same remark applies to build directories: they should be on a local filesystem (to avoid problems -that often occur when building software on a parallel filesystem like GPFS or Lustre), -which will probably be cleaned up automatically when a job fails. Here it is less easy to provide -general advice on how to deal with this, but one thing you can consider is retrying the installation -in an interactive job, so you can inspect the build directory after the installation fails. +The build directory of course also suffers from the problem of being no longer accessible if the +installation fails, but there it is not so easy to find a solution. Building on a shared file system +is not only much slower, but in particular on parallel file systems like GPFS/SpectrumScale, Lustre +or BeeGFS buiding sometimes fails in strange ways. One thing you can consider if you cannot do the +build on a login node (e.g., because the code is not suitable for cross-compiling or the configure +system does tests that would fail on the login node), is to rety the installation in an +interactive job, so you can inspect the build directory after the installation fails. ### Lock files @@ -171,37 +202,37 @@ subdirectory of ``installpath``) manually, or re-submit the job with ``eb --job As an example, we will let EasyBuild submit jobs to install ``AUGUSTUS`` with the ``foss/2020b`` toolchain. +!!! Warning "This example does not work on LUMI" + + Note that this is an example using the FOSS common toolchain. For this reason it does not work on + LUMI. + ### Configuration Before using ``--job``, let's make sure that EasyBuild is properly configured: ```shell -# use $HOME/easybuild for software, modules, sources, etc. -export EASYBUILD_PREFIX=$HOME/easybuild +# Load the EasyBuild-user module (central installations will not work at all +# using job submission) +module load LUMI/21.12 +module load partition/C +module load EasyBuild-user # use ramdisk for build directories -export EASYBUILD_BUILDPATH=/dev/shm/$USER +export EASYBUILD_BUILDPATH=/dev/shm/$USER/build +export EASYBUILD_TMPDIR=/dev/shm/$USER/tmp # use Slurm as job backend export EASYBUILD_JOB_BACKEND=Slurm ``` -In addition, add the path to the centrally installed software to ``$MODULEPATH`` via ``module use``: -```shell -module use /easybuild/modules/all -``` - -Load the EasyBuild module: - -```shell -module load EasyBuild -``` - -Let's assume that we also need to inform Slurm that jobs should be submitted into a particular account: +We will also need to inform Slurm that jobs should be submitted into a particular account, and +in a particular partition: ```shell -export SBATCH_ACCOUNT=example_project +export SBATCH_ACCOUNT=project_XXXXXXXXX +export SBATCH_PARTITION='small' ``` This will be picked up by the ``sbatch`` commands that EasyBuild will run to submit the software installation jobs. @@ -234,14 +265,14 @@ $ eb AUGUSTUS-3.4.0-foss-2020b.eb --missing Several dependencies are not installed yet, so we will need to use ``--robot`` to ensure that EasyBuild also submits jobs to install these first. -To speed up the installations a bit, we will request 10 cores for each submitted job (via ``--job-cores``). +To speed up the installations a bit, we will request 8 cores for each submitted job (via ``--job-cores``). That should be sufficient to let each installation finish in (well) under 1 hour, so we only request 1 hour of walltime per job (via ``--job-max-walltime``). In order to have some meaningful job output files, we also enable trace mode (via ``--trace``). ``` -$ eb AUGUSTUS-3.4.0-foss-2020b.eb --job --job-cores 10 --job-max-walltime 1 --robot --trace +$ eb AUGUSTUS-3.4.0-foss-2020b.eb --job --job-cores 8 --job-max-walltime 1 --robot --trace ... == resolving dependencies ... ... @@ -278,7 +309,7 @@ these jobs will be able to start. After about 20 minutes, AUGUSTUS and all missing dependencies should be installed: ``` -$ ls -lrt $HOME/easybuild/modules/all/*/*.lua | tail -11 +$ ls -lrt $HOME/EasyBuild/modules/.../*.lua | tail -11 -rw-rw----. 1 example example 1634 Mar 29 10:13 /users/example/easybuild/modules/all/HTSlib/1.11-GCC-10.2.0.lua -rw-rw----. 1 example example 1792 Mar 29 10:13 /users/example/easybuild/modules/all/SAMtools/1.11-GCC-10.2.0.lua -rw-rw----. 1 example example 1147 Mar 29 10:13 /users/example/easybuild/modules/all/BamTools/2.5.1-GCC-10.2.0.lua @@ -291,11 +322,9 @@ $ ls -lrt $HOME/easybuild/modules/all/*/*.lua | tail -11 -rw-rw----. 1 example example 1365 Mar 29 10:28 /users/example/easybuild/modules/all/SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0.lua -rw-rw----. 1 example example 2233 Mar 29 10:30 /users/example/easybuild/modules/all/AUGUSTUS/3.4.0-foss-2020b.lua -$ module use $HOME/easybuild/modules/all - $ module avail AUGUSTUS --------- /users/hkenneth/easybuild/modules/all -------- +-- EasyBuild managed user software for software stack ... -- AUGUSTUS/3.4.0-foss-2020b ``` From aff997941c01510bebd82d2fff53bc14f88f1bed Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 15:42:35 +0200 Subject: [PATCH 079/159] Some explanation in the overview of part 3 --- docs/2022-CSC_and_LO/1_02_Lmod.md | 82 +++++++- docs/2022-CSC_and_LO/3_00_part3_advanced.md | 6 + .../3_04_module_naming_schemes.md | 192 +++++++----------- 3 files changed, 152 insertions(+), 128 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md index 0caa8a47..d6d73a6c 100644 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -73,12 +73,54 @@ to a hierarchy with 3 levels: 3. The ``MPI`` level, containing libraries and applications that depend on the compiler used and the MPI implementation. +??? Example "A simple Lmod hierarchy with a single compiler" + + Here is a simple example of such a 3-level module hierarchy + (that almost could have been generated by EasyBuild): + +
+ + In this example the ``Core`` level only includes a single module `GCC/9.3.0`, + while the ``Compiler`` level includes two modules: `OpenMPI/4.0.3` and `MPICH/3.3.2`. + In the ``MPI `` level, three modules are available: one for `FFTW`, one for `ScaLAPACK`, + and one for `HDF5`. + + Initially only the modules on the top level of a module hierarchy are available for loading. + If you run "`module avail`", the command that is used to view all modules that are available + for loading, with this example module hierarchy, you will only see the `GCC/9.3.0` module. + + Some modules in the top level of the hierarchy act as a "gateway" to modules in the + next level below. + To make additional modules available for loading one of these gateway modules has to be loaded. + In our example, loading the `GCC/9.3.0` module results in two additional modules coming into + view from the ``Compiler`` level, as indicated by the arrows: the modules for `OpenMPI` and `MPICH`. + These correspond to installations of `OpenMPI` + and `MPICH` that were built using `GCC/9.3.0`. + + Similarly, the `OpenMPI/4.0.3` module serves as a gateway to the three modules in the ``MPI`` level. + Only by loading the `OpenMPI` module will these additional three modules become + available for loading. They correspond to software installations built using the ``GCC/9.3.0`` + compiler with ``OpenMPI/4.0.3``. + Now assume that we have two compilers in the hierarchy, Compiler_A and Compiler_B. Their modules would reside -at the ``Core`` level. Both copilers provide the same MPI implementation, MPI_C. So there would be two modules +at the ``Core`` level. Both compilers provide the same MPI implementation, MPI_C. So there would be two modules for ``MPI_C`` in two different subdirectories at the ``Compiler`` level. And further assume that we have an application, Appl_E, compiled with both Compiler_A and Compiler_B and using MPI_C. For that application there would also be two module files at the ``MPI`` level, one in a subdirectory corresponding ao Compiler_A and MPI_C and one -in a subdirectory corresponding to Compiler_B and MPI_C. To be able to load the module for Appl_E, a user should +in a subdirectory corresponding to Compiler_B and MPI_C. + +```mermaid +graph TD; +A[Compiler_A] --> AC[MPI_C]; +A --> AD[MPI_D] +B[Compiler_B] --> BC[MPI_C]; +AC --> ACE[Appl_E]; +AD --> ADE[Appl_E] +BC --> BCE[Appl_E]; +``` + + +To be able to load the module for Appl_E, a user should first load Compiler_A, then load MPI_C and only then is it possible to load the module for Appl_E: ```bash @@ -102,6 +144,23 @@ and depending on those adapt the path to the binaries, several very simple modul logic, and one could add an Appl_E module for a different compiler or MPI implementation without touching any of the already existing module files for that application. +Similarly, if after + +```bash +module load Compiler_A MPI_C Appl_E +``` + +one does + +```bash +module load MPI_D +``` + +thent MPI_C gets unloaded, Lmod notices that it also has to unload/deactivate Appl_E, then will load MPI_D for Compiler_A and +finally will notice that there is an equivalent Appl_E module available again, and Lmod will load that one also. However, +now loading Compiler_B will cause a warning that MPI_D and Appl_E have been deactivated as there is no module name MPI_D in +any version for Compiler_B. + ### Building blocks @@ -126,8 +185,9 @@ Some mechanisms in Lmod make implementing a hierarchy fairly easy (though there ### Implementation details -The above example could be implemented using 6 module files: One for each compiler, two for the MPI module and -two for the application module. +The above example could be implemented using 8 module files: One for each compiler, three for the MPI modules +(two for MPI_C and one for MPI_D) and +three for the application modules. ``` moduleroot @@ -139,8 +199,10 @@ moduleroot ├── Compiler │ ├── Compiler_A │ │ └── version_A -│ │ └── MPI_C -│ │ └── version_C.lua +│ │ ├── MPI_C +│ │ │ └── version_C.lua +│ │ └── MPI_D +│ │ └── version_D.lua │ └── Compiler_B │ └── version_B │ └── MPI_C @@ -148,8 +210,12 @@ moduleroot └── MPI ├── Compiler_A │ └── version_A - │ └── MPI_C - │ └── version_C + │ ├── MPI_C + │ │ └── version_C + │ │ └── Appl_E + │ │ └── version_E.lua + │ └── MPI_D + │ └── version_D │ └── Appl_E │ └── version_E.lua └── Compiler_B diff --git a/docs/2022-CSC_and_LO/3_00_part3_advanced.md b/docs/2022-CSC_and_LO/3_00_part3_advanced.md index 708c57f8..28a24990 100644 --- a/docs/2022-CSC_and_LO/3_00_part3_advanced.md +++ b/docs/2022-CSC_and_LO/3_00_part3_advanced.md @@ -2,6 +2,12 @@ *[[back to start page]](index.md)* +In this section we mostly cover "good to know that they exist" features as they are not used +on LUMI, or not really accessible to regular user installations that are performed with the +LUMI ``EasyBuild-user`` module. Hooks are used on LUMI, but it is not really advised to +overwrite the centrally defined hooks with a local file. And the whole structure of the +EasyBuild integration is also set up to make use of the GitHub integration in the future. + * [Using EasyBuild as a library](3_01_easybuild_library.md) *(hands-on)* * [Using hooks to customise EasyBuild](3_02_hooks.md) *(hands-on)* * [Submitting installations as Slurm jobs](3_03_slurm_jobs.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md index 14229d97..d0b4ebe8 100644 --- a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md +++ b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md @@ -4,21 +4,24 @@ --- -Up until now we have used EasyBuild's default module naming scheme (`EasyBuildMNS`), -which produces module files with names that closely resemble to the names of the +Up until now we have used the naming scheme in use on LUMI which is for many practical purposes +is indistinguishable from the default EasyBuild naming scheme (`EasyBuildMNS`). +It produces module files with names that closely resemble to the names of the corresponding easyconfig files. -For example, when installing `h5py-3.1.0-foss-2020b.eb` the generated module was named `h5py/3.1.0-foss-2020b`. +For example, when installing `zlib-1.2.11-cpeGNU-21.12.eb` the generated module was named +`zlib/1.2.11-cpeGNU-21.12`. EasyBuild supports several different module naming schemes: ```shell $ eb --avail-module-naming-schemes List of supported module naming schemes: - CategorizedHMNS - CategorizedModuleNamingScheme - EasyBuildMNS - HierarchicalMNS - MigrateFromEBToHMNS + EasyBuildMNS + LUMI_FlatMNS + MigrateFromEBToHMNS + HierarchicalMNS + CategorizedHMNS + CategorizedModuleNamingScheme ``` In this part of the tutorial we will take a closer look at `HierarchicalMNS`, @@ -28,52 +31,54 @@ We will also take a quick look at implementing our own custom module naming sche ## Flat vs hierarchical -The default module naming scheme `EasyBuildMNS` is an example of regular *"flat"* module naming scheme, which is characterized by: +!!! Note -* all module files are directly available for loading; -* each module name uniquely identifies a particular installation; + This text is very much based on the generic EasyBuild tutorials as we cannot demonstrate + all aspects on LUMI as it is configured today. -In contrast, a *hierarchical* module naming scheme -consists of a *hierarchy* of module files. - -The typical module hierarchy has 3 levels: + On LUMI, the complete module scheme is partly hierarchical, but EasyBuild uses a flat naming + scheme. The two levels in the hierarchy that are present to deal with different versions of + the software stack and the various types of compute nodes, are not managed by EasyBuild. -* a **core** level, where module files for software that was installed using the - [`system` toolchain](../introduction/#system-toolchain) are kept; -* a **compiler** level, where module files for software that was installed using a -*compiler-only toolchain* are stored; -* and an **MPI** level, which houses module files for software that was installed using a toolchain that includes (at least) a compiler and MPI component; +The default module naming scheme `EasyBuildMNS` +and the custom `LUMI_FlatMNS` naming scheme are both examples of regular *"flat"* +module naming schemes, which are characterized by: -Here is a simple example of such a 3-level module hierarchy: +- all module files are directly available for loading; +- each module name uniquely identifies a particular installation; -
+In contrast, a *hierarchical* module naming scheme +consists of a *hierarchy* of module files. +A fairly typical 3-level schme (``Core``, ``Compiler`` and ``MPI``) has been +discussed in the [section on Lmod](../1_02_Lmod#lmod-hierarchy). +This typical Lmod hierarcny would map very well on the EasyBuild common toolchains. -In this example the core level only includes a single module `GCC/9.3.0`, -while the compiler level includes two modules: `OpenMPI/4.0.3` and `MPICH/3.3.2`. -In the MPI level, three modules are available: one for `FFTW`, one for `ScaLAPACK`, and one for `HDF5`. +In fact, for the example -Initially only the modules on the top level of a module hierarchy are available for loading. -If you run "`module avail`" with the example module hierarchy, you will only see the `GCC/9.3.0` module. +
-Some modules in the top level of the hierarchy act as a "gateway" to modules in the -next level below. -To make additional modules available for loading one of these gateway modules has to be loaded. In our exampe, loading the `GCC/9.3.0` module results in two additional modules coming into view from the compiler level, as indicated by the arrows: the modules for `OpenMPI` and `MPICH`. These correspond to installations of `OpenMPI` -and `MPICH` that were built using `GCC/9.3.0` as a (compiler-only) toolchain. +software at the ``Core`` level would be installed with the ``GCCcore`` and ``SYSTEM`` +toolchains. Software at the ``Compiler`` level would be installed with the ``GCC`` +toolchain, while software at the ``MPI`` level after loading the ``OpenMPI`` module +would be installed using the regular ``foss`` toolchain or the ``gompi`` toolchain +(see the diagram in the +["Common toolchains" section of the page on terminology](../1_05_terminology/#common-toolchains)). -Similarly, the `OpenMPI/4.0.3` module serves as a gateway to the three modules in the MPI level. Only by loading the `OpenMPI` module will these additional three modules become -available for loading. They correspond to software installations built using the `gompi/2020a` toolchain that -consists of the `GCC/9.3.0` compiler module and the `OpenMPI/4.0.3` MPI module. Software installed using -`foss/2020a` (which is a full toolchain that also includes OpenBLAS, FFTW and ScaLAPACK) would also be stored -in this level of the module hierarchy. +On LUMI, where software is installed through the Cray Programming Environment with no real choice of +MPI implementation, a two-level arrangement would still make a lot of sense, with at the ``Core`` level +all software compiled with the SYSTEM toolchain while there could be a ``PrgEnv`` level for software +compiled with a particula programming enviornment aka cepGNU/cpeCray/cpeAOCC toolchain. Such a scheme +is used on the Cray systems at CSCS. -The characteristics of a module hierarchy are: +To recap, the characteristics of a module hierarchy are: * not all module files are directly available for loading; * some modules serve as a gateway to more modules; * to access some software installations you will first need to load one or more gateway modules in order to use them; -You can probably think of other ways to organize module files in a hierarchical module tree, but here we will stick to the standard core / compiler / MPI hierarchy. +You can probably think of other ways to organize module files in a hierarchical module tree, but here +we will stick to the standard core / compiler / MPI hierarchy. ### Pros & cons @@ -81,22 +86,24 @@ So why go through all this trouble of organizing modules hierarchically? There are a couple of advantages to this approach: -* shorter module names; +* shorter module names (or at least for the version part of the name); * less overwhelming list of available modules; * only compatible modules can be loaded together; However, the are some minor disadvantages too: -* not all existing modules are directly visible; +* not all existing modules are directly visible, so users have to learn how to find + modules using ``module spider`` etc.; * gateway modules may have little meaning to end users; #### Length of module names -When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For our `HDF5` installation for example, -we have `HDF5/1.10.6-gompi-2020a` as module name. The `-gompi-2020a` part of the name refers to the toolchain that was +When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For a `HDF5` installation +with the EasyBuild common toolchains for example, +one might have `HDF5/1.10.6-gompi-2020a` as module name. The `-gompi-2020a` part of the name refers to the toolchain that was used for this installation, but it may be confusing to some people (what kind of Pokémon is a "gompi"?!). -In the example module hierarchy shown above, the module for `HDF5` is named `HDF5/1.10.6` which is basically the bare +In the example module hierarchy shown above, the module for `HDF5` could simply be named `HDF5/1.10.6` which is basically the bare essentials: software name and version. That's way better, nice and clean! #### Amount of available modules @@ -130,8 +137,10 @@ and so these modules can not be loaded together easily. One downside of a module hierarchy is that not all existing modules are directly available for loading or are even *visible* to the user, since the output of "`module avail`" only shows a subset of all modules. -[Lmod](https://lmod.readthedocs.io) has a solution for this though: it provides a separate "`module spider`" -command to search for module files throughout the entire module hierarchy. So as long as the end users are +As we have discussed in [the Lmod section](../LMOD). +[Lmod](https://lmod.readthedocs.io) provides a powerful solution to search for modules through the +``module spider`` and to some extent also the ``module keyword`` commands. +So as long as the end users are aware of this additional command, it should not be difficult to discover which software installations exist and how they can be accessed. The "`module spider`" command will inform the user which of the gateway modules need to be loaded in order to load a specific module file. @@ -140,7 +149,7 @@ need to be loaded in order to load a specific module file. An additional potential problem of a module hierarchy is that the semantics of the gateway modules may not be clear to end users. They may wonder why they need to pick a specific compiler and MPI library, or which of the -available options is the best one. Maybe there are not even be aware what exactly a "compiler" is, or how it is +available options is the best one. Maybe they are not even aware what exactly a "compiler" is, or how it is relevant to the software they need in their bioinformatics pipeline... This can be partially resolved by loading a default compiler and MPI module so a particular set of modules @@ -148,10 +157,15 @@ is available right after login, which could be the ones used in the most recent recommended versions. More experienced users could then leverage the "`module spider`" command to navigate the module hierarchy. +On LUMI this would mean loading a default software stack, but due to the the default modules are currently +loaded on LUMI this was not possible to accomplish without losing other functionality of the module tree. + ## Using a custom module naming scheme Next to the module naming schemes that are included with EasyBuild, -you can also define your own module naming scheme (MNS), and configure EasyBuild to use it. +you can also define your own module naming scheme (MNS), and configure EasyBuild to use it +(which is eactly what has been done on LUMI to remove a feature of the default ``EasyBuildMNS`` scheme +that we do not use). ### Implementation @@ -254,7 +268,9 @@ class ExampleMNS(ModuleNamingScheme): return short_modname.startswith(name.lower().replace('-', '_') + '/') ``` -We can see what the module names with this module naming scheme would like like via ``eb -D``: +We can see what the module names with this module naming scheme would like like via ``eb -D``. +E.g., for the common toolchains (the example will not work on LUMI as the default easyconfig +files are not in the robot- and the search path): ``` $ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D @@ -270,9 +286,10 @@ $ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D ## Example module hierarchy: HDF5 -!!! Warning - **This exercise will only work if you have write access to the software installation prefix**, - because EasyBuild will try to copy the installation log file to each installlation directory. +!!! Warning "Example not suitable for LUMI" + **This exercise is meant for a system where the common toolchains can be used and requires an + indpendent EasyBuild installation in your personal file space**, + because EasyBuild will try to copy the installation log file to each installation directory. Now that we know more about hierarchical module naming schemes, let us see how EasyBuild can help us with generating a hierarchical module tree. @@ -308,7 +325,7 @@ a module. We have just made all modules unavailable, so we would have to first install EasyBuild again in our hierarchical module tree before we can continue. **We strongly recommend using an EasyBuild installation that was [installed via "`pip install`" -or "`pip3 install`"](../installation/#method-1-using-pip-recommended) in this part of the tutorial.** +or "`pip3 install`"](../1_06_installation#method-1-using-pip) in this part of the tutorial.** An easy way to do this is in the prepared environment is to run: @@ -376,7 +393,9 @@ The steps we will have to go through are: * Tell EasyBuild we want to "install" the `HDF5-1.10.7-gompi-2020b.eb` easyconfig file; * Enable dependency resolution via `--robot`; -* Instruct EasyBuild to only generate the module files, not to install the software (since it is +* Assuming the software would have been installed already with the default naming scheme + in a different module directory, instruct EasyBuild to only generate the module files, + not to install the software (since it is there already in `/easybuild/software`), via the `--module-only` option. These steps translate to this single `eb` command: @@ -544,73 +563,6 @@ Currently Loaded Modules: 4) GCC/10.2.0 8) libpciaccess/0.16 12) libfabric/1.11.0 16) HDF5/1.10.7 ``` -## Exercise - -Now it is your turn! - -Try to get a feeling for how a hierarchical module tree works by: - -* installing the missing modules for the `SciPy-bundle-2020.11-foss-2020b.eb` in the module hierarchy we - generated for HDF5; -* figure out where the `SciPy-bundle` module is located in the hierarchy, and then also load it; - -You can verify your work by running this command (since `pandas` is one of the Python packages included -in the `SciPy-bundle` installation): - -```shell -python -c 'import pandas; print(pandas.__version__)' -``` - -Start from a clean slate, by first running: - -```shell -module purge -module unuse $MODULEPATH -``` - -??? success "(click to show solution)" - - * Step 0: check which modules are still missing, using `--missing` or `-M`: - ```shell - eb SciPy-bundle-2020.11-foss-2020b.eb -M - ``` - The output should tell you that 15 out of 50 required modules are still missing. - - * Install the missing modules in the module hierarchy we have generated in `$HOME/hmns/modules`: - ```shell - eb SciPy-bundle-2020.11-foss-2020b.eb --robot --module-only - ``` - Don't forget to use both `--robot` (to enable dependency resolution) and `--module-only` - (to only run the sanity check and generate module files, not install the software again). - - * Start at the top of the module hierarchy (the `Core` level), - and run module spider to check which gateway modules to load to make `SciPy-bundle` available: - ```shell - module use $HOME/hmns/modules/all/Core - module spider SciPy-bundle/2020.11 - ``` - * Load the gateway modules: - ```shell - module load GCC/10.2.0 OpenMPI/4.0.5 - ``` - * Check that the `SciPy-bundle` module is available, and load it: - ```shell - $ module avail SciPy-bundle - ----- /home/example/hmns/modules/all/MPI/GCC/10.2.0/OpenMPI/4.0.5 ------ - SciPy-bundle/2020.11 - ``` - ``` - module load SciPy-bundle/2020.11 - ``` - * Run the test command: - ``` - $ python -c 'import pandas; print(pandas.__version__)' - 1.1.4 - ``` - -!!! Warning - **This exercise will only work if you have write access to the software installation prefix**, - because EasyBuild will try to copy the installation log file to each installlation directory. --- From 855101c989ee78e4b05aaf730f7e58f98a7bf20f Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 16:22:58 +0200 Subject: [PATCH 080/159] GitHub integration section, mostly old text and not all suited for LUMI. --- .../3_05_github_integration.md | 28 +++++++++++++++---- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_05_github_integration.md index 59dcbb67..6d04ce7f 100644 --- a/docs/2022-CSC_and_LO/3_05_github_integration.md +++ b/docs/2022-CSC_and_LO/3_05_github_integration.md @@ -4,6 +4,20 @@ --- +!!! Warning "Preliminary not for LUMI and many other Cray systems" + + Not everything in this section is as useful for EasyBuild installations that build + on the Cray Programming Environment. E.g., the default easyconfig repository does + not contain easyconfigs for the Cray PE. + + Moreover, due to the restricted operating system on the compute nodes of big + Cray systems such as the EX series (COS instead of the full SUSE linux), + the GitHub integration does not work on the compute nodes of a typical setup as + one of the required Python packages fails to work as it needs the DBus daemon. + + Since GitHub integration is not yet really used on LUMI, this section is not fully + tested but left in the tutorial for completeness. + To contribute changes to the EasyBuild code (framework or easyblocks) or easyconfigs, you will need to be a bit familiar with Git and GitHub. Or maybe not? @@ -94,11 +108,6 @@ and configuration. **Additional dependencies** Both the `GitPython` and `keyring` Python packages as well as the `keyrings.cryptfile` add-on package must be installed. -In the prepared environment, you can do this via: - -```shell -pip3 install --user GitPython keyring keyrings.cryptfile -``` !!! Note You may experiences problems installing the ``cryptography`` Python packages, @@ -294,6 +303,15 @@ will include information about your system (OS, processor, etc.) and your EasyBu ## Demo +!!! Warning "Not currently suited for LUMI" + + This is text from a previous (non-Cray) version of the tutorial. The demo does require + a number of files not installed on LUMI after going through this tutorial, + and someone who can master the repository used to clean up again. + However, having a look at the output (which was generated in the spring of 2021 on + the CSC system puhti) still offers some information, so the demo was left in this + version of the tutorial. + That is a lot to digest, so let us make this a bit more concrete with an example: we will open a pull request for the [`eb-tutorial` example software](../adding_support_software/#example) to *a fork* of the [`easybuild-easyconfigs` repository](https://github.com/easybuilders/easybuild-easyconfigs) using the `eb` command, and submit a test report in it. From 89e01df3471da1227aac1e34c33006c04a72e49b Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 16:34:33 +0200 Subject: [PATCH 081/159] Adapt the structure and include an additional reading section. --- docs/2022-CSC_and_LO/3_05_github_integration.md | 2 +- docs/2022-CSC_and_LO/index.md | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_05_github_integration.md index 6d04ce7f..6e19cb9a 100644 --- a/docs/2022-CSC_and_LO/3_05_github_integration.md +++ b/docs/2022-CSC_and_LO/3_05_github_integration.md @@ -457,4 +457,4 @@ The gist linked from this comment provides more detailed information: --- -*[[next: Overview]](index.md)* +*[[next: Additional reading]](4_00_additional_reading.md)* diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md index 58831541..b6cf98d8 100644 --- a/docs/2022-CSC_and_LO/index.md +++ b/docs/2022-CSC_and_LO/index.md @@ -78,4 +78,3 @@ freely available, so unless you have access to a system with this environment yo - [Submitting installations as Slurm jobs](3_Advanced/3_03_slurm_jobs.md) - [Module naming schemes (incl. hierarchical)](3_Advanced/3_04_module_naming_schemes.md) - [GitHub integration to facilitate contributing to EasyBuild](3_Advanced/3_05_github_integration.md) -- [**Additional reading**](4_00_additional_reading.md) From ed863b967739a44c655a8c8f66627f1c57eb8fee Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 17:43:28 +0200 Subject: [PATCH 082/159] Correcting a number of links. --- docs/2022-CSC_and_LO/1_06_installation.md | 4 ++-- docs/2022-CSC_and_LO/1_07_configuration.md | 6 +++--- docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md | 6 +++--- docs/2022-CSC_and_LO/2_03_external_modules.md | 2 +- docs/2022-CSC_and_LO/3_03_slurm_jobs.md | 6 +++--- 5 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_06_installation.md b/docs/2022-CSC_and_LO/1_06_installation.md index 6e68b500..24c71d5e 100644 --- a/docs/2022-CSC_and_LO/1_06_installation.md +++ b/docs/2022-CSC_and_LO/1_06_installation.md @@ -424,7 +424,7 @@ We do so only for the framework and easyblocks files as the easconfig files are install EasyBuild. Instead we create our own EasyConfig file for EasyBuild which contains some additional packages that enable extra features in EasyBuild and also provide more information to Lmod. Next the configuration module for EasyBuild (see the next section, -["Configuring EasyBuild"](1_06_configuring.md), for more information) and use the temporary +["Configuring EasyBuild"](1_07_configuration), for more information) and use the temporary installation of EasyBuild with our own EasyConfig file to do a proper installation of EasyBuild with module in the final location. @@ -468,7 +468,7 @@ eb --show-config ``` This should tell you that EasyBuild (ab)uses `$HOME/.local/easybuild` as a default location. -More on configuring EasyBuild in the [next part of the tutorial](configuration.md). +More on configuring EasyBuild in the [next part of the tutorial](configuration). #### System information diff --git a/docs/2022-CSC_and_LO/1_07_configuration.md b/docs/2022-CSC_and_LO/1_07_configuration.md index 17e60afe..a1c2f3de 100644 --- a/docs/2022-CSC_and_LO/1_07_configuration.md +++ b/docs/2022-CSC_and_LO/1_07_configuration.md @@ -280,7 +280,7 @@ The default `EasyBuildMNS` module naming scheme roughly corresponds to the filen and consists of the software name followed by a combination of the software version, toolchain and an optional label (which corresponds to the value of the `versionsuffix` easyconfig parameter): `/<-toolchain>`. Just like with [names of easyconfig -files](terminology.md#easyconfig-files), the `<-toolchain>` part is omitted when the `system` toolchain is used, +files](1_05_terminology#easyconfig-files), the `<-toolchain>` part is omitted when the `system` toolchain is used, and the `` value is empty by default. !!! Note "Module naming scheme on LUMI" @@ -366,9 +366,9 @@ across different software installations. They define *configuration settings*, such as the location where software should be installed, or the syntax that should be used when generating module files. -An [easyconfig file](terminology.md#easyconfig-files) (`*.eb`) on the other hand *specifies the details for one particular software installation*. +An [easyconfig file](1_05_terminology#easyconfig-files) (`*.eb`) on the other hand *specifies the details for one particular software installation*. It does this by defining a set of *easyconfig parameters*, which tell EasyBuild the name and version -of the software to install, which [toolchain](terminology.md#toolchains) and [easyblock](terminology.md#easyblocks) to use, etc. +of the software to install, which [toolchain](1_05_terminology#toolchains) and [easyblock](1_05_terminology#easyblocks) to use, etc. For each software installation performed by EasyBuild, there is a corresponding easyconfig file. There typically are only a handful of configuration files used however, for example a system-level diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md index f7d6a646..f9bec25a 100644 --- a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md @@ -1,6 +1,6 @@ # Creating easyconfig files -*[[back: Troubleshooting]](2_02_troubleshooting.md)* +*[[back: Troubleshooting]](2_02_troubleshooting)* --- @@ -58,7 +58,7 @@ Reasons to consider implementing a software-specific easyblock rather than using bootstrapping with the system compiler and then re-installing with itself.* For implementing easyblocks we refer to the -["Implementing easyblocks" section of this tutorial](2_03_implementing_easyblocks.md) +["Implementing easyblocks" section of this tutorial](2_03_implementing_easyblocks) and the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). ## Writing easyconfig files @@ -1024,4 +1024,4 @@ You can leverage the `eb-tutorial` easyconfig file we have composed in the examp --- -*[[next: Using external modules from the Cray PE]](2_03_external_modules.md)* +*[[next: Using external modules from the Cray PE]](2_03_external_modules)* diff --git a/docs/2022-CSC_and_LO/2_03_external_modules.md b/docs/2022-CSC_and_LO/2_03_external_modules.md index b19313ff..80a29712 100644 --- a/docs/2022-CSC_and_LO/2_03_external_modules.md +++ b/docs/2022-CSC_and_LO/2_03_external_modules.md @@ -1,6 +1,6 @@ # Using external modules from the Cray PE -*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* +*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files)* --- diff --git a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md index 567f8abd..283a53a0 100644 --- a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md +++ b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md @@ -37,8 +37,8 @@ will be *passed down* into the submitted job automatically, via command line opt command that is run in the job script. This includes not only command line options used next to ``--job``, but also configuration settings -that are specified via an [EasyBuild configuration file](configuration.md#configuration-files) or through -[``$EASYBUILD_*`` environment variables](configuration.md#easybuild_-environment-variables). +that are specified via an [EasyBuild configuration file](1_07_configuration#configuration-files) or through +[``$EASYBUILD_*`` environment variables](1_07_configuration#easybuild_-environment-variables). This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables that are in place in the job environment are most likely *irrelevant*, since configuration settings @@ -330,4 +330,4 @@ $ module avail AUGUSTUS --- -*[[next: Module naming schemes]](3_04_module_naming_scheme.md)* +*[[next: Module naming schemes]](3_04_module_naming_scheme)* From 557ba61657872cde5abb3479adcc615b7c6199b8 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 7 Apr 2022 17:26:56 +0200 Subject: [PATCH 083/159] Restructuring for nicer navigation bar on the left. --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 12 - .../2022-CSC_and_LO/1_01_what_is_easybuild.md | 321 ----- docs/2022-CSC_and_LO/1_02_Lmod.md | 820 ----------- docs/2022-CSC_and_LO/1_03_CPE.md | 418 ------ .../1_04_LUMI_software_stack.md | 101 -- docs/2022-CSC_and_LO/1_05_terminology.md | 294 ---- docs/2022-CSC_and_LO/1_06_installation.md | 545 -------- docs/2022-CSC_and_LO/1_07_configuration.md | 610 --------- docs/2022-CSC_and_LO/1_08_basic_usage.md | 1216 ----------------- docs/2022-CSC_and_LO/2_00_part2_using.md | 8 - docs/2022-CSC_and_LO/2_01_troubleshooting.md | 545 -------- .../2_02_creating_easyconfig_files.md | 1027 -------------- docs/2022-CSC_and_LO/2_03_external_modules.md | 85 -- .../2_04_implementing_easyblocks.md | 754 ---------- docs/2022-CSC_and_LO/3_00_part3_advanced.md | 15 - .../2022-CSC_and_LO/3_01_easybuild_library.md | 301 ---- docs/2022-CSC_and_LO/3_02_hooks.md | 213 --- docs/2022-CSC_and_LO/3_03_slurm_jobs.md | 333 ----- .../3_04_module_naming_schemes.md | 569 -------- .../3_05_github_integration.md | 460 ------- .../3_Advanced/3_04_module_naming_schemes.md | 2 +- docs/2022-CSC_and_LO/index.md | 1 + 22 files changed, 2 insertions(+), 8648 deletions(-) delete mode 100644 docs/2022-CSC_and_LO/1_00_part1_intro.md delete mode 100644 docs/2022-CSC_and_LO/1_01_what_is_easybuild.md delete mode 100644 docs/2022-CSC_and_LO/1_02_Lmod.md delete mode 100644 docs/2022-CSC_and_LO/1_03_CPE.md delete mode 100644 docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md delete mode 100644 docs/2022-CSC_and_LO/1_05_terminology.md delete mode 100644 docs/2022-CSC_and_LO/1_06_installation.md delete mode 100644 docs/2022-CSC_and_LO/1_07_configuration.md delete mode 100644 docs/2022-CSC_and_LO/1_08_basic_usage.md delete mode 100644 docs/2022-CSC_and_LO/2_00_part2_using.md delete mode 100644 docs/2022-CSC_and_LO/2_01_troubleshooting.md delete mode 100644 docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md delete mode 100644 docs/2022-CSC_and_LO/2_03_external_modules.md delete mode 100644 docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md delete mode 100644 docs/2022-CSC_and_LO/3_00_part3_advanced.md delete mode 100644 docs/2022-CSC_and_LO/3_01_easybuild_library.md delete mode 100644 docs/2022-CSC_and_LO/3_02_hooks.md delete mode 100644 docs/2022-CSC_and_LO/3_03_slurm_jobs.md delete mode 100644 docs/2022-CSC_and_LO/3_04_module_naming_schemes.md delete mode 100644 docs/2022-CSC_and_LO/3_05_github_integration.md diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md deleted file mode 100644 index 0674be92..00000000 --- a/docs/2022-CSC_and_LO/1_00_part1_intro.md +++ /dev/null @@ -1,12 +0,0 @@ -# Part I: Introduction to EasyBuild on Cray systems - -*[[back to start page]](index.md)* - -- [What is EasyBuild?](1_01_what_is_easybuild.md) -- [The Lmod module system](1_02_Lmod.md) -- [The HPE Cray Programming Environment](1_03_CPE.md) -- [LUMI software stacks](1_04_LUMI_software_stack.md) -- [Terminology](1_05_terminology.md) -- [Installation](1_06_installation.md) -- [Configuration](1_07_configuration.md) -- [Basic usage](1_08_basic_usage.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md deleted file mode 100644 index 979c6e90..00000000 --- a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md +++ /dev/null @@ -1,321 +0,0 @@ -# What is EasyBuild? - -*[[back: Overview]](index.md)* - -

EasyBuild logo

- -**[EasyBuild](http://easybuild.io)** is a **software build and installation framework** -that manages (scientific) software in an efficient way, without compromising on performance. -Easybuild is especially tailored towards [**High Performance Computing (HPC)**](https://en.wikipedia.org/wiki/Supercomputer) systems, -and provides a **command-line interface** to facilitate the installation and management of a scientific software stack. - ---- - - - - - -## Elevator pitch - -EasyBuild intends to relieve HPC support teams from manually managing -software installations while at the same time **providing a consistent and well performing -scientific software stack** to end users of HPC systems. - -It serves as a **uniform interface for installing scientific software** -and saves valuable time (and frustration) by the automation of tedious, boring and repetitive tasks. - -In addition, EasyBuild can **empower scientific researchers to self-manage their software stack**, -and it can serve as a tool that can be leveraged for **building optimized container images**. - -The project has grown to become a **platform for collaboration** among HPC sites worldwide. - ---- - -On LUMI we pay a lot of attention to empowering users to create their custom software stack which -is best suited for their project without being confronted with hundreds of modules that are not -useful to them. It also enables them to easily customize the build instructions to create a -project-specific build or even to install software with custom patches to adapt to their needs. -For this we offer both [EasyBuild](https://easybuild.io/) and [Spack](https://spack.io/) -as tools but obviously this tutorial concentrates on EasyBuild alone. - -EasyBuild can also be used to [generate container recipes](https://docs.easybuild.io/en/latest/Containers.html) -that will then use EasyBuild when building -the container to install platform-optimised software in the container. However, as that functionality -is still under development on LUMI it is not yet covered in this LUMI-specific version of the tutorial. - - ---- - -## Key features - -EasyBuild is capable of **fully autonomously installing (scientific) software**, -including making sure that all necessary dependencies are installed, -and automatically generating environment module files. - -***No*** **admin privileges are required**: it is sufficient to have write permissions -to the preferred software installation prefix. - -EasyBuild is [**highly configurable**](https://docs.easybuild.io/en/latest/Configuration.html) via configuration files, -environment variables, and command line options. Various elements of the functionality can be -[**dynamically extended**](https://docs.easybuild.io/en/latest/Including_additional_Python_modules.html) via plugins, -and [**hooks**](https://docs.easybuild.io/en/latest/Hooks.html) are available for further site-specific customizations. - -The installation procedure executed by EasyBuild is thoroughly -[**logged**](https://docs.easybuild.io/en/latest/Logfiles.html), and is fully transparent via support for -[**dry runs**](https://docs.easybuild.io/en/latest/Extended_dry_run.html) and -[**tracing**](https://docs.easybuild.io/en/latest/Tracing_progress.html) the software installation procedure -as it is performed. - -EasyBuild supports using a **custom module naming scheme**, allows for -*hierarchical* module naming schemes, and **integrates with various other tools** ranging from resource managers ([Slurm](https://slurm.schedmd.com) and [GC3Pie](https://github.com/gc3pie/gc3pie)), -container tools ([Singularity](https://github.com/hpcng/singularity) and [Docker](https://www.docker.com)), -packaging tools ([FPM](https://fpm.readthedocs.io)), and so on. - ---- - -## Focus points - -EasyBuild was created specifically for **installing scientific software on HPC systems**, -which is reflected in some of the design choices that were made. - - -### Performance - -EasyBuild strongly prefers to **build software from source code**, whenever possible. - -This is important to ensure that the binaries that are installed can maximally exploit -the capabilities of the system architecture on which the software will be run. - -For that same reason, EasyBuild out-of-the-box **optimizes software for the processor architecture of the build host** -by default, via compiler options like ``-march=native`` (GCC), ``-xHost`` (Intel compilers), etc. -This behaviour [can be changed via the ``--optarch`` configuration setting](https://docs.easybuild.io/en/latest/Controlling_compiler_optimization_flags.html). -When using the HPE Cray Programming Environment, the target modules are used instead. -In principle it is possible to cross-compile with EasyBuild, but this comes with some pitfalls. -By default EasyBuild tries to run tests if the build process of a package provides that -functionality, or can run simple tests of its own, and of course these tests will fail if -the host CPU does not support the full instruction set of the target CPU. Secondly, more -and more software has broken build processes where developers thought that it was a good -idea to hard-code options such as ``-march=native`` or ``--xHost`` in a way that EasyBuild -cannot easily overwrite those. - - -### Reproducibility - -In addition to performance, **reproducibility of installations** is a core aspect of EasyBuild. - -TODO correct link in the next lines. - -Most software installations performed with EasyBuild use a **particular compiler toolchain**, -with which we aim to be in control over the build environment and avoid relying on tools and libraries -provided by the operating system. For similar reasons, we try to **provide all required dependencies through EasyBuild** as well, -with a few notable exceptions, like ``OpenSSL`` for security reasons, and Infiniband and GPU drivers which -are too closely intertwined with the operating system. - -For both toolchains and dependencies, **fixed software versions** are specified in the -easyconfig files. That way, easyconfig files can easily be shared with others: -if they worked for you it is very likely that they will work for others too, because the vast majority of the -software stack is controlled by EasyBuild. - -As we shall see when discussing the HPE Cray Programming Environment, the reproducibility is a bit more limited -in that environment due to the way the programming environment selects shared libraries at runtime. - - -### Community effort - -In a number of different ways, we try to encourage EasyBuild users to **collaborate** and help each other out. - -We actively recommend people to report problems and bugs, to submit ideas for additional features and improvements, -and to [**contribute back**](https://docs.easybuild.io/en/latest/Contributing.html) when possible, be it -by opening pull requests to the [GitHub repositories](https://github.com/easybuilders) or the [documentation](https://docs.easybuild.io). - -The EasyBuild community also provides packages for two standard toolchains, called the -[**common toolchains**](https://easybuild.readthedocs.io/en/latest/Common-toolchains.html). -The so-called ``foss`` is based on free and open software (GNU compilers, FlexiBlas with OpenBLAS backend, LaPACK, FFTW and Open MPI) -while the ``intel`` toolchain uses the full stack of Intel compilers, libraries and MPI. Currently these toolchains -cannot be supported on LUMI though as getting Open MPI to run on LUMI is challenging and as the Intel libraries and -some MPI versions also produce slow or incorrect results on AMD CPUs (and it is not clear if Intel MPI is compatible -with SlingShot 11). -Focusing the efforts on those two popular common toolchains makes it possible to support a lot of build recipes that are -collected in the [central repository](https://github.com/easybuilders/easybuild-easyconfigs). -They can still serve as a basis to develop build recipes for the HPE Cray Programming Environment. - -Last but not least, EasyBuild provides various [**GitHub integration features**](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) -that greatly facilitate the contribution process: opening, updating, and testing pull requests, -reviewing incoming contributions, and much more can all be done directly from the EasyBuild -command line. This not only saves time, effort, brain cycles, and mouse clicks for contributors, -but it also makes the review process for *maintainers* significantly easier. -All together this leads to improved stability and consistency. - ---- - -## What EasyBuild is ***not*** - -EasyBuild is ***not*** **YABT (Yet Another Build Tool)**: it does *not* replace established build -tools like CMake or ``make``, it wraps around them. -If the installation procedure of a software package involves running some unholy trinity of tools while whispering -the correct magic incantations, EasyBuild automates this process for you. - -It is ***not*** **a replacement for traditional Linux package managers** like ``yum``, ``dnf`` or ``apt``. -EasyBuild relies on certain tools and libraries provided by the operating system. This includes glibc, OpenSSL, -drivers for Infiniband and GPUs, and so on. It is required that these tools are installed and managed by other means. -This is typically done via the package management tool that comes with your Linux distribution. - -Finally, EasyBuild is ***not*** **a magic solution to all your (software installation) problems**. -You may, and probably still will occasionally, run into compiler errors unless somebody has already taken care of the problem for you. -This is even more likely to happen in the Cray environment as builds are not often tested with the Cray compilers -or even with the AMD compilers. - ---- - -## Requirements - -Since EasyBuild is primarily intended for installing (scientific) software on HPC systems, the main target is -systems running some flavor of **GNU/Linux**. EasyBuild also works on *macOS*, -but significant effort would be required to use it extensively there. - - - -EasyBuild is implemented in [**Python**](https://www.python.org), and is compatible with Python 2.7 and 3.5+. -Since [Python 2 is no longer maintained](https://blog.python.org/2020/04/python-2718-last-release-of-python-2.html), -it is recommended to use a recent version of Python 3 for running EasyBuild. - -An [**environment modules**](https://en.wikipedia.org/wiki/Environment_Modules_(software)) tool is required for running EasyBuild. -The modern Lua-based implementation **[Lmod](https://lmod.readthedocs.io)** is strongly recommended, -but other (Tcl-based) implementations are also supported. -Both Tcl and Lua are supported as syntax for environment module files. -On LUMI Lmod is used as the module tool. - ---- - -## Development - - -EasyBuild is **actively developed** by a worldwide community under the -[**GPLv2 open source license**](https://www.gnu.org/licenses/old-licenses/gpl-2.0.html) -via the [**``easybuilders``** organisation on GitHub](https://github.com/easybuilders), in the following repositories: - -- [easybuild-framework](https://github.com/easybuilders/easybuild-framework) -- [easybuild-easyblocks](https://github.com/easybuilders/easybuild-easyblocks) -- [easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs) -- [easybuild](https://github.com/easybuilders/easybuild) (website, documentation) -- [easybuild-tutorial](https://github.com/easybuilders/easybuild-tutorial) (the generic EasyBuild tutorials) - -On LUMI the generic EasyBuild installation is used but extended and adapted using LUMI-specifics -plugins and hooks maintained in the [LUMI software stack repository](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack). - -Stable versions of EasyBuild have been released roughly every 6-8 weeks since 2012, -and are published via the *Python Package Index (PyPI)*, -see [https://pypi.org/project/easybuild](https://pypi.org/project/easybuild/). - -**Comprehensive testing** practices are applied throughout the -development cycle, with extensive suites of unit and integration tests being run in a CI environment (GitHub Actions), -consistent testing of incoming contributions (pull requests), and thorough regression testing before every release. - ---- - -## A worldwide community - -EasyBuild has grown into a **community project** since it was first released publicy in 2012, -and is used and developed by dozens (hundreds?) of HPC centres and consortia worldwide. - -Since 2016 the EasyBuild community has gathered for a yearly [**EasyBuild User Meeting**](https://easybuild.io/eum), -which is an open and highly interactive event that provides a great opportunity to meet fellow EasyBuild users, -developers, and maintainers, and to discuss related topics, learn about new aspects of the tool, and how other HPC -sites leverage it, and integrate it with other tools like [Jenkins](https://www.youtube.com/watch?v=bzW45cfRUMU), -[Ansible](https://www.youtube.com/watch?v=Qr2udsAjkG0), [CernVM-FS](https://www.youtube.com/watch?v=_0j5Shuf2uE), -[ReFrame](https://www.youtube.com/watch?v=KN9QwQIrso4) etc. - -HPC sites, consortia, and companies that use EasyBuild include (but are not limited to): - -VSC logo -JSC logo -HPCNow! logo -Fred Hutch logo -Vienna BioCenter logo -CÉCI logo -Compute Canada logo -UiO logo -BEAR logo -SNIC logo -National University of Singapore logo -IT4Innovations logo -Texas A&M logo -TU Dresden logo -University of Melbourne logo -University of Luxembourg logo -EPCC logo -Cyprus Institute logo -University of Basel logo -Microsoft logo - ---- - -## Getting help - -### *Documentation* - -If you have any questions about EasyBuild, make sure to check the EasyBuild documentation first, -which is available at [**https://docs.easybuild.io**](https://docs.easybuild.io). - -It is not unlikely that an answer to your question is available there already. -The [search box](https://docs.easybuild.io/en/latest/search.html) on the top left may be helpful. - -### GitHub - - - -You can open an issue in one of the EasyBuild repositories in [the `easybuilders` GitHub organisation](https://github.com/easybuilders/) to -ask questions, report problems or share suggestions and ideas for additional -improvements. - -Please try to post the issue in the most suitable repository. If you are -not sure which repository is most suited just go ahead and open the issue anyway, -the EasyBuild maintainers can move your issue if deemed appropriate. - -Also here you can use the search box on the top left, to see if somebody else -has already asked a similar question or reported a similar problem. - -### Mailing list - -Feel free to ask any questions or provide feedback via the EasyBuild mailing list -`easybuild@lists.ugent.be`. - -Keep in mind that **you must be subscribed to the mailing list in order to -post messages there!** You can subscribe via [this link](https://lists.ugent.be/wws/subscribe/easybuild). - -In 2016-2017 over 1,000 messages were posted to the EasyBuild mailing list. -More recently this has decreased to about 700 messages per year, mostly due -to the EasyBuild Slack channel. - -### Slack - -Slack logo - -The [EasyBuild Slack](https://easybuild.slack.com/) is a quick and easy way -to get in touch with the EasyBuild community. -Over 400 people have joined it so far, and there is usually somebody awake in there -who may be able to help you out. - -**To join the EasyBuild Slack request an invitation via -[https://easybuild-slack.herokuapp.com](https://easybuild-slack.herokuapp.com/).** - -If you are not a big fan of Slack, you can also consider joining the `#easybuild` -IRC channel (available on the [Freenode IRC network](https://freenode.net/)). -A bot relays messages between this IRC channel and the `#general` channel in the EasyBuild Slack. - - -### Conference calls - -Every other week on Wednesday an EasyBuild conference call is set up to discuss recent -developments, provide an outlook to the next EasyBuild release, and -provide answers to questions that can be raised by the people attending the call. - -These conference calls are open to anyone interested in the EasyBuild project, -and have been organised since November 2013. - -More information, including the dates and times of the next calls, is available -[on the EasyBuild Wiki](https://github.com/easybuilders/easybuild/wiki/Conference-calls). - ---- - -*[[next: The Lmod module system]](1_02_Lmod.md)* diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md deleted file mode 100644 index d6d73a6c..00000000 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ /dev/null @@ -1,820 +0,0 @@ -# The Lmod module system - -*[[back: What is EasyBuild?]](1_01_what_is_easybuild.md)* - ---- - -## Modules - -*Module* is a massively overloaded term in (scientific) software and IT in general -(kernel modules, Python modules, and so on). -In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. - -[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept -on HPC systems: it is a way to specify changes that should be made to one or more -[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a -[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file -is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or -[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, -and specifies which environment variables should be updated, and how (append, -prepend, (re)define, undefine, etc.) upon loading the environment module. -Unloading the environment module will restore the shell environment to its previous state. - -Environment module files are processed via a **modules tool**, of which there -are several conceptually similar yet slightly different implementations. -The oldest module tool still in use today is Environment Modules 3.2, implemented in C and -supporting module files written in Tcl. After a gap in developement, Xavier Delaruelle of CEA -developed [Environment Modules 4 and 5](https://sourceforge.net/projects/modules/) which is -fully implemented on Tcl. An alternative module tool is [Lmod](https://lmod.readthedocs.io), -developed by Robert McLay at TACC and implemented in LUA. This tool supports natively LUA -module files but also offers a high degree of compatibility with Tcl-based module files -developed for Environment Modules fia a translation layer and some API translation. - -The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no -packages or official support for Environment Modules 4 or 5. At the user level, -Environment Modules 3.2 and Lmod have many commands in common, but with different options. -Lmod also has some powerful features that are lacking in Environment Modules 3.2. - -!!! Note "The Cray PE on LUMI" - On LUMI, Lmod was selected as the module tool. One area where there are significant - differences between Environment Modules 3.2 (and also the newer versions) and Lmod is - in the commands for discovering modules on the system. If you are not familiar with Lmod - and its commands for users, it is worthwile to read the - [LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). - Some of those commands are also discussed on this page. - ---- - -## Lmod hierarchy - -### User view - -Lmod supports a module hierarchy. In a hierarchy, there is a distinction between the *installed -modules* and the *available modules*. Available modules are those that can be loaded directly -without first loading any other module, while the installed modules is the complete set of -modules that one could load one way or another. A typical use case -is a hierarchy to deal with different compilers on a system and different MPI implementations. -After all, it is a common practice to only link libraries and application code compiled with the -same compiler to avoid compatibility problems between compilers (and to be able to use advanced -features such as link time optimization). This is even more important for MPI, as Open MPI and -MPCIH-derived MPI implementations have incompatible Application Binary Interfaces. This would lead -to a hierarchy with 3 levels: - -1. The ``Core`` level containing the modules for the compilers themselves, e.g., one or more versions - of the GNU compiler suite and one or more versions of LLVM-based compilers. - - Loading a compiler module would then make the next level available: - -2. The ``Compiler`` level, containing modules for libraries and packages that only rely on the compilers - but do not use MPI, as well as the MPI modules, e.g., a version of Open MPI and a version of MPICH. - - Loading one of the MPI modules would then make the next level available: - -3. The ``MPI`` level, containing libraries and applications that depend on the compiler used and the MPI - implementation. - -??? Example "A simple Lmod hierarchy with a single compiler" - - Here is a simple example of such a 3-level module hierarchy - (that almost could have been generated by EasyBuild): - -
- - In this example the ``Core`` level only includes a single module `GCC/9.3.0`, - while the ``Compiler`` level includes two modules: `OpenMPI/4.0.3` and `MPICH/3.3.2`. - In the ``MPI `` level, three modules are available: one for `FFTW`, one for `ScaLAPACK`, - and one for `HDF5`. - - Initially only the modules on the top level of a module hierarchy are available for loading. - If you run "`module avail`", the command that is used to view all modules that are available - for loading, with this example module hierarchy, you will only see the `GCC/9.3.0` module. - - Some modules in the top level of the hierarchy act as a "gateway" to modules in the - next level below. - To make additional modules available for loading one of these gateway modules has to be loaded. - In our example, loading the `GCC/9.3.0` module results in two additional modules coming into - view from the ``Compiler`` level, as indicated by the arrows: the modules for `OpenMPI` and `MPICH`. - These correspond to installations of `OpenMPI` - and `MPICH` that were built using `GCC/9.3.0`. - - Similarly, the `OpenMPI/4.0.3` module serves as a gateway to the three modules in the ``MPI`` level. - Only by loading the `OpenMPI` module will these additional three modules become - available for loading. They correspond to software installations built using the ``GCC/9.3.0`` - compiler with ``OpenMPI/4.0.3``. - -Now assume that we have two compilers in the hierarchy, Compiler_A and Compiler_B. Their modules would reside -at the ``Core`` level. Both compilers provide the same MPI implementation, MPI_C. So there would be two modules -for ``MPI_C`` in two different subdirectories at the ``Compiler`` level. And further assume that we have an -application, Appl_E, compiled with both Compiler_A and Compiler_B and using MPI_C. For that application there would -also be two module files at the ``MPI`` level, one in a subdirectory corresponding ao Compiler_A and MPI_C and one -in a subdirectory corresponding to Compiler_B and MPI_C. - -```mermaid -graph TD; -A[Compiler_A] --> AC[MPI_C]; -A --> AD[MPI_D] -B[Compiler_B] --> BC[MPI_C]; -AC --> ACE[Appl_E]; -AD --> ADE[Appl_E] -BC --> BCE[Appl_E]; -``` - - -To be able to load the module for Appl_E, a user should -first load Compiler_A, then load MPI_C and only then is it possible to load the module for Appl_E: - -```bash -module load Compiler_A MPI_C Appl_E -``` - -What is interesting is what happens if the user now loads Compiler_B: - -```bash -module load Compiler_B -``` - -In a properly designed and implemented hierarchy, Lmod will unload Compiler_A which will also trigger the unloading/deactivation -of MPI_C and Appl_E. It will then load the module for Compiler_B and proceed with looking if it can find another module for -MPI_C. That will then be loaded which now makes a different module for Appl_E available, which Lmod will proceed to load. If it -cannot find an exact match for the version, Lmod will even try to locate a different version. Hence the situation after loading -Compiler_B is that now modules are loaded for Compiler_B, MPI_C for Compiler_B and Appl_E for Compiler_A with MPI_C. -All this requires very little effort from the module file programmer and very little logic in the module files. E.g., rather -then implementing a single module file for Appl_E that would require logic to see which compiler and MPI implementation is loaded -and depending on those adapt the path to the binaries, several very simple modules need to be written with very little -logic, and one could add an Appl_E module for a different compiler or MPI implementation without touching any of the already -existing module files for that application. - -Similarly, if after - -```bash -module load Compiler_A MPI_C Appl_E -``` - -one does - -```bash -module load MPI_D -``` - -thent MPI_C gets unloaded, Lmod notices that it also has to unload/deactivate Appl_E, then will load MPI_D for Compiler_A and -finally will notice that there is an equivalent Appl_E module available again, and Lmod will load that one also. However, -now loading Compiler_B will cause a warning that MPI_D and Appl_E have been deactivated as there is no module name MPI_D in -any version for Compiler_B. - - -### Building blocks - -Some mechanisms in Lmod make implementing a hierarchy fairly easy (though there are a lot of hidden pitfalls) - -- The *MODULEPATH* environment variable determines which modules are available. MODULEPATH is different from any other - path-style variable in Lmod in that any change will immediately trigger a re-evaluation of which modules are available - and trigger deactivating modules that are no longer available when a directory is removed from the MODULEPATH or - looking for alternatives for deactivated modules when a directory is added to the MODULEPATH. - -- The *"one name rule"*: Lmod cannot have two modules loaded with the same name (but a different version). By default, when loading - a module with the name of an already loaded module, Lmod will automatically swap the old one with the new one, i.e., unload the - already loaded module and load the new one. - -- The *family* concept: It is possible to declare a module to be part of a family using a command in the module file. No two modules - of the same family can be loaded at the same time, and Lmod will again by default auto-swap the already loaded one with the one - being loaded. The procedure is different though as Lmod now first has to read the new module file to discover the family, and this - may lead to more side effects. But that discussion is outside the scope of this tutorial. - - The family concept was for a long time a unique feature of Lmod, but it has been added now also to Environment Modules version 5.1. - - -### Implementation details - -The above example could be implemented using 8 module files: One for each compiler, three for the MPI modules -(two for MPI_C and one for MPI_D) and -three for the application modules. - -``` -moduleroot -├── Core -│ ├── Compiler_A -│ │ └── version_A.lua -│ └── Compiler_B -│ └── version_B.lua -├── Compiler -│ ├── Compiler_A -│ │ └── version_A -│ │ ├── MPI_C -│ │ │ └── version_C.lua -│ │ └── MPI_D -│ │ └── version_D.lua -│ └── Compiler_B -│ └── version_B -│ └── MPI_C -│ └── version_C.lua -└── MPI - ├── Compiler_A - │ └── version_A - │ ├── MPI_C - │ │ └── version_C - │ │ └── Appl_E - │ │ └── version_E.lua - │ └── MPI_D - │ └── version_D - │ └── Appl_E - │ └── version_E.lua - └── Compiler_B - └── version_B - └── MPI_C - └── version_C - └── Appl_E - └── version_E.lua -``` - -Besides the module functions needed to create the environment needed to run the compiler, the module file for -Compiler_A would need only two lines to implement the hierarchy: - -```Lua -family('Compiler') -prepend_path('MODULEPATH', 'moduleroot/Compiler/Compiler_A/version_A') -``` - -There are now two different ``version_C.lua`` files. One contains the necessary calls to module functions to -initialise the environment to use the version compiled with Compiler_A/version_A while the other contains the -necessary functions to do that for Compiler_B/version_B. Again, two more lines are needed to implement the hierarchy. -E.g., for ``moduleroot/Compiler/Compiler_A/version_A/MPI_C/version_C.lua``: - -```Lua -family('MPI') -prepend_path('MODULEPATH', 'moduleroot/MPI/Compiler_A/version_A/MPI_C/version_C') -``` - -Finally two vesions of the ``version_E.lua`` file are needed, one to prepare the environment for using the -package with Compiler_A anmd MPI_C and one for using the package with Compiler_B and MPI_C. However, these -are just regular modules and no additions are needed to work for the hierarchy. - -Both EasyBuild and Spack support Lmod hierarchies and with these tools it is also fairly automatic to create -different versions of the module files for each compiler and MPI library used to build the application. When -hand-writing modules it may be more interesting to have a generic module which would work for all those cases -and that is also possible with Lmod. Lmod does have a range of *introspection functions* that a module can use -to figure out its name, version and place in the module tree. All that would be needed is that the various -instances of the module file are at the correct location in the module tree and link to the generic file which -can be outside the module tree. In fact, this feature is used on LUMI to implement the modules that load a -particular version of the hardware for a particular section of LUMI. - ---- - -## Finding modules - -In a hierarchical setup, not all modules are available at login. This implies that a user cannot use -``module avail`` to discover which software is available on the system. To this end Lmod has powerful -search commands. It is important to understand how these commands work to ensure that the proper information -is included in the module files to improve discoverability of software. - -!!! Note "Documentation in the LUMI documentation" - Extensive information on search commands with examples of how to use them on LUMI can be found - in the [LUMI documentation]([https://)](https://docs.lumi-supercomputer.eu/), in - [the computing section, "Module environment page", "Finding modules" section](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/#finding-modules). - - -### module spider command - -The available modules at any point in time are often only a subset of all installed modules on a -system. However, Lmod provides the ``module spider`` command to search for a module with a given name -among all installed modules and to tell you how this module can be loaded (i.e., which other modules -need to be loaded to make the module available). - -The ``module spider`` command has three levels, producing different outputs: - - 1. ``module spider`` without further arguments will produce a list of all - installed software and show some basic information about those packages. - Some packages may have an ``(E)`` behind their name and will appear in blue - (in the default colour scheme) which means that they are part of a different - package. These are called *extensions* of packages or modules. - This is explained a little further in this page. - - Note that ``module spider`` will also search in packages that are hidden from - being displayed. These packages can be loaded and used. However administrators - may have decided to hide them - either because they are not useful to regular users or because they think that - they will rarely or never be directly loaded by a user and want to avoid - overloading the module display. - - 2. ``module spider `` will search for the specific package. This - can be the name of a module, but it will also search some other information - that can be included in the modules. The search is also case-insensitive. - E.g., on LUMI - ```bash - module spider GNUplot - ``` - will show something along the lines of - ``` - ------------------------------------------------------------------ - gnuplot: - ------------------------------------------------------------------ - Description: - Gnuplot is a portable command-line driven graphing utility - - Versions: - gnuplot/5.4.2-cpeCray-21.08 - gnuplot/5.4.2-cpeGNU-21.08 - ``` - so even though the capitalisation of the name was wrong, it can tell us that - there are two versions of gnuplot. The ``cpeGNU-21.08`` and ``cpeCray-21.08`` - tell that the difference is the compiler that was used to install gnuplot, - being the GNU compiler (PrgEnv-gnu) and the Cray compiler (PrgEnv-cray) - respectively. - - In some cases, if there is no ambiguity, `module spider` will actually - already produce help about the package, which is the next level. - - 3. `module spider /` will show more help information - about the package, including information on which other modules need to be - loaded to be able to load the package. E.g., - ```bash - module spider git/2.35.1 - ``` - will return something along the lines of - ``` - ------------------------------------------------------------------- - git: git/2.35.1 - ------------------------------------------------------------------- - Description: - Git is a free and open source distributed version control - system - - You will need to load all module(s) on any one of the lines below - before the "git/2.35.1" module is available to load. - - CrayEnv - LUMI/21.12 partition/C - LUMI/21.12 partition/D - LUMI/21.12 partition/G - LUMI/21.12 partition/L - - Help: - ``` - (abbreviated output). Note that it also tells you which other modules need - to be loaded. You need to choose the line which is appropriate for you and - load all modules on that line, not the whole list of in this case 9 - modules. - -!!! failure "Known issue" - The Cray PE uses Lmod in an unconventional manner with the hierarchy not - build fully in the way Lmod expects. As a consequence Lmod is not always - able to generate the correct list of modules that need to be loaded to make - a package available, and the list of ways to make a module available may - also be incomplete. - - The problem is somewhat aggrevated on LUMI because the Cray PE hierarchy sits - next to the hierarchy of the software stack as the Cray PE is installed - separately and hence cannot be integrated in the way the Lmod developer had - in mind. - - -#### Module extensions - -Certain packages, e.g., Python, Perl or R, get a lot of their functionality through -other packages that are installed together with them and extend the functionity, -e.g., NumPy and SciPy for Python. Installing all those packages as separate modules -to make it easy to see if they are installed or not on a system would lead to an -overload of modules on the system. - -Similary, admins of a software stack may chose to bundle several libraries or tools -that are often used together in a single module (and single installation directory), -e.g., to reduce module clutter but also to reduce the length of the search paths for -binaries, libraries or manual pages to speed up loading of applications. - -Lmod offers a way to make those individual packages installed in a module discoverable -by declaring them as *extensions* of the module. The ``module spider`` command will -search for those too. - - 1. ``module spider`` without further arguments: The output may contain lines similar - to - ``` - ----------------------------------------------------------------------- - The following is a list of the modules and extensions currently available: - ----------------------------------------------------------------------- - Autoconf: Autoconf/2.71 (E) - - CMake: CMake/3.21.2 (E), CMake/3.22.2 (E) - ``` - which tells that ``Autoconf`` and ``CMake`` are not available as modules themselves - but as extensions of another module, and it also tells the versions that are available, - though that list may not be complete (and is not always complete for modules either - as it is limited to one line of output). - -2. ``module spider `` will search for extensions also. E.g., - ``` - module spider CMake - ``` - on LUMI will return something along the lines of - ``` - ----------------------------------------------------------------------- - CMake: - ----------------------------------------------------------------------- - Versions: - CMake/3.21.2 (E) - CMake/3.22.2 (E) - ``` - (output abbreviated). - This tells that there is no ``CMake`` module on the system but that two versions - of ``CMake`` are provided in another module. - - 3. `module spider /` will show more information on the - extension, including which module provides the extension and which other modules - have to be loaded to make that module available. E.g., on LUMI, - ``` - module spider CMake/3.22.2 - ``` - will output something along the lines of - ``` - ----------------------------------------------------------------------- - CMake: CMake/3.22.2 (E) - ----------------------------------------------------------------------- - This extension is provided by the following modules. To access the - extension you must load one of the following modules. Note that any - module names in parentheses show the module location in the software - hierarchy. - - buildtools/21.12 (LUMI/21.12 partition/L) - buildtools/21.12 (LUMI/21.12 partition/G) - buildtools/21.12 (LUMI/21.12 partition/D) - buildtools/21.12 (LUMI/21.12 partition/C) - buildtools/21.12 (CrayEnv) - ``` - (output abbreviated and slightly reformatted for readability). This tells that - ``CMake/3.22.2`` is provided by the ``bvuildtools/21.12`` module and that there - are 5 different ways to make that package available. - -??? Bug "Restrictions with older Lmod versions" - At the time of development of this tutorial, Cray is still using the pretty old - 8.3.1 version of Lmod. Even though extensions were supported since Lmod version 8.2.5, - Lmod 8.3.1 has several problems: - - - It is not possible to hide extensions in the output of ``module avail``, a feature - that only became available in version 8.5. This may be annoying to many users as - the extension list of packages such as Python, R and Perl can be very long (the - default EasyBuild installation of R contains on the order of 600 packages). - - For that reason on LUMI extensions are only used for some modules. - - - ``module avail`` also shows extensions for modules that are not available which - makes no sense. This bug was only corrected in Lmod 8.6.13 and 8.6.14. - - -### module keyword - -Another search command that is sometimes useful is `module keyword`. It really -just searches for the given word in the short descriptions that are included in -most module files and in the name of the module. The output is not always -complete since not all modules may have a complete enough short description. - -Consider we are looking for a library or package that supports MP3 audio -encoding. -```bash -module keyword mp3 -``` -will return something along the lines of -``` ----------------------------------------------------------------- - -The following modules match your search criteria: "mp3" ----------------------------------------------------------------- - - LAME: LAME/3.100-cpeCray-21.08, LAME/3.100-cpeGNU-21.08 - LAME is a high quality MPEG Audio Layer III (mp3) encoder -``` -though the output will depend on the version of Lmod. This may not be the most -useful example on a supercomputer, but the library is in fact needed to be able -to install some other packages even though the sound function is not immediately -useful. - -??? bug "Know issue: Irrelevant output" - At the moment of the development of this tutorial, this command actually - returns a lot more output, referring to completely irrelevant extensions. - This is a bug in the HPE-Cray-provided version of Lmod (8.3.1 at the time - of development of this tutorial) that was only solved in more recent versions. - - -### module avail - -The `module avail` command is used to show only available modules, i.e., modules -that can be loaded directly without first loading other modules. It can be used -in two ways: - - 1. Without a further argument it will show an often lengthy list of all - available modules. Some modules will be marked with `(D)` which means that - they are the default module that would be loaded should you load the module - using only its name. - - 2. With the name of a module (or a part of the name) it will show all modules - that match that (part of) a name. E.g., - ```bash - module avail gnuplot - ``` - will show something along the lines of - ``` - ------ EasyBuild managed software for software stack LUMI/21.08 on LUMI-L ------ - gnuplot/5.4.2-cpeCray-21.08 gnuplot/5.4.2-cpeGNU-21.08 (D) - - Where: - D: Default Module - (output abbreviated). - ``` - but - ```bash - module avail gnu - ``` - will show you an often lengthy list that contains all packages with gnu - (case insensitive) in their name or version. - - -### Getting help - -One way to get help on a particular module has already been discussed on this -page: ``module spider /`` will produce help about the package as -soon as it can unambiguously determine the package. It is the only command that -can produce help for all installed packages. The next two commands can only -produce help about available packages. - -A second command is ``module whatis`` with the name or name and version of a -module. It will show the brief description of the module that is included in -most modules on the system. If the full version of the module is not given, it -will display the information for the default version of that module. - -The third command is ``module help``. Without any further argument it will display -some brief help about the module command. However, when used as -``module help `` or ``module help /`` it will produce help for either the -default version of the package (if the version is not specified) or the -indicated version. - - -### Implementation details - -Lmod works by executing the module file. However, the actions of all Lmod-defined -functions will depend upon the mode in which Lmod is executing the module function, -and the module file can also detect in which mode it is executing. -Modes include "load", "unload"but also "spider". E.g., when te mode is "load", the -``setenv`` function will set an environment variable to the indicated value while in -"unload" mode that environment variable will be unset, and in "spider" mode the -environment variable is left untouched. The working of ``prepend_path``, a function -that modifies PATH-style variables, depends a bit on how Lmod is configured (as it is -possible to work with reference counts), but in its most basic mode, ``prepend_path`` -will add a given directory to a given PATH-style environment variable (or move it to -the front of the PATH-style variable if the directory is already in there), while in -"unload" mode that specific directory will be removed from the PATH (but no error will -be generated should the directory that is used as the argument not be part of the path -in that PATH-style variable). When the mode is "spider", the function has special behaviour -if it is used to change the ``MODULEPATH``. It will then note the change and add that -directory to the list of directories that has to be searched for module files. -This makes ``module spider`` a very expensive command as it may have to traverse a lot -of directories and has to execute all module files in there. Therefor Lmod will build -a so-called spider cache which can be pre-built in the system for certain directories -and otherwise will be build in the user's home directory (in the ``.lmod.d/.cache`` -subdirectory). Our experience is that this cache tends to be rather fragile, -in particular on Cray systems (and that has been confirmed in discussions with -people with access to some other Cray systems) so from time to time Lmod fails to -note changes to the modules, at least when using commands such as ``module spider``. -The actual loading and unloading of the module is not based on cached information. - -Lmod has several functions that can be used in module files to provide the information -that Lmod needs for the search-related and help commands. - -The ``help`` function defines the long help text used by ``module help`` and by -``module spider`` as soon as there is no ambiguity anymore about which module is -being searched for. - -The ``whatis`` function is used to provide short information about a module. That -information is then used by ``module whatis`` and ``module keyword`` , but also -for brief information shown by ``module spider`` when multiple modules or versions -of modules are found by the command. A module file can contain multiple ``whatis`` -commands and the Lmod manuel suggests to use those lines as a kind of database -record. See, e.g., -[the Lmod manual page with module file examples](https://lmod.readthedocs.io/en/latest/100_modulefile_examples.html?highlight=whatis). -One such example is -```Lua -whatis("Name: valgrind") -whatis("Version: 3.7.0") -whatis("Category: tools") -whatis("URL: http://www.valgrind.org") -whatis("Description: memory usage tester") -``` -It is not all that important to include all those lines in a module file, but some of -those lines get a special treatment from Lmod. The line starting with ``Description`` -is used by ``module spider`` to provide some brief information about the module if it -is not totally resolved. This comes with a limitation though: It is not show for each -version of the module, so ideally all "GROMACS" modules should contain the same -description line and use other lines to provide further information about what -distinguished a particular version. -Likewise the ``Category:`` line is used by the ``spider_decoration`` hook that can be -used to add decoration to the spider level 1 output. -All in all the ``whatis`` function if often overlooked in Lmod-based module functionx -but it is a very useful function to include in the proper way in module files. - -A third function that provides information to the search commands is ``extensions``. -It can be used to list up the extensions supported by the module. The argument list -may seem strange as it takes only a single argument, a string of comma-separated ``extension/version`` -elements, but that is because the number of arguments to a function is limited in -Lua and that limit can actually be met easily by modules for Python, Perl or R packages. - - ---- - -## Some warnings about writing modulefiles - -***This section is very technical and only useful if you want to manually implement -modules that depend on each other one way or another.*** - -Lmod cannot guarantee that the order of unloading the modules will be the inverse of -the order in which they were loaded. Moreover, unloading a module is not done by reverting -stored actions done when loading the module, but by executing the modulefile again -in a mode that reverts certain actions. This can lead to subtle problems when modulefiles -communicate with each other through environment variables or by detecting which other -modules are loaded. These problems are usually solved by using a proper hierarchy -and basing actions of modulefiles on their position in the hierarchy. - -One case where passing information between modules through environment variables will -go wrong is when that environment variable is subsequently used to compute a directory -name that should be added to a PATH-like variable. Assume we have two versions of -a ``MyPython`` module, e.g., ``MyPython/2.7.18`` and ``MyPython/3.6.10``. That module then -sets an environment variable ``PYTHON_API_VERSION`` to either ``2.7`` or ``3.6``. -Next we have a module ``MyPythonPackage`` that makes a number of Python packages available -for both Python modules. However, as some Python packages have to be installed separately -for each Python version, it does so by adding a directory to the environment variable -``PYTHON_PATH`` that contains the version which it gets by using the Lua function -``os.getenv`` to request the value of ``PYTHON_API_VERSION``. - -One problem becomes clear in the following scenario: -``` bash -module load MyPython/2.7.18 -module load MyPythonPackage/1.0 -module load MyPython/3.6.10 -``` -The ``module load MyPythonPackage`` will find the environment variable ``PYTHON_PACKAGE_API`` -with the value ``2.7`` as set by ``module load MyPython/2.7.18`` and hence add the directory -for the packages for version 2.7 to ``PYTHON_PATH``. The ``module load MyPython/3.6.10`` -command will trigger two operations because of the *"one name rule"*: First it will -automatically unload ``MyPython/2.7.18`` (which will unset ``PYTHON_API_VERSIUON``) and -next it will load ``MyPython/3.6.10`` which will set ``PYTHON_API_VERSION`` to ``3.6``. -However, ``MyPythonPackage`` is not reloaded so the ``PYTHON_PATH`` variable will now point -to the wrong directory. One would be tempted to think that the easy fix for the user would -be to reload ``MyPythonPackage/1.0``: -``` bash -module load MyPythonPackage/1.0 -``` -Because of the *"one name rule"* this will again trigger an unload followed by a load -of the module. The problem is in the unload. One would expect that first unloading -``MyPythonPackage`` would remove the 2.7 directory from the ``PYTHON_PATH`` but it -will not. Lmod does not remeber that last time it loaded ``MyPythonPackage`` it added -the 2.7 directory to ``PythonPath``. Instead it will execute the commands in the -modulefile and reverse certain commands. Since ``PYTHON_API_VERSION`` has now the value -``3.6``, it will try to remove the directory for version ``3.6`` which is not in the -``PYTHON_PATH``. The subsequent load will then add the 3.6 directory to ``PYTHON_PATH`` -so the environment variable now contains both directories. - -In this simple case, a ``module purge`` after the first two ``module load`` commands would -still work as Lmod is able to figure out the right order to unload modules, but in more -complicated examples this may also go wrong. However, a ``module purge`` command after -the load of ``MyPython/3.6.10`` would also fail to clean up the environment as it would -still fail to remove the 2.7 directory from ``PYTHONPATH``. - -??? Note "Running the example" - To test this example for yourself, create a directory and add that directory to - the ``MODULEPATH`` using ``module use``. In that directory, create the following - subdirectories and files: - 1. ``MyPython/2.7.18.lua`` with content: - ``` lua - LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) - setenv( 'PYTHON_API_VERSION', '2.7' ) - ``` - 2. ``MyPython/3.6.10.lua`` with content: - ``` lua - LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) - setenv( 'PYTHON_API_VERSION', '3.6' ) - ``` - 3. ``MyPythonPackage/1.0.lua`` with content: - ``` lua - LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) - LmodMessage( 'PYTHON_API_VERSION = ' .. ( os.getenv( 'PYTHON_API_VERSION' ) or '') ) - prepend_path( 'PYTHON_PATH', 'someroot/python' .. - ( os.getenv( 'PYTHON_API_VERSION' ) or 'TTT' ) .. '/packages' ) - LmodMessage( 'PYTHON_PATH = ' .. ( os.getenv( 'PYTHON_PATH' ) or '') ) - ``` -??? Note "Solution with a hierarchy" - The better way in Lmod to implement the above scenario would be in a module hierarchy. - - Just to show the power of Lmod introspection functions combined with a proper hierarchy - we present a solution using only one version of the code for ``MyPython`` and one version - of the code for ``MyPythonPackages``. - - It is best to start from a clean directory. In that directory, create: - - 1. The files ``level1/MyPython/2.7.18.lua`` and ``level1/MyPython/3.6.10.lua``, - both with the same contents: - ``` lua - LmodMessage( '\nIn ' .. myModuleFullName() .. ' in mode ' .. mode() ) - - local api_version = myModuleVersion():match( '(%d+%.%d+)%..*' ) - - -- Set the variable PYTHON_API_VERSION but not for internal use in the modules. - setenv( 'PYTHON_API_VERSION', api_version ) - - local module_root = myFileName():match( '(.*)/level1/' .. myModuleFullName() ) - prepend_path( 'MODULEPATH', pathJoin( module_root, 'level2/PythonAPI', api_version ) ) - LmodMessage( 'MODULEPATH is now\n ' .. - os.getenv( 'MODULEPATH' ):gsub( ':', '\n ' ) ) - ``` - - 2. The files ``level2/PythonAPI/2.7/MyPythonPackage/1.0.lua`` and - ``level2/PythonAPI/3.6/MyPythonPackage/1.0.lua``, both with the contents: - ``` lua - LmodMessage( '\nIn ' .. myFileName() .. ' in mode ' .. mode() ) - - local python_api_version = myFileName():match( '.*/level2/PythonAPI/([^/]+)/.*' ) - LmodMessage( 'Detected Python API version from hierarchy: ' .. python_api_version ) - LmodMessage( 'Detected Python API version from environment: ' .. - ( os.getenv( 'PYTHON_API_VERSION' ) or '' ) ) - - prepend_path( 'PYTHON_PATH', 'someroot/python' .. python_api_version .. '/packages' ) - - LmodMessage( 'PYTHON_PATH = ' .. (os.getenv( 'PYTHON_PATH' ) or '') ) - ``` - - Now add the ``level1`` subdirectory to ``MODULEPATH``, e.g., if you're in the directory - containing the ``level1`` and ``level2`` subdirectories: - ``` bash - module use $PWD/level1 - ``` - and then try the following commands: - ``` bash - module avail - module load MyPython/2.7.18 - module avail - module load MyPythonPackage/1.0 - module load MyPython/3.6.10 - ``` - and pay attention to the output. - - Initially ``module avail`` will show none of the ``MyPythonPackage`` modules. These are - installed modules but not available modules. ``module load MyPython/2.7.18`` will set the - environment variable ``PYTHON_API_VERSION`` to ``2.7`` and also add a directory to the front - of the ``MODULEPATH`` with the directory name ending on ``level2/PythonAPI/2.7``. Now - ``module avail`` will show the ``MyPythonPackage/1.0`` module. - - The ``MyPythonPackage`` shows two ways to get the version of the Python API to use for - determining the right directory to add to ``PYTHON_PATH``. The fragile way is to enquire - the value of the environment variable ``PYTHON_API_VERSION`` set by loading ``MyPython/2.7.18``. - The more robust way is to use the Lmod introspection function ``myFileName()`` which returns - the full path and file name of the module file that is executing, and extracting the version - from the path with a pattern matching function. In this particular situation both computed - values are the same so both would have worked to correctly add - ``somedir/python2.7/packages`` to the front of ``PYTHON_PATH``. - - The next command, ``module load MyPython/3.6.10`` triggers a chain of events. - - First, Lmod notices that there is already a module loaded with the same name, so it will - unload ``MyPython/2.7.18``. This will unset the environment variable ``PYTHON_API_VERSION`` - (the inverse operation of ``setenv``) and will remove the ``.../level2/PythonAPI/2.7`` - subdirectory from the ``MODULEPATH`` (the inverse action of ``prepend_path``). - - Now due to - the change of the ``MODULEPATH`` the ``MyPythonPackage/1.0`` module which was loaded from - ``.../level2/PythonAPI/2.7`` is no longer available so Lmod will continue with unloading - that module. The interesting bit now is that ``PYTHON_API_VERSION`` is unset. So had we - computed the name of the directory to add to ``PYTHON_PATH`` using the value of that - environment variable, the module would have failed to compute the correct directory name - to remove so ``prepend_path`` would have left the ``PYTHON_PATH`` environment variable - untouched. However, by computing that value from the directory of the modulefile, we get - the right value and can correctly remove ``somedir/python2.7/packages`` from ``PYTHON_PATH``. - Lmod will also remember that the module was only unloaded due to a change in the - ``MODULEPATH`` and not because a user explicitly unloaded the module. I.e., it considers - the module as deactivated but not as unloaded. - - Lmod proceeds with loading the ``MyPython/3.6.10`` module. This will now set - ``PTHON_API_VERSION`` to ``3.6`` and add a directory with name ending on - ``level2/PythonAPI/3.6`` to ``MODULEPATH``. - - Things are not done yet though. As the ``MODULEPATH`` has changed, Lmod looks at its list - of deactivated modules and notices that a different version of ``MyPythonPackage/1.0`` is - now available. Hence it will now automatically load that module from the - ``.../level2/PythonAPI/3.6`` subdirectory so that that module now correctly detects - that ``somedir/python3.6/package`` should be added to ``PYTHON_PATH``. - - Hence at the end of the cycle we have again a correctly configured environment with no - trace of the ``2.7`` version that was loaded initially and with no action required from - the user to ensure that ``MyPythonPackage`` is unloaded and reloaded to ensure the - correct configuration. - - This idea is used on LUMI to implement the various versions of the software stack with - for each software stack also optimised binaries for each of the node types. - - ---- - -## Further reading - -- [Lmod documentation](https://lmod.readthedocs.io/en/latest/index.html) -- [Lmod on LUMI in the LUMI documentation](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/) -- [Documentation of Environment Modules 5](https://modules.readthedocs.io/en/latest/), - an alternative to Lmod (though not currently supported by HPE Cray) - - ---- - -*[[next: The Cray Programming Environment]](1_03_CPE.md)* diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md deleted file mode 100644 index d3a2ff84..00000000 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ /dev/null @@ -1,418 +0,0 @@ -# The HPE Cray Programming Environment - -*[[back: The Lmod module system]](1_02_Lmod.md)* - ---- - -On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated -as Cray PE). The environment provides several tools, including compilers, communication libraries, -optmized math libraries and various other libraries, analyzers and debuggers. - -The Cray PE is made available through *environment modules* tha allow to select particular versions of -tools and to configure the environment in a flexible way. - ---- - -## Cray PE components - -### Cray compiler environments - -Cray PE supports multiple compilers, Cray and third party compilers as well: AOCC, Cray, Intel, GNU. - -Users can access the compilers loading a programming environment module (see below). - -Compilers in the Cray Programming Environment are almost always used through the Cray compiler wrappers. In fact, -until recently, some compilers could not be used without the wrappers as they failed to even detect their own -include files and some components. The behaviour of the wrapper will depend on the other modules that are -loaded in the environment. E.g., there is no separate wrapper for MPI as the MPI header files and libraries are -automatically activated through the regular wrappers as soon as a Cray MPI module is loaded. - -The commands to invoke compiler wrappers are `ftn` (Fortran), `cc` (C), `CC` (C++). They wrap automatically -to the right compilers based on the programming environment module that is loaded and the compiler module. - -The online help can be accessed with the `-help` option. E.g.: `cc -help`, `CC -help`. -One of the most frequently used options of the compiler wrappers is `-craype-verbose`: -``` - ftn -help | grep verbose -A 1 - -craype-verbose Print the command which is forwarded - to compiler invocation -``` -More information is available with the `info` or `man` commands. -E.g.: both `info cc` or `man cc` will display the man page of the C compiler wrapper. - -The compiler wrappers call the correct compiler in the currently loaded programming environment, -with appropriate options to build and link applications with relevant libraries, as required by the modules loaded. -Besides a number of generic options, they also will pass all other options to the underlying compile, -so you can still use the regular compiler options also. - -The compiler wrappers should replace direct calls to compiler drivers in Makefiles and build scripts -to ensure that the proper libraries are used. - -!!! Note - For system libraries, only dynamic linking is supported by compiler wrappers on the Cray EX system - ---- - -#### Cray Compiling Environment (CCE) - -The Cray Compiling Environment is set by the module `PrgEnv-cray`, which is loaded by default at login. -The compiler itself is contained in the `cce` module. - -CCE consists of Cray compilers performing code analysis during compilation to generate highly optimized code. -Supported languages include Fortran, C and C++, and UPC (Unified Parallel C). -The Cray C/C++ compiler is basically the Clang/LLVM compiler with a back-end configured by HPE Cray that also -adds some Cray-specific options while the Cray Fortran compiler uses an HPE-Cray specific front-end -with a back-end based on LLVM. The Fortran compiler supports most of Fortran 2018 and tends to be -considerably stricter than the GNU or Intel Fortran compilers. - -The CCE compilers will also support OpenMP offload to NVIDIA and AMD GPUs but that is still very immature and -very much work in progress at the time of the development of this tutorial. -The Fortran compiler (but not the C/C++ compiler) also supports -OpenACC for offloading to GPU and is being updated to the newest versions of this standard. - -The classic pre-LLVM Cray C/C++ compiler is not available on Cray EX systems. - -Compiler-specific manpages can be accessed on the system with `man crayftn`, `man craycc` or `man crayCC`. - -More details are given in the -[Cray Fortran Reference Manual](https://support.hpe.com/connect/s/search?language=en_US#q=Cray%20Fortran%20Reference%20manual&t=All&sort=relevancy&numberOfResults=25) - and the - [Cray Compiling Environment Release](https://support.hpe.com/connect/s/search?language=en_US#q=Cray%20Compiling%20Environment%20Release&t=All&sort=relevancy&numberOfResults=25) -that unfortunately are hidden deep in the new [support pages of HPE](https://support.hpe.com/). -The [Clang Compiler User’s Manual](https://clang.llvm.org/docs/UsersManual.html) -is another source of information for the Cray C and C++ Clang compilers. - -For more information about compiler pragmas and directives, see `man intro_directives` on the system. - ---- - -### Third-Party Compilers - -#### GNU - -The GNU C/C++ and Fortran compilers are probably the best supported third party -compiler in the Cray PE. - -Compiler-specific manpages can be accessed on the system with `man gfortran`, `man gcc` or `man g++`. - -More details are provided by the [GCC online documentation](https://gcc.gnu.org/onlinedocs). - - -#### AOCC - -There AOCC compilers, the AMD Optimizing C/C++ Compiler and matching fortran compilers, -AMD's compiler offering for CPU-only systems, have a matching programming environment module -and are a full citizen of the Cray PE. - -Compiler-specific documentation is available in the [AOCC User Guide](https://developer.amd.com/amd-aocc/#userguide). - -Cray provides a bundled package of support libraries to install into the PE environment to enable AOCC, -and Cray PE utilities such as debuggers and performance tools work with AOCC. - - -#### AMD ROCm compilers - -The AMD ROCm compilers (in the AMD world sometimes known as AOMP) are supported on systems -with AMD CPUs. However, at the time of writing of this tutorial they are not yet available in the -LUMI environment and the integration with the Cray environment still seems to be work in progress. - - -#### Intel (not on LUMI) - -The Cray PE also provides a programming environment module to enable -the Intel® oneAPI compiler and tools. - -The documentation is available in the [Intel® oneAPI Programming Guide](https://software.intel.com/content/www/us/en/develop/documentation/oneapi-programming-guide/top/oneapi-development-environment-setup.html) - -Cray provides a bundled package of support libraries to install into the Cray PE to enable the Intel compiler, allowing -utilities such as debuggers and performance tools to work with it. - - -#### NVIDIA HPC toolkit (not on LUMI) - -The NVIDIA HPC Toolkit compilers (formerly PGI) are supported on systems with NVIDIA GPUs. - - -### Cray Scientific and Math Library - -* Manpages: `intro_libsci`, `intro_fftw3` - -The Cray Scientific and Math Libraries (CSML, also known as LibSci) are a collection of numerical routines -optimized for best performance on Cray systems. - -These libraries satisfy dependencies for many commonly used applications on Cray systems for a wide variety of domains. - -When the module for a CSML package (such as `cray-libsci` or `cray-fftw`) is loaded, -all relevant headers and libraries for these packages are added to the compile -and link lines of the `cc`, `ftn`, and `CC` compiler wrappers, so linking with them is -completely transparant (to the extent that users wonder where the libraries are). - -The CSML collection contains the following Scientific Libraries: - -* BLAS (Basic Linear Algebra Subroutines) -* BLACS (Basic Linear Algebra Communication Subprograms) -* CBLAS (Collection of wrappers providing a C interface to the Fortran BLAS library) -* IRT (Iterative Refinement Toolkit) -* LAPACK (Linear Algebra Routines) -* LAPACKE (C interfaces to LAPACK Routines) -* ScaLAPACK (Scalable LAPACK) -* `libsci_acc` (library of Cray-optimized BLAS, LAPACK, and ScaLAPACK routines) -* HDF5 -* NetCDF (Network Common Data Format) -* FFTW3 (the Fastest Fourier Transforms in the West, release 3) - - -### Cray Message Passing Toolkit - -* Manpages: `intro_mpi` - -* Website: [http://www.mpi-forum.org](http://www.mpi-forum.org) - -MPI is a widely used parallel programming model that establishes a practical, portable, efficient, -and flexible standard for passing messages between ranks in parallel processes. - -Cray MPI is derived from Argonne National Laboratory MPICH and implements the MPI-3.1 standard -as documented by the MPI Forum in MPI: A Message Passing Interface Standard, Version 3.1. - -Support for MPI varies depending on system hardware. To see which functions and environment variables the -system supports, please have a look at the corresponding man pages with `man intro_mpi` on the system. - -Note that though on LUMI at the time of the tutorial both the OpenFabric Interface (OFI) based and UCX-based -versions of the library are supported, the finalised LUMI system will only support OFI. - - -### DSMML - -* Modules: `cray-dsmml` -* -* Manpages: `intro_dsmml` -* -* Website: [https://pe-cray.github.io/cray-dsmml](https://pe-cray.github.io/cray-dsmml) - -Distributed Symmetric Memory Management Library (DSMML) is a HPE Cray proprietary memory management library. - -DSMML is a standalone memory management library for maintaining distributed shared symmetric memory -heaps for top-level PGAS languages and libraries like Coarray Fortran, UPC, and OpenSHMEM. - -DSMML allows user libraries to create multiple symmetric heaps and share information with other libraries. - -Through DSMML, interoperability can be extracted between PGAS programming models. - -Further details are available in the man page on the system with `man intro_dsmml`. - - -### Other components - -- **Cray Performance and Analysis Tools**: - - Tools to analyze the performance and behavior of programs that are run on Cray systems, and a Performance API (PAPI). - -- **Cray Debugging Support Tools**: - - Debugging tools, including `gdb4hpc` and `Valgrind4hpc`. - - ---- - -## Configuring the Cray PE through modules - -Multiple releases of the HPE Cray PE can be installed simultaneously on the system and users can mix-and-match -components. However, it is also possible to load only components for a specific release of the PE. -Cray PE versions have version numbers of the form ``yy.mm``, e.g., ``22.02`` for the version released in -February of 2022. However, each of the components have their own version number and it is not easy to see -which version of a component came with which version(s) of the Cray PE. - -Below we only discuss those modules that are important when building software with EasyBuild. Debuggers, -profilers, etc., are not included in the list. - - -### The programming environment modules - -The usual way to initialise the Cray PE is by loading one of the ``PrgEnv-*`` modules. This module will then -load the appropriate compiler, compiler wrappers and other libraries. Some of the components that will be -loaded are configured through the ``/etc/cray-pe.d/cray-pe-configuration.sh`` file, so the list of modules -may depend on the actual system that you are using. - -The Cray PE supports the following ``PrgEnv-*`` modules. On LUMI, only the first three are currently available: - -| Module | Compiler module | What? | -|:---------------|-----------------|:--------------------------------------------| -| PrgEnv-cray | cce | The Cray Compiling Environment compilers | -| PrgEnv-gnu | gcc | The GNU compilers | -| PrgEnv-aocc | aocc | AMD compilers for CPU-only systems | -| PrgEnv-amd | rocm | AMD ROCm compilers for GPU systems | -| PrgEnv-intel | intel | The Intel compilers | -| PrgEnv-nvidia | nvidia | NVIDIA HPC toolkit compilers (formerly PGI) | - -All ``PrgEnv-*`` modules belong to the same module family. Hence Lmod will automatically unload any -already loaded ``PrgEnv-*`` module when you load a different one. - - -### Selecting the version through the cpe meta-module - -The Cray PE on the EX system provides the meta-module `cpe`: the purpose of the meta-module is -similar to the scope of the `cdt` and `cdt-cuda` meta-modules available on the XC systems. - -Loading one of the ``cpe/yy.mm`` modules (e.g., ``cpe/22.02``) has the following effects: - -- It sets the default versions of each of the Cray PE modules to the version that comes with that - particular release of the Cray PE. E.g., - - ``` bash - module load cpe/22.02 - module load cce - ``` - - would load that version of the ``cce`` compiler that comes with the 22.02 release of the Cray PE. - -- It will reload all already loaded Cray PE modules and switch them over to the version corresponding - to that particular release of the Cray PE. - -???+ Bug "Limitations and bugs" - Due to the way Lmod works and implementation bugs in the ``cpe`` modules, loading the ``cpe`` module - does not always have the desired effect. - - - The Cray PE sets the default version of each module by adding a file to the list of files in the - ``LMOD_MODULERCFILE`` environment variable. This is because Lmod does not re-evaluate the visibility - of modules and the internal list of default version immediately when the value of ``LMOD_MODULERCFILE`` - is changed but only the next time the ``module`` command is executed. Hence - ``` bash - module load cpe/22.02 ; module load cce - ``` - and - ``` - module load cpe/22.02 cce - ``` - do not have the same effect. In the first version, the version of ``cce`` loaded is the version that - corresponds to the 22.02 release of the Cray PE. In the second case however the default version of the - ``cce`` module is determined by whatever list of default modules was used when calling the ``module`` - command so may or may not be the one of the 22.02 release. - - - Loading the ``cpe`` module after loading the other Cray PE modules also does not always have the - desired effect in many versions of the Cray PE. This is because of a bug in the ``cpe `` module that - reloads the modules in the wrong order which may trigger the reload of a module with whatever version - was the default when the ``module`` command was called rather than the version the the ``cpe`` module - intends to (re-)load. - - -### The compiler wrapper module craype - -The ``craype`` module loads the compiler wrappers. There is only one set of compiler wrappers for all compilers. -Which compiler will be called, which libraries will be included, but also processor and GPU target options will -be used, is all determined by other modules. Hence it is in principle possible to use a single Makefile for -a project and still reconfigure the build by loading certain modules. - - -### Target modules - -The targets for CPU and GPU optimization, the network library for MPI bt also some other compiler options, -can be set through target modules: - -- ``craype-x86-*`` (and similar options can be expected on ARM-based systems) set the target for CPU - optimisations. For LUMI, the ``crape-x86-rome``, ``craype-x86-milan`` and ``craype-x86-trento`` modules - are relevant. - - This can also be used to cross-compile to a different target architecture unless the compiler target - gets overwritten by a compiler flag added to the command line through the Makefile or other means, - something that unfortunately happens more and more often in faulty software installation procedures. - -- ``creype-accel-*`` sets the target for OpenMP offload (and likely other technologies in the future). E.g., - loading ``craype-accel-amd-gfx90a`` tells the compilers to target AMD MI200 family GPUs, while loading - ``craype-accel-host`` tells the compiler to use the CPU instead (according to the documentation, the latter - is for ``PrgEnv-cray`` only). - -- ``craype-network-*`` selects the communication library to be used by Cray MPICH. On Slingshot 11 EX systems, - only ``craype-network-ofi`` is supported, but Slingshot 10 EX systems also offer support for UCX through the - ``craype-network-ucx`` module. - -- The ``craype-hugepages*`` modules enable Cray Huge Pages support. To fully enable this support they have to - be used at link-time and at run-time. At link time, support is compiled into the binary wile at run-time they - are used to set the actual size of the huge pages. - - The ``craype-hugepages*`` modules are not supported by all compilers. E.g., the AOCC compiler does not support - huge pages at the moment, and loading the module at link time will cause an error message from the linker. - - -### The compiler modules - -The compiler modules have already been discussed with the ``PrgEnv-*`` modules above. The different regular -compiler modules also all belong to the same family so no two different compilers can be loaded simultaneously -and Lmod will automatically unload the other compiler when a new one is loaded. - - -### The MPI modules - -To load the Cray MPI libraries, both one of the ``craype-network-*`` modules and a compiler module has to be -loaded as the MPI libraries are both network- and compiler specific. - -For some unknown reason, the MPI module for the libfabric (``craype-network-ofi``) transport is called ``cray-mpich`` -while the library for the UCX transport (``craype-network-ucx``) is called ``craype-network-ucx``. As a result, -the MPI module fails to reload automatically when switching between both transports, but it does reload automatically -when switching compilers. - -Loading an MPI module will also automatically configure the regular compiler wrappers to compile with support for -that MPI module. However, the libfabric and UCX versions of the MPI library have compatible interfaces, so it is -always possible to swap between those versions at runtime. - - -### The Cray Scientific libraries - -The Cray Scientific Libraries are loaded through the ``cray-libsci`` module (or ``cray-libsci_acc`` for the GPU -versions). Loading this module makes the BLAS, LAPACK, and ScaLAPACK libraries available, and also the Cray IRT -(Iterative Refinement Toolkit), a Cray-specific library. It will also configure the compiler wrappers to link -with these libraries, so no additional include or link options are needed. - - -### The ``fftw`` module - -As FFTW is a third party library, it gets its own module (``cray-fftw``) and is not included in the ``cray-libsci`` module. - -The ``cray-fftw`` module can only be loaded if one of the processor target modules (the ``craype-x86-*`` modules) -is loaded first. - - ---- - -## Some unexpected behaviour of the Cray PE - -On Cray EX systems, dynamic linking is the preferred way of linking applications. Whith that comes some -unexpected behaviour of the Cray modules. EasyBuild users expect that at run time the versions of the -libraries that are used, are the ones from the modules that are loaded. This is not always the case -for the runtime libraries of the Cray PE. By default the Cray PE will use a default set of libraries -that is determined by the system default version of the Cray PE (which is set by the sysadmins, not -determined by the ``cpe`` module). This can only be avoided by either using\ -rpath-linking (which is also special in the Cray PE as the wrappers activate rpath linking if the -environment variable ``CRAY_ADD_RPATH`` is defined and set to ``yes``) or by manually adjusting -the library path after loading the modules: -``` bash -export LD_LIBRARY_PATH=${CRAY_LD_LIBRARY_PATH}:$LD_LIBRARY_PATH -``` -The latter cannot be easily automated in modulefiles. Any technique that can be used (without actually -reworking the Cray PE modules) has nasty side effects in some scenarios. - -The net result of this feature is that some reproducibility of the results is lost. Programs will -react differently if the system default version of the Cray PE is changed as that will change the -set of default libraries that will be used at runtime unless rpath-linking is used or users -redfine ``LD_LIBRARY_PATH``. - - ---- - -## Further reading - -- [LUMI documentation: "Developing" section](https://docs.lumi-supercomputer.eu/development/) -- The Cray PE is mostly documented through man pages. There used to be some documentation on the - Cray web site also but the documentation system got reworked after the merger with HPE. - The documentation is now in the [HPE Support Centre](https://support.hpe.com/) where it - is very difficult to find the right version of the documents. -- The [PE-Cray](https://github.com/PE-Cray) GitHub project also provides some additional - documentation, including - - [some whitepapers](https://github.com/PE-Cray/whitepapers), - - [documentation for Cray OpenSMHEMX](https://github.com/PE-Cray/cray-openshmemx) and - - [information about Cray DSMML](https://github.com/PE-Cray/cray-dsmml) - though much of that information ia also not updated regularly anymore. - - ---- - -*[[next: LUMI software stacks]](1_04_LUMI_software_stack.md)* diff --git a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md deleted file mode 100644 index 3bfae70c..00000000 --- a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md +++ /dev/null @@ -1,101 +0,0 @@ -# LUMI software stacks (technical) - -*[[back: The Cray Programming Environment]](1_03_CPE.md)* - ---- - -The user-facing documentation on how to use the LUMI software stacks is -avialable in [the LUMI documentation](https://docs.lumi-supercomputer.eu/computing/softwarestacks/). -On this page we focus more on the technical implementation behind it. - ---- - -# An overview of LUMI - -LUMI has different node types providing compute resources: - -- LUMI has 16 login nodes, though many of those are reserved for special purposes and not - available to all users. TODO - - ---- - -## CrayEnv and LUMI modules - -On LUMI, two types of software stacks are currently offered: - - - ``CrayEnv`` (module name) offers the Cray PE and enables one to use - it completely in the way intended by HPE-Cray. The environment also offers a - limited selection of additional tools, often in updated versions compared to - what SUSE Linux, the basis of the Cray Linux environment, offers. Those tools - are installed and managed via EasyBuild. However, EasyBuild is not available - in that partition. - - It also rectifies a problem caused by the fact that there is only one - configuration file for the Cray PE on LUMI, so that starting a login shell - will not produce an optimal set of target modules for all node types. - The ``CrayEnv`` module recognizes on which node type it is running and - (re-)loading it will trigger a reload of the recommended set of target - modules for that node. - - - ``LUMI`` is an extensible software stack that is mostly managed through - [EasyBuild][easybuild]. Each version of the LUMI software stack is based on - the version of the Cray Programming Environment with the same version - number. - - A deliberate choice was made to only offer a limited number of software - packages in the globally installed stack as the setup of redundancy on LUMI - makes it difficult to update the stack in a way that is guaranteed to not - affect running jobs and as a large central stack is also hard to manage. - However, the EasyBuild setup is such that users can easily install - additional software in their home or project directory using EasyBuild build - recipes that we provide or they develop, and that software will fully - integrate in the central stack (even the corresponding modules will be made - available automatically). - - Each ``LUMI`` module will also automatically activate a set of application - modules tuned to the architecture on which the module load is executed. To - that purpose, the ``LUMI`` module will automatically load the ``partition`` - module that is the best fit for the node. After loading a version of the - ``LUMI`` module, users can always load a different version of the ``partition`` - module. - -Note that the ``partition`` modules are only used by the ``LUMI`` module. In the -``CrayEnv`` environment, users should overwrite the configuration by loading their -set of target modules after loading the ``CrayEnv`` module. - - ---- - -## The ``partition`` module - -The ``LUMI`` module currently supports four partition modules, but that number may -be reduced in the future: - -| Partition | CPU target | Accelerator | -|:----------------|-----------------------|:----------------------------| -| ``partition/L`` | ``craype-x86-rome`` | ``craype-accel-host`` | -| ``partition/C`` | ``craype-x86-milan`` | ``craype-accel-host`` | -| ``partition/G`` | ``craype-x86-trento`` | ``craype-accel-amd-gfx90a`` | -| ``partition/D`` | ``craype-x86-rome`` | ``craype-accel-nvidia80`` | - -All ``partition`` modules also load `craype-network-ofi``. - -``pattition/D`` may be dropped in the future as it seems we have no working CUDA setup -and can only use the GPU nodes in the LUMI-D partition for visualisation and not with CUDA. - -Furthermore if it would turn out that there is no advantage in optimizing for Milan -specifically, or that there are no problems at all in running Milan binaries on Rome -generation CPUs, ``partition/L`` and ``partition/C`` might also be united in a single -partition. - - - - - - - ---- - -*[[next: Terminology]](1_05_terminology.md)* - diff --git a/docs/2022-CSC_and_LO/1_05_terminology.md b/docs/2022-CSC_and_LO/1_05_terminology.md deleted file mode 100644 index 8e05012f..00000000 --- a/docs/2022-CSC_and_LO/1_05_terminology.md +++ /dev/null @@ -1,294 +0,0 @@ -# Terminology - -*[[back: LUMI software stacks]](1_04_LUMI_software_stack.md)* - ---- - -Over the years, we have come up with some terminology specific to EasyBuild -to refer to particular components, which we use alongside established terminology relevant to the context -of building and installing software. - -It is important to be familiar with these terms, so we'll briefly cover them one by one. - ---- - -## Toolchains - -A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, -which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. - -We refer to the different parts of a toolchain as **toolchain components**. - -The *compiler component* typically consists of [C](https://en.wikipedia.org/wiki/C_(programming_language)), -[C++](https://en.wikipedia.org/wiki/C%2B%2B), and [Fortran](https://en.wikipedia.org/wiki/Fortran) -compilers in the context of HPC, but additional compilers (for example, -a [CUDA](https://developer.nvidia.com/cuda-zone) compiler for -[GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units) software) -can also be included. - -Additional toolchain components are usually special-purpose libraries: - -- an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/) - or vendor-specific MPI implementations like Intel MPI in a toolchain based on the Intel compilers); - -- libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), - [LAPACK](http://performance.netlib.org/lapack/), [ScaLAPACK](http://www.netlib.org/scalapack/)); - - These libraries are included because they tend to provide a common API, so mixing them in a single - executable would cause name conflicts when linking. - -- a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); - -These libraries are included because they are some of the most popular libraries in scientific software, -but also because they provide an API that is available through different implementations. Mixing -implementations in a single build is a recipe for problems as the various implementations might -define the same symbols. - -A toolchain that includes all of these libraries is referred to as a **full toolchain**, while -a **subtoolchain** is a toolchain that is missing one or more of these libraries. -A **compiler-only toolchain** only consists of compilers (no additional libraries). - -The toolchain concept also maps nicely on the Programming Environment concept in the Cray -Programming Environment. Each ``PrgEnv-*`` module in fact provides a full toolchain in a -typical Cray PE installation, except for the FFTW library. Note that systems could configure -the contents of a ``PrgEnv-*`` module differently as the MPI library and scientific library are -optional. In the Cray PE, the MPI library is provided by the ``cray-mpic`` module, -the BLAS, LAPACK and ScaLAPACK libraries by the ``cray-libsci`` module (for CPU-only nodes) -and the FFTW library through the ``cray-fftw`` module. - -Many EasBuild toolchains are organised in a hierarchy. Each toolchain can use components compiled -with itself or with one of the subtoolchains. The Cray toolchains however are not currently -organised in a hierarchy. - - -### System toolchain - -The **`system` toolchain** is a special case which corresponds to using the compilers and libraries -*provided by the operating system*, rather than using toolchain components that were installed using EasyBuild. - -It used sparingly, mostly to install software where no actual compilation is done or -to build a set of toolchain compilers and its dependencies, since the versions of the system compilers -and libraries are beyond the control of EasyBuild, which could affect the reproducibility of the installation. - -On LUMI however it is used a bit more and it takes some of the functions of the ``GCCcore`` toolchain in -other EasyBuild toolchain hierarchies. - - -### Common toolchains - -The `foss` and `intel` toolchains are also known as the *common toolchains*, -because they are widely adopted by the EasyBuild community. - -The `foss` toolchain consists of all open source components (hence the name: -"FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), -[ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). -In recent versions (since 2021a), [FlexiBLAS](https://www.mpi-magdeburg.mpg.de/projects/flexiblas) -is used as the BLAS library, with OpenBLAS and LAPACK as the backend. - -The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version -controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. - -Roughly every 6 months, a new version of these common toolchains is agreed upon -in the EasyBuild community, after extensive testing. - -The common toolchains are also organised in a toolchain hierarchy: - -```mermaid -graph TD; -A[SYSTEM] --> B[GCCcore]; -B --> C[GCC]; -B --> D[iccifort]; -C --> E[gompi: Adds OpenMPI]; -D --> F[iimpi: Adds Intel MPI]; -D --> G[imkl: Adds Intel MKL]; -E --> H[foss: Adds OpenBLAS, LAPACK, ScaLAPACK, FFTW]; -F --> I[intel]; -G --> I; -``` - -More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). - - ---- - -## EasyBuild framework - -EasyBuild is written in Python and organised in three layers, represented by the three blocks in the logo: -the EasyBuild framework, easyblocks and Easyconfig files. - -The EasyBuild *framework* consists of a set of Python modules organised in packages (``easybuild.framework``, -``easybuild.toolchains``, ``easybuild.tools``, etc.) that collectively form **the core of EasyBuild**, -and is developed in the [``easybuild-framework`` repository on GitHub](https://github.com/easybuilders/easybuild-framework). - -It implements the **common functionality that you need when building software from source**, -providing functions for unpacking source files, applying patch files, collecting the output produced -by shell commands that are being run and checking their exit code, generating environment module files, etc. - -The EasyBuild framework does *not* implement any specific installation procedure, it only provides -the necessary functionality to facilitate this. - - ---- - -## Easyblocks - -An *easyblock* is **a Python module that implements a specific software installation procedure**, -and can be viewed as a plugin to the EasyBuild framework. Easyblocks can be either *generic* or *software-specific*. - -A **generic easyblock** implements an installation procedure that can be used for -multiple different software packages. Commonly used examples include the ``ConfigureMake`` easyblock -which implements the ubiquitous ``configure``-``make``-``make install`` procedure, and the -``PythonPackage`` easyblock that can be used to install a Python package. - -*In Spack this functionality is provided in the core of the package.* - -A **software-specific easyblock** implements an installation procedure that is specific to a particular software packages. -Infamous examples include the easyblocks we have for ``GCC``, ``OpenFOAM``, ``TensorFlow``, ``WRF``, ... - -The installation procedure performed by an easyblock can be controlled by defining -**easyconfig parameters** in an easyconfig file. - -*In Spack, this functionality is provided by the ``package.py`` files for the various packages. However, EasyBuild -does not need a software-specific easyblock for every package as for many packages the combination of a -generic easyblock with the much simpler description in the easyconfig file is sufficient. -The software-specific easyblocks can be a bit troublesome on Cray PE systems. Some easyblocks contain code that -is specific for certain compilers or toolchains and do not support all compilers or all toolchains.* - -A collection of (generic and software-specific) easyblocks is developed by the EasyBuild community -in the [``easybuild-easyblocks`` repository on GitHub](https://github.com/easybuilders/easybuild-easyblocks). - ---- - -## Easyconfig parameters - -An **easyconfig parameter** specifies a particular aspect of a software installation that should be performed by -EasyBuild. - -Some easyconfig parameters are **mandatory**. The following parameters *must* be defined in *every* easyconfig file: - -* ``name`` and ``version``, which specify the name and version of the software to install; -* ``homepage`` and ``description``, which provide key metadata for the software; -* ``toolchain``, which specifies the compiler toolchain to use to install the software (see - ``toolchains`` tab); - -Other easyconfig parameters are **optional**: they can be used to provide required information, -or to control specific aspects of the installation procedure performed by the easyblock. - -Some commonly used optional easyconfig parameters include: - -* ``easyblock``, which specifies which (generic) easyblock should be used for the installation; -* ``sources`` and ``source_urls``, which specify the list of source files and where to download them; -* ``dependencies`` and ``builddependencies``, which specify the list of (build) dependencies; -* ``configopts``, ``buildopts``, and ``installopts``, which specify options for the configuration/build/install commands, respectively; - -If no value is specified for an optional easyconfig parameter, the corresponding default value will be used. - -There are two groups of easyconfig parameters. *General* easyconfig parameters can be defined for any software -package, and (usually) control a specific aspect of the installation. *Custom* easyconfig parameters are -only supported by certain easyblocks, and only make sense for particular (types of) software. - ---- - -## Easyconfig files - -*Easyconfig files* (or *easyconfigs* for short), are **simple text files written in Python syntax -that specify what EasyBuild should install**. -Each easyconfig file defines the set of **easyconfig parameters** that collectively form a complete specification -for a particular software installation. - -The **filename** of an easyconfig file usually ends with the ``.eb`` extension. -In some contexts the filename is expected to be determined by the value of a handful of key -easyconfig parameters: ``name``, ``version``, ``toolchain`` and ``versionsuffix``. The general format for -the filename of an easyconfig file is: ``--.eb``, -where the toolchain part is omitted when a ``system`` toolchain is used, and the `` can be empty. - -The filename of easyconfig files is particularly relevant when EasyBuild is searching for easyconfig files to resolve dependencies, -since it does this purely based on filenames: interpreting the contents of every (potential) easyconfig file -it encounters would be too expensive. - -In the [``easybuild-easyconfigs`` repository on GitHub](https://github.com/easybuilders/easybuild-easyconfigs), -the EasyBuild community maintains a large (and growing) collection of easyconfig files, for a wide range of -(scientific) software. - -*EasyBuild does not have an equivalent of the concretiser of Spack. EasyBuild always starts from a complete, -precise configuration of a package to install, determined by the easyconfig parameters and possibly a -custom easyblock. Both approaches have their pros and cons. The concretiser in Spack combined with the -configuration of a software package via command line parameters makes it easy to quickly generate a -number of different configurations of a package to test performance or to check which combinations work -well. But one also has to be careful to not cause an explosion of modules and software installations, and -take care of using the appropriate features to ensure that one can generate the same configurations again. -The precise configurations of EayBuild can sometimes slow down things as they may need a lot of file editing, -but when it comes to building a reproducible software stack and sharing precise building instructions, this -approach may be preferred.* - ---- - -## Easystack files - -[**Easystack files**](https://docs.easybuild.io/en/latest/Easystack-files.html) -are a new concept in EasyBuild, providing a way to define a *software stack* -that should be installed by EasyBuild. - -They are written in [YAML syntax](https://yaml.org/), and include a list of *software specifications* -which correspond to a list of easyconfig files, with support for providing specific EasyBuild -configuration options for particular software packages, and including or excluding specific software -packages based on labels. - -The support for using easystack files is currently marked as *experimental*, -which means it is subject to change in future EasyBuild releases, and may be prone to errors. - ---- - -## Extensions - -*Extensions* is the collective term we use for **additional software packages that can be installed -on top of another software package**. Common examples are *Python packages*, *R libraries*, and *Perl modules*. - -As you can tell the common terminology here is a bit messy, so we came up with a unifying term... - -Extensions can be installed in different ways: - -* *stand-alone*, as a separate installation on top of one or more other installations; -* as a part of a *bundle* of extensions that collectively form a separate installation; -* or as an actual *extension* to a specific installation to yield a "batteries included" - type of installation (for examples by adding a bunch of Python packages from PyPI into - a Python installation); - ---- - -## Dependencies - -A *dependency* is a common term in the context of software. It refers to **a software -package that is either strictly required by other software, or that can be leveraged to -enhance other software** (for example to support specific features). - -There are three main types of dependencies for computer software: - -* a **build dependency** is only required when building/installing a software package; - once the software package is installed, it is no longer needed to *use* that software - (examples: `CMake`, `pkg-config`); -* a **link-time dependency** is somewhere in between a build and runtime dependency: - it is only needed when *linking* a software package; it can become either a build or runtime - dependency, depending on exactly how the software is installed (example: `OpenBLAS`); -* a **run-time dependency** (often referred to simply as *dependency*) is a software package that is - required to *use* (or *run*) another software package (example: `Python`); - -EasyBuild currently doesn't treat link-time dependencies differently from run-time dependencies -and hence has no specific easyconfig parameter for them. - ---- - -## Bringing it all together - -The EasyBuild **framework** leverages **easyblocks** to automatically build and install -(scientific) software, potentially including additional **extensions**, using a particular compiler **toolchain**, -as specified in **easyconfig files** which each define a set of **easyconfig parameters**. - -EasyBuild ensures that the specified **(build) dependencies** are in place, -and automatically generates a set of **(environment) modules** that facilitate access to the installed software. - -An **easystack file** can be used to specify a collection of software to install with EasyBuild. - ---- - -*[[next: Installation]](1_06_installation.md)* diff --git a/docs/2022-CSC_and_LO/1_06_installation.md b/docs/2022-CSC_and_LO/1_06_installation.md deleted file mode 100644 index 24c71d5e..00000000 --- a/docs/2022-CSC_and_LO/1_06_installation.md +++ /dev/null @@ -1,545 +0,0 @@ -# Installing EasyBuild - -*[[back: Terminology]](1_05_terminology.md)* - ---- - -Before deep diving into the wonderful world of EasyBuild and getting your hands dirty with -the hands on exercises coming up in this tutorial, you will need to install EasyBuild. - -In this section we outline a couple of different ways of doing this, and also the things you should pay attention to. -By the end, you will have a functional EasyBuild installation that you can use -for the remainder of this tutorial. - -!!! Remark "EasyBuild on LUMI" - - EasyBuild is already installed for each of the LUMI software stacks. The LUMI software - stacks are based on the Cray PE (using the Cray-specific toolchains). - Moreover, in case you want to install a full shadow stack rather than build on top - of one of the LUMI software stacks, scripts are provided to initialise a new version - of the LUMI software stack based on a specific version of the Cray PE, and those - scripts also take care of the installation of EasyBuild. - - Hence this section of the tutorial is only useful for people who want more background - knowledge or who want to experiment with other EasyBuild toolchains. - - -!!! summary - - * *Requirements*: Linux, Python 2.7 or 3.5+, environment modules tool (Lmod is recommended). - * *Installation methods*: - * ``pip install easybuild`` - * installing EasyBuild with EasyBuild - * development setup - * Verify EasyBuild installation using `eb --version` and `eb --help`. - - ---- - -## Requirements - -### Linux - -The main target platform for EasyBuild is *Linux*, since it is heavily focused on installing software -on HPC system where Linux is dominant operating system (to the point where 100% of the current Top500 list of supercomputers are running Linux). - -EasyBuild is also compatible with *macOS*, but the included easyconfig files are heavily focused -on Linux so most software installations supported by EasyBuild won't work out-of-the-box on macOS. -You can still use the EasyBuild command line interface on macOS for other tasks though, -like development or contributing back to the project. - -### Python - -EasyBuild is implemented in Python, and is **compatible with both Python 2.7 and Python 3.5+** -(that is, Python 3.5 or a newer version of Python 3). - -To check which Python version you have, use: - -```shell -python -V -``` - -*No additional Python packages are required by EasyBuild*, the ones that come with the standard -Python distribution are sufficient. Some additional Python packages *can* be leveraged for specific features. -More on that later. - -### Environment modules tool - -**An *environment modules tool* is required for using EasyBuild.** - -We strongly recommend using [Lmod](https://lmod.readthedocs.io), a Lua-based modern environment -modules implementation and the most commonly used modules tool in the EasyBuild community. -Other implementations, like the original Tcl-based one, are also supported. - -To check if you have a modules tool installed, use: - -``` -module --version -``` - -If this produces output that starts with something like "`Modules based on Lua: Version 8.3.1`" you have Lmod installed, -which is the default modules tool used by EasyBuild, and you are all set for installing and using EasyBuild. -Any sufficiently recent Lmod version (8.x or even 7.x) should be fine. - -If you see output that starts with a line like "`VERSION=3.2.10`" or "`Modules Release 4.5.0`", -you have the original Tcl-based environment modules tool installed, either in its original -C-based imnplementation (``VERSION=3.2.10``) or the new Tcl-based implementation developed at CEA -(``Modules Release 4.5.0``) -and *EasyBuild will need to be configured to use it after installation*. - -If the `module` function is not defined either you do not have a modules tool installed -or your environment is not properly set up to use it. In this case, -please refer to the EasyBuild documentation here for more information. - -HPE Cray supports both the old-style Environments Modules (based on version 3.2.10) and Lmod. On LUMI, -we use Lmod as the default module system for users, and EasyBuild is configured to use that one. - ---- - -## EasyBuild as a Python package - -EasyBuild consists of a number of interdependent Python packages, -and is available via both GitHub at https://github.com/easybuilders, -as well as via the standard Python Package Index (PyPI) at -https://pypi.org/project/easybuild. - -As you may be aware the Python packaging ecosystem is bit convoluted, -which is reflected in the many different ways in which you can install a Python package. -In addition, EasyBuild is packaged in 3 components (framework, easyblocks, easyconfigs) -which slightly complicates the installation. - -
- -Nevertheless, you don't need to be a rocket scientist to install EasyBuild (and even if you are, -that's OK too), so don't worry. - -You can install EasyBuild just like you can install any other Python software that is released -via the standard *Python Package Index* (PyPI), through one of the standard Python installation tools -(like `pip`, `virtualenv`, `pipenv`, ...). -And since EasyBuild is a software installation tool in its own right, we actually have a couple -of additional tricks up our sleeve! - -### Python 2 or Python 3? - -For EasyBuild it does not matter much whether you install it on top of Python 2 or Python 3. -Since version 4.5, Python 3 does offer some optional features (requiring additional packages) -that are not available with Python 2. -Since Python 2 is end-of-life, -we strongly recommend using Python 3 if you have the choice. - -By default EasyBuild will use the `python` command to run, -but you can control this if needed via ``$EB_PYTHON``. - - -## Installing EasyBuild - -We present three methods for installing EasyBuild. -It is up to you which one you prefer, both result a fully functional EasyBuild installation. - -Time to get your hands dirty! - -* Method 1: Using ``pip`` -* Method 2: installing EasyBuild with EasyBuild -* Method 3: Development setup - ---- - -### Method 1: Using `pip` - -Since EasyBuild is released as a [Python package on PyPI](https://pypi.org/project/easybuild) -you can install it using `pip`, the most commonly used tool for installing Python packages. - -You may need to take additional steps after the installation, depending on the exact installation command. - -!!! note - There are various other ways of installing Python packages, which we won't cover here. - If you are familiar with other tools like `virtualenv` or `pipenv`, feel free to use those - instead to install EasyBuild. - -#### Running `pip install` - -Installing EasyBuild with `pip` is as simple as running the following command: - -```shell -pip install easybuild -``` - -However, you may need to slightly change this command depending on the context and your personal preferences: - -* To install EasyBuild *system-wide*, you can use `sudo` (if you have admin privileges): - ```shell - sudo pip install easybuild - ``` - -* To install EasyBuild *in your personal home directory*, you can use the `--user` option: - ```shell - pip install --user easybuild - ``` - This will result in an EasyBuild installation in `$HOME/.local/`. - -* To install EasyBuild in a *specific directory* you can use the `--prefix` option: - ```shell - pip install --prefix _PREFIX_ easybuild - ``` - In this command, you should replace '`_PREFIX_`' with the location where you want to have EasyBuild installed - (for example, `$HOME/tools` or `/tmp/$USER`). - -#### `pip` vs `pip3` - -On systems where both Python 2 and Python 3 are installed you may also have different `pip` commands -available. Or maybe `pip` is not available at all, and only "versioned" `pip` commands like `pip3` are -available. - -If you (only) have `pip3` available, you can replace `pip` with `pip3` in any of the `pip install` commands -above. - -If you want to ensure that you are using the ``pip`` installation that corresponds to the Python 3 installation -that you intend to use, you can use ``python3 -m pip`` rather than ``pip3``. - -#### Updating your environment - -If you used the `--user` or `--prefix` option in the `pip install` command, -or if you installed EasyBuild with a `pip` version that does not correspond -to your default Python installation, -you will need to update your environment to make EasyBuild ready for use. -This is not required if you did a system-wide installation in a standard location with the default Python version. - -!!! note - Keep in mind that you will have to make these environment changes again if you start a new shell session. - To avoid this, you can update one of the shell startup scripts in your home directory (`.bashrc` for example). - -#### Updating ``$PATH`` - -Update the `$PATH` environment variable to make sure the `eb` command is available: -```shell - -export PATH=_PREFIX_/bin:$PATH -``` -**Replace '`_PREFIX_`' in this command** with the directory path where EasyBuild was installed into -(use `$HOME/.local` if you used `pip install --user`). - -This is not required if you installing EasyBuild in a standard system location. - -You can check with the `which eb` command to determine whether or not you need to update the ``$PATH`` environment variable. - -#### Updating ``$PYTHONPATH`` - -If you installed EasyBuild to a non-standard location using `pip install --prefix`, -you also need to update the Python search path environment variable -[`$PYTHONPATH`](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH) to instruct Python where -it can find the EasyBuild Python packages. - -This is not required if you used the `--user` option, since Python will automatically consider -`$HOME/.local` when searching for installed Python packages, or if you installed EasyBuild in a standard -system-wide location. - -Update `$PYTHONPATH` by running a command like: - -```shell - -export PYTHONPATH=_PREFIX_/lib/pythonX.Y/site-packages:$PYTHONPATH -``` - -Here, you need to replace the `X` and `Y` with the major and minor version of your Python installation, -which you can determine by running `python -V`. -For example, if you are using Python 3.6, make sure you are using `/python3.6/` in the command to update `$PYTHONPATH`. - -And of course, you again need to **replace '`_PREFIX_`'** with the installation prefix where EasyBuild was installed -into. - -For example: - -```shell -# update $PYTHONPATH if EasyBuild was installed in $HOME/tools with Python 3.6 -export PYTHONPATH=$HOME/tools/lib/python3.6/site-packages:$PYTHONPATH -``` - -#### Setting ``$EB_PYTHON`` - -If you want to control which Python version is used to run EasyBuild, -you can specify the name or the full path to the `python` command that should be used by the `eb` command -via the `$EB_PYTHON` environment variable. - -This may be required when you installing EasyBuild with a version of `pip` that does not correspond -with the default Python version. - -For example, to ensure that `eb` uses `python3.6`: - -```shell -export EB_PYTHON=python3.6 -``` - -#### Setting ``$EB_VERBOSE`` - -To determine which `python` commands are being considered by the `eb` command, -you can define the `$EB_VERBOSE` environment variable. For example: - -```shell -$ EB_VERBOSE=1 eb --version ->> Considering 'python3.6'... ->> 'python3' version: 3.6.8, which matches Python 3 version requirement (>= 3.5) ->> Selected Python command: python3 (/usr/bin/python3.6) ->> python3.6 -m easybuild.main --version -This is EasyBuild 4.3.3 (framework: 4.3.3, easyblocks: 4.3.3) on host example -``` - -!!! Tip "Debugging startup problems" - - Using `ÈB_VERBOSE` is useful if EasyBuild fails to start up and complains it - cannot find a suitable Python executable. Rather often the error message is - caused by a failure elsewhere in EasyBuild. - - ---- - -### Method 2: Installing EasyBuild with EasyBuild - -!!! note - This section covers an alternative method for installing EasyBuild. - - If you already have EasyBuild installed through ``pip``, - you can skip ahead to the next section. - -If you prefer having EasyBuild available through an environment module file, -you can consider installing EasyBuild with EasyBuild. This can be done in 3 steps: - -* Step 1: Installing EasyBuild with ``pip`` into a temporary location (only needed if EasyBuild is not installed yet) -* Step 2: Using EasyBuild to install EasyBuild as a module -* Step 3: Loading the EasyBuild module - -!!! note - A [bootstrap script](https://docs.easybuild.io/en/latest/Installation.html#bootstrapping-easybuild) - is available that automates this procedure, - but is known to be problematic in some contexts, and is not being actively - maintained anymore. - - As a result, we do not recommend using the bootstrap script anymore. - - -#### Step 1: Installing EasyBuild into a temporary location - -If you don't have EasyBuild installed yet, you need to install it in a temporary location first. -The recommended way of doing this is using ``pip``. - -For example, to install EasyBuild into a subdirectory `/tmp/$USER` using the default Python 3 version: - -```shell -# pick installation prefix, and install EasyBuild into it -export EB_TMPDIR=/tmp/$USER/eb_tmp -python3 -m pip install --ignore-installed --prefix $EB_TMPDIR easybuild -``` - -```shell -# update environment to use this temporary EasyBuild installation -export PATH=$EB_TMPDIR/bin:$PATH -export PYTHONPATH=$(/bin/ls -rtd -1 $EB_TMPDIR/lib*/python*/site-packages | tail -1):$PYTHONPATH -export EB_PYTHON=python3 -``` - -#### Step 2: Using EasyBuild to install EasyBuild - -Once you have a working (recent) temporary EasyBuild installation, you can use it to -install EasyBuild as a module. Usually this is done in the location where you would -like to install other software too. - -You can use the ``eb --install-latest-eb-release`` command for this, -combined with the ``--prefix`` option to control which directories are used by EasyBuild for the installation. - -For example, to install the latest version of EasyBuild as a module into ``$HOME/easybuild``: - -```shell -eb --install-latest-eb-release --prefix $HOME/easybuild -``` - -!!! note - You may see a harmless deprecation warning popping up when performing this installation, just ignore it. - -#### Step 3: Loading the EasyBuild module - -Once step 2 is completed, you should be able to load the module that was generated alongside -the EasyBuild installation. You will need to do this every time you start a new shell session. - -First, make the module available by running the following command (which will update the module search path -environment variable `$MODULEPATH`): - -```shell - -module use _PREFIX_/modules/all -``` - -**Replace '`_PREFIX_`'** with the path to the directory that you used when running step 2 -(for example, ``$HOME/easybuild``). - -Then, load the `EasyBuild` module to update your environment and make EasyBuild available for use: - -```shell - -module load EasyBuild -``` - -Note that in this case, we don't need to make any changes to our environment for EasyBuild to work correctly. -The environment module file that was generated by EasyBuild specifies all changes that need to be made. - -### Method 3: Development setup - -If you are planning to make changes to EasyBuild, or if you prefer using the latest *bleeding edge* -version of EasyBuild that is being developed, you can consider *cloning* the 3 main EasyBuild repositories -from GitHub, and updating your environment to run EasyBuild from there. - -This can be done as follows (into ``$HOME/easybuild``): - -```shell -mkdir -p $HOME/easybuild -cd $HOME/easybuild -``` - -```shell -# clone EasyBuild repositories from GitHub -git clone https://github.com/easybuilders/easybuild-framework.git -git clone https://github.com/easybuilders/easybuild-easyblocks.git -git clone https://github.com/easybuilders/easybuild-easyconfigs.git -``` -```shell -# update environment for running EasyBuild from there -export PATH=$HOME/easybuild/easybuild-framework:$PATH -export PYTHONPATH=$HOME/easybuild/easybuild-framework:$HOME/easybuild/easybuild-easyblocks:$HOME/easybuild/easybuild-easyconfigs:$PYTHONPATH - -# control which Python command is used to run EasyBuild -export EB_PYTHON=python3 -``` - - -### Approach on LUMI - -*Documentation on the inner workings of the LUMI software stack can be found in the -[LUMI-SoftwareStack GitHub, docs subdirectory](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/docs).* - -To keep the different versions of the LUMI software stack as independent from one another as possible, -EasyBuild is bootstrapped for each software stack by the ``prepare_LUMI_stack.sh`` script. We use the -system Python 3 for running EasyBuild. That Python version currently doesn't have ``pip`` installed. -EasyBuild is first installed in a temporary work directory from files downloaded from PyPi. These -are each untarred and then installed into their location by running -``python3 setup.py install --prefix=...`` in their unpacked directory. -We do so only for the framework and easyblocks files as the easconfig files are not used to -install EasyBuild. Instead we create our own EasyConfig file for EasyBuild which contains -some additional packages that enable extra features in EasyBuild and also provide more information -to Lmod. Next the configuration module for EasyBuild (see the next section, -["Configuring EasyBuild"](1_07_configuration), for more information) and use the temporary -installation of EasyBuild with our own EasyConfig file to do a proper installation of EasyBuild -with module in the final location. - - -TODO: follow development setup but with downloading rather than cloning, then use that one with the -EasyBuild configuration module to install EasyBuild with itself. - - - -## Verifying the installation - -Regardless of how EasyBuild was installed, you can now run a couple of basic commands to verify the installation: - -#### Checking the version - -To check which EasyBuild version you have installed, run: - -```shell - -eb --version -``` - -The output should match with the latest EasyBuild version. - - -#### Consulting the help output - -You can consult the help output of the `eb` command, which produces a long list of available options -along with a short informative message. - -```shell -eb --help -``` - -#### Showing the default EasyBuild configuration - -To inspect the current EasyBuild configuration, you can use this command: - -```shell -eb --show-config -``` - -This should tell you that EasyBuild (ab)uses `$HOME/.local/easybuild` as a default location. -More on configuring EasyBuild in the [next part of the tutorial](configuration). - -#### System information - -You ask EasyBuild to collect and print some information about the -system you are using it on (OS, CPU, Python, etc.) using this command: - -```shell -eb --show-system-info -``` - -## Updating EasyBuild - -Before we wrap up here, a brief word about updating EasyBuild. - -Once you have EasyBuild installed, the easiest way to update to a newer version is by instructing EasyBuild -to install the latest available version as a module: - -``` - -eb --install-latest-eb-release -``` - -This will result in a *new* EasyBuild installation, which is entirely separate from the EasyBuild installation -you are currently using (so it is *not* an in-place update). -The location where this new EasyBuild version will be installed is determined by the active -EasyBuild configuration. - -If you have installed EasyBuild through ``pip``, and you prefer updating that installation, -you can use ``pip install --upgrade easybuild`` (perhaps with additional options like ``--user`` or ``--prefix``). - -!!! Note "EasyBuild upgrade policy on LUMI" - - Even minor version or patch level updates of EasyBuild may break things (and this has happned - in the past on other systems managed previously by LUMI User Support Team members). Hence a very - conservative upgrade policy is used on LUMI. - - In general we fix the version of EasyBuild within a particular version of the LUMI software stack - and only transition to the latest version when starting a new software stack. The reason is that we - want to be able to rebuild an existing software stack with as little unexpected problems as possible. - A move to a newer version of EasyBuild is acceptable should we run into major problems with an existing - version that cannot be solved in an easyconfig file or LUMI-specific custom easyblock, but in those - cases we will first rebuild the whole software stack on a test setup to ensure that no problems are - introduced by upgrading EasyBuild. - - ---- - -## Exercise (optional) - -*This exercise is not needed to proceed the course when working on LUMI as we will -use the installation that is already present.* - -Install EasyBuild in your home directory. - -Make sure that the EasyBuild installation uses the `python3` command to run, -rather than the standard `python` command. - -Choose your own adventure, or try all these installation methods! - -* install EasyBuild with `pip` (or another very similar command...) using either the `--user` or `--prefix` option; -* perform a (manual) "bootstrap" installation into `$HOME/easybuild`, as outlined in installation method 2; -* set up a development installation; - -Check that the installation works by running the verification commands outlined -[above](#verifying-the-installation). - ---- - -**You'll need a working custom EasyBuild installation for some of the -optional generic EasyBuild exercises, but in general you can use the -installation provided by LUMI.** - - -*[[next: Configuring EasyBuild]](1_07_configuration.md)* diff --git a/docs/2022-CSC_and_LO/1_07_configuration.md b/docs/2022-CSC_and_LO/1_07_configuration.md deleted file mode 100644 index a1c2f3de..00000000 --- a/docs/2022-CSC_and_LO/1_07_configuration.md +++ /dev/null @@ -1,610 +0,0 @@ -# Configuring EasyBuild - -*[[back: Installation]](1_06_installation.md)* - ---- - -After installing EasyBuild, you should **configure** it. - -EasyBuild should work fine out-of-the-box if you use Lmod as your modules tool. -If you are not using Lmod, please see here for more information. - -Nevertheless, we strongly recommend you to inspect the default configuration, -and to configure EasyBuild according to your preferences and the system on which you will use it. - -!!! Note "EasyBuild configuration on LUMI" - - On LUMI serveral configurations of EasyBuild are already available. - - The ``EasyBuild-user`` module is the most important one. It will configure EasyBuild - to install software in either a default location in the user's home directory - (``$HOME/EasyBuild``) or the location pointed to by the environment variable - ``EBU_USER_PREFIX``. The latter can be used to install software in the project directory - so that it is available to all users of the project. It enables the creation of a custom - software environment build on top of the LUMI software stack. Software in this environment is - build in exactly the same way as it would if it were installed in the central repository, but - one can now also easily create custom configurations without confusing other projects with - different or non-standard configurations of software. - - There are two more admin-only configuration modules for EasyBuild. The main one is - ``EasyBuild-production`` which is used for software installations in the central software stack. - The other one is ``EasyBuild-infrastructure`` which is used to install infrastructure modules - that need to be installed in a pure Lmod hierarchy, e.g., the modules that load the toolchains. - - ---- - -## Available configuration settings - -One of the central policies in the EasyBuild project is to avoid hardcoded settings in the codebase. -While this significantly increases the ability to configure EasyBuild to your liking, it also results -in a large amount of available configuration settings. - -The full list of configuration settings can be consulted via `eb --help`, -which shows the corresponding command line option accompanied by a short description. -At the time of writing, *over 240 different configuration settings* are supported by EasyBuild. - -For the sake of this tutorial we will focus on a specific subset of configuration settings, -and cover only the most prominent and important ones. - -We will refer to EasyBuild configuration settings using the names as they appears in the output of `eb --help`, -and omit the leading dashes (`--`) for the sake of clarity. - -Keep in mind that *every* configuration setting can be defined in 3 different ways, -see below for more details. - ---- - -### Overall prefix - -*(default: `HOME/.local/easybuild`)* - -The `prefix` configuration setting specifies the overall prefix that EasyBuild should use, -which **determines the default value for various other configuration settings**: - -* `installpath`: `` -* `buildpath`: `/build` -* `sourcepath`: `/sources` -* `repositorypath` (easyconfigs archive): `/ebfiles_repo` -* `containerpath`: `/containers` - -Here, `` represents the value of the `prefix` configuration setting. - -If one of the configuration settings affected by `prefix` is defined specifically, -the `prefix` value becomes irrelevant for that specific configuration setting. - ---- - -### Install path - -*(default: ``)* - -The **location for both the software installation directories and generated module files** -can be controlled via the `installpath` configuration setting. -Software installation directories will be placed in `/software`, while -`/modules/all` will be used for generated module files. - -The `installpath` location is usually set to a directory on a *shared filesystem* when installing -software for an HPC cluster. Of course, software can also be installed on a local filesystem, -which is particularly useful to test and evaluate software installations. - -Separate configuration settings are available for both software and modules locations, -as well as for controlling the name of the `software` and `modules/all` subdirectories. - -*The EasyBuild community recommends to only change the `installpath` configuration setting to control the location -of software installations and accompanying module files, -such that the software and modules directories are located in the same parent directory, -and the default `software` and `modules/all` names for the subdirectories are used.* - -!!! Note "Path for software and for modules on LUMI" - - On LUMI we do not follow that recommendation. There is a three for the software installations - themselves with subdirectories based on the version of the software stack and LUMI hardware - partition, and a separate tree for the modules organised in a similar way. - - This makes it slightly easier to organise the module tree with user-friendly labeling, but above - all also makes the synchronisation process of the 4 instances of the software directory more robust - as it is now easy to synchonise all modules in the last step, which is a much quicker process than - syncrhonising the software installations. - - We also use short paths for software installations (to avoid overrunning the maximum length of a - shebang line in scripts) while we use longer, more descriptive names for subdirectories in the - module tree. - - ---- - -### Build path - -*(default: `/build`)* - -For each installation it performs, EasyBuild creates a **separate build directory** where software will be compiled -before installing it. This directory is cleaned up automatically when the installation is successfully completed. -To control the location where these build directories are created, you can use the `buildpath` configuration setting. - -Keep in mind that build directories may grow out to several GBs in size during an installation, -and that the commands that run in there can be fairly I/O-intensive since they may involve -manipulating lots of small files. In addition, a build directory that corresponds to a failing installation -is *not* cleaned up automatically, but it will be cleaned up and recycled when the same installation is re-attempted. -Running out of disk space in the location where build directories are created will result in failing -installations. - -It is strongly recommend to use the path to a directory on a *local filesystem* for the value of the -`buildpath` configuration setting, since using a shared filesystem like Lustre or GPFS is known to cause -problems when building certain software packages. Using an in-memory location (like `/dev/shm/$USER`) can -significantly speed up the build process, but may also lead to problems (due to space limitations, -or specific mount options like `noexec`). - -!!! Note "buildpath on LUMI" - - The configuration modules on LUMI will use a RAM disk for the build path. On the login nodes, - ``$XDG_RUNTIME_DIR`` is used as that space is automatically cleared when the last session of a user - ends. However, on the compute nodes a job- or user-specific subdirectory of ``/dev/shm`` is currently used - as ``$XDG_RUNTIME_DIR`` does not exist. - - ---- - -### Source path - -*(default: `/sources`)* - -For most supported software, EasyBuild can **automatically download the source files** required for the installation. -Before trying to download a source file, EasyBuild will first check if it is already present in the source path. - -The locations considered by EasyBuild when checking for available source files, as well as the location to -store downloaded source files, can be controlled via the ``sourcepath`` configuration setting. - -The `sourcepath` value is a colon (`:`) separated list of directory paths. -Each of these paths will be considered in turn when checking for available source files, -until one of them provides the desired source file. Searching for source files is done -based on filename, and a couple of subdirectories are considered. -For example, for a software package named '`Example'`, EasyBuild will consider locations -like `/e/Example/`, `/Example/`, and so on. - -The first path listed in `sourcepath` is the location where EasyBuild will store downloaded source files, -organised by software name through subdirectories, so EasyBuild expects to have write permissions to this path. -For the other paths listed in `sourcepath` only read permissions are required. - -*Make sure you have write permissions to the first path listed in `sourcepath`, so EasyBuild is able -to store downloaded files there. Feel free to list additional paths if you already have a cache of downloaded -files available somewhere.* - -*Storing the downloaded files not only reduces the amount of downloads while developing new easyconfig files -but also greatly helps when recompiling a software stack, as it is not that uncommon that download sites change -of files become unavailable.* - - ---- - -### Easyconfigs archive - -*(default: `/ebfiles_repo`)* - -EasyBuild keeps track of the easyconfig files that were used for installations in the easyconfigs -archive, the location of which is specified by the `repositorypath` configuration setting. - -By default the specified path is assumed to be a regular directory, but using a Git repository -as easyconfigs archive is also supported (for more details, see -[the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Configuration.html#easyconfigs-repository-repository-repositorypath)). - - ---- - -### Modules tool & module syntax - -*(default: `Lmod` as modules tool, and `Lua` as module syntax)* - -By default, EasyBuild assumes you are using **[Lmod](https://lmod.readthedocs.io) as modules tool**. -In addition, it will generate module files in [Lua](https://www.lua.org/) syntax, as supported by Lmod -(next to `Tcl` syntax). - -To diverge from this, you can define the `modules-tool` configuration setting to indicate you -are using a different modules tool; see the output of `eb --avail-modules-tools` for a list of supported -modules tools. -Note that for anything other than Lmod, you *must* make -sure that the actual modules tool binary command is available through `$PATH` (more information -on this [in the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Configuration.html#modules-tool-modules-tool)). - -If you prefer that EasyBuild generates module files in [Tcl](https://tcl.tk/) syntax, you can instruct -it to do so via the `module-syntax` configuration setting. Note that this is *required* if you are using -a modules tool other than Lmod, since only Lmod supports module files in `Lua` syntax. - -*We recommend using EasyBuild with the defaults settings: Lmod as modules tool, and Lua as module syntax.* - ---- - -### Robot search path - -*(default: `robot-paths` specifies the location to the easyconfig files included with EasyBuild installation; -dependency resolution is* ***not*** *enabled)* - -When EasyBuild needs to locate one or more easyconfig files, it will do so via the **robot search path**. -This applies to both easyconfig files that were specified using only their filename as an argument to the -`eb` command, as well as to easyconfigs required to resolve dependencies (more on that later). - -To control the robot search path, two configuration settings are available: `robot` and `robot-paths`. -Both accept a colon-separated list of locations to consider when looking for easyconfig files, -with `robot` having a higher priority than `robot-paths` (i.e., the directories specified with `robot` -appear before those specified by `robot-paths` in the robot search path). - -The key difference between these two configuration settings is that defining `robot` also enables dependency resolution, -and hence makes EasyBuild install missing dependencies, alongside -specifying a list of paths to consider when searching for easyconfig files. -On the other hand, defining `robot-paths` does not have the side effect of enabling dependency resolution. - -In addition, you can use the `--robot` command line option without specifying any paths to it, -to only enable dependency resolution. ***This option is needed on LUMI if you want automatic -dependency resolution as it is disabled on LUMI to prevent accidental mistakes when installing -software.*** - -!!! note "Easyconfig files included with EasyBuild and the robot search path" - Keep in mind that when either of the `robot` or `robot-paths` configuration settings are - defined, the default value corresponding to the location of easyconfigs included with the - EasyBuild is *no longer considered*. - - There are ways around this however, which are outside of the scope of this tutorial. - - For more information, - [see the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Using_the_EasyBuild_command_line.html#controlling-robot-search-path). - - *On LUMI the included easyconfig files cannot be used without changes as they are for the - common toolchains which are currently not supported in the LUMI software stacks.* - - -!!! Note "Easyconfigs archive on LUMI and the robot search path" - - On LUMI we also include the easyconfigs archive at the front of the search path for easyconfig - files. This has the advantage that EasyBuild when checking dependencies will always find the - configuration that is actually installed on the system, even if the easyconfig file in one of - the regular repositories has changed. It also has the disadvantage that one may accidentally - be re-installing with the already installed configuration while one really wants to re-install - the module in a different configuration. - - - ---- - -### Module naming scheme - -*(default: `EasyBuildMNS`)* - -EasyBuild will use a specific naming scheme for the module files it generates for each of the software installations. -This ensures consistency, regardless of who uses EasyBuild to perform the installation. - -Different types of module naming schemes are supported (flat, hierarchical, ...) and you can provide an implementation of -your own custom module naming scheme if desired. A number of different naming schemes are included -with EasyBuild, which you can consult via `eb --avail-module-naming-schemes`. - -The default `EasyBuildMNS` module naming scheme roughly corresponds to the filename of easyconfig files, -and consists of the software name followed by a combination of the software version, toolchain and -an optional label (which corresponds to the value of the `versionsuffix` easyconfig parameter): -`/<-toolchain>`. Just like with [names of easyconfig -files](1_05_terminology#easyconfig-files), the `<-toolchain>` part is omitted when the `system` toolchain is used, -and the `` value is empty by default. - -!!! Note "Module naming scheme on LUMI" - - On LUMI we use a custom variant of the standard EasyBuild flat naming scheme. The EasyBuild default - naming scheme creates an ``all`` subdirectory in the module directory containing all modules, but also - a directory per category, with the categories determined from the ``moduleclass`` easyconfig parameter. - However, that choice is often rather arbitrary as modules can belong to different categories depending - on who you ask. E.g., GROMACS is used in physics, chemistry and life sciences while EasyBuild can only - assign a single category. - - Hence we do not generate the categories and drop the ``all`` level in the module subdirectory. - - ---- - -## Configuration levels - -Configuring EasyBuild can be done in different ways: - -* through one or more configuration files; -* via `$EASYBUILD_*` environment variables; -* using `eb` command line options; - -Each of the methods corresponds to a *configuration level*. - -*Every* configuration setting can be defined via one of these -mechanisms, without exception! - ---- - -### Configuration level hierarchy - -There is a strict **hierarchy between the different configuration levels** supported by EasyBuild. - -Settings defined via a *configuration file* only override default values. - -A configuration setting that is defined via the corresponding *`$EASYBUILD_*` environment variable* -takes precedence over the value specified in a configuration file (if any). - -Finally, values specified through `eb` command line options **always** win, -regardless of whether the corresponding -configuration setting was already defined some other way, be it via a configuration file or -an environment variable. - -For example, let us consider a fictional configuration setting named `magic`: - -* If a value for `magic` is specified in an EasyBuild configuration file, - then this value will only be used if the corresponding environment variable - (`$EASYBUILD_MAGIC`) is *not* defined **and** if the `--magic` command line option is *not* used; -* If the `$EASYBUILD_MAGIC` environment *is* defined however, then its value - will be used for the `this-is-magic` configuration setting; -* *Unless* the `--magic` command line option is used, - since in that case the value provided as an argument there will be used instead. - ---- - -### Configuration files - -*Configuration files* are the most basic way of configuring EasyBuild. -Two types of are supported by EasyBuild: *user-level* and *system-level* -configuration files. The output of `eb --show-default-configfiles` tells you which locations -are considered, and whether any configuration files were found. - -EasyBuild configuration files are written in the standard [INI format](https://en.wikipedia.org/wiki/INI_file), -and the configuration settings are grouped into different *sections*. - -To create an EasyBuild configuration file, the output of `eb --confighelp` is very useful: -it produces the list of all supported configuration settings which are readily grouped in sections -and with every entry commented out, along with accompanying descriptive comments mentioning the default values, -and in the correct syntax. - -*Configuration files are the recommended mechanism to define configuration settings -that should always be in place, regardless of the software you are installing.* - -#### EasyBuild configuration files vs easyconfig files - -EasyBuild configuration files are sometimes confused with easyconfig files, -due to the similar name. However, these are two entirely different concepts! - -EasyBuild configuration files (usually `*.cfg`) are a way of *configuring the general behaviour of EasyBuild* -across different software installations. They define *configuration settings*, -such as the location where software should be installed, or the syntax that should -be used when generating module files. - -An [easyconfig file](1_05_terminology#easyconfig-files) (`*.eb`) on the other hand *specifies the details for one particular software installation*. -It does this by defining a set of *easyconfig parameters*, which tell EasyBuild the name and version -of the software to install, which [toolchain](1_05_terminology#toolchains) and [easyblock](1_05_terminology#easyblocks) to use, etc. - -For each software installation performed by EasyBuild, there is a corresponding easyconfig file. -There typically are only a handful of configuration files used however, for example a system-level -configuration file, perhaps combined with a user-level one. Or there may be no configuration files involved -at all, since EasyBuild can also be configured through other mechanisms: environment variables and command line -options. - ---- - -### `$EASYBUILD_*` environment variables - -A particularly easy way to configure EasyBuild is through *environment variables*. - -At startup, EasyBuild will pick up any environment variable of which the name starts with '`EASYBUILD_`'. -For each of these, it will determine the corresponding configuration setting (or exit with an error if -none was found). - -Mapping the name of a configuration setting to the name of the corresponding environment variable is straightforward: -use capital letters, replace dashes (`-`) with underscores (`_`), and prefix with `EASYBUILD_`. - -For example: the `module-syntax` configuration setting can be specified by defining -the `$EASYBUILD_MODULE_SYNTAX` environment variable: - -```shell -export EASYBUILD_MODULE_SYNTAX=Tcl -``` - -*Configuring via environment variables is especially practical for controlling the EasyBuild configuration -in a more dynamic way. For example, you can implement a simple shell script that defines `$EASYBUILD_*` -environment variables based on the current context (user, hostname, other environment variables), and -configure EasyBuild through [sourcing](https://bash.cyberciti.biz/guide/Source_command) it.* - -!!! note - Keep in mind that environment variables are only defined for the shell session you are currently working in. - If you want to configure EasyBuild through environment variables in a more persistent way, - you can leverage one of the [shell startup scripts](https://bash.cyberciti.biz/guide/Startup_scripts) (for example `$HOME/.bash_profile` or `$HOME/.bashrc`). - ---- - -### `eb` command line options - -Finally, you can also configure EasyBuild by specifying one or options to the `eb` command. - -As mentioned earlier, the values for configuration settings defined this way override the value that -is specified through any other means. So if you want to be sure that a particular configuration setting -is defined the way you want it to be, you can use the corresponding command line option. - -There are various configuration settings for which it only makes sense to use the command line option. -An example of this is letting the `eb` command print the EasyBuild version (via `eb --version`). -Although you could configure EasyBuild to always print its version and then exit whenever the `eb` command is -run, that would not be very useful... - -*Command line options are typically used to define configuration settings that are only relevant to -that particular EasyBuild session. One example is doing a test installation into a temporary directory:* - -```shell -eb --installpath /tmp/$USER example.eb -``` - -## Inspecting the current configuration (`--show-config`) - -Given the large amount of available configuration settings in EasyBuild and the different configuration levels, -you can easily lose track of exactly how EasyBuild is configured. - -Through the `--show-config` command line option you can -easily inspect the currently active EasyBuild configuration. - -The output of `--show-config` includes a sorted list of all configuration settings that are defined to a -*non-default* value, -along with a couple of important ones that are always shown (like `buildpath`, `installpath`, `sourcepath`, and so on). -In addition, it also indicates at which configuration level each setting was defined, -so you can trace down *where* it was defined if needed. - -This is the output produced by `eb --show-config` for the default EasyBuild configuration, -where EasyBuild was installed via `pip install --user` (which results in the value shown for the -`robot-paths` configuration setting): - -```shell -# -# Current EasyBuild configuration -# (C: command line argument, D: default value, E: environment variable, F: configuration file) -# -buildpath (D) = /home/example/.local/easybuild/build -containerpath (D) = /home/example/.local/easybuild/containers -installpath (D) = /home/example/.local/easybuild -repositorypath (D) = /home/example/.local/easybuild/ebfiles_repo -robot-paths (D) = /home/example/.local/easybuild/easyconfigs -sourcepath (D) = /home/example/.local/easybuild/sources -``` - -As shown here, all configuration settings shown follow the default `prefix` value (`$HOME/.local/easybuild`), -and none of the values diverge from the default value, since all entries are marked with `(D)` for "default value"). - -**Example** - -Now let us do some basic configuring and inspect the resulting output of `--show-config`. - -First, create a user-level EasyBuild configuration file to define the `prefix` configuration setting: - -```shell -mkdir -p $HOME/.config/easybuild -echo '[config]' > $HOME/.config/easybuild/config.cfg -echo 'prefix=/apps' >> $HOME/.config/easybuild/config.cfg -``` - -In addition, define the `buildpath` configuration setting using the corresponding -environment variable: - -```shell -export EASYBUILD_BUILDPATH=/tmp/$USER -``` - -Then run `--show-config` while you specify that the `installpath` configuration -setting should be defined as `/tmp/$USER`: - -```shell -$ eb --installpath=/tmp/$USER --show-config -# -# Current EasyBuild configuration -# (C: command line argument, D: default value, E: environment variable, F: configuration file) -# -buildpath (E) = /tmp/easybuild -containerpath (F) = /apps/containers -installpath (C) = /tmp/easybuild -packagepath (F) = /apps/packages -prefix (F) = /apps -repositorypath (F) = /apps/ebfiles_repo -robot-paths (D) = /home/example/.local/easybuild/easyconfigs -sourcepath (F) = /apps/sources -``` - -The output indicates that the `installpath` setting was specified through a command line option (indicated -with `(C)`), that the `buildpath` setting was defined via an environment -variable (indicated with `(E)`), that the `robot-paths` setting still has the default value (indicated with `(D)`), and that all other configuration -settings were specified via a configuration file, some of which indirectly through the `prefix` value (indicated with -`(F)`). - ---- - -## Exercises (optional) - -*These exercises are not very relevant for LUMI as LUMI already offers a complete configuration -also for user installations of software. However, if you are a very advanced user, you may still -want to make changes to that configuration, and all three options (configuration files, -environment variables and command line parameters) are available to users in the LUMI setup.* - -***Exercise C.1* - Configure EasyBuild** - -Configure EasyBuild to use the `easybuild` subdirectory in your home directory for everything, except for: - -* The location of the build directories: use `/tmp/$USER` for this; -* The locations that should be considered when searching for source files: - include both `$HOME/easybuild/sources` and `/easybuild/sources`, but make - sure that source files that are downloaded by EasyBuild are stored in - `$HOME/easybuild/sources`. - -Leave other configuration settings set to their default value. - -??? success "(click to show solution)" - - This is pretty straightforward. - - Here we just define the corresponding environment variables: - - ```shell - export EASYBUILD_PREFIX=$HOME/easybuild - export EASYBUILD_BUILDPATH=/tmp/$USER - export EASYBUILD_SOURCEPATH=$HOME/easybuild/sources:/easybuild/sources - ``` - - The location where EasyBuild should download source files to - must be listed first in the `sourcepath` configuration setting. - - The output of `--show-config` should look like this: - - ```shell - buildpath (E) = /tmp/example - containerpath (E) = /home/example/easybuild/containers - installpath (E) = /home/example/easybuild - packagepath (E) = /home/example/easybuild/packages - prefix (E) = /home/example/easybuild - repositorypath (E) = /home/example/easybuild/ebfiles_repo - robot-paths (D) = /home/example/easybuild/easyconfigs - sourcepath (E) = /home/example/easybuild/sources:/easybuild/sources - ``` - ---- - -***Exercise C.2* - Install a trivial software package with EasyBuild** - -Try running the following command: - -```shell -eb bzip2-1.0.6.eb -``` - -Where do you expect to find the installation? - -??? success "(click to show solution)" - - The software was installed in `$HOME/easybuild`, - since that's how we configured EasyBuild in *Exercise 3.1*: - - ```shell - $ ls $HOME/easybuild - ebfiles_repo modules software sources - ``` - - The actual installation is in `$HOME/easybuild/software`, - while the module file was generated in `$HOME/easybuild/modules/all`: - - ```shell - $ ls $HOME/easybuild/software - bzip2 - $ ls $HOME/easybuild/software/bzip2 - 1.0.6 - $ ls $HOME/easybuild/software/bzip2/1.0.6 - bin easybuild include lib man - ``` - - ```shell - $ ls $HOME/easybuild/modules/all - bzip2 - $ ls $HOME/easybuild/modules/all/bzip2 - 1.0.6.lua - ``` - - The source file for bzip2 1.0.6 was downloaded to `$HOME/easybuild/sources`: - - ```shell - $ ls $HOME/easybuild/sources/b/bzip2 - bzip2-1.0.6.tar.gz - ``` - - We will discuss this in more detail in the next part of the tutorial. - ---- - -*[[next: Basic usage]](1_08_basic_usage.md)* diff --git a/docs/2022-CSC_and_LO/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_08_basic_usage.md deleted file mode 100644 index f5031e48..00000000 --- a/docs/2022-CSC_and_LO/1_08_basic_usage.md +++ /dev/null @@ -1,1216 +0,0 @@ -# Basic usage - -*[[back: Configuring EasyBuild]](1_07_configuration.md)* - ---- - -Now that we have installed and configured EasyBuild, we can start using it for what it is intended for: -getting scientific software installed without breaking a sweat, or having to resist the urge to -shout out four-letter words. - -We will look at the high-level workflow first, and then cover each aspect in more detail. - -A couple of exercises are available at the end to help to make you more familiar with the -EasyBuild command line interface, so pay attention! - ---- - -## Overall workflow - -Installing software with EasyBuild is as easy as specifying to the **`eb` command** what we -want to install, and then sitting back to enjoy a coffee or tea (or whatever beverage you prefer). - -This is typically done by **specifying the name of one or more easyconfig files**, often combined -with the `--robot` (or `-r`) option to let EasyBuild also install missing dependencies. - -It is recommended to first assess the current situation before letting EasyBuild install the software, -and to check which **dependencies** are already installed and which are still missing. In addition, -you may want to inspect the specifics of the **installation procedure** that will be performed by EasyBuild, -and ensure that the active EasyBuild configuration is what it should be. - ---- - -## Specifying easyconfigs - -Letting EasyBuild know what should be installed can be done by specifying one or more easyconfig files, -which is also the most common way. Alternative methods like using the `--software-name` option won't be -covered in this tutorial, since they are not commonly used. - -Arguments passed to the `eb` command, being anything that is *not* an option (which starts with `-` or `--`) or -is a value for a preceding configuration option, are assumed to refer to easyconfig files (with some exceptions). -These could be: - -* the *(absolute or relative) path* to an easyconfig file; -* the *name* of an easyconfig file; -* the path to a *directory* containing easyconfig files; - -Specified paths to files must of course point to existing files; if not, EasyBuild will print an appropriate error message: - -```shell -$ eb /tmp/does_not_exist.eb -ERROR: Can't find path /tmp/does_not_exist.eb -``` - -When only the *name* of an easyconfig file is specified, EasyBuild will automatically try and locate it. -First, it will consider the *current directory*. If no file with the specified name is found there, -EasyBuild will search for the easyconfig file in the [robot search path](../1_07_configuration/#robot-search-path). - -If the path to an existing *directory* is provided, EasyBuild will walk through the entire directory -(including all subdirectories), retain all files of which the name ends with `.eb`, and (try to) use these -as easyconfig files. - - -#### Example command - -Suppose we have the current situation in our home directory: - -* two (easyconfig) files named `example1.eb` and `example2.eb`; -* a subdirectory named `some_deps`, which has two easyconfig files `dep1.eb` and `dep2.eb` - alongside a text file named `list.txt`; -* a subdirectory named `more_deps` located *in* the `some_deps` subdirectory, - which contains another easyconfig file `dep3.eb`; - -Or, visually represented: - -```shell -example1.eb -example2.eb -some_deps -├── deb2.eb -├── dep1.eb -├── list.txt -└── more_deps - └── dep3.eb -``` - -In this context, we run the following EasyBuild command from our home directory: - -```shell -eb example1.eb bzip2-1.0.6.eb $HOME/example2.eb some_deps -``` - -EasyBuild will interpret each of these arguments as follows: - -* `example1.eb` is the name of a file in the current directory, so it can be used directly; -* `bzip2-1.0.6.eb` is the name of an easyconfig file to locate via the robot search path - (since it does not exist in the current directory); -* `$HOME/example2.eb` specifies the full path to an existing file, which can be used directly; -* `some_deps` is the relative path to an existing directory, so EasyBuild will scan it and find three - easyconfig files: `some_deps/dep1.eb`, `some_deps/dep2.eb` and `some_deps/more_deps/dep3.eb`, - The `list.txt` file will be ignored since its name does not end with `.eb`. - ---- - -### Easyconfig filenames - -Note that the `eb` command does not care how easyconfig files are named, at least to some extent: the `.eb` -file extension *does* matter when easyconfig files are being picked up in subdirectories. - -File names for easyconfigs being mostly irrelevant is only correct with respect to the arguments passed to -the `eb` command however. As we will learn soon, the name of easyconfig files *does* matter (a lot) -when EasyBuild needs to locate easyconfigs that can be used to resolve a specified dependency -(see [here](#enabling-dependency-resolution)). - -This explains why easyconfig files usually adher to a very specific naming scheme, -corresponding to `--.eb`, where: - -* `` represents the software name; -* `` represents the software version; -* `` represents the toolchain used in the easyconfig file, which consists of the toolchain name - and version separated with a dash (`-`), and which is omitted (including the preceding `-`) when the - [`system` toolchain](../1_05_terminology#system-toolchain) is used; -* `` represents the value of the `versionsuffix` easyconfig parameter, - which is sometimes used to distinguish multiple variants of particular software installations - (and is empty by default); - ---- - -### Searching for easyconfigs - -You will frequently need to determine the exact name of an easyconfig file you want to install, -or just check which easyconfigs are available for a given software package. -This can be done by searching for easyconfigs using **`eb --search`** or **`eb -S`**. - -By default all directories listed in the [robot search path](../1_07_configuration#robot-search-path) will be -searched. If you want to search in additional directories without changing the robot search path, -you can use the `search-paths` configuration setting, or you can change the robot search path via either -the `robot` or `robot-paths` configuration options. - -Both the `--search` and `-S` options trigger the same search operation, but yield different output: -`eb --search` will print the full path to each easyconfig file that matches the specified search pattern, -while `eb -S` produces a more concise output. - -For example, let's check which easyconfig files are available for OpenFOAM 8 with a `foss` toolchain: - -```shell -$ eb --search openfoam-8-foss - * /home/example/.local/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-8-foss-2020a.eb - * /home/example/.local/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-8-foss-2020b.eb -``` - -The output is a bit more condensed when using `eb -S`: - -```shell -$ eb -S openfoam-8-foss -CFGS1=/home/example/.local/easybuild/easyconfigs/o/OpenFOAM - * $CFGS1/OpenFOAM-8-foss-2020a.eb - * $CFGS1/OpenFOAM-8-foss-2020b.eb -``` - -Note that the search is performed *case-insensitive*. - -The search pattern can include wildcards like `.*` and/or character groups like `[0-9]`, -or other special characters like `^` or `$` to mark the start/end of the filename, -but you need to be careful that `bash` does not expand these before the `eb` command is started, -so it is recommended to wrap the search pattern in single quotes (`'...'`) when using wildcards. - -For example, to check which easyconfigs are available to install TensorFlow 2.4.1 with the `2020b` version of a toolchain: - -```shell -$ eb -S '^tensorflow-2.4.1.*2020b' -CFGS1=/home/example/.local/easybuild/easyconfigs/t/TensorFlow - * $CFGS1/TensorFlow-2.4.1-foss-2020b.eb - * $CFGS1/TensorFlow-2.4.1-fosscuda-2020b.eb -``` - ---- - -#### Search index - -When searching for easyconfig files, you may see a message like this pop up: - -``` -== found valid index for , so using it... -``` - -This indicates that a **search index** was used for this particular directory which significantly speeds -up the search procedure, especially when the easyconfig files are located on a shared filesystem where metadata -operations involving lots of (small) files can be slow. - -For the easyconfig files included with an EasyBuild release, a search index is readily provided. -For other directories, you can create a search index using `eb --create-index `. -[See the EasyBuild documentation for more information](https://easybuild.readthedocs.io/en/latest/Easyconfigs_index.html). - ---- - -## Inspecting easyconfigs - -Once you have determined the name of the easyconfig file that corresponds to the software you want to install, -you may want to take a closer look at its contents before employing it. - -Since easyconfig files are simple text files (in Python syntax), you could use the ubiquitous `cat` command -or your favorite text editor (`vim`, what else). To avoid that you need to locate the easyconfig file first -and copy-paste the full path to it, you can use **`eb --show-ec`**. - -!!! Hint - To follow the examples below on LUMI, load ``LUMI/21.12`` and ``EasyBuild-user`` - (though results may differ or the examples not work anymore as the software installation - on LUMI evolves). - -For example, let's inspect the contents of the `bzip2-1.0.6.eb` easyconfig file: - -```shell -$ eb --show-ec bzip2-1.0.8-cpeCray-21.12.eb -== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-53o823qb/easybuild-xn6nmt61.log -== Contents of /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeCray-21.12.eb: -# Contributed by Kurt Lust, LUMI project & UAntwerpen - -local_bzip2_version = '1.0.8' # http://www.bzip.org/downloads.html - -name = 'bzip2' -version = local_bzip2_version - -homepage = 'https://www.sourceware.org/bzip2/' - -whatis = [ - 'Description: bzip2 is a freely available, patent free, high-quality data compressor.', - 'The module contains both executables and libraries.' - 'Keywords: BZ2', -] - -description = """ -bzip2 is a freely available, patent free, high-quality data compressor. It -typically compresses files to within 10% to 15% of the best available techniques -(the PPM family of statistical compressors), whilst being around twice as fast -at compression and six times faster at decompression. It is based on the -Burrows-Wheeler block-sorting text compression algorithm and Huffman coding. -""" - -usage = """ -Check the man pages for the available commands or the web-based documentation for the -library functions. -""" - -docurls = [ - 'Web-based documentation: http://www.bzip.org/docs.html', - 'Man pages available for bzcmp, bzdiff, bzegrep, bzfgrep, bzgrep, bzip2, bunzip2, bzless and bzmore', -] - -toolchain = {'name': 'cpeCray', 'version': '21.12'} -toolchainopts = {'pic': True} - -source_urls = ['https://sourceware.org/pub/%(name)s/'] -sources = [SOURCE_TAR_GZ] -patches = ['bzip2-%(version)s-pkgconfig-manpath.patch'] -checksums = [ - 'ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269', # bzip2-1.0.8.tar.gz - 'de11269dc6e4917023af4cee9ff83b204953ad9cde561dbc9d6fc70d9b9578e3', # bzip2-1.0.8-pkgconfig-manpath.patch -] - -builddependencies = [ # Create a reproducible build environment. - ('buildtools', '%(toolchain_version)s', '', True), -] - -local_bzip2_major_minor = '.'.join(local_bzip2_version.split('.')[:2]) - -sanity_check_paths = { - 'files': [ 'lib/pkgconfig/bzip2.pc', 'lib/libbz2.a', 'lib/libbz2.%s' % SHLIB_EXT, - 'lib/libbz2.%s.%s' % (SHLIB_EXT, local_bzip2_major_minor), - 'lib/libbz2.%s.%s' % (SHLIB_EXT, local_bzip2_version), - 'include/bzlib.h' ] + - [ 'bin/b%s' % x for x in ['unzip2', 'zcat', 'zdiff', 'zgrep', 'zip2', 'zip2recover', 'zmore'] ] + - [ 'share/man/man1/bz%s.1' % x for x in ['cmp', 'diff', 'egrep', 'fgrep', 'grep', 'ip2', 'less', 'more'] ], - 'dirs': [] -} - -sanity_check_commands = [ - 'bzip2 --help', - 'pkg-config --libs bzip2', -] - -moduleclass = 'tools' - -== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-53o823qb/easybuild-xn6nmt61.log* have been removed. -== Temporary directory /run/user/10012026/easybuild/tmp/eb-53o823qb has been removed. -``` -The output may actually be longer for an easyconfig file that is already installed on the system -as a new easyconfig file is generated in the repository with some information about the installation -added to it and as on LUMI these are at the front of the robot search path to ensure that the system -finds the right easyconfig file matching with a module on the system. - -We'll get back to what all of this means later... - ---- - -## Checking dependencies - -!!! note - In some of the examples below, we assume that some software is already installed with EasyBuild. - - If you are following hands-on in a prepared environment, make sure these installations are visible - in the output of "`module avail`". - - -Before kicking off an installation, it is good practice to check which of the required dependencies -are already installed, and which ones are still missing. - -This can be helpful to ensure that your EasyBuild configuration is set up correctly, -and to prevent from accidentally installing an entirely new software stack from scratch -in an unintended location (like `$HOME/.local/easybuild`, the -[default installation path](../configuration/#install-path)). - ---- - -### Dry run - -To get a complete overview of all required dependencies, and see which ones are already installed -and which ones aren't yet, you can use **`eb --dry-run`**. - -Since `--dry-run` produces rather verbose output including the full path to each easyconfig file, -there is a more concise equivalent available as well: `eb --dry-run-short`, which is equivalent with **`eb -D`**. - -For example, to check which of the required dependencies for `SAMtools-1.11-GCC-10.2.0.eb` are already installed: - -```shell -$ eb SAMtools-1.14-cpeGNU-21.12.eb -D -== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-oo0lj9lq/easybuild-2cyomy8v.log -Dry run: printing build status of easyconfigs and dependencies -CFGS=/appl/lumi - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeGNU/cpeGNU-21.12.eb (module: cpeGNU/21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ncurses/ncurses-6.2-cpeGNU-21.12.eb (module: ncurses/6.2-cpeGNU-21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/zlib/zlib-1.2.11-cpeGNU-21.12.eb (module: zlib/1.2.11-cpeGNU-21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeGNU-21.12.eb (module: bzip2/1.0.8-cpeGNU-21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12-minimal.eb (module: gettext/0.21-cpeGNU-21.12-minimal) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/XZ/XZ-5.2.5-cpeGNU-21.12.eb (module: XZ/5.2.5-cpeGNU-21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/Brotli/Brotli-1.0.9-cpeGNU-21.12.eb (module: Brotli/1.0.9-cpeGNU-21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cURL/cURL-7.78.0-cpeGNU-21.12.eb (module: cURL/7.78.0-cpeGNU-21.12) - * [ ] $CFGS/LUMI-EasyBuild-contrib/easybuild/easyconfigs/h/HTSlib/HTSlib-1.14-cpeGNU-21.12.eb (module: HTSlib/1.14-cpeGNU-21.12) - * [ ] $CFGS/LUMI-EasyBuild-contrib/easybuild/easyconfigs/s/SAMtools/SAMtools-1.14-cpeGNU-21.12.eb (module: SAMtools/1.14-cpeGNU-21.12) -== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-oo0lj9lq/easybuild-2cyomy8v.log* have been removed. -== Temporary directory /run/user/10012026/easybuild/tmp/eb-oo0lj9lq has been removed. -``` - -This output tells you that most of the dependencies required by ``SAMtools-1.14-cpeGNU-21.12.eb`` are -already installed, since they are marked with ``[x]``. However, the easyconfig files for -``HTSLib-1.14-cpeGNU-21.12.eb`` and SAMtools itself are not installed yet, denoted by the -lack of an ``x`` in ``[ ]``. - - ---- - -### Missing dependencies - -If you are only interested in which dependencies are still *missing*, -you can consult the output of **`eb --missing`**, or the equivalent **`eb -M`**. - -For example, for the SAMtools easyconfig file used in the previous example we get (with -some lines removed from the output): - -```shell -$ eb SAMtools-1.14-cpeGNU-21.12.eb -M -2 out of 11 required modules missing: - -* HTSlib/1.14-cpeGNU-21.12 (HTSlib-1.14-cpeGNU-21.12.eb) -* SAMtools/1.14-cpeGNU-21.12 (SAMtools-1.14-cpeGNU-21.12.eb) -``` - -That should be pretty self-explanatory: out of the 113 required dependencies (which includes the `cpeGNU` toolchain -and everything needed to install it), only 2 dependencies (including SAMtools itself) are missing. Great! - ---- - -## Inspecting install procedures - -Finally, before actually installing something you may want to assess *how* exactly EasyBuild is going -to install the software. - -Perhaps you don't trust EasyBuild yet (you will eventually though, hopefully), -or maybe you just want to double check that you have made the right choice before going through with -the actual installation. - -Using **`eb --extended-dry-run`**, or just **`eb -x`** for short, -you can get a **detailed overview of the installation procedure that would be performed by EasyBuild**, -**in a matter of seconds**. - -By means of example, let's inspect some parts of the installation procedure for ``HTSLib-1.14-cpeGNU-21.12.eb``: - -```shell -$ eb HTSlib-1.14-cpeGNU-21.12.eb -x -... - -*** DRY RUN using 'ConfigureMake' easyblock (easybuild.easyblocks.generic.configuremake @ /appl/lumi/SW/LUMI-21.12/common/EB/EasyBuild/4.5.3/lib/python3.6/site-packages/easybuild/easyblocks/generic/configuremake.py) *** - -== building and installing HTSlib/1.14-cpeGNU-21.12... -fetching files... [DRY RUN] - -... - -[prepare_step method] -Defining build environment, based on toolchain (options) and specified dependencies... - -Loading toolchain module... - -module load cpeGNU/21.12 - -Loading modules for dependencies... - -module load buildtools/21.12 -module load zlib/1.2.11-cpeGNU-21.12 -module load bzip2/1.0.8-cpeGNU-21.12 -module load XZ/5.2.5-cpeGNU-21.12 -module load cURL/7.78.0-cpeGNU-21.12 - -... - -Defining build environment... - -... - - export CC='cc' - export CFLAGS='-O2 -ftree-vectorize -fno-math-errno' - -... - -configuring... [DRY RUN] - -[configure_step method] - running command "./configure --prefix=/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/HTSlib/1.14-cpeGNU-21.12" - (in /run/user/10012026/easybuild/build/HTSlib/1.14/cpeGNU-21.12/HTSlib-1.14) - -building... [DRY RUN] - -[build_step method] - running command "make -j 256" - (in /run/user/10012026/easybuild/build/HTSlib/1.14/cpeGNU-21.12/HTSlib-1.14) - -testing... [DRY RUN] - -[test_step method] - -installing... [DRY RUN] - -... - -sanity checking... [DRY RUN] - -[sanity_check_step method] -Sanity check paths - file ['files'] - * bin/bgzip - * bin/tabix - * lib/libhts.so -Sanity check paths - (non-empty) directory ['dirs'] - * include -Sanity check commands - * bgzip --version - * htsfile --version - * tabix --version - -... -``` - -We've obviously trimmed the generated output a bit, but it should be sufficient. - -An overview of the installation procedure is shown, following the installation steps as they would be -performed by EasyBuild. The output above shows: - -* how the build environment will be set up during the `prepare` step, by loading the module for both the - toolchains and the dependencies, and defining a set of environment variables like `$CC`, `$CFLAGS`, etc. -* which command will be executed during the configuration step, and in which directory; -* the list of files and directories that will be checked during the sanity check step; - -If you were concerned about EasyBuild being too much of a black box, that is hopefully resolved now. - -!!! note - It is important to highlight here that the reported installation procedure *may* not be 100% correct, - since the [easyblock](1_05_terminology.md#easyblocks) can change its mind based on the output of shell commands - that were executed, or based on the contents of a file that was generated during the installation. - Since all "actions" that would be performed during the installation are actually skipped when using `eb -x`, - the reported installation procedure could be partially incorrect. - - In addition, the easyblock may trip over the fact that the installation procedure is not actually being - executed, which sometimes leads to an unexpected error. These situations are handled gracefully by `eb -x` - though, which will report any errors that occurred and then continue anyway with inspecting the remainder of the - installation procedure. Although this obviously limits the value of the generated output, - it doesn't make it completely useless. - ---- - -## Installing software - -You should now be able to make an informed decision on which easyconfig file you should use to -make EasyBuild install the software you require. - -As mentioned before, installing an easyconfig is as simple as passing it to the `eb` command. - -So, let's try to install libdap version 3.20.9 with the cpeGNU/21.12 toolchain. - -Let's first check if it has any dependencies that still need to be installed: - -```shell -$ eb libdap-3.20.9-cpeGNU-21.12.eb -D -== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-wm_bk3j6/easybuild-puyu_559.log -Dry run: printing build status of easyconfigs and dependencies - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeGNU/cpeGNU-21.12.eb (module: cpeGNU/21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/zlib/zlib-1.2.11-cpeGNU-21.12.eb (module: zlib/1.2.11-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libtirpc/libtirpc-1.3.2-cpeGNU-21.12.eb (module: libtirpc/1.3.2-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/Brotli/Brotli-1.0.9-cpeGNU-21.12.eb (module: Brotli/1.0.9-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ICU/ICU-69.1-cpeGNU-21.12.eb (module: ICU/69.1-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cURL/cURL-7.78.0-cpeGNU-21.12.eb (module: cURL/7.78.0-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeGNU-21.12.eb (module: bzip2/1.0.8-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ncurses/ncurses-6.2-cpeGNU-21.12.eb (module: ncurses/6.2-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libreadline/libreadline-8.1-cpeGNU-21.12.eb (module: libreadline/8.1-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12-minimal.eb (module: gettext/0.21-cpeGNU-21.12-minimal) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/PCRE2/PCRE2-10.37-cpeGNU-21.12.eb (module: PCRE2/10.37-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/XZ/XZ-5.2.5-cpeGNU-21.12.eb (module: XZ/5.2.5-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libxml2/libxml2-2.9.12-cpeGNU-21.12.eb (module: libxml2/2.9.12-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libxslt/libxslt-1.1.34-cpeGNU-21.12.eb (module: libxslt/1.1.34-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12.eb (module: gettext/0.21-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/file/file-5.41-cpeGNU-21.12.eb (module: file/5.41-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/util-linux/util-linux-2.37.1-cpeGNU-21.12.eb (module: util-linux/2.37.1-cpeGNU-21.12) - * [ ] /pfs/lustrep3/users/kurtlust/LUMI/LUMI-EasyBuild-contrib/easybuild/easyconfigs/l/libdap/libdap-3.20.9-cpeGNU-21.12.eb (module: libdap/3.20.9-cpeGNU-21.12) -== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-wm_bk3j6/easybuild-puyu_559.log* have been removed. -== Temporary directory /run/user/10012026/easybuild/tmp/eb-wm_bk3j6 has been removed. -``` - -and now instal the library: - -```shell -$ eb libdap-3.20.9-cpeGNU-21.12.eb -== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-kfphjoi8/easybuild-kcs00ai5.log -== processing EasyBuild easyconfig -/pfs/lustrep3/users/kurtlust/LUMI/LUMI-EasyBuild-contrib/easybuild/easyconfigs/l/libdap/libdap-3.20.9-cpeGNU-21.12.eb -== building and installing libdap/3.20.9-cpeGNU-21.12... -== fetching files... -== ... (took 2 secs) -== creating build dir, resetting environment... -== unpacking... -== patching... -== preparing... -== ... (took 6 secs) -== configuring... -== ... (took 1 min 6 secs) -== building... -== ... (took 53 secs) -== testing... -== installing... -== ... (took 5 secs) -== taking care of extensions... -== restore after iterating... -== postprocessing... -== sanity checking... -== ... (took 3 secs) -== cleaning up... -== creating module... -== ... (took 2 secs) -== permissions... -== packaging... -== COMPLETED: Installation ended successfully (took 2 mins 20 secs) -== Results of the build can be found in the log file(s) -/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/libdap/3.20.9-cpeGNU-21.12/easybuild/easybuild-libdap-3.20.9-20220329.154535.log -== Build succeeded for 1 out of 1 -== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-kfphjoi8/easybuild-kcs00ai5.log* have been removed. -== Temporary directory /run/user/10012026/easybuild/tmp/eb-kfphjoi8 has been removed. -``` - -That was... easy. Is that really all there is to it? Well, almost... - -### Enabling dependency resolution - -The libdap installation worked like a charm, but remember that all required dependencies were already -available (see [above](#dry-run)). - -If we try this with the `SAMtools-1.14-cpeGNU-21.12.eb`, for which the required `HTSlib` dependencies is not available yet, it's less successful: - -```shell -$ eb SAMtools-1.14-cpeGNU-21.12.eb -M - -2 out of 11 required modules missing: - -* HTSlib/1.14-cpeGNU-21.12 (HTSlib-1.14-cpeGNU-21.12.eb) -* SAMtools/1.14-cpeGNU-21.12 (SAMtools-1.14-cpeGNU-21.12.eb) -``` - -```shell -$ eb SAMtools-1.14-cpeGNU-21.12.eb -... -== preparing... -== FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12): build failed -(first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12 (took 3 secs) -== Results of the build can be found in the log file(s) -/run/user/10012026/easybuild/tmp/eb-rgj1v43y/easybuild-SAMtools-1.14-20220329.155911.ZtDcX.log - -ERROR: Build of /appl/lumi/LUMI-EasyBuild-contrib/easybuild/easyconfigs/s/SAMtools/SAMtools-1.14-cpeGNU-21.12.eb failed (err: 'build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12') -``` - -Oh my, what's this all about? - -If we filter the output a bit and focus on the actual error, the problem is clear: - -``` -Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12') -``` - -The required dependency `HTSlib/1.14-cpeGNU-21.12` is not installed yet, -and EasyBuild does not automatically install missing dependencies unless it is told to do so -(which we didn't do in the configuration for LUMI). - -It helpfully suggests to use the `--robot` (or '-r') command line option, so let's try that: - -```shell -$ eb SAMtools-1.14-cpeGNU-21.12.eb --robot -... -== resolving dependencies ... -... -== building and installing HTSlib/1.14-cpeGNU-21.12... -... -== COMPLETED: Installation ended successfully (took 13 sec) -... -== building and installing SAMtools/1.14-cpeGNU-21.12... -... -== COMPLETED: Installation ended successfully (took 8 sec) -... -== Build succeeded for 2 out of 2 -``` - -With dependency resolution enabled the `HTSlib` module gets installed first, -before EasyBuild proceeds with installing `SAMtools`. Great! - ---- - -### Trace output - -As you may have noticed if you tried the previous example hands-on, -some installations take a while. -An installation can be spending quite a bit of time during the build step, but what is actually going on there? - -To provide some more feedback as the installation progresses, you can enable the "`trace`" configuration setting. -Let's do this by defining the `$EASYBUILD_TRACE` environment variable, just to avoid having to type `--trace` -over and over again. - -We will redo the installation of `SAMtools-1.14-cpeGNU-21.12.eb` by passing the `--rebuild` -option to the `eb` command (try yourself what happens if you don't use the `--rebuild` option!): - -```shell -$ export EASYBUILD_TRACE=1 -$ eb SAMtools-1.14-cpeGNU-21.12.eb --rebuild -... -== configuring... - >> running command: - [started at: 2022-03-29 18:46:31] - [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] - [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-g7vd83qv.log] - /users/kurtlust/LUMI-user-appl/sources/generic/eb_v4.5.3/ConfigureMake/config.guess - >> command completed: exit 0, ran in < 1s - >> running command: - [started at: 2022-03-29 18:46:31] - [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] - [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-k0etfv8i.log] - ./configure --prefix=/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/SAMtools/1.14-cpeGNU-21.12 --build=x86_64-pc-linux-gnu ---host=x86_64-pc-linux-gnu --with-htslib=$EBROOTHTSLIB - >> command completed: exit 0, ran in 00h00m03s -== ... (took 3 secs) -== building... - >> running command: - [started at: 2022-03-29 18:46:34] - [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] - [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-svcps0yj.log] - make -j 256 CC="cc" CXX="CC" CFLAGS="-O2 -ftree-vectorize -fno-math-errno -fPIC" CXXFLAGS="-O2 -ftree-vectorize -fno-math-errno --fPIC" - >> command completed: exit 0, ran in 00h00m06s -== ... (took 6 secs) -``` - -That's a bit more comforting to stare at... - -SAMtools uses a custom easyblock that is derived from the generic `ConfigureMake` easyblock. -During the *configure* step, the `./configure` command is run with `--build` and `--host` -options added by the generic ConfigureMake easyblock (and the other command, `config.guess` plays -a role in determining the value of those flags). The `--with-htslib=$EBROOTHTSLIB` flag is added -via our easyconfig file to tell SAMtools to use an already available version of HTSlib rather -than the built-in one. - -During the *build* step, the software is actually being compiled -by running the `make` command. EasyBuild automatically uses the available cores on the system (in this case 256). - -We even get a pointer to a log file that contains the output of the command being run, -so we can use `tail -f` to see in detail how it progresses. - -Once the `make` command completes, we get a message that the command completed with a exit code 0 -(implying success), and that it took 3 seconds to run. That's good to know. - -Later during the installation, we now also see this output during the sanity check step: - -``` -== sanity checking... - >> file 'bin/blast2sam.pl' found: OK - >> file 'bin/bowtie2sam.pl' found: OK - >> file 'bin/export2sam.pl' found: OK - >> file 'bin/interpolate_sam.pl' found: OK -... - >> loading modules: SAMtools/1.14-cpeGNU-21.12... - >> running command 'samtools version' ... - >> result for command 'samtools version': OK -``` - -Thanks to enabling trace mode, EasyBuild tells us which files (& directories, but there are non in this case) -it is checking for -in the installation, and which command it is trying to run before declaring it a success. Nice! - -The extra output you get when trace mode is enabled is concise and hence not overwhelming, -while it gives a better insight into what is going on during the installation. -It may also help to spot unexpected actions being taken during the installation early on, -so you can interrupt the installation before it completes, if deemed necessary. - ---- - -## Using installed software - -So far, we have already installed 4 different software packages (SAMtools, HTSlib, libdap); -we even installed SAMtools twice! - -A lot was going on underneath the covers: locating and unpacking -the source tarballs, setting up the build environment, configuring the build, compiling, -creating and populating the installation directory, performing a quick sanity check on the installation, -cleaning things up, and finally generating the environment module file corresponding to the installation. - -That's great, but how do we now actually *use* these installations? - -This is where the generated module files come into play: they form the access portal to the software -installations, and we'll use the ubiquitous `module` command to digest them. - -First, we need to make sure that the modules tool is aware of where the module files for -our installations are located. On LUMI, when using the EasyBuild-user module to configure EasyBuild, -everything is taken care of for you and the LUMI modules will also automatically add the -suitable module directories for user-installed software to the search path for modules. -By default, EasyBuild-config will install in `$HOME/EasyBuild`, but it is possible to -build the installation in a different directory by pointing to it with the environment -variable `EBU_USER_PREFIX`. Of course this variable needs to be set before loading the `LUMI` -module. (Note that one reason why we don't load a software stack by default is that in -the current setup of LUMI this module would be loaded before the user gets the chance to set -that environment variable in `.bash_profile` or `.bashrc`.) - -When loading the `EasyBuild-user` module, the module command will show you were EasyBuild -will install the software and put the modules, and also put its repository of -processed easyconfig file. - -```shell -ml EasyBuild-user - -EasyBuild configured to install software from the LUMI/21.12 software stack for the LUMI/L -partition in the user tree at /users/kurtlust/LUMI-user-appl. - * Software installation directory: /users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L - * Modules installation directory: /users/kurtlust/LUMI-user-appl/modules/LUMI/21.12/partition/L - * Repository: /users/kurtlust/LUMI-user-appl/ebrepo_files/LUMI-21.12/LUMI-L - * Work directory for builds and logs: /run/user/10012026/easybuild - Clear work directory with clear-eb -``` - -EasyBuild will copy each easyconfig file it installs to -the repository and add some lines to it with information about the installation. It -also has some options that may edit the source easyconfig, e.g., when asking EasyBuild -to try to build with another toolchain. - -You can always check where EasyBuild is installing stuff by checking the output of -`eb --show-config`. - -If you're unsure where EasyBuild is installing stuff at this point, -check the output of `eb --show-config`; the value of the `installpath` configuration setting is what we are interested in now: - -```shell -$ eb --show-config -... -buildpath (E) = /run/user/XXXXXXXX/easybuild/build -... -installpath-modules (E) = /users/XXXXXXXX/LUMI-user/modules/LUMI/21.12/partition/L -installpath-software (E) = /users/XXXXXXXX/LUMI-user/SW/LUMI-21.12/L... -... -repositorypath (E) = /users/XXXXXXXX/LUMI-user/ebrepo_files/LUMI-21.12/LUMI-L -... -sourcepath (E) = /users/XXXXXXXX/LUMI-user/sources:/appl/lumi/sources/easybuild -... -``` - -This is sligthly different from the default EasyBuild setup, where the modules, software, -repository and sources would be installed in respectively the subdirectories `modules`, -`software`, `ebfiles_repo` and `sources` of the directory pointed to by the `installpath` -line. - -The modules directory is also a simplified one from the standard EasyBuild one as that also -provides a module categorisation besides a directory containing all modules. As this categorisation -is largely arbitrary and hard to use in the module system, we decided simply not to use it in -our installation and use a custom naming scheme. - -However, if you would be using EasyBuild on another system with its default configuration, the -above setup would be used. For more information, we refer to the generic EasyBuild tutorials on the -[EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). - -Now the modules tool should be aware of our brand new installations: - -```shell -$ module avail -... --- EasyBuild managed user software for software stack LUMI/21.12 on LUMI-L --- - HTSlib/1.14-cpeGNU-21.12 libdap/3.20.9-cpeGNU-21.12 - SAMtools/1.14-cpeGNU-21.12 - ------ EasyBuild managed software for software stack LUMI/21.12 on LUMI-L ----- -... -``` - -This output shows both the modules for our own installations as well as the "central" installations -(which we omitted for brevity). - -Now we can load these modules and start using these software installations. - -Let's test this for SAMtools. In our current environment, the `samtools` command is not available yet: - -```shell -$ module list - -Currently Loaded Modules: - 1) perftools-base/21.12.0 - 2) cce/13.0.0 - 3) craype/2.7.13 - 4) cray-dsmml/0.2.2 - 5) cray-mpich/8.1.12 - 6) cray-libsci/21.08.1.2 - 7) PrgEnv-cray/8.2.0 - 8) ModuleLabel/label (S) - 9) init-lumi/0.1 (S) - 10) craype-x86-rome - 11) craype-accel-host - 12) libfabric/1.11.0.4.106 - 13) craype-network-ofi - 14) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta - 15) partition/L (S) - 16) LUMI/21.12 (S) - - Where: - S: Module is Sticky, requires --force to unload or purge -$ samtools --bash: samtools: command not found -``` - -Loading the module for SAMtools changes that: - -```shell -$ module load SAMtools/1.14-cpeGNU-21.12 - -Lmod is automatically replacing "cce/13.0.0" with "gcc/11.2.0". -Lmod is automatically replacing "PrgEnv-cray/8.2.0" with "cpeGNU/21.12". - -Due to MODULEPATH changes, the following have been reloaded: - 1) cray-mpich/8.1.12 - -$ module list - -Currently Loaded Modules: - 1) perftools-base/21.12.0 - 2) ModuleLabel/label (S) - 3) init-lumi/0.1 (S) - 4) craype-x86-rome - 5) craype-accel-host - 6) libfabric/1.11.0.4.106 - 7) craype-network-ofi - 8) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta - 9) partition/L (S) - 10) LUMI/21.12 (S) - 11) gcc/11.2.0 - 12) craype/2.7.13 - 13) cray-mpich/8.1.12 - 14) cray-libsci/21.08.1.2 - 15) cray-dsmml/0.2.2 - 16) cpeGNU/21.12 - 17) ncurses/6.2-cpeGNU-21.12 - 18) zlib/1.2.11-cpeGNU-21.12 - 19) bzip2/1.0.8-cpeGNU-21.12 - 20) gettext/0.21-cpeGNU-21.12-minimal - 21) XZ/5.2.5-cpeGNU-21.12 - 22) Brotli/1.0.9-cpeGNU-21.12 - 23) cURL/7.78.0-cpeGNU-21.12 - 24) HTSlib/1.14-cpeGNU-21.12 - 25) SAMtools/1.14-cpeGNU-21.12 - - Where: - S: Module is Sticky, requires --force to unload or purge - -$ samtools --version -samtools 1.14 -Using htslib 1.14 -Copyright (C) 2021 Genome Research Ltd.... -... -``` - -Note that the modules for the required dependencies, including the compiler toolchain (which provides runtime libraries -like `libstdc++.so`), are loaded automatically. The "`module load`" command changes the active environment, -by updating environment variables like `$PATH` for example, to make the software available for use. - -##### Resetting your environment - -To restore your environment to a pristine state in which no modules are loaded, you can either -unload the loaded modules one by one using "`module unload`", or you can unload all of them at once using -"`module purge`". On LUMI, `module purge` will unload all application modules but will not undo the selection -of the software stack. It will reset the software stack though to use those modules that fit best -with the hardware of the current node (i.e., you may find a different `partition` module). - -```shell -$ module purge -The following modules were not unloaded: - (Use "module --force purge" to unload all): - - 1) LUMI/21.12 6) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta - 2) craype-x86-rome 7) partition/L - 3) craype-accel-host 8) init-lumi/0.1 - 4) libfabric/1.11.0.4.106 9) ModuleLabel/label - 5) craype-network-ofi -$ module list - -Currently Loaded Modules: - 1) LUMI/21.12 (S) - 2) craype-x86-rome - 3) craype-accel-host - 4) libfabric/1.11.0.4.106 - 5) craype-network-ofi - 6) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta - 7) partition/L (S) - 8) init-lumi/0.1 (S) - 9) ModuleLabel/label (S) - - Where: - S: Module is Sticky, requires --force to unload or purge -``` - -Runnin `module --force purge` instead will remove all modules, including the `init-lumi` -module which does part of the initialisation. You will not be able to use the software -stacks completely as before without first loading `init-lumi` in its most recent (or default) -version again! - -```shell -$ module --force purge -$ module list -No modules loaded -``` - ---- - -## Stacking software - -Maybe you have overlooked how the software we are playing around with was not only installed across multiple -different installation directories per software, we are also "stacking" our own installations (in `$HOME/EasyBuild` -or `$EBU_USER_PREFIX`) on top of installations that are provided in a totally different location (`/appl/lumi`). - -**EasyBuild doesn't care *where* software is installed: as long as the module file that provides access to it -is available, it is happy to pick it up and use it when required.** - -This implies that end users of LUMI can easily install their -own small software stack on top of what is provided centrally by the LUMI User Support, -for example. They can even -"replace" a central software installation for their purposes if they need to, since the modules tool will -load the first module file that matches the request being made (there are some caveats with this, but we -won't go into those here). - ---- - -## Exercises - -TODO: Rework for LUMI and provide the necessary EasyConfig files. - -***Guidelines*** - -Do yourself a favor: don't peek at the solution until you have made an attempt to solve the exercise yourself! - -Please do not spoil solutions to others before they have been discussed by the tutorial organisers. - -The exercises are based on the easyconfig files included with EasyBuild 4.3.3. - ---- - -***Exercise U.0**** - Making installed software available* - -Before working on the exercises for this part of the tutorial, -make sure that the software that is already installed in the prepared environment is available. - -We will assume that you have a small software stack installed using the `2020b` version of the `foss` toolchain. - -**Tip:** execute a "`module use`" command, and verify with "`module avail`" that a bunch of software modules -are available for loading. - -??? success "(click to show solution)" - - Use the following command to make the modules for the software available that is pre-installed - in the prepared environment: - ```shell - module use /easybuild/modules/all - ``` - - If software is installed in a different location than `/easybuild/` in your environment, - you should adjust the command accordingly. - - ---- - -***Exercise U.1**** - Searching easyconfigs* - -See if EasyBuild provides any easyconfig files for installing GROMACS version 2020/5. - -??? success "(click to show solution)" - To check for available easyconfig files, we can use `eb --search` or `eb -S`: - ```shell - $ eb -S gromacs-2020.5 - == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... - CFGS1=/home/example/.local/easybuild/easyconfigs/g/GROMACS - * $CFGS1/GROMACS-2020.5-fosscuda-2020a-Python-3.8.2.eb - * $CFGS1/GROMACS-2020.5_fix_threads_gpu_Gmxapitests.patch - ``` - This actually shows one easyconfig file but also a patch file. We can also search specifically - for GROMACS 2020.5 in the `foss` and `fosscuda` toolchains using - ```shell - $ eb -S gromacs-2020.5-foss - == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... - CFGS1=/home/example/.local/easybuild/easyconfigs/g/GROMACS - * $CFGS1/GROMACS-2020.5-fosscuda-2020a-Python-3.8.2.eb - ``` - and now we find a single easyconfig file. - ---- - -***Exercise U.2**** - Checking dependencies* - -Check which dependencies are missing to install QuantumESPRESSO version 6.6 with the `2020b` version of the `foss` toolchain. - -??? success "(click to show solution)" - First, we need to determine the name of the easyconfig file for QuantumESPRESSO version 6.6: - ```shell - $ eb -S 'QuantumESPRESSO-6.6.*foss-2020b' - == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... - CFGS1=/home/example/.local/easybuild/easyconfigs/q/QuantumESPRESSO - * $CFGS1/QuantumESPRESSO-6.6-foss-2020b.eb - ``` - To determine which dependencies are missing to install this QuantumESPRESSO easyconfig file, we can use `--missing`: - ```shell - $ eb QuantumESPRESSO-6.6-foss-2020b.eb --missing - - 3 out of 58 required modules missing: - - * libxc/4.3.4-GCC-10.2.0 (libxc-4.3.4-GCC-10.2.0.eb) - * ELPA/2020.11.001-foss-2020b (ELPA-2020.11.001-foss-2020b.eb) - * QuantumESPRESSO/6.6-foss-2020b (QuantumESPRESSO-6.6-foss-2020b.eb) - ``` - (some nonessential output removed). - ---- - -***Exercise U.3**** - Performing a dry run* - -Figure out which command EasyBuild would use to compile -the software provided by the `Bowtie2-2.4.2-GCC-9.3.0.eb` easyconfig file, -without actually installing `Bowtie2`. - -Also, which binaries will EasyBuild check for to sanity check the installation? - -??? success "(click to show solution)" - To inspect the installation procedure, we can use `eb -x Bowtie2-2.4.2-GCC-9.3.0.eb`. - - The output for the build step shows the actual compilation command that would be performed (`make ...`): - - ```shell - [build_step method] - >> running command: - [started at: 2021-03-08 20:15:08] - [working dir: /local_scratch/hkenneth/eb-1wodfohg/__ROOT__/local_scratch/hkenneth/Bowtie2/2.4.2/GCC-9.3.0/Bowtie2-2.4.2] - [output logged in /local_scratch/hkenneth/eb-1wodfohg/easybuild-run_cmd-haojzisn.log] - make -j 48 CC="gcc" CPP="g++" CXX="g++" RELEASE_FLAGS="-O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -std=gnu++98" - (in /local_scratch/hkenneth/Bowtie2/2.4.2/GCC-9.3.0/Bowtie2-2.4.2) - ``` - - If the output you get is less detailed, you may not have set `export EASYBUILD_TRACE=1`. - - The output for the sanity check step shows which binaries are expected to be installed: - ``` - [sanity_check_step method] - Sanity check paths - file ['files'] - * bin/bowtie2 - * bin/bowtie2-align-l - * bin/bowtie2-align-s - * bin/bowtie2-build - * bin/bowtie2-build-l - * bin/bowtie2-build-s - * bin/bowtie2-inspect - * bin/bowtie2-inspect-l - * bin/bowtie2-inspect-s - ``` - ---- - -***Exercise U.4**** - Installing software* - -Install version 3.1.0 of the `h5py` Python package and all missing dependencies, -using the `foss/2020b` toolchain, into `/tmp/$USER/easybuild`, -while leveraging the already installed software available from `/easybuild`. - -Enable trace output so you can see which parts of the installation take a while. - -??? success "(click to show solution)" - First, determine the easyconfig file we can use for this: - ```shell - $ eb -S 'h5py-3.1.0.*foss-2020b' - CFGS1=/home/example/.local/easybuild/easyconfigs/h/h5py - * $CFGS1/h5py-3.1.0-foss-2020b.eb - ``` - - Make sure the pre-install software in `/easybuild/` is available: - ```shell - module use /easybuild/modules/all - ``` - - Check which dependencies are missing to install this `h5py` easyconfig: - ```shell - $ eb h5py-3.1.0-foss-2020b.eb --missing - - 2 out of 63 required modules missing: - - * pkgconfig/1.5.1-GCCcore-10.2.0-python (pkgconfig-1.5.1-GCCcore-10.2.0-python.eb) - * h5py/3.1.0-foss-2020b (h5py-3.1.0-foss-2020b.eb) - ``` - - Install `h5py` by specifying the easyconfig file and enabling dependency resolution via `--robot`, - while indicating that we want to install the software into `/tmp/$USER/easybuild` using the `--installpath` - option. Also make sure that trace mode is enabled by defining the `$EASYBUILD_TRACE` environment variable. - ```shell - $ export EASYBUILD_TRACE=1 - $ eb h5py-3.1.0-foss-2020b.eb --robot --installpath /tmp/$USER/easybuild - ... - == building and installing pkgconfig/1.5.1-GCCcore-10.2.0-python... - ... - == building and installing h5py/3.1.0-foss-2020b... - ... - == building... - >> running command: - [started at: 2020-06-10 21:47:32] - [working dir: /tmp/example/h5py/3.1.0/foss-2020b/h5py-3.1.0] - [output logged in /tmp/eb-rjjkbqe1/easybuild-run_cmd-d_dkc4iz.log] - python setup.py configure --mpi --hdf5=$EBROOTHDF5 && /easybuild/software/Python/3.8.6-GCCcore-10.2.0/bin/python setup.py build - >> command completed: exit 0, ran in 00h01m27s - ... - == COMPLETED: Installation ended successfully (took 2 min 46 sec) - ... - == Build succeeded for 2 out of 2 - ``` - - The trace output shows that most time is spent in the build command, - which runs both `python setup.py configure` and `python setup.py build`. - ---- - -***Exercise U.5**** - Using installed software* - -Using the `h5py` installation from the previous exercise to create an empty HDF5 file, -using the following Python statements: - -```python -import h5py -f = h5py.File("empty.hdf5", "w") -f.close() -``` - -Check the resulting file using the `h5stat` command. - -??? success "(click to show solution)" - First, we need to make the modules tool aware of the module files that were installed into `/tmp/$USER/easybuild`: - ```shell - module use /tmp/$USER/easybuild/modules/all - ``` - - Then we can check the `h5py` module is available, and load it: - ```shell - $ module avail h5py - ------------ /tmp/example/easybuild/modules/all ------------ - h5py/3.1.0-foss-2020b - ``` - - ```shell - module load h5py/3.1.0-foss-2020b - ``` - - The Python code snippet can be run directly on the command line using "`python -c '...'`", since it's tiny: - ```shell - python -c 'import h5py; f = h5py.File("empty.hdf5", "w"); f.close()' - ``` - Of course you can also copy the Python code snippet in a file named `test_h5py.py`, - and then run it with `python test_h5py.py`. - - Checking with the `h5stat` command shows that the resulting `empty.hdf5` is indeed a valid HDF5 file: - ```shell - $ ls -l empty.hdf5 - -rw-rw-r-- 1 example example 800 Jun 10 21:54 empty.hdf5 - - $ h5stat empty.hdf5 - Filename: empty.hdf5 - File information - # of unique groups: 1 - # of unique datasets: 0 - ... - ``` - ---- - -If you've made it through the hands-on exercises, congratulations! - -If not, don't worry too much about it. We covered a lot of ground here, -and it's a lot to take in at once, take your time... - ---- - -*[[next: Part 2: Using EasyBuild]](2_00_part2_using.md)* diff --git a/docs/2022-CSC_and_LO/2_00_part2_using.md b/docs/2022-CSC_and_LO/2_00_part2_using.md deleted file mode 100644 index c50036be..00000000 --- a/docs/2022-CSC_and_LO/2_00_part2_using.md +++ /dev/null @@ -1,8 +0,0 @@ -# Part II: Using EasyBuild - -*[[back to start page]](index.md)* - -* [Troubleshooting](2_01_troubleshooting.md) *(hands-on)* -* [Creating easyconfig files](2_02_creating_easyconfig_files.md) *(hands-on)* -* [Using external modules from the Cray PE](2_03_external_modules.md) -* [Implementing easyblocks](2_04_implementing_easyblocks.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/2_01_troubleshooting.md b/docs/2022-CSC_and_LO/2_01_troubleshooting.md deleted file mode 100644 index 30f2f5cc..00000000 --- a/docs/2022-CSC_and_LO/2_01_troubleshooting.md +++ /dev/null @@ -1,545 +0,0 @@ -# Troubleshooting - -*[[back: Overview]](index.md)* - ---- - -Whatever tool you use, when installing scientific software you'll -be running into problems rather sooner than later. - -In this part we take a look at how you can **troubleshoot a failing installation**, -and focus on a couple of EasyBuild aspects that can be helpful in that context, -like the **error messages produced by EasyBuild**, the detailed **log file** that is -available for the installation and the **build directory** where most software is -being compiled before it actually gets installed. - -At the end of this part an exercise is available in which you get -to fix a failing installation, so pay attention! - -## EasyBuild error messages - -When EasyBuild detects that something went wrong, it will usually produce a -short (and hopefully helpful) error message. - -Things that could go wrong during an installation include: - -* missing source or patch files; -* a checksum error on a downloaded source or patch file; -* required dependencies that are not specified in the easyconfig file; -* failing shell commands; -* running out of available memory or disk space; -* a segmentation fault caused by a flipped bit triggered by a cosmic ray ([really, it happens!](https://blogs.oracle.com/linux/attack-of-the-cosmic-rays-v2)); - -Unfortunately this is not an exhaustive list, there are plenty of other -potential problems that could result in a failing installation... - -For each of the shell commands that EasyBuild executes during an -installation, it will check the exit status. -If the exit status is zero, EasyBuild will usually assume that the shell command -ran correctly, and it will continue with the rest of the installation procedure. -If the exit status is anything but zero, EasyBuild will assume that a problem -has occurred, and the installation will be interrupted. - -### Example - -Here is an example of an EasyBuild error message (slightly reformatted for clarity): - -``` -$ eb example.eb -... -== building... -== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): -build failed (first 300 chars): cmd "make" exited with exit code 2 and output: -/usr/bin/g++ -O2 -ftree-vectorize -march=znver2 -fno-math-errno -c -o core.o core.cpp -cc1plus: error: bad value (‘znver2’) for ‘-march=’ switch (took 1 sec) -== Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log -ERROR: Build of /home/easybuild/subread.eb failed (err: ...) -``` - -Let's break this down a bit: during the `build` step of the installation -procedure EasyBuild was running `make` as a shell command, which -failed (exit code 2, so not zero). -The `make` command tripped over the compilation of `core.cpp` that failed because -`-march=znver2` is not a known option to the `g++` command. - -OK fine, but now what? Can you spot something suspicious here? -Wait a minute... Why is `make` using `/usr/bin/g++` for the compilation?! -That's not where our toolchain compiler is installed, -that's somewhere under `/opt/cray/pe/gcc`. - -Let's see what `/usr/bin/g++` is: - -``` -$ /usr/bin/g++ --version -g++ (SUSE Linux) 7.5.0 -``` - -Oh my that's an ancient compiler (7.5 was released on November 14, 2019, -a few months after the release of the Zen2 architecture, but -the base version, 7.1, is really from May 2, 2017, long before the Zen2 architecture was around) -That could definitely explain why it doesn't know about the Zen2 architecture yet... - -Your next step in this case should probably be figuring -out why `/usr/bin/g++` is being used rather than just `g++`, which would -result in using the right compiler version because EasyBuild sets up the build -environment carefully. - -This is a fictitious example of course, but hopefully it gives you a feeling -of how errors that occur during the installation are handled. - -## EasyBuild log files - -Finding the cause of a problem that made the installation fail is, unfortunately, not always that straightforward... - -EasyBuild includes the first 300 characters of the output produced by a failing -shell command in the error message, which is a simple way to try include -actionable information in the error message while not flooding your terminal -with the full output of the command that failed. - -In some cases there won't be any useful information in there however, -since the actual error message(s) could only appear way later, perhaps even after -the command was already running for several minutes. - -In that case, you will have to dive into the log file that is created by EasyBuild for -every installation, which is located in the unique temporary directory for the EasyBuild session. - -See for example this output line from our earlier example error message: - -``` -== Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log -``` - -You can open this file with your favorite text editor or a tool like `less` -to take a look at the information collected in the log file, which includes -things like: - -* informative messages produced by both the EasyBuild framework and the easyblock - describing how the installation is progressing; -* how the build environment was set up: which modules were loaded, which environment variables were set; -* the exact shell commands that were executed, and in which directory they were run; -* the full output produced by these commands, and their exit code; - -Note that the installation log is also copied into each software installation -directory for successful installation, into the `easybuild` subdirectory. -For example: - -``` -/appl/lumi/SW/LUMI-21.12/L/EB/ncurses/6.2-cpeGNU-21.12/easybuild/easybuild-ncurses-6.2-20220302.110244.log -``` - -### Last log - -The `eb` command supports a handy little option that prints the location -to the most recently updated build log. You can leverage this to quickly -open the build log of the last ***failed*** EasyBuild session in an editor: - -``` -vim $(eb --last-log) -``` - -### Navigating log files - -Usually you want to go to the end of the log file and then work your way up, -either by scrolling or by searching for specific patterns. Here are a couple -of suggestions of patterns you can use to locate errors: - -* `ERROR` -* `Error 1` -* `error: ` -* `failure` -* `not found` -* `No such file or directory` -* `bazel` -* `Segmentation fault` - -Using "`error`" as a search pattern is not very useful: you will hit a lot of log lines -that are not actually errors at all (like the compilation of an `error.c` file), -and you'll miss others that do include errors but mention `ERROR` or `Error` -rather than `error`. - -When using `less` to view a log file, you can navigate it by: - -* hitting '`$`' followed by '`G`' to go to the end of the log file; -* using your arrow keys to scroll up/down; -* typing '`?`' followed by some text and Enter to search backwards for a particular -pattern ('`/`' to search forwards, '`n`' for next match); -* hitting '`q`' to exit; - -It can also be helpful to zoom in on a specific step of the installation procedure, -which you can do by looking for step markers like these: - -``` -== 2020-06-13 01:34:48,816 example INFO configuring... -== 2020-06-13 01:34:48,817 example INFO Starting configure step -... -== 2020-06-13 01:34:48,823 main.EB_HPL INFO Running method configure_step part of step configure -``` - -If you want to look at the start of the output produced by a particular command, -you can look for the log message that looks like this (this is from the installation -log for `HDF5`): - -``` -== 2020-06-09 13:11:19,968 run.py:222 INFO running cmd: make install -== 2020-06-09 13:11:25,186 run.py:538 INFO cmd " make install " exited with exit code 0 and output: -Making install in src -make[1]: Entering directory `/tmp/example/HDF5/1.10.7/gompi-2020b/hdf5-1.10.7/src' -... -``` - -It can be useful to look for the *first* error that occurred in the output of a command, since subsequent errors are -often fallout from earlier errors. You can do this by first navigating -to the start of the output for a command using "`INFO running cmd`" as a search pattern, and then looking for patterns -like "`error:`" from there. - -## Inspecting the build directory - -When an installation fails the corresponding build directory is *not* cleaned up -automatically, that is only done for successful installations. -This allows you to dive in and check for clues in the files that are stored there. - -The location of the build directory is mentioned in the EasyBuild error message: - -``` -== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): ... -``` - -For software using a classic `configure` script, you may have to locate -and inspect the `config.log` file in the build directory to determine the underlying cause of an error. For software using CMake as a configuration tool you often have to check in -`CMakeOutput.log` or `CMakeError.log` for clues, which are sneakily hidden by CMake in a `CMakeFiles` subdirectory of the build directory. - -As a side note here: as EasyBuild does not clean out old and failed builds you will need to eventually manually remove these build directories from the `buildpath` directory. On the login nodes of LUMI this is currently not much of an issue as `$XDG_RUNTIME_DIR` is (ab)used for build and -temporary files and that directory is cleaned automatically. However, when building on compute nodes, where `$XDG_RUNTIME_DIR` is -not available, it is essential to manually clean that directory as the space is not automatically cleaned when your session -ends. The `EasyBuild-user` module does define the bash function `clear-eb` that can be used to clear that space. - -## Exercise - -Let's work our way through a less smooth software installation, -using the easyconfig file that is provided below. - -Don't worry if most of this is still unclear to you, we'll get -to writing easyconfig files from scratch [later in this tutorial](2_02_creating_easyconfig_files). - -```python -easyblock = 'MakeCp' - -name = 'Subread' -version = '2.0.1' - -homepage = 'http://subread.sourceforge.net' -description = "High performance read alignment, quantification and mutation discovery" - -toolchain = {'name': 'PrgEnv-gnu', 'version': '21.10'} - -# download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz -sources = ['subread-%(version)s-source.tar.gz'] -checksums = ['d808eb5b1823c572cb45a97c95a3c5acb3d8e29aa47ec74e3ca1eb345787c17b'] - -start_dir = 'src' - -# -fcommon is required to compile Subread 2.0.1 with GCC 10, -# which uses -fno-common by default (see https://www.gnu.org/software/gcc/gcc-10/porting_to.html) -buildopts = '-f Makefile.Linux CFLAGS="-fast -fcommon"' - -files_to_copy = ['bin'] - -sanity_check_paths = { - 'files': ['bin/featureCounts', 'bin/subread-align'], - 'dirs': ['bin/utilities'], -} - -sanity_check_commands = ["featureCounts --version"] - -moduleclass = 'bio' -``` - -Do you spot any potential problems yet with this easyconfig file? - -***Preparation*** - -Start by copying the text above in a file named `subread.eb` -(which does not follow the EasyBuild conventions but that is not a problem for this exercise), -so you can gradually fix the problem you'll encounter. - -Also make sure that the pre-installed software stack is available, -and that the EasyBuild-user module is loaded (unless you installed EasyBuild -yourself): - -``` -module load LUMI/21.12 -module load EasyBuild-user -``` - -This will configure EasyBuild correctly for this exercise, though if you already have -an existing EasyBuild user installation you may want to work in a different one -by pointing `$EBU_USER_PREFIX` to the desired work directory before loading -`LUMI/21.12`. - -Check your configuration via `eb --show-config`. - -Strictly speaking the configuration doesn't matter much for the sake of this -exercise, but it may help with the step-wise approach we'll take and -grasping the solutions. - -Remember though: *no peeking* before you tried to solve each step yourself! - ---- - -***Exercise T.1**** - Sources* - -Try to install the `subread.eb` easyconfig file, see what happens. - -Can you fix the problem you run into, perhaps without even changing -the easyconfig file? - -??? success "(click to show solution)" - - The installation fails because the source file `subread-2.0.1-source.tar.gz` - is not found: - ``` - $ eb subread.eb - ... - == FAILED: Installation ended unsuccessfully (build directory: /run/user/XXXXXXXX/easybuild/build/Subread/2.0.1/cpeGNU-21.12): build failed (first 300 chars): - Couldn't find file subread-2.0.1-source.tar.gz anywhere, and downloading it didn't work either... - Paths attempted (in order): ... - ``` - - In this case, the problem is that the easyconfig file does not specify - where the sources can be downloaded from. Not automatically at least, - but there is a helpful comment included: - ```python - # download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz - sources = ['subread-%(version)s-source.tar.gz'] - ``` - - We can download the source tarball ourselves, - and move it to the location where EasyBuild expects to find it - (in the `sourcepath` directory): - ``` - curl -OL https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz - mkdir -p $EBU_USER_PREFIX/sources/s/Subread - mv subread-2.0.1-source.tar.gz $EBU_USER_PREFIX/sources/s/Subread/ - ``` - - Or, we can change the easyconfig file to specify the location where - the easyconfig file can be downloaded from: - ```python - source_urls = ['https://download.sourceforge.net/subread/'] - sources = ['subread-%(version)s-source.tar.gz'] - ``` - Note that the `source_urls` value is a *list* of candidate download URLs, - *without* the filename of the source file itself. - - This way, EasyBuild will download the source file when running `eb subread.eb`. - - The source tarball is fairly large (23MB), so don't be alarmed if the download takes a little while. - - ```shell - $ ls -lh $EBU_USER_PREFIX/sources/s/Subread - total 23M - -rw-rw-r-- 1 XXXXXXXX XXXXXXXX 23M Mar 30 16:08 subread-2.0.1-source.tar.gz - ``` - ---- - -***Exercise T.2**** - Toolchain* - -After fixing the problem with missing source file, try the installation again. - -What's wrong now? How can you fix it quickly? - -Take into account that we just want to get this software package installed, -we don't care too much about details like the version of the dependencies or -the toolchain here... - - -??? success "(click to show solution)" - - The installation fails because the easyconfig specifies that GCC 8.5.0 - should be used as toolchain: - - ``` - $ eb subread.eb - ... - ERROR: Failed to process easyconfig /pfs/lustrep3/users/kurtlust/easybuild-tutorial/Troubleshooting/subread.eb: Toolchain PrgEnv-gnu not found, - available toolchains: ... - ... - ``` - - `PrgEnv-gnu` is an HPE Cray PE module that may look like a toolchain - it certainly has - the same function: provide compiler, MPI and basic math libraries - but it is not - recognised as a toolchain by EasyBuild. EasyBuild prefers to manage its own modules so that it knows - well what is in it which is not the case with the `PrgEnv-*` modules from the Cray PE - as the content may differe between systems and as the versions of the compilers etc. that - are loaded differ on other modules that are loaded. Hence we created Cray-specific toolchains. - You'll actually find two series of Cray toolchains in the list of available toolchains. The - `CrayGNU`, `CrayIntel`, `CrayPGI` and `CrayCCE` are included with the EasyBuild distribution - and where developed by CSCS for their systems using Environment Modules. These were not compatible - with the initial releases of the Cray PE with Lmod modules so new ones were developed on which we - also built for the LUMI toolchains. Those are called `cpeCray`, `cpeGNU`, `cpeAOCC` and `cpeAMD` - and are maintained by LUST and available via the LUMI repositories. - - Changing the toolchain name to `cpeGNU` is not enough to solve all problems though: - - ``` - $ eb subread.eb - ... - ERROR: Build of /pfs/lustrep3/users/kurtlust/easybuild-tutorial/Troubleshooting/subread.eb failed (err: 'build failed (first 300 chars): - No module found for toolchain: cpeGNU/21.10') - ... - ``` - - We don't have this `cpeGNU` version installed, but we do have GCC 21.12: - - ```shell - $ module avail cpeGNU/ - ----- Infrastructure modules for the software stack LUMI/21.12 on LUMI-L ----- - cpeGNU/21.12 - ... - ``` - - So let's try using that instead. - - Edit the easyconfig file so it contains this: - - ```python - toolchain = {'name': 'cpeGNU', 'version': '21.12'} - ``` ---- - -***Exercise T.3**** - Build step* - -With the first two problems fixed, now we can actually try to build the software. - -Can you fix the next problem you run into? - -??? success "(click to show solution)" - - The compilation fails, but the error message we see is incomplete due to - EasyBuild truncating the command output (only the 300 first characters of the output are shown): - ``` - == FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12): build failed - (first 300 chars): cmd " make -j 256 -f Makefile.Linux CFLAGS="-fast -fcommon"" exited with exit code 2 and output: - gcc -mtune=core2 -O3 -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\""2.0.1"\" -D_FILE_OFFSET_BITS=64 -fmessage-length=0 - -ggdb -fast -fcommon -I/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86 (took 4 secs) - ``` - - If you open the log file (e.g., with `view $(eb --last-log)`) and scroll to the end, - the error is pretty obvious: - ``` - gcc: error: unrecognized command line option -fast; did you mean -Ofast? - make: *** [core.o] Error 1 - ``` - - The easyconfig file hard specifies the `-fast` compiler flag via the `CFLAGS` argument to the build command: - ```python - # -fcommon is required to compile Subread 2.0.1 with GCC 10, - # which uses -fno-common by default (see https://www.gnu.org/software/gcc/gcc-10/porting_to.html) - buildopts = '-f Makefile.Linux CFLAGS="-fast -fcommon"' - ``` - - EasyBuild sets up the build environment, so there should be no need - to hard specify compiler flags (certainly not incorrect ones). - The comment above the `buildopts` definition makes it clear that the `-fcommon` - flag *is* required though, because GCC 10 became a bit stricter by - using `-fno-common` by default (and we're using GCC 11 in `cpeGNU/21.12`). - Note that we are using `-fcommon` - as an escape mechanism here: it would be better to fix the source code - and create a patch file instead. - - An easy way to fix this problem is to replace the `-fast` with `-Ofast`, - as the compiler error suggests. - - In this case it is advised to change the `CFLAGS` argument that is added - to be build command to replace the `-fast` with `$CFLAGS`, - which is defined in the build environment by EasyBuild. - - ```python - buildopts = '-f Makefile.Linux CFLAGS="$CFLAGS -fcommon"' - ``` - - Note that we need to be careful with quotes here: we use inner double quotes - to ensure that `$CFLAGS` will be expanded to its value when the build command is run. - As you can see after the change by doing a dry-run: - - ``` - $ eb subread.eb -x - ... - - Defining build environment... - - export BLAS_INC_DIR='/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/include' - ... - export CC='cc' - export CFLAGS='-O2 -ftree-vectorize -fno-math-errno' - ... - [build_step method] - running command "make -j 256 -f Makefile.Linux CFLAGS="$CFLAGS -fcommon"" - (in /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12/Subread-2.0.1/src) - ... - ``` - - EasyBuild will launch the command - ``` - make -j 256 -f Makefile.Linux CFLAGS="$CFLAGS -fcommon" - ``` - in a shell where `CFLAGS` is defined and set to an appropriate value (determined by - defaults in EasyBuild, settings in the EasyBuild configuration and settings in the - easyconfig file that we shall discuss later). - ---- - -***Exercise T.4**** - Sanity check* - -After fixing the compilation issue, you're really close to getting the installation working, we promise! - -Don't give up now, try one last time and fix the last problem that occurs... - -??? success "(click to show solution)" - - Now the installation itself works but the sanity check fails, - and hence the module file does not get generated: - ``` - $ eb subread.eb - ... - == FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12): - build failed (first 300 chars): Sanity check failed: sanity check command featureCounts --version exited with code 255 - (output: featureCounts: unrecognized option '--version' - ... - ... - ``` - - If you look at the full output in the log file you can see - that the correct option to check the version of the `featureCounts` command - is "`-v`" rather than "`--version`", so we need to fix this in the easyconfig file. - - Make the following change in the easyconfig file: - ```python - sanity_check_commands = ["featureCounts -v"] - ``` - - After doing so, **you don't have to redo the installation - from scratch**, you can use the `--module-only` option to only run the - sanity check and generate the module file again: - ``` - eb subread.eb --module-only - ``` - ---- - -In the end, you should be able to install Subread 2.0.1 with the cpeGNU 21.12 toolchain by fixing the problems with the `subread.eb` easyconfig file. - -Check your work by manually loading the module and checking the version -via the `featureCounts` command, which should look like this: - -```shell -$ module load Subread/2.0.1-cpeGNU-21.12 -... -$ featureCounts -v -featureCounts v2.0.1 -``` - ---- - -*[[next: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md deleted file mode 100644 index f9bec25a..00000000 --- a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md +++ /dev/null @@ -1,1027 +0,0 @@ -# Creating easyconfig files - -*[[back: Troubleshooting]](2_02_troubleshooting)* - ---- - -We already know how to install easyconfig files that are provided by LUMI, -but what about installing software for which no easyconfig is available yet, -or for which we need to adapt an existing easyconfig? - -To do this we will need to **create additional easyconfig files**, -since every software installation performed by EasyBuild is done based on an easyconfig file. - -In this part of the tutorial we will look at the guts of easyconfig files and even create some ourselves! - -## Easyconfigs vs easyblocks - -Before we dive into writing [easyconfig files](../1_05_terminology/#easyconfig-files), -let us take a brief look at how they relate to [easyblocks](../1_05_terminology/#easyblocks). - -As we discussed [earlier](../1_05_terminology/#terminology), an easyconfig file (`*.eb`) is required -for each installation -performed by EasyBuild which specifies the details of the installation (which software -version, toolchain, etc.), while the installation procedure is implemented -in an easyblock (a Python module). - -When can we leverage a *generic easyblock*, perhaps via a "fat" easyconfig file that includes -a lot of carefully defined easyconfig parameters, and when should we use a minimal easyconfig file -together with a custom *software-specific* easyblock? - -This is not an easy question to answer in a general sense, since it depends on several factors: -the complexity of the software you want to get installed, how much flexibility you want, -how "intelligent" the installation procedure should be with respect to the compiler toolchain and dependencies -that are used for the installation, whether you want an easyconfig that is more user-focused (with easy-to-understand -parameters) or one that is more focused towards people who understand installation commands, etc. - -In a nutshell, custom software-specific easyblocks are "do once and forget": they are central solution to peculiarities in the installation procedure of a particular software package. However, they also hide a lot from direct view, so if things go -wrong, it is often harder to debug the exact problem. And since a single easyblock has to cover -multiple software versions, multiple toolchains and multiple possible configurations of the package, -they are often harder to develop and certainly harder to test. As a result many of the easyblocks -included with EasyBuild work poorly on HPE Cray systems, e.g., because they want to add compiler flags -specific for a certain compiler and don't recognise the Cray compilers. - -Reasons to consider implementing a software-specific easyblock rather than using a generic easyblock include: - -- 'critical' values for easyconfig parameters required to make installation succeed; -- toolchain-specific aspects of the build and installation procedure (e.g., configure options); -- interactive commands that need to be run; - *For example: The easyblock to [install maple](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/m/maple.py) - responds to a number of interactive questions.* -- custom (configure) options for dependencies; - *For example, the [easyblock for VMD](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/v/vmd.py) - will add configure options depending on the dependency list of the package.* -- having to create or adjust specific (configuration) files; -- 'hackish' usage of a generic easyblock; -- complex or very non-standard installation procedure; - *For example, the [easyblock to install the gcc compilers from source](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/g/gcc.py), - bootstrapping with the system compiler and then re-installing with itself.* - -For implementing easyblocks we refer to the -["Implementing easyblocks" section of this tutorial](2_03_implementing_easyblocks) -and the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). - -## Writing easyconfig files - -Writing an easyconfig file boils down to defining a set of easyconfig parameters in a text file, -which we give a '`.eb`' extension by convention. The name of an easyconfig file doesn't matter -when using it directly to install software, but it does matter when EasyBuild needs to find it -to resolve a dependency for example (as we [discussed earlier](../1_08_basic_usage/#easyconfig-filenames)). - -The syntax for easyconfig files is *Python syntax*: you are basically defining a bunch of Python variables -that correspond to easyconfig parameters. - -The order in which the easyconfig parameters are defined doesn't matter, but we generally try to stick to a particular -order which roughly corresponds to the order in which the easyconfig parameters are used during the installation. -That is mostly helpful for humans staring at easyconfig files or comparing them. - -### Mandatory parameters - -A limited number of easyconfig parameters are *mandatory*, they must be defined in every easyconfig file: - -- `name`: the name of the software to install; -- `version`: the version of the software to install; -- `homepage`: a URL to the website of the software; -- `description`: a description of the software; -- `toolchain`: the compiler toolchain to use for the installation; - -#### `name`, `version` - -It should be no surprise that specifying the name and version of the software you want to install is mandatory. -This information may influence the value of several other easyconfig parameters (like the name of the source file), and is also used to the determine the name of the module file to install. - -```python -name = 'example' -version = '1.0' -``` - -#### `homepage`, `description` - -The homepage and description are included in the generated module file for the installation. -That way the "`module show`" command provides some useful high-level information about the installation. - -```python -homepage = 'https://example.org' -description = "This is just an example." -``` - -Usually it does not matter whether you use single or double quotes to specify string values. -There are cases where it is important to use -the right type of quotes however, we will get back to that later (keep it in mind for the exercises!). - -For multi-line descriptions, you will need to use "triple quoting" (which is standard Python syntax): - -```python -description = """This is an example - of a multi-line description. - It is spread across multiple lines.""" -``` - -If no homepage is known for the package, the convention in the EasyBuild community is to use -`(none)` as the value. - -The `description` field is used in two different places in the LUA module file generated by -EasyBuild: -- In the help block, for the first section of the help information -- If there is no `whatis` parameter in the easyconfig file, it is also used to generate - a whatis-command with the description in the module file. - -*Note:* We ask people who contribute easyconfigs to LUMI to instead use a decent description -that is useful in the help information, and to specify a short description via the `whatis` -parameter: - -```python -whatis = ['Description: Blosc is an extremely fast, multi-threaded, meta-compressor library'] -``` - -Note that in this case the word "Description:" has to be explicitly specified. - - -#### `toolchain` - -EasyBuild also requires that the [compiler toolchain](../1_05_terminology/#toolchains) is specified, via the `toolchain` -easyconfig parameter. - -This can either be the [`system` toolchain](../1_05_terminology/#system-toolchain), for which a constant named `SYSTEM` is available: - -```python -toolchain = SYSTEM -``` - -Usually we specify a 'proper' toolchain like the `cpeGNU/21.12` toolchain we have used bvefore. -The name and version of the toolchain can be specified using a small Python dictionary, -for example: - -```python -toolchain = {'name': 'cpeGNU', 'version': '21.12'} -``` - -### Commonly used parameters - -You will often need to specify additional easyconfig parameters to get something useful done. -We will cover the most commonly used ones here, but keep in mind that these are *not* mandatory. - -A full overview of all known easyconfig parameters can be obtained via "`eb --avail-easyconfig-params`" -or just "`eb -a`" for short, or can be consulted in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_parameters.html). - -#### Sources, patches, and checksums - -In most easyconfig files you will see that a list of source files is specified via the `sources` -easyconfig parameter, usually combined -with one or more URLs where these sources can be downloaded specified via `source_urls`. -There also may be patch files listed (specified via `patches`), -and checksums for both the source files and patches (specified via `checksums`). - -The `sources` easyconfig parameter is commonly defined but it is *not* mandatory, -because some easyconfig files only specify bundles of software packages and hence only -serve to generate a module file. - -Here is an example of how these easyconfig parameters can be specified: - -```python -source_urls = [ - 'https://example.org/download/', - 'https://example.org/download/archive/', -] -sources = ['example-1.0-src.tar.gz'] -patches = ['example-fix.patch'] -checksums = [ - '9febae18533d035ac688d977cb2ca050e6ca8379311d7a14490ad1ef948d45fa', - '864395d648ad9a5b75d1a745c8ef82b78421d571584037560a22a581ed7a261c', -] -``` - -Each of these require a *list* of values, so even if there is only a single source file or download URL -you must use square brackets as shown in the example. The default value for each of these is an empty list (`[]`). - -Some things worth pointing out here: - -* The download URLs specified via `source_urls` do *not* include the name of the file, that is added - automatically by EasyBuild when it tries to download the file (only if it's not available already.) -* If multiple download URLs are specified, they are each tried once in order until the download of the source file was - successful. This can be useful to include backup locations where source files can be downloaded from. -* Names of source files and patches should not include hardcoded software versions, they usually use a - template value like `%(version)s` instead: - ```python - sources = ['example-%(version)s-src.tar.gz'] - ``` - EasyBuild will use the value of the `version` easyconfig parameter to determine the actual name of the source - file. This way the software version is only specified in one place and the easyconfig file is easier to - update to other software versions. A list of template values can be consulted via the EasyBuild command - line via the `--avail-easyconfig-templates` option, or in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_templates.html). -* Source files can also be specified in ways other than just using a filename, see the - [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-param-sources-alt) for more information. - It is also possible to download a given commit from a GitHub repository. -* Specified checksums are usually SHA256 checksum values, but - [other types are also supported](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html?highlight=checksums#checksums). - - -#### Easyblock - -The easyblock that should be used for the installation can be specified via the `easyblock` easyconfig parameter. - -This is not mandatory however, because by default EasyBuild will determine the easyblock to use based on the -name of the software. If '`example`' is specified as software name, EasyBuild will try to locate a -software-specific easyblock named `EB_example` (in a Python module named `example.py`). Software-specific -easyblocks follow the convention that the class name starts with `'EB_`', followed by the software name -(where some characters are replaced, like '`-`' with '`_minus_`'). It is possible to use different -naming conventions for software-specific easyblocks, but then EasyBuild will not automatically detect -that there is one for the package and it will also need to be specified via the `easyblock` parameter. - -**Generic easyblocks** - -Usually the `easyblock` value is the name of a *generic* easyblock, if it is specified. The name of -a generic easyblock does *not* start with '`EB_`', so you can easily distinguish it from a software-specific -easyblock. - -Here are a couple of commonly used generic easyblocks: - -* `ConfigureMake`: implements the standard `./configure`, `make`, `make install` installation procedure; -* `CMakeMake`: same as `ConfigureMake`, but with `./configure` replaced with `cmake` for the configuration step; -* `PythonPackage`: implements the installation procedure for a single Python package, by default using - "`python setup.py install`" but other methods like using "`pip install`" are also supported; -* `Bundle`: a simple generic easyblock to bundle a set of software packages together in a single installation directory; -* `PythonBundle`: a customized version of the `Bundle` generic easyblock to install a bundle of Python packages - in a single installation directory; - -A full overview of the available generic easyblocks is available in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/generic_easyblocks.html). You can also consult the output of -`eb --list-easyblocks`, which gives an overview of *all* known easyblocks, and how they relate to each other. - -**Custom easyconfig parameters** - -Most generic easyblocks provide additional easyconfig parameters to steer their behaviour. -You can consult these via "`eb -a --easyblock`" or just "`eb -a -e`", which results in an -additional "`EASYBLOCK-SPECIFIC`" section to be added. See the (partial) output of this command for example: - -```shell -$ eb -a -e ConfigureMake -Available easyconfig parameters (* indicates specific to the ConfigureMake easyblock): -... -EASYBLOCK-SPECIFIC ------------------- -build_cmd* Build command to use [default: "make"] -build_type* Value to provide to --build option of configure script, e.g., x86_64-pc-linux-gnu (determined by config.guess shipped with EasyBuild if None, False implies to leave it up to the configure script) [default: None] -configure_cmd* Configure command to use [default: "./configure"] -configure_cmd_prefix* Prefix to be glued before ./configure [default: ""] -host_type* Value to provide to --host option of configure script, e.g., x86_64-pc-linux-gnu (determined by config.guess shipped with EasyBuild if None, False implies to leave it up to the configure script) [default: None] -install_cmd* Build command to use [default: "make install"] -prefix_opt* Prefix command line option for configure script ('--prefix=' if None) [default: None] -tar_config_opts* Override tar settings as determined by configure. [default: False] -``` - -#### Dependencies - -You will often need to list one or more [dependencies](../1_05_terminology/#dependencies) that are required -to install or run the software. -We distinguish between two main different types of dependencies: runtime dependencies and build dependencies. - -*Runtime dependencies* are required for using the installed software, and may also have to be available -during the installation. These dependencies can be specified via the `dependencies` easyconfig parameter. -EasyBuild will load the modules for these dependencies when setting up the build environment, -and will include load statements for them in the generated module file. - -*Build dependencies* are only required during the installation of the software, not for using the -software once it is installed. The modules for these dependencies will be loaded in the build environment -set up by EasyBuild during the installation, but they will *not* be loaded by the generated module file. -You can specify build dependencies via the `builddependencies` easyconfig parameter. -One typical example of a build dependency is `CMake`, which is only needed for configuring -the build. On LUMI we define a `buildtools` module for each version of the `LUMI` software stack -which contains most popular build tools. This ensures that we use a consistent set of build tools -and that they do not change over the life time of a stack, also in case of OS upgrades, to -increase the reproducibility of the build process - -Here is a simple example of specifying dependencies: - -```python -builddependencies = [ - ('buildtools', '%(toolchain_version)s', '', True) -] - -dependencies = [ - ('Python', '3.8.2'), - ('HDF5', '1.10.6'), - ('SciPy-bundle', '2020.03', '-Python-%(pyver)s'), -] -``` - -Both `builddependencies` and `dependencies` require a list of tuples, -each of which specifying one dependency. -The name and version of a dependency is specified with a 2-tuple (a tuple with two string values). - -In some cases additional information may have to be provided, as is shown in the example above for the `SciPy-bundle` -dependency where a 3rd value is specified corresponding to the `versionsuffix` value of this dependency. -If this is not specified, it is assumed to be the empty string (`''`). - -Note how we use the '`%(pyver)s'` template value in the `SciPy-bundle` dependency -specification, to avoid hardcoding the Python version in different places. (Though this -specific parameter is less useful on LUMI as we currently try to build on top of `cray-python`.) - -The `buildtools` build dependency shows that there is a fourth parameter specifying the toolchain -used for that dependency and is needed if that toolchain is different from the one used in the example. -As it is not possible to load several Cray toolchains together (they are not in a hierachical relation) -the only useful value on LUMI is `True` which tells that `buildtools` is build with the `SYSTEM` -toolchain. Here also we use a template, `%(toolchain_version)s` which - as its name suggests - expands -to the version of the toolchain, as we version our `buildtools` modules after the version of the Cray -toolchains for which they are intended. - -See also the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#dependencies) -for additional options on specifying dependencies. That page specifies two more dependency types: - -* `hiddendependencies` are currently not used on LUMI, and if we would use them in the future, it will likely - be through a way that does not require this parameter. -* `osdependencies` can be used to let EasyBuild check if certain needed OS packages are installed. - See, e.g., the [easyconfigs for the `buildtools` package](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/easybuild/easyconfigs/b/buildtools) - on LUMI. - - -#### Version suffix - -In some cases you may want to build a particular software package in different configurations, -or include a label in the module name to highlight a particular aspect -of the installation. - -The `versionsuffix` easyconfig parameter can be used for this purpose. -The name of this parameter implies that this label will be added after the -software version (and toolchain label) in the standard module naming scheme. - -If you are configuring the software to build with a particular non-default value, -you can indicate this as follows for example: - -```python -versionsuffix = '-example-label' -``` - -This mechanism is used frequently to indicate that a software installation depends -on a particular version of Python, where the `%(pyver)s` template comes in useful again: - -```python -versionsuffix = '-Python-%(pyver)s' -... -dependencies = [('Python', '2.7.18')] -``` - -Even though Python 2 is officially dead and -buried some scientific software still requires it, and mixing modules where -some use Python 2 and other use Python 3 doesn't work well. -The `versionsuffix` label is helpful to inform the user that a particular Python version is -required by the installation. - -#### Customizing configure, build, test, and install commands - -When using a generic easyblock like `ConfigureMake` or `CMakeMake`, you will often -find yourself having to specify options to the configure, build, test, or install commands, or to inject additional commands right before them. - -For this the following standard easyconfig parameters are available: - -* `preconfigopts`: string value that is glued *before* the configure command; -* `configopts`: string value that is added *after* the configure command, which can be used to specify configuration options; - -Equivalent easyconfig parameters are available for the `build`, `test` and `install` steps: `prebuildopts`, `buildopts`, `pretestopts`, `testopts`, `preinstallopts`, and `installopts`. - -Here is a fictitious example of how they can be used: - -```python -easyblock = 'ConfigureMake' -... -dependencies = [('HDF5', '1.10.6')] -... -configopts = '--enable-hdf5-support' - -prebuildopts = 'export HDF5_PREFIX="$EBROOTHDF5" && ' - -installopts = "PREFIX='%(installdir)s'" -``` -(again an imaginary example as on LUMI we advise to use the Cray-provided HDF5 modules.) - -Here we are: - -* Adding the `--enable-hdf5-support` configure option, to convince the `ConfigureMake` easyblock to run the following - command during the configure step: - ```shell - ./configure --prefix ... --enable-hdf5-support - ``` - (where the '`...`' represents the path to installation directory where the software should be installed). - -* Specifying that an additional command has to be run before running `make` in the build step. We use '`&&`' to glue the - command to the `make` command, so `make` will only be run if the command we specified ran correctly. So, the build step will run something like: - ```shell - export HDF5_PREFIX="$EBROOTHDF5" && make -j 4 - ``` - The '`4`' value passed to the `-j` option shown here, which specifies how many commands `make` can run in parallel, is automatically determined by EasyBuild based on the number of available cores (taking into account `ulimit` settings, and cpuset and cgroup restrictions). - -* Passing the location where the software should be installed via the `PREFIX` argument to the `make install` command during the installation step. This results in the - following command being run: - ```shell - make install PREFIX=... - ``` - (where the '`...`' again represents the path to installation directory). - Even though the - installation directory is already specified in the configure command, it is - apparently blatantly ignored by the software we are installing here, and we are expected to specify it - this way instead. How rude! - -The `$EBROOTHDF5` environment variable that we are using in `prebuildopts` corresponds to the path of -the installation directory of the HDF5 dependency. EasyBuild includes a statement -to define an `$EBROOT*` environment variable -like this in every environment module file it generates (see the output of "`module show HDF5`"). - -#### Sanity check - -One seemingly trivial yet important aspect of the installation procedure that EasyBuild performs -is the sanity check step. - -By default EasyBuild does a simple sanity check that verifies whether there is a non-empty `bin` subdirectory -in the installation, next to a non-empty `lib` or `lib64` directory (either is sufficient). - -It is recommended to customize the sanity check and check for something more specific, like a particular -binary or directory, or making sure that a trivial command (like `example -V` or `example --help`) -runs correctly. - -To specify a custom set of files and/or directories to check, -you can use the `sanity_check_paths` easyconfig parameter. The expected value is Python dictionary -with two keys: `files` and `dirs`. For example: - -```python -sanity_check_paths = { - 'files': ['bin/example'], - 'dirs': ['examples/one', 'examples/two'], -} -``` - -In addition, you can specify one or more commands that should be working without a problem (that is, have a zero exit status) via the `sanity_check_commands` easyconfig parameter. -These commands will be run just like a user would: after loading the module that was generated for this installation. -Here is an example: - -```python -sanity_check_commands = [ - "example --version", - "example --help", -] -``` - - -#### Module class - -Finally, you will usually see the `moduleclass` easyconfig parameter to be defined as well, for example: - -```python -moduleclass = 'lib' -``` - -This is done to categorize software, and it is used to group the generated module files into smaller sets -([remember what we saw when installing software earlier](../1_08_basic_usage/#using-installed-software)). - -This is currently not used on LUMI since we feel that (a) it is not easy to explain to users how they can -then only make certain classes that are useful to them visible and more importantly (b) since it is not -always intuitive to decide which moduleclass should be used for a package, or from a user 's perspective, -in which category to look for a package. - -## Generating tweaked easyconfigs - -Sometimes you may want to install software that differs only slightly from an -existing easyconfig file, like a newer software version or using a different -compiler toolchain. Do we need to create an easyconfig file for this too? - -We do, but EasyBuild does provide some help so you don't need to *manually* -create the easyconfig file. You can use one of the `--try-*` options provided -by the `eb` command to make EasyBuild *generate* a new easyconfig file based on -an existing one. - -For example, to try installing a different software version you can use the `--try-software-version` option: - -```shell -eb example-1.2.3.eb --try-software-version 1.2.4 -``` - -Or, to try using a different compiler toolchain you can use `--try-toolchain`: - -```shell -eb example-1.2.3-foss-2020b.eb --try-toolchain cpeCray,21.12 -``` - -It is important to keep in mind the *"try"* aspect here: while easyconfigs that -are generated by EasyBuild via a `--try-*` option often do work fine, there is -no strong guarantee they will. Newer software versions may come with changes to -the installation procedure, additional dependencies that are required, etc. -Using a different compiler toolchain may be as simple as just switching one for -another, but it may require additional changes to be made to configure options, for example. - -## Copying easyconfigs - -One additional handy command line option we want to highlight is `--copy-ec`, which can be used to -copy easyconfig files to a specific location. That may sound trivial, but -keep in mind that you can specify easyconfigs to the `eb` command using only -the filename, and letting the robot search mechanism locate them. - -So to copy an easyconfig file, we would have to use `eb --search` first to -get the full location to it, copy-paste that, and then use the `cp` command. - -It is a lot easier with `--copy-ec`: - -```shell -$ eb --copy-ec SAMtools-1.11-GCC-10.2.0.eb SAMtools.eb -... -SAMtools-1.10-GCC-10.2.0.eb copied to SAMtools.eb -``` - -If you omit the target location, the easyconfig file will simply be copied -to the current working directory, retaining the original filename. - -You can copy multiple easyconfig files at once, as long as the target location -is an existing directory. - -## Example - -By means of example, we are going to puzzle together an easyconfig file to install the -example software package `eb-tutorial`. - -The sources for `eb-tutorial` version 1.0.1 are available at: - -``` -https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.0.1.tar.gz -``` - -You can consult the unpacked sources at [https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1). - -### Preparation - -Make sure that a proper version of the `LUMI` software stack is loaded -(we recommend `LUMI/21.12` for the session for which this tutorial was designed) -and also `EasyBuild-user` is loaded to configure EasyBuild to install in -`$HOME/EasyBuild` or `$EBU_USER_PREFIX`. - -``` shell -module load LUMI/21.12 -module load EasyBuild-user` -``` - - -### Easyblock - -Let's start by getting the mandatory easyconfig parameters defined in the easyconfig file: - -```python -name = 'eb-tutorial' -version = '1.0.1' - -homepage = 'https://easybuilders.github.io/easybuild-tutorial' - -whatis = [ 'Description: EasyBuild tutorial example'] - -description = """ -This is a short C++ example program that can be buid using CMake. -""" -``` - -Let's see what EasyBuild does with this very minimal easyconfig file: - -```shell -$ eb eb-tutorial.eb -== Temporary log file in case of crash /run/user/XXXXXXXXX/easybuild/tmp/eb-k_82wotb/easybuild-wg0k_reb.log -ERROR: Failed to process easyconfig /pfs/lustrep4/users/XXXXXXXX/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v1.eb: -No software-specific easyblock 'EB_eb_minus_tutorial' found for eb-tutorial -``` - -It is not mandatory to specify an easyblock in the easyconfig. However, in the absense of that -specification, EasyBuild goes looking for an application-specific easyblock with the standard name, -in this case `EB_eb_minus_tutorial`, which it does not have. Does that mean we have to implement an easyblock? - -In this simple case it doesn't, since we can leverage one of the available *generic easyblocks*. -But, which one? - -Build instructions are usually included in a `README` file, or in the documentation. -In this case, there's indeed a minimal [`README` -file](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1/README) available, -which tells us that we should use the `cmake` command to configure the installation, followed by `make` and `make install`. - -[We briefly discussed](#easyblock) a generic easyblock that does exactly this: `CMakeMake`. - -```python -easyblock = 'CMakeMake' -``` - -The "`easyblock =`" line is usually at the top of the easyconfig file, but strictly speaking -the order of the parameter definitions doesn't matter (unless one is defined in terms of another one). - - -### Mandatory parameters - -When trying this improved easyconfig file, EasyBuild will inform us that we failed to specify -one of the mandatory easyconfig parameters: `toolchain`: - -``` -$ eb example.eb -== Temporary log file in case of crash /run/user/XXXXXXXXX/easybuild/tmp/eb-22om7hut/easybuild-ipozjipf.log -ERROR: Failed to process easyconfig /pfs/lustrep4/users/XXXXXXXX/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v2.eb: -mandatory parameters not provided in pyheader: toolchain -``` - -We will use `cpeCray/21.12` as toolchain, so we also define the `toolchain` easyconfig parameter: - -```python -toolchain = {'name': 'cpeCray', 'version': '21.12'} -``` - -In addition, we'll also specify the `moduleclass`. -This is not required, but it is usually set to a sensible value: - -```python -moduleclass = 'tools' -``` - -The default value is '`base`', at least '`tools`' has *some* meaning. - -### CMake build dependency - -The `CMakeMake` easyblock needs the `cmake` command. On LUMI we are lucky as -`cmake` is already installed in the OS. It may be a somewhat older version, but for this -program is is probably enough. Yet in general it is better to use sufficiently recent -build tools, and `cmake` is one of those tools that is typically entered as a build -dependency. After all, the less you rely on the OS, the more likely it becomes that -your easyconfig is useful for other sites also. - -In the [section on Lmod](../1_02_Lmod#module-extensions) we've already seen that on LUMI -the `cmake` command is available through the `buildtools` modules, and as discussed in -other examples on this page, LUMI has one for every `LUMI` software stack with its version -number the same as the stack and corresponding toolchains. It is a good practice to -add this module as a build dependency: - -```python -builddependencies = [ - ('buildtools', '%(toolchain_version)s', '', True) -] -``` - -In a more traditional EasyBuild setup with the common toolchains, there is usually no need to specify -the toolchain for (build) dependencies. EasyBuild will automatically consider -[subtoolchains](../1_05_terminology/#toolchains) compatible with the specified toolchain to locate -modules for the dependencies. However, the Cray PE toolchains on LUMI are currently not part of -such a hierarchy and the `SYSTEM` toolchain we used for `buildtools` is not automatically considered -which is why we need the 4-element version of the dependency specification. - -You can verify that EasyBuild now locates the dependency via `eb -D` (equivalent with `eb --dry-run`): - -``` -$ eb eb-tutorial.eb -D - ... - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeCray/cpeCray-21.12.eb (module: cpeCray/21.12) - * [ ] /pfs/lustrep4/users/kulust/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v4.eb (module: eb-tutorial/1.0.-cpeCray-21.12) -``` - -### Sources - -If you try again after adding `buildtools` as a build dependency, you will see the installation fail again in the -configuration step. Inspecting the log file reveals this: - -``` -CMake Error: The source directory "/run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12" does not appear to contain CMakeLists.txt. -``` - -Wait, but there *is* a `CMakeLists.txt`, we can see it in the -[unpacked sources](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1)! - -Let's inspect the build directory: - -``` -$ ls /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12 -easybuild_obj -$ ls /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12/easybuild_obj -$ -``` - -There's nothing there at all! And that's not strange because we didn't actually -specify any sources in our easyconfig file... - -The `sources` easyconfig parameter is commonly defined but it is *not* mandatory, -because some easyconfig files only specify bundles of software packages and hence only -serve to generate a module file. - -So we need to specify one or more source files that should be used, -via the `sources` easyconfig parameter which specifies a *list* of -names of source files: - -```python -sources = ['eb-tutorial-1.0.1.tar.gz'] -``` - -We can avoid hardcoding the version number here by using a *template value*: - -```python -sources = ['eb-tutorial-%(version)s.tar.gz'] -``` - -And since this is a standard way of naming software files, there's -even a constant available that we can use: - -```python -sources = [SOURCE_TAR_GZ] -``` - -That way, we only have the software version specified *once* in the easyconfig file, -via the `version` easyconfig parameter. That will come in useful later (see [Exercise 7.2](#exercises))... - -If now we try installing the easyconfig file again, EasyBuild complains -that it can't find the specified source file anywhere: - -``` -Couldn't find file eb-tutorial-1.0.1.tar.gz anywhere, and downloading it didn't work either... -Paths attempted (in order):... -``` - -To let EasyBuild automatically download the source file if it is not available yet, -we have to specify *where* it can be downloaded. This is done via `source_urls`: - -```python -source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] -sources = [SOURCE_TAR_GZ] -``` - -### Required configure option - -With `sources` and `source_urls` defined, we can try again. Yet again we see the configure step fail. -Is this a ruse to make you hate CMake with a passion? Maybe... - -Here's what we find in the log file: - -``` -CMake Error at CMakeLists.txt:7 (message): - EBTUTORIAL_MSG is not set! -``` - -Apparently the `eb-tutorial` software has a required configure option. It's almost as if that -was done on purpose, how silly! - -Options to the configure command can be specified by the `configopts` easyconfig parameter. -To define the value of a CMake option, we need to use `-DNAME_OF_OPTION`, so: - -```python -configopts = "-DEBTUTORIAL_MSG='Hello from the EasyBuild tutorial!' " -``` - -We need to be a little bit careful with quotes here. If we use outer double quotes, -we have to use single quotes to specify the actual value for the `EBTUTORIAL_MSG` configure option. -That works fine here, but that's not always the case! -In some cases we will have to use inner doubles quotes, for example to get environment variables -expanded when the configure command is run (see [Exercise 7.1](#exercises)). - -### Sanity check - -Hopefully that brings us closer to getting the installation to work... - -``` -$ eb eb-tutorial.eb -.... -== sanity checking... -== ... (took 2 secs) -== FAILED: Installation ended unsuccessfully (build directory: /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12): -build failed (first 300 chars): Sanity check failed: no (non-empty) directory found at 'lib' or 'lib64' in -/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 (took 7 secs) -``` - -It got all the way to the sanity check step, that's great! - -The sanity check failed because no '`lib`' or `'lib64'` directory was found. -Indeed: - -``` -$ ls /users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 -bin -$ ls /users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 -eb-tutorial -``` - -There is only a binary named `eb-tutorial` in the `bin` subdirectory. -So we need to customize the standard sanity check: - -```python -sanity_check_paths = { - 'files': ['bin/eb-tutorial'], - 'dirs': [], -} -``` - -Since we want to obtain a *working* installation, we might as well try to run this `eb-tutorial` command as well: - -```python -sanity_check_commands = ['eb-tutorial'] -``` - -Let us now retry, but use `--module-only` rather than redoing the whole installation. -`--module-only` still sanity checks the installation, so if it creates -a module, we know it will work as expected. -By enabling trace mode via `--trace` we can get some more information too: - -```shell -$ eb eb-tutorial.eb --module-only --trace -... -== sanity checking... - >> file 'bin/eb-tutorial' found: OK - >> loading modules: eb-tutorial/1.0.1-cpeCray-21.12... - >> running command 'eb-tutorial' ... - >> result for command 'eb-tutorial': OK -... -== COMPLETED: Installation ended successfully (took 4 sec) -``` - -Yes, great success! - -To convince yourself that the installation works as intended, try to load the `eb-tutorial` module and -run the `eb-tutorial` command yourself: - -``` -$ module load eb-tutorial -$ eb-tutorial -Hello from the EasyBuild tutorial! -``` - -### Complete easyconfig - -Here is the complete easyconfig we puzzled together for this example: - -```python -easyblock = 'CMakeMake' - -name = 'eb-tutorial' -version = '1.0.1' - -homepage = 'https://easybuilders.github.io/easybuild-tutorial' -description = "EasyBuild tutorial example" - -toolchain = {'name': 'GCC', 'version': '10.2.0'} - -source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] -sources = [SOURCE_TAR_GZ] -checksums = ['d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9'] - -builddependencies = [('CMake', '3.18.4')] - -configopts = "-DEBTUTORIAL_MSG='Hello from the EasyBuild tutorial!' " - -sanity_check_paths = { - 'files': ['bin/eb-tutorial'], - 'dirs': [], -} - -sanity_check_commands = ['eb-tutorial'] - -moduleclass = 'tools' -``` - -One additional easyconfig parameter we defined here is `checksums`, -which specifies SHA256 checksums for source (and patch) files, -so EasyBuild can verify them before performing an installation. - -You can let EasyBuild determine *and* inject these SHA256 checksums -automatically via `eb --inject-checksums`: - -``` -$ eb eb-tutorial.eb --inject-checksums -... -== injecting sha256 checksums for sources & patches in example.eb... -== * eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 -``` - -Finally, we should consider changing the name of the easyconfig that we just developed -to align with the EasyBuild conventions as otherwise it would not be found when used as -a dependency of another package. In this case, the name should be -`eb-tutorial-1.0.0-cpeCray-21.12.eb`. In fact, EasyBuild stored a processed version -of our easyconfig with that name in the repository: - -``` -$ ls $EASYBUILD_REPOSITORYPATH -eb-tutorial -$ ls $EASYBUILD_REPOSITORYPATH/eb-tutorial -eb-tutorial-1.0.1-cpeCray-21.12.eb -$ cat $EASYBUILD_REPOSITORYPATH/eb-tutorial/eb-tutorial-1.0.1-cpeCray-21.12.eb -... -``` - -Note that EasyBuild has added an additional parameter to the easyconfig file, `buildstats`, -with a lot of information about how easybuild was called and properties of the node on which -it was run. This file is still a valid easyconfig file though from which we can build the -program again. - -Let's also inspect the installation directory when the whole build process has finished successfully. -This is very easy after loading the module, as EasyBuild-generated modules define a number of environment -variables for each module: - -``` -$ module load eb-tutorial -$ env | grep TUTORIAL -EBVERSIONEBMINTUTORIAL=1.0.1 -EBDEVELEBMINTUTORIAL=/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12/easybuild/eb-tutorial-1.0.1-cpeCray-21.12-easybuild-devel -EBROOTEBMINTUTORIAL=/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 -``` -The most interesting one of those variables is the `EBROOT` variable which points to the installation directory. -As variable names cannot contain minus signs, the minus in the module name is replaced with `MIN` in the name -of the variable (which is not the most consistent thing however as in the name of an easyblock it is replaced -with `minus`). - -Let's have a look in that directory: - -``` -$ ls $EBROOTEBMINTUTORIAL -bin easybuild -$ ls EBROOTEBMINTUTORIAL/easybuild -easybuild-eb-tutorial-1.0.1-20220401.184518.log -easybuild-eb-tutorial-1.0.1-20220401.184518_test_report.md -eb-tutorial-1.0.1-cpeCray-21.12-easybuild-devel -eb-tutorial-1.0.1-cpeCray-21.12.eb -reprod -$ ls EBROOTEBMINTUTORIAL/easybuild/reprod -easyblocks -eb-tutorial-1.0.1-cpeCray-21.12.eb -eb-tutorial-1.0.1-cpeCray-21.12.env -hooks -ls $EBROOTEBMINTUTORIAL/easybuild/reprod/easyblocks -cmakemake.py configuremake.py -``` - -As you can see, EasyBuild has also created the `easybuild` subdirectory (and it actually told us about that -at the end of the installation) which contains a lot of information about the build, also to make it easier -to reproduce a build process afterwards. - - ---- - -## Exercises - ---- - -***Exercise E.1**** - Making `eb-tutorial` a bit more personal* - -Change the easyconfig file for `eb-tutorial` to make the message printed by the `eb-tutorial` command -a bit more personal: include the username of the account that was used to install the software in it -(using the `$USER` environment variable). - -??? success "(click to show solution)" - For this we need to change the value that is passed to the `EBTUTORIAL_MSG` configure option: - ```python - configopts = '-DEBTUTORIAL_MSG="Hello from the EasyBuild tutorial! I was installed by $USER." ' - ``` - Here we have to use inner double quotes, to ensure that the `$USER` environment variable is expanded - by the shell when running the `cmake` configure command. - - When you run the `eb-tutorial` command yourself, you should get output like this (not a message that - includes a literal '`$USER`' string): - - ```shell - Hello from the EasyBuild tutorial! I was installed by example. - ``` - - To re-install the `example.eb` easyconfig, you will need to use `eb --rebuild` or `eb --force`. - EasyBuild will also complain about modules that are loaded already if you tested the program above. - A good way to get rid of all those modules on LUMI is to simply use `module purge`. You don't need - to reload the software stack, but you will need to load `EasyBuild-user` again. - ---- - -***Exercise E.2**** - Installing eb-tutorial version 1.1.0* - -Install version 1.1.0 of the `eb-tutorial` example software, -which is a trivial version bump compared to version 1.0.1. - -The sources are available via: - -``` -https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.1.0.tar.gz -``` - -You can leverage the `eb-tutorial` easyconfig file we have composed in the example above, -**but you should not make any manual changes to it!** - -??? success "(click to show solution)" - You can use the `--try-software-version` option for this: - ```shell - $ eb eb-tutorial.eb --try-software-version 1.1.0 - ... - == building and installing eb-tutorial/1.1.0-cpeCray-21.12... - ... - == COMPLETED: Installation ended successfully (took 4 sec) - ``` - To test: - ``` - $ module load eb-tutorial/1.1.0-cpeCray-21.12 - $ eb-tutorial - I have a message for you: - Hello from the EasyBuild tutorial! I was installed by XXXXXXXX. - ``` - (`eb-tutorial` version 1.0.1 doesn't print "`I have a message for you:`") - - EasyBuild has also created a new easyconfig for this configuration and stored - in the repository and the `easybuild` subdirectory from the installation - directory. As on LUMI the repository is in the search path we can actually copy - the file back to the current directory: - ``` - eb --copy-ec eb-tutorial-1.1.0-cpeCray-21.12. ``` - Some of the formatting is lost though and the checksum is still missing, so you may want - to do some cleaning up. - ``` - eb eb-tutorial-1.1.0-cpeCray-21.12.eb --inject-checksum - ``` - - ---- - -*[[next: Using external modules from the Cray PE]](2_03_external_modules)* diff --git a/docs/2022-CSC_and_LO/2_03_external_modules.md b/docs/2022-CSC_and_LO/2_03_external_modules.md deleted file mode 100644 index 80a29712..00000000 --- a/docs/2022-CSC_and_LO/2_03_external_modules.md +++ /dev/null @@ -1,85 +0,0 @@ -# Using external modules from the Cray PE - -*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files)* - ---- - -## What are external modules? - -EasyBuild supports the use of modules that were not installed via EasyBuild. -We refer to such modules as [external modules](https://docs.easybuild.io/en/latest/Using_external_modules.html). -External modules do not define the `EBROOT*` and `EBVERSION*` environment variables that EasyBuild would define -and uses internally in several easyblocks and some easyconfig files. - -External modules are used extensively on Cray systems to interface with the Cray PE (which comes with its own -modules and cannot be installed via EasyBuild): -[external modules can be used as dependencies](https://docs.easybuild.io/en/latest/Using_external_modules.html#using-external-modules-as-dependencies), -by including the module name in the dependencies list, -along with the `EXTERNAL_MODULE` constant marker. - -For example, to specify the module `cray-fftw` as a dependency, you should write the following in your easyconfig file: -``` python -dependencies = [('cray-fftw', EXTERNAL_MODULE)] -``` - -For such dependencies, EasyBuild will: - -* load the module before initiating the software build and install procedure - -* include a `module load` statement in the generated module file (for runtime dependencies) - -!!! Note - The default version of the external module will be loaded unless a specific version is given as dependency, - and here that version needs to be given as part of the name of the module and not as the second element in the - tuple. - - ```python - dependencies = [('cray-fftw/3.3.8.12', EXTERNAL_MODULE)] - ``` - -If the specified module is not available, EasyBuild will exit with an error message stating that the dependency -can not be resolved because the module could not be found, without searching for a matching easyconfig file -from which it could generate the module. - - ---- - -## EasyBuild Metadata for external modules - -[Metadata](https://docs.easybuild.io/en/latest/Using_external_modules.html#metadata-for-external-modules) -can be supplied to EasyBuild for external modules: using the `--external-modules-metadata` -configuration option, the location of one or more metadata files can be specified. - -The files are expected to be in INI format, with a section per module name -and key-value assignments specific to that module. - -The external modules metadata file can be also defined with the corresponding environment variable: -``` -echo $EASYBUILD_EXTERNAL_MODULES_METADATA -/apps/common/UES/jenkins/production/easybuild/cpe_external_modules_metadata-21.04.cfg -``` - -The following keys are -[supported by EasyBuild](https://docs.easybuild.io/en/latest/Using_external_modules.html#supported-metadata-values): - -* name: software name(s) provided by the module -* version: software version(s) provided by the module -* prefix: installation prefix of the software provided by the module - -For instance, the external module version loaded by the dependency `cray-fftw` can be specified as follows: -```ini -[cray-fftw] -name = FFTW -prefix = FFTW_DIR/.. -version = 3.3.8.10 -``` - -The environment variable `$EBROOTFFTW` will also be defined according to the `prefix` specified in the metadata file. - -On LUMI, users in generally don't need to be too concerned about the metadata file as the EasyBuild-user (and other hidden -EasyBuild configuration modules) take care of pointing to the right metadata file, which is specific for each version of the -Cray PE and hence each version of the LUMI software stack. - ---- - -*[[next: Implementing easyblocks]](2_04_implementing_easyblocks.md)* diff --git a/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md deleted file mode 100644 index a91b0dc8..00000000 --- a/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md +++ /dev/null @@ -1,754 +0,0 @@ -# Implementing easyblocks - -*[[back: Using external modules from the Cray PE]](2_03_external_modules.md)* - ---- - -## The basics - -An *easyblock* is a Python module that implements a software build and installation procedure. - -This concept is essentially implemented as a Python script that plugs into the EasyBuild framework. - -EasyBuild will leverage easyblocks as needed, depending on which software packages it needs to install. Which easyblock is required is determined by the ``easyblock`` easyconfig parameter, if it is present, or by the software name. - - -## Generic vs software-specific easyblocks - -Easyblocks can either be *generic* or *software-specific*. - -*Generic* easyblocks implement a "standard" software build and installation procedure that is used by multiple different -software packages. -A commonly used example is the -[``ConfigureMake``](https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/generic/configuremake.py) -generic easyblock, which implements the standard ``configure`` - ``make`` - ``make install`` installation procedure used -by most GNU software packages. - -*Software-specific* easyblocks implement the build and installation procedure for a particular software package. -Typically this involves highly customised steps, for example specifying dedicated configuration options, creating -or adjusting specific files, executing non-standard shell commands, etc. Usually a custom implementation of the -sanity check is also included. Much of the work done in software-specific easyblocks can often also be done -in generic easyblocks using parameters such as ``confdigopts`` etc., but a software-specific easyblock can -hide some of that complexity from the user. Other software-specific easyblocks implement very specific -installation procedures that do not fit in one of the generic ones. - -Using a generic easyblock requires specifying the ``easyblock`` parameter in the easyconfig file. -If it is not specified, EasyBuild will try and find the software-specific easyblock derived from the software name. - -The distinction between generic and software-specific easyblocks can be made based on the naming scheme that is used -for an easyblock (see below). - - -## Naming - -Easyblocks need to follow a strict naming scheme, to ensure that EasyBuild can pick them up automatically as needed. -This involves two aspects: - -* the name of the Python class; -* the name and location of the Python module file. - -### Python class name - -The name of the Python class is determined by the *software name* for software-specific easyblocks. -It consists of a prefix '``EB_``', followed by the (encoded) software name. - -Because of limitations in Python on characters allowed in names of Python classes, -only alphanumeric characters and underscores (``_``) are allowed. Any other characters are replaced following an encoding scheme: - -* spaces are replaced by underscores (``_``); -* dashes ``-`` are replaced by ``_minus_`` (note the inconsistency with the naming of ``EBROOT`` and ``EBVERSION`` variables); -* underscores are replaced by ``_underscore_``; - -The ``encode_class_name`` function provided in ``easybuild.tools.filetools`` returns the expected class name -for a given software name; for example: - -```shell -$ python3 -c "from easybuild.tools.filetools import encode_class_name; print(encode_class_name('netCDF-Fortran'))" -EB_netCDF_minus_Fortran -``` - -**Python class name for *generic* easyblocks** - -For *generic* easyblocks, the class name does *not* include an ``EB_`` prefix (since there is no need for an escaping -mechanism) and hence the name is fully free to choose, taking into account the restriction to alphanumeric characters -and underscores. - -For code style reasons, the class name should start with a capital letter and use CamelCasing. - -Examples include ``Bundle``, ``ConfigureMake``, ``CMakePythonPackage``. - -### Python module name and location - -The *filename* of the Python module is directly related to the name of Python class it provides. - -It should: - -* *not* include the ``EB_`` prefix of the class name for software-specific easyblocks; -* consists only of lower-case alphanumeric characters (``[a-z0-9]``) and underscores (``_``); - * dashes (``-``) are replaced by underscores (``_``); - * any other non-alphanumeric characters (incl. spaces) are simply dropped; - -Examples include: - -* ``gcc.py`` (for *GCC*) -* ``netcdf_fortran.py`` (for *netCDF-Fortran*) -* ``gamess_us.py`` (for *GAMESS (US)*) - -The ``get_module_path`` function provided by the EasyBuild framework in the -``easybuild.framework.easyconfig.easyconfig`` module returns the (full) -module location for a particular software name or easyblock class name. For example: - -```python ->>> from easybuild.framework.easyconfig.easyconfig import get_module_path ->>> get_module_path('netCDF-Fortran') -'easybuild.easyblocks.netcdf_fortran' ->>> get_module_path('EB_netCDF_minus_Fortran') -'easybuild.easyblocks.netcdf_fortran' -``` - -The location of the Python module is determined by whether the easyblock is generic or software-specific. -Generic easyblocks are located in the ``easybuid.easyblocks.generic`` namespace, while software-specific easyblocks -live in the ``easybuild.easyblocks`` namespace directly. - -To keep things organised, the actual Python module files -for software-specific easyblocks are kept in 'letter' subdirectories, -rather than in one large '``easyblocks``' directory -(see -[https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/](https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/)), -but this namespace is collapsed transparently by EasyBuild (you don't need to import from letter subpackages). - -To let EasyBuild pick up one or more new or customized easyblocks, you can use the [``--include-easyblocks``](https://docs.easybuild.io/en/latest/Including_additional_Python_modules.html#including-additional-easyblocks-include-easyblocks) -configuration option. As long as both the filename of the Python module and the name of the Python class -are correct, EasyBuild will use these easyblocks when needed. - -On LUMI, the EasyBuild configuration modules take care of setting this parameter (using the corresponding environment -variable), pointing to custom easyblocks in the LUMI software stack itself and a repo (with a fixed name) that users -can create themselves. At this moment it does not yet include possible other easyblock repositories in other repositories. - - -## Structure of an easyblock - -The example below shows the overal structure of an easyblock: - -```python -from easybuild.framework.easyblock import EasyBlock -from easybuild.tools.run import run_cmd - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def configure_step(self): - """Custom implementation of configure step for Example""" - - # run configure.sh to configure the build - run_cmd("./configure.sh --install-prefix=%s" % self.installdir) -``` - - -Each easyblock includes an implementation of a ``class`` that (directly or indirectly) derives from the abstract -``EasyBlock`` class. - -Typically some useful functions provided by the EasyBuild framework are imported at the top of the Python module. - -In the class definition, one or more '``*_step``' methods (and perhaps a couple of others) are redefined, -to implement the corresponding step in the build and installation procedure. - -Each easyblock *must* implement the ``configure``, ``build`` and ``install`` steps, since these are not implemented -in the abstract ``EasyBlock`` class. This could be done explicitly by redefining the corresponding ``*_step`` methods, -or implicitly by deriving from existing (generic) easyblocks. - -The full list of methods that can be redefined in an easyblock can be consulted in -the [API documentation](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyblock.html). - -## Deriving from existing easyblocks - -When implementing an easyblock, it is common to derive from an existing (usually generic) easyblock, -and to leverage the functionality provided by it. This approach is typically used when only a specific part -of the build and installation procedure needs to be customised. - -In the (fictitious) example below, we derive from the generic ``ConfigureMake`` easyblock to redefine the ``configure`` -step. In this case, we are *extending* the ``configure`` step as implemented by ``ConfigureMake`` rather than -redefining it entirely, since we call out to the original ``configure_step`` method at the end. - -```python -from easybuild.easyblocks.generic.configuremake import ConfigureMake -from easybuild.tools.filetools import copy_file - - -class EB_Example(ConfigureMake): - """Custom easyblock for Example""" - - def configure_step(self): - """Custom implementation of configure step for Example""" - - # use example make.cfg for x86-64 - copy_file('make.cfg.x86', 'make.cfg') - - # call out to original configure_step implementation of ConfigureMake easyblock - super(EB_Example, self).configure_step() -``` - -## Easyconfig parameters - -All of the easyconfig parameters that are defined in an easyconfig file -are available via the ``EasyConfig`` instance that can be accessed through ``self.cfg`` in an easyblock. -For instance, if the easyconfig file specifies - -```python -name = 'example' -version = '2.5.3' -versionsuffix = '-Python-3.7.4' -``` - -then these three parameters are accessible within an easyblock via ``self.cfg['name']``, ``self.cfg['version']`` -and ``self.cfg['versionsuffix']``. - -A few of the most commonly used parameters can be referenced directly: - -* **``self.name``** is equivalent with ``self.cfg['name']``; -* **``self.version``** is equivalent with ``self.cfg['version']``; -* **``self.toolchain``** is equivalent with ``self.cfg['toolchain']``; - - -### Updating parameters - -You will often find that you need to *update* some easyconfig parameters in an easyblock, -for example ``configopts`` which specifies options for the configure command. - -Because of implementation details (related to -how template values like ``%(version)s`` are handled), you need to be a bit careful here... - -To completely redefine the value of an easyconfig parameter, you can use simple assignment. For example: - -```python -self.cfg['example'] = "A new value for the example easyconfig parameter." -``` - -If want to *add* to the existing value however, you *must* use the ``self.cfg.update`` method. For example: - -```python -self.cfg.update('some_list', 'example') -``` - -This will ***not*** work (because ``self.cfg['some_list']`` does not return a reference to the original value, -but to a *temporary copy* thereof): - -```python -# anti-pattern, this does NOT work as expected! -self.cfg['some_list'].append('example') -``` - -### Custom parameters - -Additional custom easyconfig parameters can be defined in an easyblock to steer its behaviour. -This is done via the ``extra_options`` *static* method. Custom parameters can be specified to be mandatory. - -The example below shows how this can be implemented: - -```python -from easybuild.easyblocks.generic.configuremake import ConfigureMake -from easybuild.framework.easyconfig import CUSTOM, MANDATORY - - -class EB_Example(ConfigureMake): - """Custom easyblock for Example""" - - @staticmethod - def extra_options(): - """Custom easyconfig parameters for Example""" - extra_vars = { - 'required_example_param': [None, "Example required custom parameter", MANDATORY], - 'optional_example_param': [None, "Example optional custom parameter", CUSTOM], - } - return ConfigureMake.extra_options(extra_vars) -``` - -The first element in the list of a defined custom parameter corresponds to the default value for that parameter -(both ``None`` in the example above). The second element provides some informative help text -(which can then be displayed with ``eb -a -e ``, eg, ``eb -a -e EB_GCC``), -and the last element -indicates whether the parameter is mandatory (``MANDATORY``) or just an optional custom parameter (``CUSTOM``). - -## Easyblock constructor - -In the ``class`` constructor of the easyblock, i.e. the ``__init__`` method, one or more class variables -can be initialised. These can be used for sharing information between different ``*_step`` methods in the easyblock. - -For example: - -```python -from easybuild.framework.easyblock import EasyBlock - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def __init__(self, *args, **kwargs): - """Constructor for Example easyblock, initialises class variables.""" - - # call out to original constructor first, so 'self' (i.e. the class instance) is initialised - super(EB_Example, self).__init__(*args, **kwargs) - - # initialise class variables - self.example_value = None - self.example_list = [] -``` - -## File operations - -File operations is a common use case for implementing easyblocks, hence the EasyBuild framework provides a -number of useful functions related to this, including: - -* ``read_file()``: read file at a specified location and returns its contents; - -* ``write_file(, )`` at a specified location with provided contents; - to append to an existing file, use ``append=True`` as an extra argument; - -* ``copy_file(, )`` to copy an existing file; - -* ``apply_regex_substitutions(, )`` to patch an existing file; - -All of these functions are provided by the [``easybuild.tools.filetools``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html) module. - -## Executing shell commands - -For executing shell commands two functions are provided by the -[``easybuild.tools.run``](https://docs.easybuild.io/en/latest/api/easybuild.tools.run.html) module: - -* ``run_cmd()`` to run a non-interactive shell command; - -* ``run_cmd_qa(, )`` to run an interactive shell command; - -Both of these accept a number of optional arguments: - -* ``simple=True`` to just return ``True`` or ``False`` to indicate a successful execution, - rather than the default return value, i.e., a tuple that provides the command output and the exit code (in that - order); - -* ``path=`` to run the command in a specific subdirectory; - -The ``run_cmd_qa`` function takes two additional specific arguments: - -* ``no_qa=`` to specify a list of patterns to recognize non-questions; - -* ``std_qa=`` to specify regular expression patterns for common questions, and the matching answer; - -## Manipulating environment variables - -To (re)define environment variables, the ``setvar`` function provided by the -[``easybuild.tools.environment``](https://docs.easybuild.io/en/latest/api/easybuild.tools.environment.html) -module should be used. - -This makes sure that the changes being made to the specified environment variable are kept track of, -and that they are handled correctly under ``--extended-dry-run``. - -## Logging and errors - -It is good practice to include meaningful log messages in the ``*_step`` methods being customised in the easyblock, -to enrich the EasyBuild log with useful information for later debugging or diagnostics. - -For logging, the provided ``self.log`` logger class should be used. -You can use the ``self.log.info`` method to log an informative message. -Similar methods are available for logging debug messages (``self.log.debug``), which are -only emitted when ``eb`` is run with debugging mode enabled (``--debug`` or ``-d``), -and for logging warning messages (``self.log.warning``). - -If something goes wrong, you can raise an ``EasyBuildError`` instance to report the error. - -For example: - -```python -from easybuild.framework.easyblock import EasyBlock -from easybuild.tools.build_log import EasyBuildError -from easybuild.tools.run import run_cmd - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def configure_step(self): - """Custom implementation of configure step for Example""" - - cmd = "./configure --prefix %s" % self.installdir) - out, ec = run_cmd(cmd) - - success = 'SUCCESS' - if success in out: - self.log.info("Configuration command '%s' completed with success." % cmd) - else: - raise EasyBuildError("Pattern '%s' was not found in output of '%s'." % (success, cmd)) -``` - -## Custom sanity check - -For software-specific easyblocks, a custom sanity check is usually included to verify that the installation was -successful or not. - -This is done by redefining the ``sanity_check_step`` method in the easyblock. For example: - -```python -from easybuild.framework.easyblock import EasyBlock - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def sanity_check_step(self): - """Custom sanity check for Example.""" - - custom_paths = { - 'files': ['bin/example'], - 'dirs': ['lib/examples/'], - } - custom_commands = ['example --version'] - - # call out to parent to do the actual sanity checking, pass through custom paths and commands - super(EB_Example, self).sanity_check_step(custom_paths=custom_paths, custom_commands=custom_commands) -``` - -You can both specify file paths and subdirectories to check for, which are specified relative to the installation directory, -as well as simple commands that should execute successfully after completing the installation and loading the generated module file. - -It is up to you how extensive you make the sanity check, but it is recommended to make the check as complete -as possible to catch any potential build or installation problems that may occur, while ensuring that it can -run relatively quickly (in seconds, or at most a couple of minutes). - -## Version-specific parts - -In some cases version-specific actions or checks need to be included in an easyblock. -For this, it is recommended to use ``LooseVersion`` rather than directly comparing version numbers using string values. - -For example: - -```python -from distutils.version import LooseVersion -from easybuild.framework.easyblock import EasyBlock - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def sanity_check_step(self): - """Custom sanity check for Example.""" - - custom_paths = { - 'files': [], - 'dirs': [], - } - - # in older versions, the binary used to be named 'EXAMPLE' rather than 'example' - if LooseVersion(self.version) < LooseVersion('1.0'): - custom_paths['files'].append('bin/EXAMPLE') - else: - custom_paths['files'].append('bin/example') - - super(EB_Example, self).sanity_check_step(custom_paths=custom_paths) -``` - -## Compatibility with ``--extended-dry-run`` and ``--module-only`` - -Some special care must be taken to ensure that an easyblock is fully compatible with ``--extended-dry-run`` / ``-x`` -(see [Inspecting install procedures](../basic_usage/#inspecting-install-procedures)) and ``--module-only``. - -For compatibility with ``--extended-dry-run``, you need to take into account that specified operations -like manipulating files or running shell commands will not actually be executed. You can check -whether an easyblock is being run in dry run mode via ``self.dry_run``. - -For example: - -```python -from easybuild.framework.easyblock import EasyBlock -from easybuild.tools.build_log import EasyBuildError -from easybuild.tools.run import run_cmd - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def configure_step(self): - """Custom implementation of configure step for Example""" - - cmd = "./configure --prefix %s" % self.installdir) - out, ec = run_cmd(cmd) - - success = 'SUCCESS' - if success in out: - self.log.info("Configuration command '%s' completed with success." % cmd) - - # take into account that in dry run mode we won't get any output at all - elif self.dry_run: - self.log.info("Ignoring missing '%s' pattern since we're running in dry run mode." % success) - - else: - raise EasyBuildError("Pattern '%s' was not found in output of '%s'." % (success, cmd)) -``` - -For ``--module-only``, you should make sure that both the ``make_module_step``, including the ``make_module_*`` submethods, -and the ``sanity_check_step`` methods do not make any assumptions about the presence of certain environment variables, or that class variables have been defined already. - -This is required because under ``--module-only`` the large majority of the ``*_step`` functions are -simply *skipped* entirely. So, if the ``configure_step`` method is responsible for defining class variables that are -picked up in ``sanity_check_step``, the latter may run into unexpected initial values like ``None``. -A possible workaround is to define a separate custom method to define the class variables, and to call out to this -method from ``configure_step`` and ``sanity_check_step`` (for the latter, conditionally, i.e., only if the class -variables still have the initial values). - -For example: - -```python -from easybuild.framework.easyblock import EasyBlock - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def __init__(self, *args, **kwargs): - """Easyblock constructor.""" - super(EB_Example, self).__init__(*args, **kwargs) - - self.command = None - - def set_command(self): - """Initialize 'command' class variable.""" - # $CC environment variable set by 'prepare' step determines exact command - self.command = self.name + '-' + os.getenv('CC') - - def configure_step(self): - """Custom configure step for Example.""" - - self.set_command() - self.cfg.update('configopts', "COMMAND=%s" % self.command) - - super(EB_Example, self).configure_step() - - def sanity_check_step(self): - """Custom implementation of configure step for Example""" - - if self.command is None: - self.set_command() - - super(EB_Example, self).sanity_check_step(custom_commands=[self.command]) -``` - - -## Easyblocks in the Cray ecosystem - -The generic easyblocks are usually rather independent of compilers etc. and tend to work well with all toolchains. -However, software-specific easyblocks may contain code that is specific for certain toolchains and are often only -tested with the common toolchains (foss and intel and their subtoolchains). Many of those easyblocks will fail -on Cray systems (or any system that uses other toolchains) as they don't recognise the compiler and rather than -implementing some generic behaviour that may or may not work, produce an error message instead that the compiler -toolchain is not supported. - -Several packages on LUMI therefore use generic easyblocks rather than the software-specific easyblocks that may -exist for those applications. Adapting those software-specific easyblocks for LUMI poses an interesting maintenance -problem. Either one could decide to not contribute back to the community, but this implies then that all modifications -made to the corresponding easyblocks in the EasyBuild distribution should be monitored and implemented in the custom -easyblocks for Cray also. On the other hand, contributing back to the community also poses two problems. First it -would also require to implement the Cray toolchains as used on LUMI in the core of EasyBuild (which already contains -a different set of toolchains targeted more at how the Cray PE works with the regular environment modules), and that -only makes sense if these toolchains are first extended to not only cover the programming environments supported on -LUMI but also the Intel and NVIDIA programming environments. Second, the EasyBuild community has no easy way of testing -any modification made to such an easyblock on a Cray PE system. Hence every update made in the community may break -the Cray PE support again. - - -## Exercise - -### Exercise I.1 - -Try implementing a new custom easyblock for ``eb-tutorial``, which derives directly -from the base ``EasyBlock`` class. - -Your easyblock should: - -* define a custom mandatory easyconfig parameter named ``message``; -* run `cmake` to configure the installation, which includes at least: - * specifying the correct installation prefix (using the `-DCMAKE_INSTALL_PREFIX=...` option); - * passing down the value of ``message`` easyconfig parameter via `-DEBTUTORIAL_MSG=...` -* run `make` to build `eb-tutorial`; -* run `make install` to install the generated binary; -* perform a custom sanity check to ensure the installation is correct; -* pick up on commonly used easyconfig parameters like `configopts` and `preinstallopts` where appropriate; - -??? success "(click to show solution)" - - Here's a complete custom easyblock for ``eb-tutorial`` that derives from the base ``EasyBlock`` class, - which should be included in a file named ``eb_tutorial.py``. - - We need to implement the ``configure_step``, ``build_step``, and ``install_step`` methods in - order to have a fully functional easyblock. - - The configure, build, and install steps take into account the corresponding easyconfig - parameters that allow customizing these commands from an easyconfig file. - - ```python - from easybuild.framework.easyblock import EasyBlock - from easybuild.framework.easyconfig import MANDATORY - from easybuild.tools.run import run_cmd - - - class EB_eb_minus_tutorial(EasyBlock): - """Custom easyblock for eb-tutorial.""" - - @staticmethod - def extra_options(): - extra = EasyBlock.extra_options() - extra.update({ - 'message': [None, "Message that eb-tutorial command should print", MANDATORY], - }) - return extra - - def configure_step(self): - """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" - - cmd = ' '.join([ - self.cfg['preconfigopts'], - 'cmake', - '-DCMAKE_INSTALL_PREFIX=\'%s\'' % self.installdir, - '-DEBTUTORIAL_MSG="%s"' % self.cfg['message'], - self.cfg['configopts'], - ]) - run_cmd(cmd) - - def build_step(self): - """Build step for eb-tutorial""" - - cmd = ' '.join([ - self.cfg['prebuildopts'], - 'make', - self.cfg['buildopts'], - ]) - run_cmd(cmd) - - def install_step(self): - """Install step for eb-tutorial""" - - cmd = ' '.join([ - self.cfg['preinstallopts'], - 'make install', - self.cfg['installopts'], - ]) - run_cmd(cmd) - - def sanity_check_step(self): - custom_paths = { - 'files': ['bin/eb-tutorial'], - 'dirs': [], - } - custom_commands = ['eb-tutorial'] - return super(EB_eb_minus_tutorial, self).sanity_check_step(custom_paths=custom_paths, - custom_commands=custom_commands) - ``` - - We also need to adapt our easyconfig file for ``eb-tutorial``: - - - The ``easyblock`` line is no longer needed as we will rely on the automatic selection of the - software-specific easyblock. - - We don't need to define the message through ``configopts`` but via the easyblock-specific - configuration parameter ``message``. In fact, we were so careful when implementing the ``configure_step`` - that even variable expansion will still work so we can still include ``$USER`` in the message. - - The sanity check is also no longer needed as it is done by the software-specific easyblock. - - So the easyconfig file simplifies to: - - ```python - name = 'eb-tutorial' - version = "1.1.0" - - homepage = 'https://easybuilders.github.io/easybuild-tutorial' - - whatis = [ 'Description: EasyBuild tutorial example'] - - description = """ - This is a short C++ example program that can be buid using CMake. - """ - - toolchain = {'name': 'cpeCray', 'version': '21.12'} - - builddependencies = [ - ('buildtools', '%(toolchain_version)s', '', True) - ] - - source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] - sources = [SOURCE_TAR_GZ] - checksums = ['def18b69b11a3ec34ef2a81752603b2118cf1a57e350aee41de9ea13c2e6a7ef'] - - message = 'Hello from the EasyBuild tutorial! I was installed by $USER.' - - moduleclass = 'tools' - - ``` - - Running this example on LUMI is a little tricky as using ``--include-easyblocks`` to point EasyBuild to - our new easyblock interfers with settings already made by the EasyBuild configuration modules (``EasyBuild-user``) - and causes error messages about the toolchains. So either the easyblock needs to be copied to the user location - that can be found by looking at the output of ``eb --show-config`` or we simply need to extend the list of - easyblocks that EasyBuild searches with the easyblocks in the current directory: - - ``` shell - EASYBUILD_INCLUDE_EASYBLOCKS="$EASYBUILD_INCLUDE_EASYBLOCKS,./*.py" - ``` - - - - -### Exercise I.2 - -Try implementing another new custom easyblock for ``eb-tutorial``, -which derives from the generic ``CMakeMake`` easyblock. - -Your easyblock should only: - -* define a custom mandatory easyconfig parameter named ``message``; -* pass down the value of ``message`` easyconfig parameter via `-DEBTUTORIAL_MSG=...` -* perform a custom sanity check to ensure the installation is correct; - -??? success "(click to show solution)" - - When deriving from the ``CMakeMake`` generic easyblock, there is a lot less to worry about. - - We only need to customize the ``configure_step`` method to ensure that the ``-DEBTUTORIAL_MSG`` configuration - option is specified; the ``CMakeMake`` easyblock already takes care of specifying the location of - the installation directory (and a bunch of other configuration options, like compiler commands and flags, etc.). - - Implementing the ``build_step`` and ``install_step`` methods is no longer needed, - the standard procedure that is run by the ``CMakeMake`` generic easyblock is fine, - and even goes beyond what we did in the previous exercise (like building in parallel with ``make -j``). - - ```python - from easybuild.easyblocks.generic.cmakemake import CMakeMake - from easybuild.framework.easyconfig import MANDATORY - from easybuild.tools.run import run_cmd - - class EB_eb_minus_tutorial(CMakeMake): - """Custom easyblock for eb-tutorial.""" - - @staticmethod - def extra_options(): - extra = CMakeMake.extra_options() - extra.update({ - 'message': [None, "Message that eb-tutorial command should print", MANDATORY], - }) - return extra - - def configure_step(self): - """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" - self.cfg.update('configopts', '-DEBTUTORIAL_MSG="%s"'% self.cfg['message']) - - super(EB_eb_minus_tutorial, self).configure_step() - - def sanity_check_step(self): - custom_paths = { - 'files': ['bin/eb-tutorial'], - 'dirs': [], - } - custom_commands = ['eb-tutorial'] - return super(EB_eb_minus_tutorial, self).sanity_check_step(custom_paths=custom_paths, - custom_commands=custom_commands) - ``` - - This is a much simpler easyblock as we already use all the logic that has been written for us to build - with CMake. - -*[[next: Part 3: Overview]](3_00_part3_advanced.md)* diff --git a/docs/2022-CSC_and_LO/3_00_part3_advanced.md b/docs/2022-CSC_and_LO/3_00_part3_advanced.md deleted file mode 100644 index 28a24990..00000000 --- a/docs/2022-CSC_and_LO/3_00_part3_advanced.md +++ /dev/null @@ -1,15 +0,0 @@ -# Part III: Advanced topics - -*[[back to start page]](index.md)* - -In this section we mostly cover "good to know that they exist" features as they are not used -on LUMI, or not really accessible to regular user installations that are performed with the -LUMI ``EasyBuild-user`` module. Hooks are used on LUMI, but it is not really advised to -overwrite the centrally defined hooks with a local file. And the whole structure of the -EasyBuild integration is also set up to make use of the GitHub integration in the future. - -* [Using EasyBuild as a library](3_01_easybuild_library.md) *(hands-on)* -* [Using hooks to customise EasyBuild](3_02_hooks.md) *(hands-on)* -* [Submitting installations as Slurm jobs](3_03_slurm_jobs.md) *(hands-on)* -* [Module naming schemes (incl. hierarchical)](3_04_module_naming_schemes.md) *(hands-on)* -* [GitHub integration to facilitate contributing to EasyBuild](3_05_github_integration.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/3_01_easybuild_library.md b/docs/2022-CSC_and_LO/3_01_easybuild_library.md deleted file mode 100644 index f0c255eb..00000000 --- a/docs/2022-CSC_and_LO/3_01_easybuild_library.md +++ /dev/null @@ -1,301 +0,0 @@ -# Using EasyBuild as a library - -*[[back: Configuring EasyBuild]](1_07_configuration.md)* - ---- - -You can use EasyBuild as a Python library and implement a script to automate a particular task. - -All the functionality provided by the EasyBuild framework can be leveraged, -so you only have to glue things together and add the missing pieces. - - -## Setting up the EasyBuild configuration - -Before you call any of the functions provided by the EasyBuild framework, -you should initialize EasyBuild by **setting up the configuration**. -Some of the EasyBuild framework functions assume that EasyBuild has been initialized properly, -and calling them without doing so will result in errors. - -To set up the EasyBuild configuration, you should call the ``set_up_configuration`` function -that is provided by the ``easybuild.tools.options`` module. - -This function takes a couple of optional arguments that are useful in the context of -a Python script that uses EasyBuild as a library: - -* Via ``args`` you can provide a list of command line arguments that should be passed to the option parser. - By default, the standard command line arguments are picked up via ``sys.args``, which may not be what you want. -* By passing ``silent=True`` you can specify that the ``set_up_configuration`` function should not print - anything when it is called. By default, it will print the location to the temporary EasyBuild log file. - -### Cleaning up the temporary directory - -When EasyBuild is configured, a temporary directory specific to that EasyBuild session will be created automatically. -You should clean up that directory, especially if you will be creating temporary files, or if the script will be run -often. - -Note that cleaning up the temporary directory implies removes the temporary log files, -so you probably only want to do this if no errors occurred. - -### Minimal example script - -Here is a minimal (and pretty useless) example Python script, which sets up the EasyBuild configuration -and cleans up the temporary directory: - -```python -#!/usr/bin/env python3 -from easybuild.tools.filetools import remove_dir -from easybuild.tools.options import set_up_configuration - -opts, _ = set_up_configuration(args=[], silent=True) - -remove_dir(opts.tmpdir) -``` - - -## Example use cases - -Once the EasyBuild configuration has been set up, the functions provided by the EasyBuild framework -can be called from a Python script (or directly from the Python interpreter). - -A full overview of all functions is available via the [EasyBuild API documentation](https://docs.easybuild.io/en/latest/api/easybuild.html). - -We highlight a couple commonly used functions in the sections below. - -### File operations - -The [``easybuild.tools.filetools``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html) -module provides a bunch of functions for performing file operations. - -That includes straightforward things like reading, writing, and copying files -(see [``read_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.read_file), [``write_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.write_file), [``copy_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.copy_file)), but also more specific functionality like applying a patch file ([``apply_patch``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.apply_patch)), -computing different types of checksums for a file ([``compute_checksum``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.compute_checksum)), and downloading files -([``download_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.download_file)). - -Here is a trivial example where we download a file from a specified URL to a temporary directory, -compute the SHA256 checksum, and then copy the downloaded file to the current working directory: - -```python -#!/usr/bin/env python3 -import os -import sys -import tempfile - -from easybuild.tools.filetools import download_file, compute_checksum, copy_file, remove_dir -from easybuild.tools.options import set_up_configuration - -url = sys.argv[1] - -opts, _ = set_up_configuration() - -fn = os.path.basename(url) -tmp_path = os.path.join(opts.tmpdir, fn) -download_file(fn, url, tmp_path) - -sha256 = compute_checksum(tmp_path, checksum_type='sha256') -print("SHA256 checksum for %s: %s" % (fn, sha256)) - -target_dir = os.getcwd() -copy_file(tmp_path, target_dir) -print("%s copied to %s" % (fn, target_dir)) - -# cleanup temp dir -remove_dir(opts.tmpdir) -``` - -Example output: -```shell -$ export TEST_URL='https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.0.1.tar.gz' -$ python3 download_and_copy.py $TEST_URL -== Temporary log file in case of crash /tmp/eb-or_xhaw8/easybuild-ewooa00c.log -SHA256 checksum for eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 -eb-tutorial-1.0.1.tar.gz copied to /home/example -``` -We are deliberately *not* specifying the `args` optional argument to the `set_up_configuration` call here, -so we can pass EasyBuild configuration options as command line arguments to this script. -Of course, only those configuration options that are taken into account by the EasyBuild -framework functions being called have any impact. For example: - -```shell -# wait for max. 1h for the download to complete -python3 download_and_copy.py $TEST_URL --download-timeout 3600 -``` - -### Running shell commands - -To run shell commands, we can leverage the ``run_cmd`` functions from the ``easybuild.tools.run`` module. - -Here is a simple example for running the '`make`' command via a Python script: - -```python -#!/usr/bin/env python3 -import sys - -from easybuild.tools.filetools import remove_dir, which -from easybuild.tools.run import run_cmd -from easybuild.tools.options import set_up_configuration - -opts, _ = set_up_configuration(args=[]) - -cmd = 'make' -cmd_path = which(cmd) -if cmd_path: - print(">>> '%s' command found at %s" % (cmd, cmd_path)) -else: - sys.stderr.write("ERROR: '%s' command not found!\n" % cmd) - sys.exit(1) - -cmd = ' '.join(["make"] + sys.argv[1:]) -out, ec = run_cmd(cmd) - -print("\n>>> Output of '%s' (exit code %s):\n\n%s" % (cmd, ec, out)) - -remove_dir(opts.tmpdir) -``` - -Note that in this case it is important to use `args=[]` when calling `set_up_configuration`, -to avoid that the additional script arguments which are options for the '`make`' command -are picked up by the EasyBuild option parser. - -Example usage: - -```shell -$ python3 run_make.py --version -== Temporary log file in case of crash /tmp/eb-zfvbceg6/easybuild-7tynetaj.log - ->> Output of 'make --version' (exit code 0): - -GNU Make 3.81 -``` - -### Interacting with the modules tool - -You can interact with the environment modules tool in a Python script using the -[``easybuild.tools.modules``](https://docs.easybuild.io/en/latest/api/easybuild.tools.modules.html) -Python module that is a part of the EasyBuild framework. -The modules tool that will be used is determined by the active EasyBuild configuration. - -Here is an example script that checks which modules are available and currently loaded, -loads the default module file for ``bzip2``, and inspects the resulting changes to the environment. - -```python -#!/usr/bin/env python3 -import os -import sys - -from easybuild.tools.filetools import remove_dir -from easybuild.tools.modules import get_software_root_env_var_name, modules_tool -from easybuild.tools.options import set_up_configuration - -opts, _ = set_up_configuration() - -# obtain ModulesTool instance for preferred modules tool (determined by active EasyBuild configuration) -mod_tool = modules_tool() -print("Active modules tool: %s version %s" % (mod_tool.NAME, mod_tool.version)) - -avail_modules = mod_tool.available() -print("Found %d available modules in total" % len(avail_modules)) - -avail_eb_modules = mod_tool.available('EasyBuild') -print("Found %d available modules for EasyBuild: %s" % (len(avail_eb_modules), ', '.join(avail_eb_modules))) - -loaded_modules = mod_tool.loaded_modules() -print("%d modules are currently loaded: %s" % (len(loaded_modules), ', '.join(loaded_modules))) - -# load default module for bzip2, check changes to environment -name = 'bzip2' -env_var_name = get_software_root_env_var_name(name) -if any(m.startswith(name + '/') for m in avail_modules): - - print("Current $%s value: %s" % (env_var_name, os.getenv(env_var_name, '(no set)'))) - print("Loading (default) '%s' module..." % name) - - mod_tool.load([name]) - print("Loaded modules: %s" % ', '.join(mod_tool.loaded_modules())) - - # inspect $_LMFILES_ environment variable to determine path to loaded bzip2 module file - for mod_file_path in os.getenv('_LMFILES_').split(':'): - if name in mod_file_path: - print("Path to loaded %s module: %s" % (name, mod_file_path)) - break - - # $EBROOTBZIP2 should be set now (if the bzip2 module was installed with EasyBuild) - print("Current $%s value: %s" % (env_var_name, os.getenv(env_var_name, '(no set)'))) -else: - sys.stderr.write("No modules available for %s\n" % name) - sys.exit(1) - -remove_dir(opts.tmpdir) -``` - -### Parsing easyconfig files - -Here is another small example Python script, which uses the EasyBuild framework functionality -to locate and parse an easyconfig file, and inspect the value of specific easyconfig parameters. - -We define a small helper function named ``parse_easyconfig``, because the EasyBuild framework API -is a bit awkward to use for this simple use case. - -```python -#!/usr/bin/env python3 -import sys - -from easybuild.framework.easyconfig.tools import det_easyconfig_paths, parse_easyconfigs -from easybuild.tools.options import set_up_configuration - - -def parse_easyconfig(ec_fn): - """ - Helper function: find and parse easyconfig with specified filename, - and return parsed easyconfig file (an EasyConfig instance). - """ - # determine path to easyconfig file - ec_path = det_easyconfig_paths([ec_fn])[0] - - # parse easyconfig file; - # the 'parse_easyconfigs' function expects a list of tuples, - # where the second item indicates whether or not the easyconfig file was - # automatically generated or not - ec_dicts, _ = parse_easyconfigs([(ec_path, False)]) - - # only retain first parsed easyconfig, ignore any others (which are unlikely - # anyway) - return ec_path, ec_dicts[0]['ec'] - - -# check whether required arguments are provided -if len(sys.argv) < 3: - sys.stderr.write("ERROR: Usage: %s ") - sys.exit(1) - -ec_fn = sys.argv[1] -keys = sys.argv[2:] - -set_up_configuration(args=[], silent=True) - -ec_path, ec = parse_easyconfig(ec_fn) - -print("Inspecting %s ..." % ec_path) -for key in keys: - print("%s: %s" % (key, ec[key])) -``` - -Example usage (ensure that ``EasyBuild-user`` is loaded as EasyBuild should be -configured properly!): - -``` -$ ./inspect_easyconfig.py GMP-6.2.1-cpeCray-21.12.eb name version sources sanity_check_paths -Inspecting /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/GMP/GMP-6.2.1-cpeCray-21.12.eb ... -name: GMP -version: 6.2.1 -sources: ['gmp-6.2.1.tar.bz2'] -sanity_check_paths: {'files': ['include/gmp.h', 'include/gmpxx.h', 'lib/libgmp.a', 'lib/libgmp.la', 'lib/libgmp.so', 'lib/libgmpxx.a', 'lib/libgmpxx.la', 'lib/libgmpxx.so', 'lib/pkgconfig/gmp.pc', 'lib/pkgconfig/gmpxx.pc'], 'dirs': ['share/info']} -``` - -This feature can be used, e.g., to extract information from easyconfig files to generate documentation -for the library of easyconfig files. - - ---- - -*[[next: Using hooks to customise EasyBuild]](3_02_hooks.md)* diff --git a/docs/2022-CSC_and_LO/3_02_hooks.md b/docs/2022-CSC_and_LO/3_02_hooks.md deleted file mode 100644 index f91f7d7c..00000000 --- a/docs/2022-CSC_and_LO/3_02_hooks.md +++ /dev/null @@ -1,213 +0,0 @@ -# Using hooks to customise EasyBuild - -*[[back: Using EasyBuild as a library]](3_01_easybuild_library.md)* - ---- - -Sometimes you may have the need to customise the behaviour of EasyBuild, -while you want to avoid copying easyconfig files in order to make small changes -(like adding an additional configuration option), having to maintain a set -of modified easyblocks, or using a patched version of the EasyBuild framework. - -EasyBuild provides support for implementing so-called *hooks*, -which are small functions that are triggered automatically at specific times. - -Through these hook functions you can take additional actions, or even tweak -internal data structures, to change the software installation procedure that EasyBuild -would normally execute. - -This is a very powerful feature, but it is important to aware of some details in order -to make good use of it. - -Support for hooks is also -[covered in detail in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Hooks.html). - -## Why hooks? - -Hooks can be implemented for various reasons. Some examples used by some sites in the EasyBuild -community: - -- Enforce some site policies on easyconfig files. A practical example for LUMI could be imposing - the use of the `whatis` parameter and ensuring that it contains a `Description` line to have a - more consistent format for module files. -- Ensuring that some information is always in the module file. E.g., on LUMI the `site_contacts` - parameter is added automatically when installing in the central stack if it is not already in - the easyconfig file to refer to the LUST support pages. -- Modify the behaviour of standard easyconfig files that come with EasyBuild to adapt them to the system - while allowing users to simply use the standard easyconfig files. E.g., we could consider this option - to offer the common EasyBuild FOSS toolchain and its subtoolchains to users on LUMI-C, while using hooks to - adapt, e.g., the easyconfig files for Open MPI to work on LUMI. - -Some site have really large hook files to implement policies and modify standard EasyBuild build recipes, e.g., -- [The hooks file from Jülich Supercomputing Centre](https://github.com/easybuilders/JSC/blob/2022/Custom_Hooks/eb_hooks.py) -- ComputeCanada [cc_hooks_common.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_common.py) - and [cc_hooks_gento.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_gentoo.py) -- [The hooks file from the EESSI software stack](https://github.com/EESSI/software-layer/blob/main/eb_hooks.py). - [EESSI](https://www.eessi-hpc.org/) is an effort to build a software stack distributed via - [CernVM-FS](https://cernvm.cern.ch/fs/) using EasyBuild to build all software. - -## Implementing and using hooks - -Using hooks is done by implementing functions with specific names in a Python module, -and configuring EasyBuild to use them by specifying the path to the file that contains them -via the ``hooks`` configuration option. - -For example, if the file named ``my_eb_hooks.py`` in your home directory contains the implementation -of your hooks, you can configure EasyBuild to use them by defining the ``$EASYBUILD_HOOKS`` environment -variable: - -```shell -export EASYBUILD_HOOKS=$HOME/my_eb_hooks.py -``` - -## Available hooks - -EasyBuild supports various hooks, the full list can be consulted via ``eb --avail-hooks``. - -There are 3 different types of hooks: - -- start/end hooks -- parse hook -- pre/post-step hooks - -These are discussed in more details in the sections below. - -### Start/end hooks - -The first type of hooks are the ``start`` and ``end`` hooks, which are respectively triggered -at the very start of the EasyBuild session (right after setting up the EasyBuild configuration), -and at the very end of the session (right before stopping the logger and doing final cleanup). - -These hooks are only called *once* for every EasyBuild session, no matter how many installations -are performed during that session. - -The ``end`` hook is *not* triggered if an error occurred during one of the installations, -only on successful completion of the EasyBuild session. - -These hooks can be implementing by defining a function named ``start_hook`` and ``end_hook`` -in the Python module file that is provided via the ``hooks`` configuration option. -No arguments are provided when calling these hooks. - -### Parse hook - -The ``parse`` hook is triggered right after an easyconfig file is being parsed, -*before* EasyBuild sets up its internal data structures based on the parsed easyconfig. - -If you want to dynamically change one or more easyconfig parameters without changing the corresponding -easyconfig files, using this hook may be appropriate. - -Note that parsing an easyconfig file can happen for a variety of reasons, -not only when the easyconfig file will actually be installed. EasyBuild will also -parse easyconfig files to check whether they resolve required dependencies, -to check whether the corresponding module file is already installed, etc. - -This hook can be implemented via a function named ``parse_hook``, and exactly one -argument is provided when it is called: the [``EasyConfig``](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyconfig.easyconfig.html#easybuild.framework.easyconfig.easyconfig.EasyConfig) -instance that represents the parsed easyconfig file. - -### Pre/post-step hooks - -The third type of hooks are *pre/post-step* hooks, which are triggered right before or -after a particular installation step is executed. - -EasyBuild performs each software installation by stepping through over a dozen different methods, -and for each of these steps there a pre- and post-hook is triggered, which results in over 30 -additional hooks. - -To use any of these hooks, you need to implement a function that follow a strict naming scheme: -``__hook``. For example, the hook that is triggered right before the ``configure`` -step is run is a function named ``pre_configure_hook``. - -Every time these hooks are called, a single argument is provided: an [``EasyBlock``](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyblock.html#easybuild.framework.easyblock.EasyBlock) -instance that represents the easyblock that is being used to perform the installation. -The parsed easyconfig file can be accessed via the ``cfg`` class variable of the ``EasyBlock`` instance. - -These hooks are useful for influencing the installation procedure at a particular stage. - -## Caveats - -There are a couple of important caveats to take into account when implementing hooks. - -### Breaking EasyBuild with hooks - -Since hooks allow you to inject custom code into EasyBuild at runtime, -it is also easy to break EasyBuild by using hooks... - -Make sure to carefully test your hook implementations, and constrain the actions -you take a much as possible, for example by adding conditions to control for which -software names you will actually modify the installation procedure, etc. - -Any errors that are triggered or raised while a hook function is running -will interrupt the EasyBuild session. - -So don't forget: with great power comes great responsibility! - -### Template values - -Depending on the type of hook, you may observe "raw" values of easyconfig parameters where -template values have not been resolved yet, or values in which template values have been resolved already. - -In the ``parse`` hook, you will always see unresolved template values. - -In the pre/post-step hooks you will see resolved template values, -unless you explicitly disable templating. - -To obtain easyconfig parameter values with unresolved template values in step hooks, -you can use the ``disable_templating`` [context manager](https://docs.python.org/3/reference/compound_stmts.html#with). -For example: - -```python -from easybuild.framework.easyconfig.easyconfig import disable_templating -from easybuild.tools.build_log import print_warning - -def pre_source_hook(eb): - """Print warning when software version was found in 'raw' name of source file.""" - with disable_templating(eb.cfg): - for src in eb.cfg['sources']: - if eb.version in src: - msg = "Software version '%s' found in name of source file (%s), " % (eb.version, src) - msg += "please use %(version)s template value instead!" - print_warning(msg) -``` - - -### Manipulating easyconfig parameters - -If you want update a particular easyconfig parameter without overwriting the existing value, -a bit of care has to be taken: you should use the ``update`` method of the ``EasyConfig`` instance -for this, unless you disable template resolution. This is particularly important when -updating easyconfig parameters that have *mutable* value (like a ``list`` or ``dict``). - -Here's a correct example of a pre-install hook: - -```python -def pre_install_hook(eb): - if eb.name == 'pigz': - # always copy the README directory too when installing pigz - eb.cfg.update('files_to_copy', 'README') -``` - -This seemingly equivalent implementation will ***not*** work (the value of the `files_to_copy` -easyconfig parameter will *not* be updated): - -```python -def pre_install_hook(eb): - if eb.name == 'pigz': - # incorrect way of adding 'README' to 'files_to_copy' (DON'T USE THIS!) - eb.cfg['files_to_copy'].append('README') -``` - -To use this coding style successfully, you have to disable the templating mechanism -when updating the easyconfig parameter: - -```python -def pre_install_hook(eb): - if eb.name == 'pigz': - # this works, but it is better to use the 'update' method instead... - with disable_templating(eb.cfg): - eb.cfg['files_to_copy'].append('README') -``` - ---- - -*[[next: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* diff --git a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md deleted file mode 100644 index 283a53a0..00000000 --- a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md +++ /dev/null @@ -1,333 +0,0 @@ -# Submitting installations as Slurm jobs - -*[[back: Using hooks to customise EasyBuild]](3_02_hooks.md)* - ---- - -EasyBuild can submit jobs to different backends including Slurm to install software, -to *distribute* the often time-consuming installation of a set of software applications and -the dependencies they require to a cluster. Each individual package is installed in a separate -job and job dependencies are used to manage the dependencies between package so that no build -is started before the dependencies are in place. - -This is done via the ``--job`` command line option. - -It is important to be aware of some details before you start using this, which we'll cover here. - -!!! Warning "This section is not supported on LUMI, use at your own risk" - - EasyBuild on LUMI is currently not fully configured to support job submission via Slurm. Several - changes would be needed to the configuration of EasyBuild, including the location of the - temporary files and build directory. Those have to be made by hand. - - Due to the setup of the central software stack, this feature is currently useless to install - the central stack. For user installations, there are also limitations as the enviornment - on the compute nodes is different from the login nodes so, e.g., different locations for - temporary files are being used. These would only be refreshed if the EasyBuild configuration - modules are reloaded on the compute nodes which cannot be done currently in the way Slurm - job submission is set up in EasyBuild. - - Use material in this section with care; it has not been completely tested. - - -## Configuration - -The EasyBuild configuration that is active at the time that ``eb --job`` is used -will be *passed down* into the submitted job automatically, via command line options to the ``eb`` -command that is run in the job script. - -This includes not only command line options used next to ``--job``, but also configuration settings -that are specified via an [EasyBuild configuration file](1_07_configuration#configuration-files) or through -[``$EASYBUILD_*`` environment variables](1_07_configuration#easybuild_-environment-variables). - -This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables -that are in place in the job environment are most likely *irrelevant*, since configuration settings -they specify they will most likely be overruled by the corresponding command line options. -It does also imply however that the EasyBuild configuration that is in place when ``eb --job`` is used -does also work on the compute nodes to which the job is submitted. - - -## Using ``eb --job`` - -### Job backend - -The default job backend in EasyBuild v4.x is [``GC3Pie``](https://gc3pie.readthedocs.io). -To let EasyBuild submit jobs to Slurm instead, you should set the ``job-backend`` configuration setting -to ``Slurm``, for example by setting the corresponding environment variable: - -```shell -export EASYBUILD_JOB_BACKEND='Slurm' -``` - -On LUMI this is taken care of in the EasyBuild configuration modules such as ``EasyBuild-user``. - - -### Job resources - -To submit an installation as a job, simply use ``eb --job``: - -```shell -eb example.eb --job -``` - -By default, EasyBuild will submit single-core jobs requesting for 24 hours of walltime. -You can tweak the requested resources via the ``job-cores`` and ``job-max-walltime`` configuration options. -For example: - -```shell -# submit job to install example, using 5 cores and 2 hours of max. walltime -eb example.eb --job --job-cores 5 --job-max-walltime 2 -``` - -Note that not all ``job-*`` configuration settings apply to all job backends, -see the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Submitting_jobs.html) for more details. - -### Controlling Slurm submission options - -When using Slurm as a job backend, EasyBuild will automatically generate job scripts which -use the ``eb`` command to perform a single installation. These scripts will be submitted -using the ``sbatch`` command. - -EasyBuild currently doesn't provide away to customize the Slurm submission options, -for example to submit to a particular partition, or to use a particular account, -build you can set the corresponding ``$SBATCH_*`` environment variables prior to running ``eb --job``. - -For example, to specify a particular account that should be used for the jobs submitted by EasyBuild -(equivalent with using the ``-A`` or ``--account`` command line option for ``sbatch``): - -```shell -export SBATCH_ACCOUNT='project_XXXXXXXXX' -``` - -Or to submit to a particular Slurm partition (equivalent with the ``-p`` or ``--partition`` option for ``sbatch``): - -```shell -export SBATCH_PARTITION='small' -``` - -For more information about supported ``$SBATCH_*`` environment variables, -see the [Slurm documentation](https://slurm.schedmd.com/sbatch.html#lbAJ). - -## Combining ``--job`` and ``--robot`` - -If one or more dependencies are still missing for the software you want to install, -you can combine ``--job`` and ``--robot`` to get EasyBuild to submit a *separate* job -for each of the installations. These jobs will *not* ``--robot``, they will each only -perform a single installation. - -Dependencies between jobs will be "registered" at submission time, so Slurm will put jobs -on hold until the jobs that install the required (build) dependencies have completed successfully, -and cancel jobs if the job to install a dependency failed for some reason. - -## Attention points - -There are a couple of important things to keep an eye on when submitting installations as jobs... - -### Differences on cluster workernodes - -Sometimes the resources available on the login nodes and cluster workernodes are slightly different, -and you may need to take this into account in your EasyBuild configuration. - -For example, plenty of disk space may be available in the `/tmp` temporary filesystem on a login node, -while the workernodes require you to use a different location for temporary files and directories. -As a result, you may need to slightly change your EasyBuild configuration when submitting installations -as jobs, to avoid that they fail almost instantly due to a lack of disk space. - -Keep in mind that the active EasyBuild configuration is passed down into the submitted jobs, -so any configuration that is present on the workernodes may not have any effect. - -For example, on LUMI it is possible to use ``$XDG_RUNTIME_DIR`` on the login nodes which has -the advantage that any leftovers of failed builds will be cleaned up when the user ends their last -login session on that node, but it is not possible to do so on the compute nodes. - -```shell -# EasByuild is configured to use /tmp/$USER on the login node -uan01 $ eb --show-config | grep buildpath -buildpath (E) = /run/user/XXXXXXXX/easybuild/build - -# use /dev/shm/$USER for build directories when submitting installations as jobs -login01 $ eb --job --buildpath /dev/shm/$USER/easybuild example.eb --robot -``` - - -### Temporary log files and build directories - -The problems for the temporary log files are twofold. First, they may end up in a place -that is not available on the compute nodes. E.g., for the same reasons as for the build -path, the LUMI EasyBuild configuration will place the temporary files in a subdirectory of -``$XDG_RUNTIME_DIR`` on the loginnodes but a subdirectory of ``/dev/shm/$USER`` on the -compute nodes. The second problem however is that if an installation fails, those log files are -not even accessible anymore which may leave you wondering about the actual cause of the failing -installation... - -To remedy this, there are a couple of EasyBuild configuration options you can use: - -* You can use ``--tmp-logdir`` to specify a different location where EasyBuild should store temporary log files, - for example: - ```shell - $ eb --job example.eb --tmp-logdir $HOME/eb_tmplogs - ``` - This will move at least the log file to a suitable place. - -* If you prefer having the entire log file stored in the Slurm job output files, - you can use ``--logtostdout`` when submitting the jobs. This will result in extensive logging - to your terminal window when submitting the jobs, but it will also make EasyBuild - log to ``stdout`` when the installation is running in the job, and hence the log messages will be - captured in the job output files. - -The build directory of course also suffers from the problem of being no longer accessible if the -installation fails, but there it is not so easy to find a solution. Building on a shared file system -is not only much slower, but in particular on parallel file systems like GPFS/SpectrumScale, Lustre -or BeeGFS buiding sometimes fails in strange ways. One thing you can consider if you cannot do the -build on a login node (e.g., because the code is not suitable for cross-compiling or the configure -system does tests that would fail on the login node), is to rety the installation in an -interactive job, so you can inspect the build directory after the installation fails. - -### Lock files - -EasyBuild creates [locks](https://docs.easybuild.io/en/latest/Locks.html) -to prevent that the same installation is started multiple times on -different system to the same installation directory. - -If an installation fails or gets interrupted, EasyBuild cleans up those locks automatically. - -However, if a Slurm job that is using EasyBuild to install software gets cancelled (because it -ran out of walltime, tried to consume too much memory, through an ``scancel`` command, etc.), -EasyBuild will not get the chance to clean up the lock file. - -If this occurs you will need to either clean up the lock file (which is located in the `software/.locks` -subdirectory of ``installpath``) manually, or re-submit the job with ``eb --job --ignore-locks``. - -## Example - -As an example, we will let EasyBuild submit jobs to install ``AUGUSTUS`` with the ``foss/2020b`` toolchain. - -!!! Warning "This example does not work on LUMI" - - Note that this is an example using the FOSS common toolchain. For this reason it does not work on - LUMI. - -### Configuration - -Before using ``--job``, let's make sure that EasyBuild is properly configured: - -```shell -# Load the EasyBuild-user module (central installations will not work at all -# using job submission) -module load LUMI/21.12 -module load partition/C -module load EasyBuild-user - -# use ramdisk for build directories -export EASYBUILD_BUILDPATH=/dev/shm/$USER/build -export EASYBUILD_TMPDIR=/dev/shm/$USER/tmp - -# use Slurm as job backend -export EASYBUILD_JOB_BACKEND=Slurm -``` - - -We will also need to inform Slurm that jobs should be submitted into a particular account, and -in a particular partition: - -```shell -export SBATCH_ACCOUNT=project_XXXXXXXXX -export SBATCH_PARTITION='small' -``` - -This will be picked up by the ``sbatch`` commands that EasyBuild will run to submit the software installation jobs. - - -### Submitting jobs to install AUGUSTUS - -Now we can let EasyBuild submit jobs for AUGUSTUS. - -Let's first check what is still missing: - -```shell -$ eb AUGUSTUS-3.4.0-foss-2020b.eb --missing -... -11 out of 61 required modules missing: - -* HTSlib/1.11-GCC-10.2.0 (HTSlib-1.11-GCC-10.2.0.eb) -* lpsolve/5.5.2.11-GCC-10.2.0 (lpsolve-5.5.2.11-GCC-10.2.0.eb) -* Boost/1.74.0-GCC-10.2.0 (Boost-1.74.0-GCC-10.2.0.eb) -* GSL/2.6-GCC-10.2.0 (GSL-2.6-GCC-10.2.0.eb) -* SAMtools/1.11-GCC-10.2.0 (SAMtools-1.11-GCC-10.2.0.eb) -* BCFtools/1.11-GCC-10.2.0 (BCFtools-1.11-GCC-10.2.0.eb) -* METIS/5.1.0-GCCcore-10.2.0 (METIS-5.1.0-GCCcore-10.2.0.eb) -* BamTools/2.5.1-GCC-10.2.0 (BamTools-2.5.1-GCC-10.2.0.eb) -* MPFR/4.1.0-GCCcore-10.2.0 (MPFR-4.1.0-GCCcore-10.2.0.eb) -* SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0.eb) -* AUGUSTUS/3.4.0-foss-2020b (AUGUSTUS-3.4.0-foss-2020b.eb) -``` - -Several dependencies are not installed yet, so we will need to use ``--robot`` to ensure that -EasyBuild also submits jobs to install these first. - -To speed up the installations a bit, we will request 8 cores for each submitted job (via ``--job-cores``). -That should be sufficient to let each installation finish in (well) under 1 hour, -so we only request 1 hour of walltime per job (via ``--job-max-walltime``). - -In order to have some meaningful job output files, we also enable trace mode (via ``--trace``). - -``` -$ eb AUGUSTUS-3.4.0-foss-2020b.eb --job --job-cores 8 --job-max-walltime 1 --robot --trace -... -== resolving dependencies ... -... -== List of submitted jobs (11): Boost-1.74.0-GCC-10.2.0 (Boost/1.74.0-GCC-10.2.0): 1000011; GSL-2.6-GCC-10.2.0 (GSL/2.6-GCC-10.2.0): 1000004; SAMtools-1.11-GCC-10.2.0 (SAMtools/1.11-GCC-10.2.0): 1000005; HTSlib-1.11-GCC-10.2.0 (HTSlib/1.11-GCC-10.2.0): 1000006; BCFtools-1.11-GCC-10.2.0 (BCFtools/1.11-GCC-10.2.0): 1000001; lpsolve-5.5.2.11-GCC-10.2.0 (lpsolve/5.5.2.11-GCC-10.2.0): 1000007; BamTools-2.5.1-GCC-10.2.0 (BamTools/2.5.1-GCC-10.2.0): 1000008; METIS-5.1.0-GCCcore-10.2.0 (METIS/5.1.0-GCCcore-10.2.0): 1000009; MPFR-4.1.0-GCCcore-10.2.0 (MPFR/4.1.0-GCCcore-10.2.0): 1000010; SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0): 1000002; AUGUSTUS-3.4.0-foss-2020b (AUGUSTUS/3.4.0-foss-2020b): 1000003 -== Submitted parallel build jobs, exiting now -``` - -### Inspecting the submitted jobs - -Once EasyBuild has submitted the jobs, we can inspect them via Slurm's ``squeue`` command: - -``` -$ squeue -u $USER -la - JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON) -1000001 small BCFtools user123 PENDING 0:00 2:00:00 1 (Dependency) -1000002 small SuiteSpa user123 PENDING 0:00 2:00:00 1 (Dependency) -1000003 small AUGUSTUS user123 PENDING 0:00 2:00:00 1 (Dependency) -1000004 small GSL-2.6- user123 RUNNING 0:21 2:00:00 1 node003 -1000005 small SAMtools user123 RUNNING 0:21 2:00:00 1 node007 -1000006 small HTSlib-1 user123 RUNNING 0:21 2:00:00 1 node007 -1000007 small lpsolve- user123 RUNNING 0:21 2:00:00 1 node011 -1000008 small BamTools user123 RUNNING 0:21 2:00:00 1 node011 -1000009 small METIS-5. user123 RUNNING 0:21 2:00:00 1 node013 -1000010 small MPFR-4.1 user123 RUNNING 0:21 2:00:00 1 node029 -1000011 small Boost-1. user123 RUNNING 0:24 2:00:00 1 node029 -``` - -Note that 3 jobs can not be started yet, because those installations require on one or more -missing dependencies. As soon as the jobs for those dependencies (successfully) complete, -these jobs will be able to start. - -### Final result - -After about 20 minutes, AUGUSTUS and all missing dependencies should be installed: - -``` -$ ls -lrt $HOME/EasyBuild/modules/.../*.lua | tail -11 --rw-rw----. 1 example example 1634 Mar 29 10:13 /users/example/easybuild/modules/all/HTSlib/1.11-GCC-10.2.0.lua --rw-rw----. 1 example example 1792 Mar 29 10:13 /users/example/easybuild/modules/all/SAMtools/1.11-GCC-10.2.0.lua --rw-rw----. 1 example example 1147 Mar 29 10:13 /users/example/easybuild/modules/all/BamTools/2.5.1-GCC-10.2.0.lua --rw-rw----. 1 example example 957 Mar 29 10:13 /users/example/easybuild/modules/all/lpsolve/5.5.2.11-GCC-10.2.0.lua --rw-rw----. 1 example example 1549 Mar 29 10:13 /users/example/easybuild/modules/all/METIS/5.1.0-GCCcore-10.2.0.lua --rw-rw----. 1 example example 1525 Mar 29 10:14 /users/example/easybuild/modules/all/GSL/2.6-GCC-10.2.0.lua --rw-rw----. 1 example example 1221 Mar 29 10:15 /users/example/easybuild/modules/all/MPFR/4.1.0-GCCcore-10.2.0.lua --rw-rw----. 1 example example 1678 Mar 29 10:15 /users/example/easybuild/modules/all/BCFtools/1.11-GCC-10.2.0.lua --rw-rw----. 1 example example 1292 Mar 29 10:21 /users/example/easybuild/modules/all/Boost/1.74.0-GCC-10.2.0.lua --rw-rw----. 1 example example 1365 Mar 29 10:28 /users/example/easybuild/modules/all/SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0.lua --rw-rw----. 1 example example 2233 Mar 29 10:30 /users/example/easybuild/modules/all/AUGUSTUS/3.4.0-foss-2020b.lua - -$ module avail AUGUSTUS - --- EasyBuild managed user software for software stack ... -- - AUGUSTUS/3.4.0-foss-2020b -``` - ---- - -*[[next: Module naming schemes]](3_04_module_naming_scheme)* diff --git a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md deleted file mode 100644 index d0b4ebe8..00000000 --- a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md +++ /dev/null @@ -1,569 +0,0 @@ -# Module naming schemes - -*[[back: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* - ---- - -Up until now we have used the naming scheme in use on LUMI which is for many practical purposes -is indistinguishable from the default EasyBuild naming scheme (`EasyBuildMNS`). -It produces module files with names that closely resemble to the names of the -corresponding easyconfig files. -For example, when installing `zlib-1.2.11-cpeGNU-21.12.eb` the generated module was named -`zlib/1.2.11-cpeGNU-21.12`. - -EasyBuild supports several different module naming schemes: - -```shell -$ eb --avail-module-naming-schemes -List of supported module naming schemes: - EasyBuildMNS - LUMI_FlatMNS - MigrateFromEBToHMNS - HierarchicalMNS - CategorizedHMNS - CategorizedModuleNamingScheme -``` - -In this part of the tutorial we will take a closer look at `HierarchicalMNS`, -which is the standard **hierarchical** module naming scheme included with EasyBuild. - -We will also take a quick look at implementing our own custom module naming scheme. - -## Flat vs hierarchical - -!!! Note - - This text is very much based on the generic EasyBuild tutorials as we cannot demonstrate - all aspects on LUMI as it is configured today. - - On LUMI, the complete module scheme is partly hierarchical, but EasyBuild uses a flat naming - scheme. The two levels in the hierarchy that are present to deal with different versions of - the software stack and the various types of compute nodes, are not managed by EasyBuild. - -The default module naming scheme `EasyBuildMNS` -and the custom `LUMI_FlatMNS` naming scheme are both examples of regular *"flat"* -module naming schemes, which are characterized by: - -- all module files are directly available for loading; -- each module name uniquely identifies a particular installation; - -In contrast, a *hierarchical* module naming scheme -consists of a *hierarchy* of module files. -A fairly typical 3-level schme (``Core``, ``Compiler`` and ``MPI``) has been -discussed in the [section on Lmod](../1_02_Lmod#lmod-hierarchy). -This typical Lmod hierarcny would map very well on the EasyBuild common toolchains. - -In fact, for the example - -
- -software at the ``Core`` level would be installed with the ``GCCcore`` and ``SYSTEM`` -toolchains. Software at the ``Compiler`` level would be installed with the ``GCC`` -toolchain, while software at the ``MPI`` level after loading the ``OpenMPI`` module -would be installed using the regular ``foss`` toolchain or the ``gompi`` toolchain -(see the diagram in the -["Common toolchains" section of the page on terminology](../1_05_terminology/#common-toolchains)). - -On LUMI, where software is installed through the Cray Programming Environment with no real choice of -MPI implementation, a two-level arrangement would still make a lot of sense, with at the ``Core`` level -all software compiled with the SYSTEM toolchain while there could be a ``PrgEnv`` level for software -compiled with a particula programming enviornment aka cepGNU/cpeCray/cpeAOCC toolchain. Such a scheme -is used on the Cray systems at CSCS. - -To recap, the characteristics of a module hierarchy are: - -* not all module files are directly available for loading; -* some modules serve as a gateway to more modules; -* to access some software installations you will first need to load one or more gateway modules in order - to use them; - -You can probably think of other ways to organize module files in a hierarchical module tree, but here -we will stick to the standard core / compiler / MPI hierarchy. - -### Pros & cons - -So why go through all this trouble of organizing modules hierarchically? - -There are a couple of advantages to this approach: - -* shorter module names (or at least for the version part of the name); -* less overwhelming list of available modules; -* only compatible modules can be loaded together; - -However, the are some minor disadvantages too: - -* not all existing modules are directly visible, so users have to learn how to find - modules using ``module spider`` etc.; -* gateway modules may have little meaning to end users; - -#### Length of module names - -When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For a `HDF5` installation -with the EasyBuild common toolchains for example, -one might have `HDF5/1.10.6-gompi-2020a` as module name. The `-gompi-2020a` part of the name refers to the toolchain that was -used for this installation, but it may be confusing to some people (what kind of Pokémon is a "gompi"?!). - -In the example module hierarchy shown above, the module for `HDF5` could simply be named `HDF5/1.10.6` which is basically the bare -essentials: software name and version. That's way better, nice and clean! - -#### Amount of available modules - -The output of "`module avail`" can be quite overwhelming if lots of module files -are installed and a flat module naming scheme is used, since *all* modules are -*always* available. -EasyBuild makes it very easy to install lots of software, -so the number of installed modules can easily grow into the hundreds or even thousands... - -This often explosive growth of modules is less of an issue when using a hierarchical module naming scheme, since -initially only a small set of modules are available, and relatively limited -groups of additional modules become available as gateway modules are loaded. - -#### Loading compatible modules - -Since all modules are available at once when using a flat module naming scheme, you can easily load -modules together that are not compatible with each other. - -Imagine loading two modules that were built with a different compiler toolchain (different compiler, -different MPI library). That's likely to end in tears, unless you have the necessary technical expertise -to fully understand what is going on, *and* you are being very careful... - -In a module hierarchy this can be prevented, since modules for software that were installed with a -different compiler and/or a different MPI library are located in a different part of the module hierarchy, -and so these modules can not be loaded together easily. - - -#### Visibility of existing modules - -One downside of a module hierarchy is that not all existing modules are directly available for loading -or are even *visible* to the user, since the output of "`module avail`" only shows a subset of all modules. - -As we have discussed in [the Lmod section](../LMOD). -[Lmod](https://lmod.readthedocs.io) provides a powerful solution to search for modules through the -``module spider`` and to some extent also the ``module keyword`` commands. -So as long as the end users are -aware of this additional command, it should not be difficult to discover which software installations exist -and how they can be accessed. The "`module spider`" command will inform the user which of the gateway modules -need to be loaded in order to load a specific module file. - -#### Semantics of gateway modules - -An additional potential problem of a module hierarchy is that the semantics of the gateway modules may not be clear -to end users. They may wonder why they need to pick a specific compiler and MPI library, or which of the -available options is the best one. Maybe they are not even aware what exactly a "compiler" is, or how it is -relevant to the software they need in their bioinformatics pipeline... - -This can be partially resolved by loading a default compiler and MPI module so a particular set of modules -is available right after login, which could be the ones used in the most recent toolchain, or the -recommended versions. More experienced users could then leverage the "`module spider`" command to navigate -the module hierarchy. - -On LUMI this would mean loading a default software stack, but due to the the default modules are currently -loaded on LUMI this was not possible to accomplish without losing other functionality of the module tree. - -## Using a custom module naming scheme - -Next to the module naming schemes that are included with EasyBuild, -you can also define your own module naming scheme (MNS), and configure EasyBuild to use it -(which is eactly what has been done on LUMI to remove a feature of the default ``EasyBuildMNS`` scheme -that we do not use). - -### Implementation - -To use a custom module naming scheme, you need to implement a Python module file, -where you define a Python class that derives from the general [``ModuleNamingScheme``](https://docs.easybuild.io/en/latest/api/easybuild.tools.module_naming_scheme.mns.html#easybuild.tools.module_naming_scheme.mns.ModuleNamingScheme) class. - -For a flat module naming scheme, it is sufficient to implement the ``det_full_module_name`` method, -which should return a string value (the full module name). -You may also need to customize the ``is_short_modname_for`` method, which verifies whether -a given (short) module name is for the software with a particular given name (or not). - -The argument provided to ``det_full_module_name`` can be a small Python dictionary which only specifies -the value of a handful of easyconfig parameters (`name`, `version`, `toolchain`, and `versionsuffix`), -or an ``EasyConfig`` instance which represents a parsed easyconfig file and contains values for *all* -known easyconfig parameters. - -For simple module naming schemes, just have ``name``, ``version``, ``toolchain``, and ``versionsuffix`` -available is sufficient. If it is not, you can list which additional easyconfig parameters are required for -your module naming scheme via a class constant ``REQUIRED_KEYS``. - -A fairly straightforward example of a *flat* module name scheme is the default EasyBuild module naming scheme -[``EasyBuildMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/easybuild_mns.py). - -For a *hierarchical* module naming scheme, various additional methods have to be implemented. - -Some of these, like ``det_module_subdir`` and ``det_short_module_name``, determine which part of -the (full) module name determines the location of the module in the module hierarchy, and which part -is the user-facing (or "short") module name. Others, like ``det_modpath_extensions``, -determine which modules are *gateway* modules that open up an additional level of the hierarchy -(by *extending* the ``$MODULEPATH``). - -A typical example of a hierarchical module naming scheme is [``HierarchicalMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/hierarchical_mns.py) that comes with EasyBuild (which is quite complex because it includes quite a bit of special handling for particular compiler -toolchains), which implements the traditional ``Core``-``Compiler``-``MPI`` module hierarchy we discussed -above. - -### Configuring EasyBuild - -To let EasyBuild use a custom module naming scheme, you need to: - -* specify the path to the Python module file implementing it via the ``include-module-naming-schemes`` - EasyBuild configuration option; - -* indicate that you also want to *use* this custom module naming scheme via the ``module-naming-scheme`` - EasyBuild configuration option. - -For example: - -```shell -export EASYBUILD_INCLUDE_MODULE_NAMING_SCHEMES=$HOME/easybuild/example_mns.py -export EASYBUILD_MODULE_NAMING_SCHEME=ExampleMNS -``` - -#### Example custom module naming scheme - -Here is an example of a custom module naming scheme, where: - -* the ``versionsuffix`` goes directly after the ``version``; -* all dashes are replaced by underscores; -* all module names are lowercase; - -Note that we also need to customise the ``is_short_modname_for`` method, -to make sure it returns ``True`` when the EasyBuild framework checks whether -``scipy_bundle/2020.11_foss_2020b`` is a module name for ``SciPy-bundle``. - -```python -import os - -from easybuild.tools.module_naming_scheme.mns import ModuleNamingScheme - - -class ExampleMNS(ModuleNamingScheme): - - REQUIRED_KEYS = ['name', 'version', 'versionsuffix', 'toolchain'] - - def det_full_module_name(self, ec): - """ - Determine full module name: - - all lowercase - - replace all dashes with underscores - """ - parts = [ec['version']] - - # versionsuffix directly after version (but only if it's not empty) - if ec['versionsuffix']: - parts.append(ec['versionsuffix']) - - # only add toolchain name/version for non-system toolchain - tc = ec['toolchain'] - if tc['name'].lower() != 'system': - parts.extend([tc['name'], tc['version']]) - - modname = ec['name'] + '/' + '_'.join(parts) - - modname = modname.replace('-', '_').replace('__', '_') - - return modname.lower() - - def is_short_modname_for(self, short_modname, name): - """Determine whether short module name is a module for the software with specified name.""" - return short_modname.startswith(name.lower().replace('-', '_') + '/') -``` - -We can see what the module names with this module naming scheme would like like via ``eb -D``. -E.g., for the common toolchains (the example will not work on LUMI as the default easyconfig -files are not in the robot- and the search path): - -``` -$ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D - ... - * [ ] $CFGS/g/GCC/GCC-10.2.0.eb (module: gcc/10.2.0) - ... - * [ ] $CFGS/p/Python/Python-2.7.18-GCCcore-10.2.0.eb (module: python/2.7.18_gcccore_10.2.0) - ... - * [ ] $CFGS/o/OpenMPI/OpenMPI-4.0.5-GCC-10.2.0.eb (module: openmpi/4.0.5_gcc_10.2.0) - ... - * [ ] $CFGS/s/SciPy-bundle/SciPy-bundle-2020.03-foss-2020a-Python-2.7.18.eb (module: scipy_bundle/2020.03_python_2.7.18_foss_2020a) -``` - -## Example module hierarchy: HDF5 - -!!! Warning "Example not suitable for LUMI" - **This exercise is meant for a system where the common toolchains can be used and requires an - indpendent EasyBuild installation in your personal file space**, - because EasyBuild will try to copy the installation log file to each installation directory. - -Now that we know more about hierarchical module naming schemes, -let us see how EasyBuild can help us with generating a hierarchical module tree. - -In this example we will use EasyBuild to generate modules organised in a hierarchy -for some of the software that is already installed in the prepared environment. - -The good news is that the existing installations can be reused. There is absolutely no need -to reinstall the software, we are just creating a different "view" on these software installations. - -### Preparing the environment - -Before running EasyBuild to generate a hierarchical module tree, we have to be a bit careful -with preparing our environment. - -**We must absolutely avoid mixing modules from a flat and hierarchical module naming scheme!** - -Some module files will have the same name in both module trees (like `GCC/10.2.0` for example), -but their contents will be different. -Mixing modules from a flat and hierarchical module tree *will* trigger problems... - -So we have to make sure that the module files we already have in `/easybuild` are *not* visible. -The easiest way to do this is to unload all modules (using "`module purge`") -and resetting the module search path to be empty, which we can do with "`module unuse $MODULEPATH`". - -```shell -module purge -module unuse $MODULEPATH -``` - -In this part of the tutorial, we are assuming you are *not* using an EasyBuild installation provided through -a module. We have just made all modules unavailable, so we would have to first -install EasyBuild again in our hierarchical module tree before we can continue. - -**We strongly recommend using an EasyBuild installation that was [installed via "`pip install`" -or "`pip3 install`"](../1_06_installation#method-1-using-pip) in this part of the tutorial.** - -An easy way to do this is in the prepared environment is to run: - -```shell -pip3 install --user easybuild -export PATH=$HOME/.local/bin:$PATH -export EB_PYTHON=python3 -``` - -### Configuring EasyBuild - -First of all, we need to make sure that EasyBuild is properly configured. -We can do this by defining this set of environment variables: - -```shell -export EASYBUILD_PREFIX=$HOME/easybuild -export EASYBUILD_BUILDPATH=/tmp/$USER -export EASYBUILD_INSTALLPATH_SOFTWARE=/easybuild/software -export EASYBUILD_MODULE_NAMING_SCHEME=HierarchicalMNS -export EASYBUILD_INSTALLPATH_MODULES=$HOME/hmns/modules -``` - -To make sure we didn't make any silly mistakes, we double check using `eb --show-config`: - - -```shell -$ eb --show-config -# -# Current EasyBuild configuration -# (C: command line argument, D: default value, E: environment variable, F: configuration file) -# -buildpath (E) = /tmp/example -containerpath (E) = /home/example/easybuild/containers -installpath (E) = /home/example/easybuild -installpath-modules (E) = /home/example/hmns/modules -installpath-software (E) = /easybuild/software -module-naming-scheme (E) = HierarchicalMNS -packagepath (E) = /home/example/easybuild/packages -prefix (E) = /home/example/easybuild -repositorypath (E) = /home/example/easybuild/ebfiles_repo -robot-paths (D) = /home/example/.local/easybuild/easyconfigs -sourcepath (E) = /home/example/easybuild/sources -``` - -There are a couple of things worth pointing out here: - -* We have defined the `module-naming-scheme` configuration setting to `HierarchicalMNS`, - which makes EasyBuild use the included standard hierarchical module naming scheme (the classic - core / compiler / MPI one we discussed above). -* We have specified *different* locations for the software (via `installpath-software`) - and the module files (via `installpath-modules`). This is important because we want to - reuse the software that is already installed in `/easybuild/software` while we want to - generate an entirely new module tree for it (in `$HOME/hmns/modules`). - -The other configuration settings are the same as before, and mostly irrelevant for this example. - -### Generating modules for HDF5 - -Let us now generate a hierarchical module tree for `HDF5` and all of its dependencies, -including the toolchain. That sounds complicated, and it sort of is since there are -a lot of details you have to get right for the module hierarchy to works as intended, -but EasyBuild can do all the hard work for us. - -The steps we will have to go through are: - -* Tell EasyBuild we want to "install" the `HDF5-1.10.7-gompi-2020b.eb` easyconfig file; -* Enable dependency resolution via `--robot`; -* Assuming the software would have been installed already with the default naming scheme - in a different module directory, instruct EasyBuild to only generate the module files, - not to install the software (since it is - there already in `/easybuild/software`), via the `--module-only` option. - -These steps translate to this single `eb` command: - -``` -$ eb HDF5-1.10.7-gompi-2020b.eb --robot --module-only -... -== building and installing MPI/GCC/10.2.0/OpenMPI/4.0.5/HDF5/1.10.7... -... -== sanity checking... -== cleaning up [skipped] -== creating module... -... -== COMPLETED: Installation ended successfully (took 9 sec) -... -== Build succeeded for 41 out of 41 -``` - -This should take a couple of minutes in total, for generating 41 modules. -Remember that this also includes generating module files for the toolchain and -all of its components. - -In addition, there is a bit more going on one that just generating module files, -since the sanity check step is still being run for each of the installations -when using `--module-only` to ensure the installation is actually functional. -After all, there is no point in generating a module for an obviously broken -installation... - -### Loading the HDF5 module - -After generating the hierarchical module tree for HDF5, how do we access the HDF5 installation through it? - -Here's what the module tree looks like on disk: - -``` -$ ls $HOME/hmns/modules/all -Compiler Core MPI -``` - -Those are basically the 3 levels in the module hierarchy we showed in our example earlier. - -The starting point is the top level of the module hierarchy named `Core`: - -``` -module use $HOME/hmns/modules/all/Core -``` - -Let us see what that gives us in terms of available modules: - -``` -$ module avail - ---------------------- /home/easybuild/hmns/modules/all/Core --------------------- - Bison/3.5.3 GCCcore/10.2.0 flex/2.6.4 help2man/1.47.4 - Bison/3.7.1 (D) M4/1.4.18 gettext/0.21 ncurses/6.2 - GCC/10.2.0 binutils/2.35 gompi/2020b zlib/1.2.11 -``` - - -Nice and short module names, but only a limited set of them. - -We know a module file exists for `HDF5`, but we can't see it yet (and hence -we can't load it either). - -``` -$ module avail HDF5 -No module(s) or extension(s) found! -Use "module spider" to find all possible modules and extensions. -``` - -Let us see if `module spider` is of any help, as "`module avail`" so kindly suggests: - -``` -$ module spider HDF5 -... - - You will need to load all module(s) on any one of the lines below - before the "HDF5/1.10.7" module is available to load. - - GCC/10.2.0 OpenMPI/4.0.5 -``` - -This tells us we need to load two gateway modules before we can load the module -for HDF5. - -Let us start with loading the `GCC` compiler module: - -``` -module load GCC/10.2.0 -``` - -And then check again which modules are available: - -``` -$ module avail - --------------- /home/easybuild/hmns/modules/all/Compiler/GCC/10.2.0 -------------- - OpenMPI/4.0.5 - ------------- /home/easybuild/hmns/modules/all/Compiler/GCCcore/10.2.0 ------------ - Autoconf/2.69 UCX/1.9.0 libpciaccess/0.16 - ... - Szip/2.1.1 libfabric/1.11.0 zlib/1.2.11 (L,D) - Autoconf/2.69 XZ/5.2.5 libtool/2.4.6 - ---------------------- /home/easybuild/hmns/modules/all/Core --------------------- - Bison/3.5.3 GCCcore/10.2.0 (L) flex/2.6.4 help2man/1.47.4 - Bison/3.7.1 (D) M4/1.4.18 gettext/0.21 ncurses/6.2 - GCC/10.2.0 (L) binutils/2.35 gompi/2020b zlib/1.2.11 -``` - -Good news, we now have additional modules available! - -The compiler level of our hierarchy actually consists of two directories here: `Compiler/GCCcore/10.2.0` -and `Compiler/GCC/10.2.0`. The modules in the `GCCcore` directory are ones we can use in other compiler -toolchains that use GCC 10.2.0 as a base compiler (the details of that are out of scope here). - -The module we are interested in is `OpenMPI/4.0.5`, which is another gateway module. - -Remember that the "`module spider`" output told us that there does indeed exist a module for `HDF5`, but that -we need to load *both* the `GCC/10.2.0` and `OpenMPI/4.0.5` modules first. - -So, let us do exactly that (remember that `GCC/10.2.0` is already loaded): - -``` -module load OpenMPI/4.0.5 -``` - -If you now check the output of "`module avail`" again, you should see the `HDF5/1.10.7` module: - -``` -$ module avail - --------- /home/easybuild/hmns/modules/all/MPI/GCC/10.2.0/OpenMPI/4.0.5 ------- - HDF5/1.10.7 - ------------- /home/easybuild/hmns/modules/all/Compiler/GCC/10.2.0 ------------ - OpenMPI/4.0.5 (L) - -... -``` - -To use HDF5, we need to load this `HDF5/1.10.7` module. We can verify that the installation works -using one of the commands provided by HDF5, `h5dump` for example: - -``` -module load HDF5/1.10.7 -``` - -``` -$ h5dump --version -h5dump: Version 1.10.7 -``` - -If you now check which modules are loaded via "`module list`", you will notice that all module names -and nice and short now, which is one of the advantages of using a hierarchical module tree: - -```shell -$ module list - -Currently Loaded Modules: - 1) GCCcore/10.2.0 5) numactl/2.0.13 9) hwloc/2.2.0 13) PMIx/3.1.5 - 2) zlib/1.2.11 6) XZ/5.2.5 10) libevent/2.1.12 14) OpenMPI/4.0.5 - 3) binutils/2.35 7) libxml2/2.9.10 11) UCX/1.9.0 15) Szip/2.1.1 - 4) GCC/10.2.0 8) libpciaccess/0.16 12) libfabric/1.11.0 16) HDF5/1.10.7 -``` - - ---- - -*[[next: GitHub integration]](3_05_github_integration.md)* diff --git a/docs/2022-CSC_and_LO/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_05_github_integration.md deleted file mode 100644 index 6e19cb9a..00000000 --- a/docs/2022-CSC_and_LO/3_05_github_integration.md +++ /dev/null @@ -1,460 +0,0 @@ -# GitHub integration to facilitate contributing to EasyBuild - -*[[back: Module naming schemes]](3_04_module_naming_schemes.md)* - ---- - -!!! Warning "Preliminary not for LUMI and many other Cray systems" - - Not everything in this section is as useful for EasyBuild installations that build - on the Cray Programming Environment. E.g., the default easyconfig repository does - not contain easyconfigs for the Cray PE. - - Moreover, due to the restricted operating system on the compute nodes of big - Cray systems such as the EX series (COS instead of the full SUSE linux), - the GitHub integration does not work on the compute nodes of a typical setup as - one of the required Python packages fails to work as it needs the DBus daemon. - - Since GitHub integration is not yet really used on LUMI, this section is not fully - tested but left in the tutorial for completeness. - -To contribute changes to the EasyBuild code (framework or easyblocks) or easyconfigs, -you will need to be a bit familiar with Git and GitHub. Or maybe not? - -## Manual contribution procedure - -0) Create and setup a [GitHub account](https://github.com/join) (and register your SSH public key); - -1) Clone and fork the appropriate GitHub repository, for example when contributing an easyconfig file: - -```shell -git clone git@github.com:easybuilders/easybuild-easyconfigs.git -cd easybuild-easyconfigs -git remote add my_fork git@github.com:your_github_account/easybuild-easyconfigs.git -``` - -**You should change '`your_github_account`' in the last line to your own GitHub user name!** - -2) Create and check out a new branch, starting from the (up-to-date) ``develop`` branch: - -``` -git checkout develop -git pull origin develop -git checkout -b example -``` - -3) Stage the changes you want to contribute, after you make sure that your easyconfig file has the -[correct filename](../basic_usage/#easyconfig-filenames), and that it's located in the appropriate directory. - -```shell -mkdir -p easybuild/easyconfigs/e/example/ -mv example.eb easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb -git add easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb -``` - -4) Commit those changes with a sensible commit message: - -```shell -git commit -m "This is just an example" -``` - -5) Push your branch to your fork of the repository on GitHub: - -```shell -git push my_fork example -``` - -6) Open the pull request through the GitHub web interface, making sure that: - -* the target branch is correct (should be `develop`); -* an appropriate title is used; -* a short description of the changes is provided; -* the changes are indeed the ones you want to propose; -* clicking the (correct) green button; - -
Pfft
- -That didn't exactly motivate you to contribute, did it... - -## Github integration features - -Over the years we noticed that some people were keen on contributing to EasyBuild, -but they were not very familiar with Git or GitHub. That meant they had to overcome a -relatively steep learning curve before they could contribute... - -
Gandalf vs Git
- -In addition, the contribution workflow can be a bit daunting and time consuming, -even if you're already familiar with the procedure. You will have dozens of -branches flying around in no time, and if you get stuck in a weird corner -with `git` you may quickly end up demotivated. - -This is frustrating not only for the people who wanted to contribute but -also for the EasyBuild maintainers, and it doesn't agree with the philosophy of -a project that aims to *automate* tedious software installation procedures. - -At the end of 2015 efforts were made to tackle this issue by implementing -GitHub integration features in EasyBuild, which automate the contribution -workflow by running `git` commands and interacting with the [GitHub API](https://developer.github.com/v3/). - -We will briefly go over some of these features here, but they are also covered in detail [in the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html). - -### Requirements & configuration - -First of all, the GitHub integration features impose a couple of additional [requirements](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) -and configuration. - - -**Additional dependencies** - -Both the `GitPython` and `keyring` Python packages as well as the `keyrings.cryptfile` add-on package must be installed. - -!!! Note - You may experiences problems installing the ``cryptography`` Python packages, - which is a dependency of keyring. The underlying cause is that you need to have - the [``Rust``](https://www.rust-lang.org/) compiler installed to install the latest version - of ``cryptography`` (see [here](https://github.com/pyca/cryptography/issues/5771)). - - You can work around this issue using: - - ```shell - pip3 install --user 'cryptography<3.4' - ``` - -**SSH public key in GitHub account** - -You need to have a GitHub account that has your SSH public key registered in it -(via [https://github.com/settings/keys](https://github.com/settings/keys)). - -If you need to generate an SSH key pair, you can run the following command: - -```shell -ssh-keygen -t rsa -b 4096 -``` - -You can copy the SSH public key from the output of this command: - -```shell -cat .ssh/id_rsa.pub -``` - - -**Forked repository in GitHub** - -In addition, you must have *forked* the EasyBuild repository you want to contribute to -(for example [https://github.com/easybuilders/easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs)). - -**EasyBuild configuration, incl. GitHub token** - -You also have to configure EasyBuild a bit more, so it knows about your -GitHub user name *and* has a GitHub token available in order to perform actions -in GitHub with your credentials. - -To do this, you should define the `github-user` configuration option and -run the "`eb --install-github-token`" command: - -```shell -# replace 'ebtutorial' with your own GitHub username! -$ export EASYBUILD_GITHUB_USER=ebtutorial -$ eb --install-github-token -``` - -To create a GitHub token: - -* Visit [https://github.com/settings/tokens](https://github.com/settings/tokens). -* Click *"Personal access tokens"*. -* Click followed by *"Generate new token"*. -* Give the token a name (for example *"Token for EasyBuild"*). -* Select both the '`repo`' and '`gist`' scopes. -* Click the green *"Generate token"* button. -* Copy the generated token. -* Paste the token when asked by `--install-github-token` (and hit *Enter*). -* Enter a password to encrypt your GitHub token. - -The output should look something like this: - -```shell -$ eb --install-github-token -== temporary log file in case of crash /tmp/eb-9z0bdve9/easybuild-hfpti62w.log -Token: -Validating token... -Token seems to be valid, installing it. -Please set a password for your new keyring: -Please confirm the password: -Token 'fed..987' installed! -``` - - -**Checking status of GitHub integration** - -You can check the status of the GitHub integration using "`eb --check-github`": - -```shell -$ eb --check-github -== temporary log file in case of crash /tmp/eb-4ckdlyfy/easybuild-gp69ev2w.log - -Checking status of GitHub integration... - -Making sure we're online...OK - -* GitHub user...ebtutorial => OK -Please enter password for encrypted keyring: -* GitHub token...fed..987 (len: 40) => OK (validated) -* git command...OK ("git version 1.8.3.1; ") -* GitPython module...OK (GitPython version 3.1.3) -* push access to ebtutorial/easybuild-easyconfigs repo @ GitHub...OK -* creating gists...OK -* location to Git working dirs... not found (suboptimal) - -All checks PASSed! - -Status of GitHub integration: -* --from-pr: OK -* --new-pr: OK -* --review-pr: OK -* --update-pr: OK -* --upload-test-report: OK -``` - -If you see '`OK`' for each of the status checks, you're all set -to try out the GitHub integration features! - -!!! Note - If your SSH private key is protected with a password, you may need - to enter your password a couple of times when running "`eb --check-github`". - - You can avoid this by [using an SSH agent](https://help.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent). - -!!! Note - If you see the ``push access`` check fail with ``Failed to fetch branch 'main'``, - you will need to rename the ``master`` branch in your fork of the ``easybuild-easyconfigs`` - repository from ``master`` to ``main`` (this is required since EasyBuild v4.3.3). - - This can be done via the *pencil* icon at - [https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches](https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches) - (replace ``YOUR_GITHUB_ACCOUNT`` with the name of your GitHub account in this URL). - -### Creating pull requests - -The GitHub integration in EasyBuild allows you to **create pull requests -using the `eb` command**, without even leaving your shell environment. -How cool is that‽ - -To create a pull request to the `easybuild-easyconfigs` repository, -you can either do it in a single go by -running "`eb --new-pr`" and passing it one or more easyconfig files to add -into the pull request. - -The more detailed option is to first create a branch in your repository fork -in GitHub via "`eb --new-branch-github`" and then later open the pull request -via "`eb --new-pr-from-branch`". This method can be useful when preparing multiple -interdependent pull requests, or to check whether your changes pass the unit tests -(which are run automatically in the GitHub Actions CI environment for -all branches pushed to your fork). - -The `--new-pr` option can also be used to open pull requests to the easyblocks -and framework repositories, and it will even automatically determine the target -repository based on the contents of the files you provide. Whoa! - -You can control the target repository for your pull request using -`--pr-target-account` (default is `easybuilders`) and `--pr-target-repo`. - -If you want you can carefully double check your work before actually -opening the pull request by doing a dry run via "`eb --dry-run --new-pr`" -or "`eb -D --new-pr`". - -Finally, you can use "`eb --preview-pr`" to see how the easyconfig files -for which you plan to create a pull request differ from existing easyconfig -files. - -### Updating pull requests - -To update an existing pull request with additional changes -you can use "`eb --update-pr`" and pass the pull request ID, -alongside the paths to the updated files. - -If you have only created a branch (for example via `eb --new-branch-github`) -you can update it via `--update-branch-github` in the same way, -passing the branch name instead of a pull request ID. - -### Using a pull request - -Next to creating and updating branches and pull requests -you can also *use* easyconfig files and easyblocks from a pull request, -regardless of its status (open, merged, or closed). This is particularly -useful when testing contributions, or to install software for which -support is not yet included in the latest EasyBuild release. - -Using the `--from-pr` option you can install easyconfig files from the -pull request with specified ID. By default all easyconfig files that are -touched by the pull request will be installed, but you can specify -particular ones to use as well. It is generally advised to also use the -`--robot` option to ensure that the easyconfig files are installed in the -correct order with respect to dependencies. - -Similarly, using a new or updated easyblock from a pull request is as simple -as using the `--include-easyblocks-from-pr` option. And of course you can -combine it with `--from-pr`! - -Via `--upload-test-report` you can let EasyBuild submit a comment into the -easyconfig pull request to show that the installation worked on your system. This is -useful for others to know, in particular EasyBuild maintainers, since the comment -will include information about your system (OS, processor, etc.) and your EasyBuild configuration. - -## Demo - -!!! Warning "Not currently suited for LUMI" - - This is text from a previous (non-Cray) version of the tutorial. The demo does require - a number of files not installed on LUMI after going through this tutorial, - and someone who can master the repository used to clean up again. - However, having a look at the output (which was generated in the spring of 2021 on - the CSC system puhti) still offers some information, so the demo was left in this - version of the tutorial. - -That is a lot to digest, so let us make this a bit more concrete with an example: -we will open a pull request for the [`eb-tutorial` example software](../adding_support_software/#example) to *a fork* of the [`easybuild-easyconfigs` repository](https://github.com/easybuilders/easybuild-easyconfigs) using the `eb` command, -and submit a test report in it. - -!!! Note - Make sure that you have correctly configured the GitHub integration, - [see above](#requirements-configuration). - -### Creating pull request - -We first configure EasyBuild to target the `ebtutorial` GitHub account rather -than the default `easybuilders` GitHub organisation, -by defining the `pr-target-account` configuration setting: - -```shell -export EASYBUILD_PR_TARGET_ACCOUNT=ebtutorial -``` - -In the output of "`eb --show-config`" you should see a line like this: - -``` -pr-target-account (E) = ebtutorial -``` - -We only do this to avoid that lots of pull requests for the `eb-tutorial` -example software are opened in the [central easyconfigs repository](https://github.com/easybuilders/easybuild-easyconfigs). - -Opening a pull request is as simple as running "`eb --new-pr`" and passing -the easyconfig file: - -```shell -$ eb --new-pr example.eb -== temporary log file in case of crash /tmp/eb-ggr6scbq/easybuild-hnk271xj.log -== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... -== fetching branch 'develop' from https://github.com/ebtutorial/easybuild-easyconfigs.git... -== copying files to /tmp/eb-ggr6scbq/git-working-dirxwk1fzaw/easybuild-easyconfigs... -== pushing branch '20200622095415_new_pr_eb-tutorial100' to remote 'github_ebtutorial_qgtfU' (git@github.com:ebtutorial/easybuild-easyconfigs.git) -Enter passphrase for key '/home/example/.ssh/id_rsa': -Please enter password for encrypted keyring: - -Opening pull request -* target: ebtutorial/easybuild-easyconfigs:develop -* from: ebtutorial/easybuild-easyconfigs:20200622095415_new_pr_eb-tutorial100 -* title: "{tools}[GCC/10.2.0] eb-tutorial v1.0.1" -* labels: new -* description: -""" -(created using `eb --new-pr`) - -""" -* overview of changes: - easybuild/easyconfigs/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb | 26 ++++++++++++++++++++++++++ - 1 file changed, 26 insertions(+) - -Opened pull request: https://github.com/ebtutorial/easybuild-easyconfigs/pull/ -== Temporary log file(s) /tmp/eb-ggr6scbq/easybuild-hnk271xj.log* have been removed. -== Temporary directory /tmp/eb-ggr6scbq has been removed. -``` - -Take a moment to grasp what we did here: we ran **a single `eb` command** which -took care of the **[whole contribution procedure](#contribution-procedure)** for us, including: - -* Cloning the `easybuilders/easybuild-easyconfigs` repository and checking out the `develop` branch (in a temporary - directory); -* Picking a sensible name for a branch and creating it; -* Adding the `eb-tutorial` easyconfig file to the branch, in the correct location - (`easybuild/easyconfigs/e/eb-tutorial/`) and with the correct filename (`eb-tutorial-1.0.1-GCC-10.2.0.eb`); -* Pushing the branch to our fork (`example/easybuild-easyconfigs`); -* Actually opening the pull request, using an informative title. - -That is so... easy! - -This feature not only *significantly* lowers the bar for contributing, -it also saves quite a bit of time since you don't need to double check -various details (like targeting the `develop` branch) or spend time on -coming up with a nice looking title or funny branch name (although you -still can if you really want to). - -There are a couple of nice side effects too, like not having any local branches -to tidy up on once the pull request gets merged (since `--new-pr` created the -branch only in a temporary directory). - -If many contributions are made via `--new-pr` it also simplifies the task -of EasyBuild maintainers, since pull requests opened this way have a particular -structure to them and thus are easier to digest because they look familiar. - -### Uploading test report - -After opening the pull request, we should also upload a test report to show that the installation is working. -This is just as easy as creating the pull request. - -First make sure that the pre-installed software in the prepared environment -is available, since the required dependencies for `eb-tutorial` are already -installed there: - -```shell -module use /easybuild/modules/all -``` - -You can verify which dependencies are still missing using `--from-pr` combined with `--missing`: - -```shell -# change '1' to the ID of your own pull request (see output of --new-pr) -$ eb --from-pr 1 --missing -== temporary log file in case of crash /tmp/eb-ioi9ywm1/easybuild-e3v0xa1b.log -Please enter password for encrypted keyring: -== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... - -1 out of 20 required modules missing: - -* eb-tutorial/1.0.1-GCC-10.2.0 (eb-tutorial-1.0.1-GCC-10.2.0.eb) -``` - -Uploading a test report boils down to combining `--from-pr` with `--upload-test-report`: - -```shell -# change '1' to the ID of your own pull request (see output of --new-pr) -$ eb --rebuild --from-pr 1 --upload-test-report -Please enter password for encrypted keyring: -... -== processing EasyBuild easyconfig /tmp/eb-bnb1pv3n/files_pr65/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb -== building and installing eb-tutorial/1.0.1-GCC-10.2.0... -... -== COMPLETED: Installation ended successfully (took 2 sec) -... -Adding comment to easybuild-easyconfigs issue #65: 'Test report by @ebtutorial -**SUCCESS** -Build succeeded for 1 out of 1 (1 easyconfigs in this PR) -example - Linux centos linux 7.8.2003, x86_64, Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz (haswell), Python 3.6.8 -See https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 for a full test report.' -== Test report uploaded to https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 and mentioned in a comment in easyconfigs PR#1 -``` - -Note that we may need to use `--rebuild` here since `eb-tutorial` may already be installed. - -This results in a comment being added to the pull request: - -
Test report comment
- -The gist linked from this comment provides more detailed information: - -
Test report gist
- ---- - -*[[next: Additional reading]](4_00_additional_reading.md)* diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md index 0a6b712d..ca706c01 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md @@ -67,7 +67,7 @@ would be installed using the regular ``foss`` toolchain or the ``gompi`` toolcha On LUMI, where software is installed through the Cray Programming Environment with no real choice of MPI implementation, a two-level arrangement would still make a lot of sense, with at the ``Core`` level all software compiled with the SYSTEM toolchain while there could be a ``PrgEnv`` level for software -compiled with a particula programming environment aka cepGNU/cpeCray/cpeAOCC toolchain. Such a scheme +compiled with a particular programming environment aka cpeGNU/cpeCray/cpeAOCC toolchain. Such a scheme is used on the Cray systems at CSCS. To recap, the characteristics of a module hierarchy are: diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md index b6cf98d8..58831541 100644 --- a/docs/2022-CSC_and_LO/index.md +++ b/docs/2022-CSC_and_LO/index.md @@ -78,3 +78,4 @@ freely available, so unless you have access to a system with this environment yo - [Submitting installations as Slurm jobs](3_Advanced/3_03_slurm_jobs.md) - [Module naming schemes (incl. hierarchical)](3_Advanced/3_04_module_naming_schemes.md) - [GitHub integration to facilitate contributing to EasyBuild](3_Advanced/3_05_github_integration.md) +- [**Additional reading**](4_00_additional_reading.md) From df0d8e5b99aab9c4e6a7ed392d8591f7d3a9cbf4 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 19 May 2022 16:54:20 +0200 Subject: [PATCH 084/159] Some improvments to the module naming schemes section based on the ISC'22 tutorial. --- .../3_Advanced/3_04_module_naming_schemes.md | 90 ++++++++++--------- 1 file changed, 47 insertions(+), 43 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md index ca706c01..ce8a6eff 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md @@ -55,7 +55,7 @@ This typical Lmod hierarcny would map very well on the EasyBuild common toolchai In fact, for the example -
+
software at the ``Core`` level would be installed with the ``GCCcore`` and ``SYSTEM`` toolchains. Software at the ``Compiler`` level would be installed with the ``GCC`` @@ -102,10 +102,10 @@ However, the are some minor disadvantages too: When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For a `HDF5` installation with the EasyBuild common toolchains for example, -one might have `HDF5/1.10.6-gompi-2020a` as module name. The `-gompi-2020a` part of the name refers to the toolchain that was +one might have `HDF5/1.12.1-gompi-2021b` as module name. The `-gompi-2021b` part of the name refers to the toolchain that was used for this installation, but it may be confusing to some people (what kind of Pokémon is a "gompi"?!). -In the example module hierarchy shown above, the module for `HDF5` could simply be named `HDF5/1.10.6` which is basically the bare +In the example module hierarchy shown above, the module for `HDF5` could simply be named `HDF5/1.12.1` which is basically the bare essentials: software name and version. That's way better, nice and clean! #### Amount of available modules @@ -329,9 +329,10 @@ install EasyBuild again in our hierarchical module tree before we can continue. **We strongly recommend using an EasyBuild installation that was [installed via "`pip install`" or "`pip3 install`"](../../1_Intro/1_06_installation#method-1-using-pip) in this part of the tutorial.** -An easy way to do this is in the prepared environment is to run: +An easy way to do this is in the prepared environment used for regular EasyBuild tutorials is to run: ```shell +unset PIP_PREFIX pip3 install --user easybuild export PATH=$HOME/.local/bin:$PATH export EB_PYTHON=python3 @@ -393,7 +394,7 @@ but EasyBuild can do all the hard work for us. The steps we will have to go through are: -* Tell EasyBuild we want to "install" the `HDF5-1.10.7-gompi-2020b.eb` easyconfig file; +* Tell EasyBuild we want to "install" the `HDF5-1.12.1-gompi-2021b.eb` easyconfig file; * Enable dependency resolution via `--robot`; * Assuming the software would have been installed already with the default naming scheme in a different module directory, instruct EasyBuild to only generate the module files, @@ -403,9 +404,9 @@ The steps we will have to go through are: These steps translate to this single `eb` command: ``` -$ eb HDF5-1.10.7-gompi-2020b.eb --robot --module-only +$ eb HDF5-1.12.1-gompi-2021b.eb --robot --module-only ... -== building and installing MPI/GCC/10.2.0/OpenMPI/4.0.5/HDF5/1.10.7... +== building and installing MPI/GCC/11.2.0/OpenMPI/4.1.1/HDF5/1.12.1... ... == sanity checking... == cleaning up [skipped] @@ -450,10 +451,10 @@ Let us see what that gives us in terms of available modules: ``` $ module avail ---------------------- /home/easybuild/hmns/modules/all/Core --------------------- - Bison/3.5.3 GCCcore/10.2.0 flex/2.6.4 help2man/1.47.4 - Bison/3.7.1 (D) M4/1.4.18 gettext/0.21 ncurses/6.2 - GCC/10.2.0 binutils/2.35 gompi/2020b zlib/1.2.11 +------------------------ /home/easybuild/hmns/modules/all/Core ------------------------- + binutils/2.37 GCC/11.2.0 gompi/2021b OpenSSL/1.1 + Bison/3.8.2 GCCcore/11.2.0 M4/1.4.19 pkg-config/0.29.2 + flex/2.6.4 gettext/0.21 ncurses/6.2 zlib/1.2.11 ``` @@ -475,9 +476,9 @@ $ module spider HDF5 ... You will need to load all module(s) on any one of the lines below - before the "HDF5/1.10.7" module is available to load. + before the "HDF5/1.12.1" module is available to load. - GCC/10.2.0 OpenMPI/4.0.5 + GCC/11.2.0 OpenMPI/4.1.1 ``` This tells us we need to load two gateway modules before we can load the module @@ -486,7 +487,7 @@ for HDF5. Let us start with loading the `GCC` compiler module: ``` -module load GCC/10.2.0 +module load GCC/11.2.0 ``` And then check again which modules are available: @@ -494,62 +495,65 @@ And then check again which modules are available: ``` $ module avail --------------- /home/easybuild/hmns/modules/all/Compiler/GCC/10.2.0 -------------- - OpenMPI/4.0.5 +--------------------------- /home/easybuild/hmns/modules/all/Compiler/GCC/11.2.0 --------------------------- + OpenMPI/4.1.1 ------------- /home/easybuild/hmns/modules/all/Compiler/GCCcore/10.2.0 ------------ - Autoconf/2.69 UCX/1.9.0 libpciaccess/0.16 - ... - Szip/2.1.1 libfabric/1.11.0 zlib/1.2.11 (L,D) - Autoconf/2.69 XZ/5.2.5 libtool/2.4.6 +------------------------- /home/easybuild/hmns/modules/all/Compiler/GCCcore/11.2.0 ------------------------- + Autoconf/2.71 flex/2.6.4 (D) libreadline/8.1 pkg-config/0.29.2 (D) + Automake/1.16.4 groff/1.22.4 libtool/2.4.6 PMIx/4.1.0 + Autotools/20210726 help2man/1.48.3 libxml2/2.9.10 Szip/2.1.1 + binutils/2.37 (L,D) hwloc/2.5.0 M4/1.4.19 (D) UCX/1.11.2 + Bison/3.7.6 libevent/2.1.12 ncurses/6.2 (D) xorg-macros/1.19.3 + DB/18.1.40 libfabric/1.13.2 numactl/2.0.14 XZ/5.2.5 + expat/2.4.1 libpciaccess/0.16 Perl/5.34.0 zlib/1.2.11 (L,D) ---------------------- /home/easybuild/hmns/modules/all/Core --------------------- - Bison/3.5.3 GCCcore/10.2.0 (L) flex/2.6.4 help2man/1.47.4 - Bison/3.7.1 (D) M4/1.4.18 gettext/0.21 ncurses/6.2 - GCC/10.2.0 (L) binutils/2.35 gompi/2020b zlib/1.2.11 +---------------------------------- /home/easybuild/hmns/modules/all/Core ----------------------------------- + binutils/2.37 GCC/11.2.0 (L) gompi/2021b OpenSSL/1.1 + Bison/3.8.2 (D) GCCcore/11.2.0 (L) M4/1.4.19 pkg-config/0.29.2 + flex/2.6.4 gettext/0.21 ncurses/6.2 zlib/1.2.11 ``` Good news, we now have additional modules available! -The compiler level of our hierarchy actually consists of two directories here: `Compiler/GCCcore/10.2.0` -and `Compiler/GCC/10.2.0`. The modules in the `GCCcore` directory are ones we can use in other compiler +The compiler level of our hierarchy actually consists of two directories here: `Compiler/GCCcore/11.2.0` +and `Compiler/GCC/11.2.0`. The modules in the `GCCcore` directory are ones we can use in other compiler toolchains that use GCC 10.2.0 as a base compiler (the details of that are out of scope here). -The module we are interested in is `OpenMPI/4.0.5`, which is another gateway module. +The module we are interested in is `OpenMPI/4.1.1`, which is another gateway module. Remember that the "`module spider`" output told us that there does indeed exist a module for `HDF5`, but that -we need to load *both* the `GCC/10.2.0` and `OpenMPI/4.0.5` modules first. +we need to load *both* the `GCC/11.2.0` and `OpenMPI/4.1.1` modules first. -So, let us do exactly that (remember that `GCC/10.2.0` is already loaded): +So, let us do exactly that (remember that `GCC/11.2.0` is already loaded): ``` -module load OpenMPI/4.0.5 +module load OpenMPI/4.1.1 ``` -If you now check the output of "`module avail`" again, you should see the `HDF5/1.10.7` module: +If you now check the output of "`module avail`" again, you should see the `HDF5/1.12.1` module: ``` $ module avail --------- /home/easybuild/hmns/modules/all/MPI/GCC/10.2.0/OpenMPI/4.0.5 ------- - HDF5/1.10.7 +-------- /home/easybuild/hmns/modules/all/MPI/GCC/11.2.0/OpenMPI/4.1.1 ------- + HDF5/1.12.1 ------------- /home/easybuild/hmns/modules/all/Compiler/GCC/10.2.0 ------------ - OpenMPI/4.0.5 (L) +------------ /home/easybuild/hmns/modules/all/Compiler/GCC/11.2.0 ------------ + OpenMPI/4.1.1 (L) ... ``` -To use HDF5, we need to load this `HDF5/1.10.7` module. We can verify that the installation works +To use HDF5, we need to load this `HDF5/1.12.1` module. We can verify that the installation works using one of the commands provided by HDF5, `h5dump` for example: ``` -module load HDF5/1.10.7 +module load HDF5/1.12.1 ``` ``` $ h5dump --version -h5dump: Version 1.10.7 +h5dump: Version 1.12.1 ``` If you now check which modules are loaded via "`module list`", you will notice that all module names @@ -559,10 +563,10 @@ and nice and short now, which is one of the advantages of using a hierarchical m $ module list Currently Loaded Modules: - 1) GCCcore/10.2.0 5) numactl/2.0.13 9) hwloc/2.2.0 13) PMIx/3.1.5 - 2) zlib/1.2.11 6) XZ/5.2.5 10) libevent/2.1.12 14) OpenMPI/4.0.5 - 3) binutils/2.35 7) libxml2/2.9.10 11) UCX/1.9.0 15) Szip/2.1.1 - 4) GCC/10.2.0 8) libpciaccess/0.16 12) libfabric/1.11.0 16) HDF5/1.10.7 + 1) GCCcore/11.2.0 5) numactl/2.0.14 9) hwloc/2.5.0 13) libfabric/1.13.2 17) HDF5/1.12.1 + 2) zlib/1.2.11 6) XZ/5.2.5 10) OpenSSL/1.1 14) PMIx/4.1.0 + 3) binutils/2.37 7) libxml2/2.9.10 11) libevent/2.1.12 15) OpenMPI/4.1.1 + 4) GCC/11.2.0 8) libpciaccess/0.16 12) UCX/1.11.2 16) Szip/2.1.1 ``` From 36934df4c5fc178853a3bbb0ffd69df396bf9f2c Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 2 Jun 2022 14:36:06 +0200 Subject: [PATCH 085/159] Several minor corrections, including a new least of EasyBuild community membewrs. --- .../1_Intro/1_01_what_is_easybuild.md | 21 ++++---- docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md | 52 +++++++++---------- docs/2022-CSC_and_LO/index.md | 18 +++---- 3 files changed, 46 insertions(+), 45 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md index 3059cd82..fa5075f5 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md @@ -131,8 +131,8 @@ The EasyBuild community also provides packages for two standard toolchains, call The so-called ``foss`` is based on free and open software (GNU compilers, FlexiBlas with OpenBLAS backend, LaPACK, FFTW and Open MPI) while the ``intel`` toolchain uses the full stack of Intel compilers, libraries and MPI. Currently these toolchains cannot be supported on LUMI though as getting Open MPI to run on LUMI is challenging and as the Intel libraries and -some MPI versions also produce slow or incorrect results on AMD CPUs (and it is not clear if Intel MPI is compatible -with SlingShot 11). +some MPI versions also produce slow or incorrect results on AMD CPUs (and it is not entirely clear if Intel MPI is compatible +with SlingShot 11, though in principle it should). Focusing the efforts on those two popular common toolchains makes it possible to support a lot of build recipes that are collected in the [central repository](https://github.com/easybuilders/easybuild-easyconfigs). They can still serve as a basis to develop build recipes for the HPE Cray Programming Environment. @@ -228,23 +228,24 @@ HPC sites, consortia, and companies that use EasyBuild include (but are not limi VSC logo JSC logo HPCNow! logo -Fred Hutch logo -Vienna BioCenter logo +Fred Hutch logo +Vienna BioCenter logo +Compute Canada logo +LUMI logo CÉCI logo -Compute Canada logo UiO logo BEAR logo -SNIC logo +SNIC logo National University of Singapore logo IT4Innovations logo Texas A&M logo TU Dresden logo University of Melbourne logo University of Luxembourg logo -EPCC logo -Cyprus Institute logo -University of Basel logo -Microsoft logo +EPCC logo +Cyprus Institute logo +University of Basel logo +Microsoft logo --- diff --git a/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md index fed386d1..6913ab11 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md @@ -156,7 +156,7 @@ one does module load MPI_D ``` -thent MPI_C gets unloaded, Lmod notices that it also has to unload/deactivate Appl_E, then will load MPI_D for Compiler_A and +then MPI_C gets unloaded, Lmod notices that it also has to unload/deactivate Appl_E, then will load MPI_D for Compiler_A and finally will notice that there is an equivalent Appl_E module available again, and Lmod will load that one also. However, now loading Compiler_B will cause a warning that MPI_D and Appl_E have been deactivated as there is no module name MPI_D in any version for Compiler_B. @@ -416,31 +416,31 @@ search for those too. This tells that there is no ``CMake`` module on the system but that two versions of ``CMake`` are provided in another module. - 3. `module spider /` will show more information on the - extension, including which module provides the extension and which other modules - have to be loaded to make that module available. E.g., on LUMI, - ``` - module spider CMake/3.22.2 - ``` - will output something along the lines of - ``` - ----------------------------------------------------------------------- - CMake: CMake/3.22.2 (E) - ----------------------------------------------------------------------- - This extension is provided by the following modules. To access the - extension you must load one of the following modules. Note that any - module names in parentheses show the module location in the software - hierarchy. - - buildtools/21.12 (LUMI/21.12 partition/L) - buildtools/21.12 (LUMI/21.12 partition/G) - buildtools/21.12 (LUMI/21.12 partition/D) - buildtools/21.12 (LUMI/21.12 partition/C) - buildtools/21.12 (CrayEnv) +3. ``module spider /`` will show more information on the + extension, including which module provides the extension and which other modules + have to be loaded to make that module available. E.g., on LUMI, + ``` + module spider CMake/3.22.2 ``` - (output abbreviated and slightly reformatted for readability). This tells that - ``CMake/3.22.2`` is provided by the ``bvuildtools/21.12`` module and that there - are 5 different ways to make that package available. + will output something along the lines of + ``` + ----------------------------------------------------------------------- + CMake: CMake/3.22.2 (E) + ----------------------------------------------------------------------- + This extension is provided by the following modules. To access the + extension you must load one of the following modules. Note that any + module names in parentheses show the module location in the software + hierarchy. + + buildtools/21.12 (LUMI/21.12 partition/L) + buildtools/21.12 (LUMI/21.12 partition/G) + buildtools/21.12 (LUMI/21.12 partition/D) + buildtools/21.12 (LUMI/21.12 partition/C) + buildtools/21.12 (CrayEnv) + ``` + (output abbreviated and slightly reformatted for readability). This tells that + ``CMake/3.22.2`` is provided by the ``bvuildtools/21.12`` module and that there + are 5 different ways to make that package available. ??? Bug "Restrictions with older Lmod versions" At the time of development of this tutorial, Cray is still using the pretty old @@ -787,7 +787,7 @@ still fail to remove the 2.7 directory from ``PYTHONPATH``. the module as deactivated but not as unloaded. Lmod proceeds with loading the ``MyPython/3.6.10`` module. This will now set - ``PTHON_API_VERSION`` to ``3.6`` and add a directory with name ending on + ``PYTHON_API_VERSION`` to ``3.6`` and add a directory with name ending on ``level2/PythonAPI/3.6`` to ``MODULEPATH``. Things are not done yet though. As the ``MODULEPATH`` has changed, Lmod looks at its list diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md index 58831541..b45a740b 100644 --- a/docs/2022-CSC_and_LO/index.md +++ b/docs/2022-CSC_and_LO/index.md @@ -9,7 +9,7 @@ See that site for [credits to the contributors of those tutorials](https://easyb This is an introductory tutorial to [EasyBuild](https://easybuild.io), a command line tool for installing (scientific) software on High Performance Computing (HPC) systems. -These tutorials are specifically for EasyBuild as implemented on [LUMI](https://lumi-supercomputer.eu) +This tutorial is specifically for EasyBuild as implemented on [LUMI](https://lumi-supercomputer.eu) and has been developed for CSC and the Local Organisations within the LUMI consortium. Yet much of the material is useful to a broader community of EasyBuild users on Cray systems or even EasyBuild users in general. @@ -32,23 +32,23 @@ with the project it could be interesting to step through it. Our main target audience includes: -- Application experts in LUST and the local organizations who want to contribute to the - software stack on LUMI or support their users -- Developers who want to make their developments available to LUMI users -- Advanced users who want to customize available build recipes or develop their own recipes +- application experts in LUST and the local organizations who want to contribute to the + software stack on LUMI or support their users; +- developers who want to make their developments available to LUMI users; +- advanced users who want to customize available build recipes or develop their own recipes. ## Prerequisites We expect you to be (a little bit) familiar with: -- using a Linux command line interface -- the (absolute) basics of compiling software from source +- using a Linux command line interface; +- the (absolute) basics of compiling software from source. EasyBuild requires: -- GNU/Linux (any distribution) -- Python 2.7 or 3.5+ +- GNU/Linux (any distribution), +- Python 2.7 or 3.5+, though a fairly recent version of Python 3 is highly recommended, - an environment modules tool (see the ``module`` command). On LUMI we use [Lmod](https://lmod.readthedocs.io), a modern environment modules tool implemented in Lua. From f62ed3192fe9e9af49912492761b5cb336d1483f Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 3 Jun 2022 11:55:53 +0200 Subject: [PATCH 086/159] Minor corrections to the Lmod section. --- docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md | 55 ++++++++++++----------- 1 file changed, 30 insertions(+), 25 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md index 6913ab11..ee0e5856 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_02_Lmod.md @@ -437,10 +437,10 @@ search for those too. buildtools/21.12 (LUMI/21.12 partition/D) buildtools/21.12 (LUMI/21.12 partition/C) buildtools/21.12 (CrayEnv) - ``` - (output abbreviated and slightly reformatted for readability). This tells that - ``CMake/3.22.2`` is provided by the ``bvuildtools/21.12`` module and that there - are 5 different ways to make that package available. + ``` + (output abbreviated and slightly reformatted for readability). This tells that + ``CMake/3.22.2`` is provided by the ``bvuildtools/21.12`` module and that there + are 5 different ways to make that package available. ??? Bug "Restrictions with older Lmod versions" At the time of development of this tutorial, Cray is still using the pretty old @@ -553,7 +553,9 @@ and the module file can also detect in which mode it is executing. Modes include "load", "unload" but also "spider". E.g., when the mode is "load", the ``setenv`` function will set an environment variable to the indicated value while in "unload" mode that environment variable will be unset, and in "spider" mode the -environment variable is left untouched. The working of ``prepend_path``, a function +environment variable is left untouched. + +The working of ``prepend_path``, a function that modifies PATH-style variables, depends a bit on how Lmod is configured (as it is possible to work with reference counts), but in its most basic mode, ``prepend_path`` will add a given directory to a given PATH-style environment variable (or move it to @@ -563,6 +565,7 @@ be generated should the directory that is used as the argument not be part of th in that PATH-style variable). When the mode is "spider", the function has special behaviour if it is used to change the ``MODULEPATH``. It will then note the change and add that directory to the list of directories that has to be searched for module files. + This makes ``module spider`` a very expensive command as it may have to traverse a lot of directories and has to execute all module files in there. Therefore Lmod will build a so-called spider cache which can be pre-built in the system for certain directories @@ -598,14 +601,18 @@ whatis("Description: memory usage tester") It is not all that important to include all those lines in a module file, but some of those lines get a special treatment from Lmod. The line starting with ``Description`` is used by ``module spider`` to provide some brief information about the module if it -is not totally resolved. This comes with a limitation though: It is not show for each +is not totally resolved. This comes with a limitation though: It is not shown for each version of the module, so ideally all "GROMACS" modules should contain the same description line and use other lines to provide further information about what distinguished a particular version. Likewise the ``Category:`` line is used by the ``spider_decoration`` hook that can be used to add decoration to the spider level 1 output. -All in all the ``whatis`` function if often overlooked in Lmod-based module functionx +All in all the ``whatis`` function is often overlooked in Lmod-based module functionx but it is a very useful function to include in the proper way in module files. +The EasyBuild support for the ``whatis`` lines is also far from ideal. It will autogenerate +certain lines from information specified in the EasyBuild recipes, but it also allows to +specify ``whatis`` lines yourself via a parameter in the EasyBuild recipes. However, +as soon as you specify the parameter, it will no longer auto-generate the other lines. A third function that provides information to the search commands is ``extensions``. It can be used to list up the extensions supported by the module. The argument list @@ -613,7 +620,6 @@ may seem strange as it takes only a single argument, a string of comma-separated elements, but that is because the number of arguments to a function is limited in Lua and that limit can actually be met easily by modules for Python, Perl or R packages. - --- ## Some warnings about writing modulefiles @@ -631,13 +637,13 @@ and basing actions of modulefiles on their position in the hierarchy. One case where passing information between modules through environment variables will go wrong is when that environment variable is subsequently used to compute a directory -name that should be added to a PATH-like variable. Assume we have two versions of +name that is then added to a PATH-like variable. Assume we have two versions of a ``MyPython`` module, e.g., ``MyPython/2.7.18`` and ``MyPython/3.6.10``. That module then sets an environment variable ``PYTHON_API_VERSION`` to either ``2.7`` or ``3.6``. Next we have a module ``MyPythonPackage`` that makes a number of Python packages available for both Python modules. However, as some Python packages have to be installed separately for each Python version, it does so by adding a directory to the environment variable -``PYTHON_PATH`` that contains the version which it gets by using the Lua function +``PYTHONPATH`` that contains the version which it gets by using the Lua function ``os.getenv`` to request the value of ``PYTHON_API_VERSION``. One problem becomes clear in the following scenario: @@ -648,11 +654,11 @@ module load MyPython/3.6.10 ``` The ``module load MyPythonPackage`` will find the environment variable ``PYTHON_PACKAGE_API`` with the value ``2.7`` as set by ``module load MyPython/2.7.18`` and hence add the directory -for the packages for version 2.7 to ``PYTHON_PATH``. The ``module load MyPython/3.6.10`` +for the packages for version 2.7 to ``PYTHONPATH``. The ``module load MyPython/3.6.10`` command will trigger two operations because of the *"one name rule"*: First it will automatically unload ``MyPython/2.7.18`` (which will unset ``PYTHON_API_VERSIUON``) and next it will load ``MyPython/3.6.10`` which will set ``PYTHON_API_VERSION`` to ``3.6``. -However, ``MyPythonPackage`` is not reloaded so the ``PYTHON_PATH`` variable will now point +However, ``MyPythonPackage`` is not reloaded so the ``PYTHONPATH`` variable will now point to the wrong directory. One would be tempted to think that the easy fix for the user would be to reload ``MyPythonPackage/1.0``: ``` bash @@ -660,12 +666,12 @@ module load MyPythonPackage/1.0 ``` Because of the *"one name rule"* this will again trigger an unload followed by a load of the module. The problem is in the unload. One would expect that first unloading -``MyPythonPackage`` would remove the 2.7 directory from the ``PYTHON_PATH`` but it +``MyPythonPackage`` would remove the 2.7 directory from the ``PYTHONPATH`` but it will not. Lmod does not remember that last time it loaded ``MyPythonPackage`` it added the 2.7 directory to ``PythonPath``. Instead it will execute the commands in the modulefile and reverse certain commands. Since ``PYTHON_API_VERSION`` has now the value ``3.6``, it will try to remove the directory for version ``3.6`` which is not in the -``PYTHON_PATH``. The subsequent load will then add the 3.6 directory to ``PYTHON_PATH`` +``PYTHONPATH``. The subsequent load will then add the 3.6 directory to ``PYTHONPATH`` so the environment variable now contains both directories. In this simple case, a ``module purge`` after the first two ``module load`` commands would @@ -692,9 +698,9 @@ still fail to remove the 2.7 directory from ``PYTHONPATH``. ``` lua LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) LmodMessage( 'PYTHON_API_VERSION = ' .. ( os.getenv( 'PYTHON_API_VERSION' ) or '') ) - prepend_path( 'PYTHON_PATH', 'someroot/python' .. + prepend_path( 'PYTHONPATH', 'someroot/python' .. ( os.getenv( 'PYTHON_API_VERSION' ) or 'TTT' ) .. '/packages' ) - LmodMessage( 'PYTHON_PATH = ' .. ( os.getenv( 'PYTHON_PATH' ) or '') ) + LmodMessage( 'PYTHONPATH = ' .. ( os.getenv( 'PYTHONPATH' ) or '') ) ``` ??? Note "Solution with a hierarchy" The better way in Lmod to implement the above scenario would be in a module hierarchy. @@ -731,9 +737,9 @@ still fail to remove the 2.7 directory from ``PYTHONPATH``. LmodMessage( 'Detected Python API version from environment: ' .. ( os.getenv( 'PYTHON_API_VERSION' ) or '' ) ) - prepend_path( 'PYTHON_PATH', 'someroot/python' .. python_api_version .. '/packages' ) + prepend_path( 'PYTHONPATH', 'someroot/python' .. python_api_version .. '/packages' ) - LmodMessage( 'PYTHON_PATH = ' .. (os.getenv( 'PYTHON_PATH' ) or '') ) + LmodMessage( 'PYTHONPATH = ' .. (os.getenv( 'PYTHONPATH' ) or '') ) ``` Now add the ``level1`` subdirectory to ``MODULEPATH``, e.g., if you're in the directory @@ -758,13 +764,13 @@ still fail to remove the 2.7 directory from ``PYTHONPATH``. ``module avail`` will show the ``MyPythonPackage/1.0`` module. The ``MyPythonPackage`` shows two ways to get the version of the Python API to use for - determining the right directory to add to ``PYTHON_PATH``. The fragile way is to enquire + determining the right directory to add to ``PYTHONPATH``. The fragile way is to enquire the value of the environment variable ``PYTHON_API_VERSION`` set by loading ``MyPython/2.7.18``. The more robust way is to use the Lmod introspection function ``myFileName()`` which returns the full path and file name of the module file that is executing, and extracting the version from the path with a pattern matching function. In this particular situation both computed values are the same so both would have worked to correctly add - ``somedir/python2.7/packages`` to the front of ``PYTHON_PATH``. + ``somedir/python2.7/packages`` to the front of ``PYTHONPATH``. The next command, ``module load MyPython/3.6.10`` triggers a chain of events. @@ -777,11 +783,11 @@ still fail to remove the 2.7 directory from ``PYTHONPATH``. the change of the ``MODULEPATH`` the ``MyPythonPackage/1.0`` module which was loaded from ``.../level2/PythonAPI/2.7`` is no longer available so Lmod will continue with unloading that module. The interesting bit now is that ``PYTHON_API_VERSION`` is unset. So had we - computed the name of the directory to add to ``PYTHON_PATH`` using the value of that + computed the name of the directory to add to ``PYTHONPATH`` using the value of that environment variable, the module would have failed to compute the correct directory name - to remove so ``prepend_path`` would have left the ``PYTHON_PATH`` environment variable + to remove so ``prepend_path`` would have left the ``PYTHONPATH`` environment variable untouched. However, by computing that value from the directory of the modulefile, we get - the right value and can correctly remove ``somedir/python2.7/packages`` from ``PYTHON_PATH``. + the right value and can correctly remove ``somedir/python2.7/packages`` from ``PYTHONPATH``. Lmod will also remember that the module was only unloaded due to a change in the ``MODULEPATH`` and not because a user explicitly unloaded the module. I.e., it considers the module as deactivated but not as unloaded. @@ -794,7 +800,7 @@ still fail to remove the 2.7 directory from ``PYTHONPATH``. of deactivated modules and notices that a different version of ``MyPythonPackage/1.0`` is now available. Hence it will now automatically load that module from the ``.../level2/PythonAPI/3.6`` subdirectory so that that module now correctly detects - that ``somedir/python3.6/package`` should be added to ``PYTHON_PATH``. + that ``somedir/python3.6/package`` should be added to ``PYTHONPATH``. Hence at the end of the cycle we have again a correctly configured environment with no trace of the ``2.7`` version that was loaded initially and with no action required from @@ -804,7 +810,6 @@ still fail to remove the 2.7 directory from ``PYTHONPATH``. This idea is used on LUMI to implement the various versions of the software stack with for each software stack also optimised binaries for each of the node types. - --- ## Further reading From 28c00baae7365bb6e33db79c9eb76be93da60964 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 7 Mar 2022 18:30:05 +0100 Subject: [PATCH 087/159] Updated the "What is EasyBuild" page of the tutorial for LUMI." --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 12 + .../2022-CSC_and_LO/1_01_what_is_easybuild.md | 321 ++++++++++++++++++ 2 files changed, 333 insertions(+) create mode 100644 docs/2022-CSC_and_LO/1_00_part1_intro.md create mode 100644 docs/2022-CSC_and_LO/1_01_what_is_easybuild.md diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md new file mode 100644 index 00000000..0bfd0ad5 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -0,0 +1,12 @@ +# Part I: Introduction to EasyBuild on Cray systems + +*[[back to start page]](index.md)* + +*(Tue March 4th 2021, 9am-12 CET)* + +- [What is EasyBuild?](1_01_what_is_easybuild.md) + +- [Terminology](terminology.md) +- [Installation](installation.md) *(hands-on)* +- [Configuration](configuration.md) *(hands-on)* +- [Basic usage](basic_usage.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md new file mode 100644 index 00000000..678c8c4a --- /dev/null +++ b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md @@ -0,0 +1,321 @@ +# What is EasyBuild? + +*[[back: Overview]](index.md)* + +

EasyBuild logo

+ +**[EasyBuild](http://easybuild.io)** is a **software build and installation framework** +that manages (scientific) software in an efficient way, without compromising on performance. +Easybuild is especially tailored towards [**High Performance Computing (HPC)**](https://en.wikipedia.org/wiki/Supercomputer) systems, +and provides a **command-line interface** to facilitate the installation and management of a scientific software stack. + +--- + + + + + +### *Elevator pitch* + +EasyBuild intends to relieve HPC support teams from manually managing +software installations while at the same time **providing a consistent and well performing +scientific software stack** to end users of HPC systems. + +It serves as a **uniform interface for installing scientific software** +and saves valuable time (and frustration) by the automation of tedious, boring and repetitive tasks. + +In addition, EasyBuild can **empower scientific researchers to self-manage their software stack**, +and it can serve as a tool that can be leveraged for **building optimized container images**. + +The project has grown to become a **platform for collaboration** among HPC sites worldwide. + +--- + +On LUMI we pay a lot of attention to empowering users to create their custom software stack which +is best suited for their project without being confronted with hundreds of modules that are not +useful to them. It also enables them to easily customize the build instructions to create a +project-specific build or even to install software with custom patches to adapt to their needs. +For this we offer both [EasyBuild](https://easybuild.io/) and [Spack](https://spack.io/) +as tools but obviously this tutorial concentrates on EasyBuild alone. + +EasyBuild can also be used to [generate container recipes](https://docs.easybuild.io/en/latest/Containers.html) +that will then use EasyBuild when building +the container to install platform-optimised software in the container. However, as that functionality +is still under development on LUMI it is not yet covered in this LUMI-specific version of the tutorial. + + +--- + +### *Key features* + +EasyBuild is capable of **fully autonomously installing (scientific) software**, +including making sure that all necessary dependencies are installed, +and automatically generating environment module files. + +***No*** **admin privileges are required**: it is sufficient to have write permissions +to the preferred software installation prefix. + +EasyBuild is [**highly configurable**](https://docs.easybuild.io/en/latest/Configuration.html) via configuration files, +environment variables, and command line options. Various elements of the functionality can be +[**dynamically extended**](https://docs.easybuild.io/en/latest/Including_additional_Python_modules.html) via plugins, +and [**hooks**](https://docs.easybuild.io/en/latest/Hooks.html) are available for further site-specific customizations. + +The installation procedure executed by EasyBuild is thoroughly +[**logged**](https://docs.easybuild.io/en/latest/Logfiles.html), and is fully transparent via support for +[**dry runs**](https://docs.easybuild.io/en/latest/Extended_dry_run.html) and +[**tracing**](https://docs.easybuild.io/en/latest/Tracing_progress.html) the software installation procedure +as it is performed. + +EasyBuild supports using a **custom module naming scheme**, allows for +*hierarchical* module naming schemes, and **integrates with various other tools** ranging from resource managers ([Slurm](https://slurm.schedmd.com) and [GC3Pie](https://github.com/gc3pie/gc3pie)), +container tools ([Singularity](https://github.com/hpcng/singularity) and [Docker](https://www.docker.com)), +packaging tools ([FPM](https://fpm.readthedocs.io)), and so on. + +--- + +### *Focus points* + +EasyBuild was created specifically for **installing scientific software on HPC systems**, +which is reflected in some of the design choices that were made. + + +#### Performance + +EasyBuild strongly prefers to **build software from source code**, whenever possible. + +This is important to ensure that the binaries that are installed can maximally exploit +the capabilities of the system architecture on which the software will be run. + +For that same reason, EasyBuild out-of-the-box **optimizes software for the processor architecture of the build host** +by default, via compiler options like ``-march=native`` (GCC), ``-xHost`` (Intel compilers), etc. +This behaviour [can be changed via the ``--optarch`` configuration setting](https://docs.easybuild.io/en/latest/Controlling_compiler_optimization_flags.html). +When using the HPE Cray Programming Environment, the target modules are used instead. +In principle it is possible to cross-compile with EasyBuild, but this comes with some pitfalls. +By default EasyBuild tries to run tests if the build process of a package provides that +functionality, or can run simple tests of its own, and of course these tests will fail if +the host CPU does not support the full instruction set of the target CPU. Secondly, more +and more software has broken build processes where developers thought that it was a good +idea to hard-code options such as ``-march=native`` or ``--xHost`` in a way that EasyBuild +cannot easily overwrite those. + + +#### Reproducibility + +In addition to performance, **reproducibility of installations** is a core aspect of EasyBuild. + +TODO correct link in the next lines. + +Most software installations performed with EasyBuild use a **particular compiler toolchain**, +with which we aim to be in control over the build environment and avoid relying on tools and libraries +provided by the operating system. For similar reasons, we try to **provide all required dependencies through EasyBuild** as well, +with a few notable exceptions, like ``OpenSSL`` for security reasons, and Infiniband and GPU drivers which +are too closely intertwined with the operating system. + +For both toolchains and dependencies, **fixed software versions** are specified in the +easyconfig files. That way, easyconfig files can easily be shared with others: +if they worked for you it is very likely that they will work for others too, because the vast majority of the +software stack is controlled by EasyBuild. + +As we shall see when discussing the HPE Cray Programming Environment, the reproducibility is a bit more limited +in that environment due to the way the programming environment selects shared libraries at runtime. + + +#### Community effort + +In a number of different ways, we try to encourage EasyBuild users to **collaborate** and help each other out. + +We actively recommend people to report problems and bugs, to submit ideas for additional features and improvements, +and to [**contribute back**](https://docs.easybuild.io/en/latest/Contributing.html) when possible, be it +by opening pull requests to the [GitHub repositories](https://github.com/easybuilders) or the [documentation](https://docs.easybuild.io). + +The EasyBuild community also provides packages for two standard toolchains, called the +[**common toolchains**](https://easybuild.readthedocs.io/en/latest/Common-toolchains.html). +The so-called ``foss`` is based on free and open software (GNU compilers, FlexiBlas with OpenBLAS backend, LaPACK, FFTW and Open MPI) +while the ``intel`` toolchain uses the full stack of Intel compilers, libraries and MPI. Currently these toolchains +cannot be supported on LUMI though as getting Open MPI to run on LUMI is challenging and as the Intel libraries and +some MPI versions also produce slow or incorrect results on AMD CPUs (and it is not clear if Intel MPI is compatible +with SlingShot 11). +Focusing the efforts on those two popular common toolchains makes it possible to support a lot of build recipes that are +collected in the [central repository](https://github.com/easybuilders/easybuild-easyconfigs). +They can still serve as a basis to develop build recipes for the HPE Cray Programming Environment. + +Last but not least, EasyBuild provides various [**GitHub integration features**](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) +that greatly facilitate the contribution process: opening, updating, and testing pull requests, +reviewing incoming contributions, and much more can all be done directly from the EasyBuild +command line. This not only saves time, effort, brain cycles, and mouse clicks for contributors, +but it also makes the review process for *maintainers* significantly easier. +All together this leads to improved stability and consistency. + +--- + +### *What EasyBuild is* ***not*** + +EasyBuild is ***not*** **YABT (Yet Another Build Tool)**: it does *not* replace established build +tools like CMake or ``make``, it wraps around them. +If the installation procedure of a software package involves running some unholy trinity of tools while whispering +the correct magic incantations, EasyBuild automates this process for you. + +It is ***not*** **a replacement for traditional Linux package managers** like ``yum``, ``dnf`` or ``apt``. +EasyBuild relies on certain tools and libraries provided by the operating system. This includes glibc, OpenSSL, +drivers for Infiniband and GPUs, and so on. It is required that these tools are installed and managed by other means. +This is typically done via the package management tool that comes with your Linux distribution. + +Finally, EasyBuild is ***not*** **a magic solution to all your (software installation) problems**. +You may, and probably still will occasionally, run into compiler errors unless somebody has already taken care of the problem for you. +This is even more likely to happen in the Cray environment as builds are not often tested with the Cray compilers +or even with the AMD compilers. + +--- + +### *Requirements* + +Since EasyBuild is primarily intended for installing (scientific) software on HPC systems, the main target is +systems running some flavor of **GNU/Linux**. EasyBuild also works on *macOS*, +but significant effort would be required to use it extensively there. + + + +EasyBuild is implemented in [**Python**](https://www.python.org), and is compatible with Python 2.7 and 3.5+. +Since [Python 2 is no longer maintained](https://blog.python.org/2020/04/python-2718-last-release-of-python-2.html), +it is recommended to use a recent version of Python 3 for running EasyBuild. + +An [**environment modules**](https://en.wikipedia.org/wiki/Environment_Modules_(software)) tool is required for running EasyBuild. +The modern Lua-based implementation **[Lmod](https://lmod.readthedocs.io)** is strongly recommended, +but other (Tcl-based) implementations are also supported. +Both Tcl and Lua are supported as syntax for environment module files. +On LUMI Lmod is used as the module tool. + +--- + +### *Development* + + +EasyBuild is **actively developed** by a worldwide community under the +[**GPLv2 open source license**](https://www.gnu.org/licenses/old-licenses/gpl-2.0.html) +via the [**``easybuilders``** organisation on GitHub](https://github.com/easybuilders), in the following repositories: + +- [easybuild-framework](https://github.com/easybuilders/easybuild-framework) +- [easybuild-easyblocks](https://github.com/easybuilders/easybuild-easyblocks) +- [easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs) +- [easybuild](https://github.com/easybuilders/easybuild) (website, documentation) +- [easybuild-tutorial](https://github.com/easybuilders/easybuild-tutorial) (the generic EasyBuild tutorials) + +On LUMI the generic EasyBuild installation is used but extended and adapted using LUMI-specifics +plugins and hooks maintained in the [LUMI software stack repository](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack). + +Stable versions of EasyBuild have been released roughly every 6-8 weeks since 2012, +and are published via the *Python Package Index (PyPI)*, +see [https://pypi.org/project/easybuild](https://pypi.org/project/easybuild/). + +**Comprehensive testing** practices are applied throughout the +development cycle, with extensive suites of unit and integration tests being run in a CI environment (GitHub Actions), +consistent testing of incoming contributions (pull requests), and thorough regression testing before every release. + +--- + +### *A worldwide community* + +EasyBuild has grown into a **community project** since it was first released publicy in 2012, +and is used and developed by dozens (hundreds?) of HPC centres and consortia worldwide. + +Since 2016 the EasyBuild community has gathered for a yearly [**EasyBuild User Meeting**](https://easybuild.io/eum), +which is an open and highly interactive event that provides a great opportunity to meet fellow EasyBuild users, +developers, and maintainers, and to discuss related topics, learn about new aspects of the tool, and how other HPC +sites leverage it, and integrate it with other tools like [Jenkins](https://www.youtube.com/watch?v=bzW45cfRUMU), +[Ansible](https://www.youtube.com/watch?v=Qr2udsAjkG0), [CernVM-FS](https://www.youtube.com/watch?v=_0j5Shuf2uE), +[ReFrame](https://www.youtube.com/watch?v=KN9QwQIrso4) etc. + +HPC sites, consortia, and companies that use EasyBuild include (but are not limited to): + +VSC logo +JSC logo +HPCNow! logo +Fred Hutch logo +Vienna BioCenter logo +CÉCI logo +Compute Canada logo +UiO logo +BEAR logo +SNIC logo +National University of Singapore logo +IT4Innovations logo +Texas A&M logo +TU Dresden logo +University of Melbourne logo +University of Luxembourg logo +EPCC logo +Cyprus Institute logo +University of Basel logo +Microsoft logo + +--- + +### *Getting help* + +#### *Documentation* + +If you have any questions about EasyBuild, make sure to check the EasyBuild documentation first, +which is available at [**https://docs.easybuild.io**](https://docs.easybuild.io). + +It is not unlikely that an answer to your question is available there already. +The [search box](https://docs.easybuild.io/en/latest/search.html) on the top left may be helpful. + +#### GitHub + + + +You can open an issue in one of the EasyBuild repositories in [the `easybuilders` GitHub organisation](https://github.com/easybuilders/) to +ask questions, report problems or share suggestions and ideas for additional +improvements. + +Please try to post the issue in the most suitable repository. If you are +not sure which repository is most suited just go ahead and open the issue anyway, +the EasyBuild maintainers can move your issue if deemed appropriate. + +Also here you can use the search box on the top left, to see if somebody else +has already asked a similar question or reported a similar problem. + +#### Mailing list + +Feel free to ask any questions or provide feedback via the EasyBuild mailing list +`easybuild@lists.ugent.be`. + +Keep in mind that **you must be subscribed to the mailing list in order to +post messages there!** You can subscribe via [this link](https://lists.ugent.be/wws/subscribe/easybuild). + +In 2016-2017 over 1,000 messages were posted to the EasyBuild mailing list. +More recently this has decreased to about 700 messages per year, mostly due +to the EasyBuild Slack channel. + +#### Slack + +Slack logo + +The [EasyBuild Slack](https://easybuild.slack.com/) is a quick and easy way +to get in touch with the EasyBuild community. +Over 400 people have joined it so far, and there is usually somebody awake in there +who may be able to help you out. + +**To join the EasyBuild Slack request an invitation via +[https://easybuild-slack.herokuapp.com](https://easybuild-slack.herokuapp.com/).** + +If you are not a big fan of Slack, you can also consider joining the `#easybuild` +IRC channel (available on the [Freenode IRC network](https://freenode.net/)). +A bot relays messages between this IRC channel and the `#general` channel in the EasyBuild Slack. + + +#### Conference calls + +Every other week on Wednesday an EasyBuild conference call is set up to discuss recent +developments, provide an outlook to the next EasyBuild release, and +provide answers to questions that can be raised by the people attending the call. + +These conference calls are open to anyone interested in the EasyBuild project, +and have been organised since November 2013. + +More information, including the dates and times of the next calls, is available +[on the EasyBuild Wiki](https://github.com/easybuilders/easybuild/wiki/Conference-calls). + +--- + +*[[next: Terminology]](terminology.md)* From 20868966a6e82bcf415fc5f288937a698a85e6e0 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 9 Mar 2022 10:58:10 +0100 Subject: [PATCH 088/159] Updated the structure, initial updates of terminology and a new page about the Cray PE and Lmod. --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 4 +- .../2022-CSC_and_LO/1_01_what_is_easybuild.md | 34 +- docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md | 71 ++++ docs/2022-CSC_and_LO/1_03_terminology.md | 348 ++++++++++++++++++ 4 files changed, 438 insertions(+), 19 deletions(-) create mode 100644 docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md create mode 100644 docs/2022-CSC_and_LO/1_03_terminology.md diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md index 0bfd0ad5..e381f45d 100644 --- a/docs/2022-CSC_and_LO/1_00_part1_intro.md +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -5,8 +5,8 @@ *(Tue March 4th 2021, 9am-12 CET)* - [What is EasyBuild?](1_01_what_is_easybuild.md) - -- [Terminology](terminology.md) +- [Terminology](1_02_terminology.md) + - [Installation](installation.md) *(hands-on)* - [Configuration](configuration.md) *(hands-on)* - [Basic usage](basic_usage.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md index 678c8c4a..5ccdb1db 100644 --- a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md @@ -15,7 +15,7 @@ and provides a **command-line interface** to facilitate the installation and man -### *Elevator pitch* +## Elevator pitch EasyBuild intends to relieve HPC support teams from manually managing software installations while at the same time **providing a consistent and well performing @@ -46,7 +46,7 @@ is still under development on LUMI it is not yet covered in this LUMI-specific v --- -### *Key features* +## Key features EasyBuild is capable of **fully autonomously installing (scientific) software**, including making sure that all necessary dependencies are installed, @@ -73,13 +73,13 @@ packaging tools ([FPM](https://fpm.readthedocs.io)), and so on. --- -### *Focus points* +## Focus points EasyBuild was created specifically for **installing scientific software on HPC systems**, which is reflected in some of the design choices that were made. -#### Performance +### Performance EasyBuild strongly prefers to **build software from source code**, whenever possible. @@ -99,7 +99,7 @@ idea to hard-code options such as ``-march=native`` or ``--xHost`` in a way that cannot easily overwrite those. -#### Reproducibility +### Reproducibility In addition to performance, **reproducibility of installations** is a core aspect of EasyBuild. @@ -120,7 +120,7 @@ As we shall see when discussing the HPE Cray Programming Environment, the reprod in that environment due to the way the programming environment selects shared libraries at runtime. -#### Community effort +### Community effort In a number of different ways, we try to encourage EasyBuild users to **collaborate** and help each other out. @@ -148,7 +148,7 @@ All together this leads to improved stability and consistency. --- -### *What EasyBuild is* ***not*** +## What EasyBuild is ***not*** EasyBuild is ***not*** **YABT (Yet Another Build Tool)**: it does *not* replace established build tools like CMake or ``make``, it wraps around them. @@ -167,7 +167,7 @@ or even with the AMD compilers. --- -### *Requirements* +## Requirements Since EasyBuild is primarily intended for installing (scientific) software on HPC systems, the main target is systems running some flavor of **GNU/Linux**. EasyBuild also works on *macOS*, @@ -187,7 +187,7 @@ On LUMI Lmod is used as the module tool. --- -### *Development* +## Development EasyBuild is **actively developed** by a worldwide community under the @@ -213,7 +213,7 @@ consistent testing of incoming contributions (pull requests), and thorough regre --- -### *A worldwide community* +## A worldwide community EasyBuild has grown into a **community project** since it was first released publicy in 2012, and is used and developed by dozens (hundreds?) of HPC centres and consortia worldwide. @@ -250,9 +250,9 @@ HPC sites, consortia, and companies that use EasyBuild include (but are not limi --- -### *Getting help* +## Getting help -#### *Documentation* +### *Documentation* If you have any questions about EasyBuild, make sure to check the EasyBuild documentation first, which is available at [**https://docs.easybuild.io**](https://docs.easybuild.io). @@ -260,7 +260,7 @@ which is available at [**https://docs.easybuild.io**](https://docs.easybuild.io) It is not unlikely that an answer to your question is available there already. The [search box](https://docs.easybuild.io/en/latest/search.html) on the top left may be helpful. -#### GitHub +### GitHub @@ -275,7 +275,7 @@ the EasyBuild maintainers can move your issue if deemed appropriate. Also here you can use the search box on the top left, to see if somebody else has already asked a similar question or reported a similar problem. -#### Mailing list +### Mailing list Feel free to ask any questions or provide feedback via the EasyBuild mailing list `easybuild@lists.ugent.be`. @@ -287,7 +287,7 @@ In 2016-2017 over 1,000 messages were posted to the EasyBuild mailing list. More recently this has decreased to about 700 messages per year, mostly due to the EasyBuild Slack channel. -#### Slack +### Slack Slack logo @@ -304,7 +304,7 @@ IRC channel (available on the [Freenode IRC network](https://freenode.net/)). A bot relays messages between this IRC channel and the `#general` channel in the EasyBuild Slack. -#### Conference calls +### Conference calls Every other week on Wednesday an EasyBuild conference call is set up to discuss recent developments, provide an outlook to the next EasyBuild release, and @@ -318,4 +318,4 @@ More information, including the dates and times of the next calls, is available --- -*[[next: Terminology]](terminology.md)* +*[[next: The HPE Cray Programming Environment and Lmod]](1_02_CPE_and_Lmod.md)* diff --git a/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md b/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md new file mode 100644 index 00000000..01812057 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md @@ -0,0 +1,71 @@ +# The HPE Cray Programming Environment and Lmod + +*[[back: What is EasyBuild?]](1_01_what_is_easybuild.md)* + +--- + +On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated +as Cray PE). The environment provides several tools, including compilers, communication libraries, +optmized math libraries and various other libraries, analyzers and debuggers. + +The Cray PE is made available through *environment modules* tha allow to select particular versions of +tools and to configure the environment in a flexible way. + +--- + +## Modules + +*Module* is a massively overloaded term in (scientific) software and IT in general +(kernel modules, Python modules, and so on). +In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. + +[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept +on HPC systems: it is a way to specify changes that should be made to one or more +[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a +[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file +is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or +[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, +and specifies which environment variables should be updated, and how (append, +prepend, (re)define, undefine, etc.) upon loading the environment module. +Unloading the environment module will restore the shell environment to its previous state. + +Environment module files are processed via a **modules tool**, of which there +are several conceptually similar yet slightly different implementations. +The oldest module tool still in use today is Environment Modules 3.2, implemented in C and +supporting module files written in Tcl. After a gap in developement, Xavier Delaruelle of CEA +developed [Environment Modules 4 and 5](https://sourceforge.net/projects/modules/) which is +fully implemented on Tcl. An alternative module tool is [Lmod](https://lmod.readthedocs.io), +developed by Robert McLay at TACC and implemented in LUA. This tool supports natively LUA +module files but also offers a high degree of compatibility with Tcl-based module files +developed for Environment Modules fia a translation layer and some API translation. + +The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no +packages or official support for Environment Modules 4 or 5. On LUMI, Lmod was selected as +the module tool. At the user level, Lmod and the various versions of Emvironment Modules +have very simmilar commands for managing the environment, but with different options. +The commands for searching for modules are very different though so if you are not familiar +with Lmod and its commands for users, it is worthwile to read the +[LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). + +--- + +## Lmod features + + +--- + +## Cray PE components + + +--- + +## Configuring the Cray PE through modules + +--- + +## Further reading + + +--- + +*[[next: Terminology]](1_03_terminology.md)* diff --git a/docs/2022-CSC_and_LO/1_03_terminology.md b/docs/2022-CSC_and_LO/1_03_terminology.md new file mode 100644 index 00000000..ec085b0c --- /dev/null +++ b/docs/2022-CSC_and_LO/1_03_terminology.md @@ -0,0 +1,348 @@ +# Terminology + +*[[back: The HPE Cray Programming Environment and Lmod]](1_02_CPE_and_Lmod.md)* + +--- + +Over the years, we have come up with some terminology specific to EasyBuild +to refer to particular components, which we use alongside established terminology relevant to the context +of building and installing software. + +It is important to be familiar with these terms, so we'll briefly cover them one by one. + +--- + +## *Toolchains* + +A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, +which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. + +We refer to the different parts of a toolchain as **toolchain components**. + +The *compiler component* typically consists of [C](https://en.wikipedia.org/wiki/C_(programming_language)), +[C++](https://en.wikipedia.org/wiki/C%2B%2B), and [Fortran](https://en.wikipedia.org/wiki/Fortran) +compilers in the context of HPC, but additional compilers (for example, +a [CUDA](https://developer.nvidia.com/cuda-zone) compiler for +[GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units) software) +can also be included. + +Additional toolchain components are usually special-purpose libraries: + +- an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/)); + +- libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), + [LAPACK](http://performance.netlib.org/lapack/)); + + These libraries are included because they tend to provide a common API, so mixing them in a single + executable would cause name conflicts when linking. + +- a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); + +A toolchain that includes all of these libraries is referred to as a **full toolchain**, while +a **subtoolchain** is a toolchain that is missing one or more of these libraries. +A **compiler-only toolchain** only consists of compilers (no additional libraries). + +Many EasBuild toolchains are organised in a hierarchy. Each toolchain can use components compiled +with itself or with one of the subtoolchains. + + +### System toolchain + +The **`system` toolchain** is a special case which corresponds to using the compilers and libraries +*provided by the operating system*, rather than using toolchain components that were installed using EasyBuild. + +It used sparingly, mostly to install software where no actual compilation is done or +to build a set of toolchain compilers and its dependencies, since the versions of the system compilers +and libraries are beyond the control of EasyBuild, which could affect the reproducibility of the installation. + +On LUMI however it is used a bit more and it takes some of the functions of the ``GCCcore`` toolchain in +other EasyBuild toolchain hierarchies. + + +### Common toolchains + +The `foss` and `intel` toolchains are also known as the *common toolchains*, +because they are widely adopted by the EasyBuild community. + +The `foss` toolchain consists of all open source components (hence the name: +"FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), +[ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). + +The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version +controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. + +Roughly every 6 months, a new version of these common toolchains is agreed upon +in the EasyBuild community, after extensive testing. + +More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). + +--- + +## *Modules* + +*Module* is a massively overloaded term in (scientific) software and IT in general +(kernel modules, Python modules, and so on). +In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. + +[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept +on HPC systems: it is a way to specify changes that should be made to one or more +[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a +[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file +is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or +[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, +and specifies which environment variables should be updated, and how (append, +prepend, (re)define, undefine, etc.) upon loading the environment module. +Unloading the environment module will restore the shell environment to its previous state. + +Environment module files are processed via a **modules tool**, of which there +are several conceptually similar yet slightly different implementations. +The Tcl-based [Environment Modules](https://sourceforge.net/projects/modules/) implementation, and +[Lmod](https://lmod.readthedocs.io), a more recent Lua-based implementation (which also supports module +files written in Tcl syntax), are the most commonly used ones. + +**Environment module files are automatically generated for each software installation** by EasyBuild, +and *loading* a module results in changes being made to the environment of the current shell +session such that the corresponding software installation can be used. + +--- + +## *EasyBuild framework* + +The EasyBuild *framework* consists of a set of Python modules organised in packages (``easybuild.framework``, +``easybuild.toolchains``, ``easybuild.tools``, etc.) that collectively form **the core of EasyBuild**, +and is developed in the [``easybuild-framework`` repository on GitHub](https://github.com/easybuilders/easybuild-framework). + +It implements the **common functionality that you need when building software from source**, +providing functions for unpacking source files, applying patch files, collecting the output produced +by shell commands that are being run and checking their exit code, generating environment module files, etc. + +The EasyBuild framework does *not* implement any specific installation procedure, it only provides +the necessary functionality to facilitate this. + + +--- + +## *Easyblocks* + +An *easyblock* is **a Python module that implements a specific software installation procedure**, +and can be viewed as a plugin to the EasyBuild framework. Easyblocks can be either *generic* or *software-specific*. + +A **generic easyblock** implements an installation procedure that can be used for +multiple different software packages. Commonly used examples include the ``ConfigureMake`` easyblock +which implements the ubiquitous ``configure``-``make``-``make install`` procedure, and the +``PythonPackage`` easyblock that can be used to install a Python package. + +A **software-specific easyblock** implements an installation procedure that is specific to a particular software packages. +Infamous examples include the easyblocks we have for ``GCC``, ``OpenFOAM``, ``TensorFlow``, ``WRF``, ... + +The installation procedure performed by an easyblock can be controlled by defining +**easyconfig parameters** in an easyconfig file. + +A collection of (generic and software-specific) easyblocks is developed by the EasyBuild community +in the [``easybuild-easyblocks`` repository on GitHub](https://github.com/easybuilders/easybuild-easyblocks). + +--- + +## *Easyconfig parameters* + +An **easyconfig parameter** specifies a particular aspect of a software installation that should be performed by +EasyBuild. + +Some easyconfig parameters are **mandatory**. The following parameters *must* be defined in *every* easyconfig file: + +* ``name`` and ``version``, which specify the name and version of the software to install; +* ``homepage`` and ``description``, which provide key metadata for the software; +* ``toolchain``, which specifies the compiler toolchain to use to install the software (see + ``toolchains`` tab); + +Other easyconfig parameters are **optional**: they can be used to provide required information, +or to control specific aspects of the installation procedure performed by the easyblock. + +Some commonly used optional easyconfig parameters include: + +* ``easyblock``, which specifies which (generic) easyblock should be used for the installation; +* ``sources`` and ``source_urls``, which specify the list of source files and where to download them; +* ``dependencies`` and ``builddependencies``, which specify the list of (build) dependencies; +* ``configopts``, ``buildopts``, and ``installopts``, which specify options for the configuration/build/install commands, respectively; + +If no value is specified for an optional easyconfig parameter, the corresponding default value will be used. + +There are two groups of easyconfig parameters. *General* easyconfig parameters can be defined for any software +package, and (usually) control a specific aspect of the installation. *Custom* easyconfig parameters are +only supported by certain easyblocks, and only make sense for particular (types of) software. + +--- + +## *Easyconfig files* + +*Easyconfig files* (or *easyconfigs* for short), are **simple text files written in Python syntax +that specify what EasyBuild should install**. +Each easyconfig file defines the set of **easyconfig parameters** that collectively form a complete specification +for a particular software installation. + +The **filename** of an easyconfig file usually ends with the ``.eb`` extension. +In some contexts the filename is expected to be determined by the value of a handful of key +easyconfig parameters: ``name``, ``version``, ``toolchain`` and ``versionsuffix``. The general format for +the filename of an easyconfig file is: ``--.eb``, +where the toolchain part is omitted when a ``system`` toolchain is used, and the `` can be empty. + +The filename of easyconfig files is particularly relevant when EasyBuild is searching for easyconfig files to resolve dependencies, +since it does this purely based on filenames: interpreting the contents of every (potential) easyconfig file +it encounters would be too expensive. + +In the [``easybuild-easyconfigs`` repository on GitHub](https://github.com/easybuilders/easybuild-easyconfigs), +the EasyBuild community maintains a large (and growing) collection of easyconfig files, for a wide range of +(scientific) software. + +--- + +## *Easystack files* + +[**Easystack files**](https://docs.easybuild.io/en/latest/Easystack-files.html) +are a new concept in EasyBuild, providing a way to define a *software stack* +that should be installed by EasyBuild. + +They are written in [YAML syntax](https://yaml.org/), and include a list of *software specifications* +which correspond to a list of easyconfig files, with support for providing specific EasyBuild +configuration options for particular software packages, and including or excluding specific software +packages based on labels. + +The support for using easystack files is currently marked as *experimental*, +which means it is subject to change in future EasyBuild releases, and may be prone to errors. + +--- + +## *Extensions* + +*Extensions* is the collective term we use for **additional software packages that can be installed +on top of another software package**. Common examples are *Python packages*, *R libraries*, and *Perl modules*. + +As you can tell the common terminology here is a bit messy, so we came up with a unifying term... + +Extensions can be installed in different ways: + +* *stand-alone*, as a separate installation on top of one or more other installations; +* as a part of a *bundle* of extensions that collectively form a separate installation; +* or as an actual *extension* to a specific installation to yield a "batteries included" + type of installation (for examples by adding a bunch of Python packages from PyPI into + a Python installation); + +--- + +## *Dependencies* + +A *dependency* is a common term in the context of software. It refers to **a software +package that is either strictly required by other software, or that can be leveraged to +enhance other software** (for example to support specific features). + +There are three main types of dependencies for computer software: + +* a **build dependency** is only required when building/installing a software package; + once the software package is installed, it is no longer needed to *use* that software + (examples: `CMake`, `pkg-config`); +* a **run-time dependency** (often referred to simply as *dependency*) is a software package that is + required to *use* (or *run*) another software package (example: `Python`); +* a **link-time dependency** is somewhere in between a build and runtime dependency: + it is only needed when *linking* a software package; it can become either a build or runtime + dependency, depending on exactly how the software is installed (example: `OpenBLAS`); + +The distinction between link-time and run-time dependencies is mostly irrelevant for this tutorial, +but we will discriminate build-only dependencies. + +--- + +## *Toolchains* + +A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, +which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. + +We refer to the different parts of a toolchain as **toolchain components**. + +The *compiler component* typically consists of [C](https://en.wikipedia.org/wiki/C_(programming_language)), +[C++](https://en.wikipedia.org/wiki/C%2B%2B), and [Fortran](https://en.wikipedia.org/wiki/Fortran) +compilers in the context of HPC, but additional compilers (for example, +a [CUDA](https://developer.nvidia.com/cuda-zone) compiler for +[GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units) software) +can also be included. + +Additional toolchain components are usually special-purpose libraries: + +* an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/)); +* libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), + [LAPACK](http://performance.netlib.org/lapack/)); +* a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); + +A toolchain that includes all of these libraries is referred to as a **full toolchain**, while +a **subtoolchain** is a toolchain that is missing one or more of these libraries. +A **compiler-only toolchain** only consists of compilers (no additional libraries). + +### System toolchain + +The **`system` toolchain** is a special case which corresponds to using the compilers and libraries +*provided by the operating system*, rather than using toolchain components that were installed using EasyBuild. + +It used sparingly, mostly to install software where no actual compilation is done or +to build a set of toolchain compilers and its dependencies, since the versions of the system compilers +and libraries are beyond the control of EasyBuild, which could affect the reproducibility of the installation. + +### Common toolchains + +The `foss` and `intel` toolchains are also known as the *common toolchains*, +because they are widely adopted by the EasyBuild community. + +The `foss` toolchain consists of all open source components (hence the name: +"FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), +[ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). + +The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version +controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. + +Roughly every 6 months, a new version of these common toolchains is agreed upon +in the EasyBuild community, after extensive testing. + +More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). + +--- + +## *Modules* + +*Module* is a massively overloaded term in (scientific) software and IT in general +(kernel modules, Python modules, and so on). +In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. + +[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept +on HPC systems: it is a way to specify changes that should be made to one or more +[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a +[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file +is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or +[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, +and specifies which environment variables should be updated, and how (append, +prepend, (re)define, undefine, etc.) upon loading the environment module. +Unloading the environment module will restore the shell environment to its previous state. + +Environment module files are processed via a **modules tool**, of which there +are several conceptually similar yet slightly different implementations. +The Tcl-based [Environment Modules](https://sourceforge.net/projects/modules/) implementation, and +[Lmod](https://lmod.readthedocs.io), a more recent Lua-based implementation (which also supports module +files written in Tcl syntax), are the most commonly used ones. + +**Environment module files are automatically generated for each software installation** by EasyBuild, +and *loading* a module results in changes being made to the environment of the current shell +session such that the corresponding software installation can be used. + +--- + +## *Bringing it all together* + +The EasyBuild **framework** leverages **easyblocks** to automatically build and install +(scientific) software, potentially including additional **extensions**, using a particular compiler **toolchain**, +as specified in **easyconfig files** which each define a set of **easyconfig parameters**. + +EasyBuild ensures that the specified **(build) dependencies** are in place, +and automatically generates a set of **(environment) modules** that facilitate access to the installed software. + +An **easystack file** can be used to specify a collection of software to install with EasyBuild. + +--- + +*[[next: Installation]](installation.md)* From c4d368ff551d64aca9a20131a0c96708e31bc040 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 10 Mar 2022 14:37:51 +0100 Subject: [PATCH 089/159] Additions to the Lmod section and some restructuring, and better information in the README for the repository. --- .../2022-CSC_and_LO/1_01_what_is_easybuild.md | 2 +- docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md | 71 ------ docs/2022-CSC_and_LO/1_02_Lmod.md | 211 ++++++++++++++++++ docs/2022-CSC_and_LO/1_03_CPE.md | 30 +++ ..._03_terminology.md => 1_04_terminology.md} | 2 +- 5 files changed, 243 insertions(+), 73 deletions(-) delete mode 100644 docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md create mode 100644 docs/2022-CSC_and_LO/1_02_Lmod.md create mode 100644 docs/2022-CSC_and_LO/1_03_CPE.md rename docs/2022-CSC_and_LO/{1_03_terminology.md => 1_04_terminology.md} (99%) diff --git a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md index 5ccdb1db..979c6e90 100644 --- a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md @@ -318,4 +318,4 @@ More information, including the dates and times of the next calls, is available --- -*[[next: The HPE Cray Programming Environment and Lmod]](1_02_CPE_and_Lmod.md)* +*[[next: The Lmod module system]](1_02_Lmod.md)* diff --git a/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md b/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md deleted file mode 100644 index 01812057..00000000 --- a/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md +++ /dev/null @@ -1,71 +0,0 @@ -# The HPE Cray Programming Environment and Lmod - -*[[back: What is EasyBuild?]](1_01_what_is_easybuild.md)* - ---- - -On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated -as Cray PE). The environment provides several tools, including compilers, communication libraries, -optmized math libraries and various other libraries, analyzers and debuggers. - -The Cray PE is made available through *environment modules* tha allow to select particular versions of -tools and to configure the environment in a flexible way. - ---- - -## Modules - -*Module* is a massively overloaded term in (scientific) software and IT in general -(kernel modules, Python modules, and so on). -In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. - -[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept -on HPC systems: it is a way to specify changes that should be made to one or more -[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a -[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file -is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or -[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, -and specifies which environment variables should be updated, and how (append, -prepend, (re)define, undefine, etc.) upon loading the environment module. -Unloading the environment module will restore the shell environment to its previous state. - -Environment module files are processed via a **modules tool**, of which there -are several conceptually similar yet slightly different implementations. -The oldest module tool still in use today is Environment Modules 3.2, implemented in C and -supporting module files written in Tcl. After a gap in developement, Xavier Delaruelle of CEA -developed [Environment Modules 4 and 5](https://sourceforge.net/projects/modules/) which is -fully implemented on Tcl. An alternative module tool is [Lmod](https://lmod.readthedocs.io), -developed by Robert McLay at TACC and implemented in LUA. This tool supports natively LUA -module files but also offers a high degree of compatibility with Tcl-based module files -developed for Environment Modules fia a translation layer and some API translation. - -The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no -packages or official support for Environment Modules 4 or 5. On LUMI, Lmod was selected as -the module tool. At the user level, Lmod and the various versions of Emvironment Modules -have very simmilar commands for managing the environment, but with different options. -The commands for searching for modules are very different though so if you are not familiar -with Lmod and its commands for users, it is worthwile to read the -[LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). - ---- - -## Lmod features - - ---- - -## Cray PE components - - ---- - -## Configuring the Cray PE through modules - ---- - -## Further reading - - ---- - -*[[next: Terminology]](1_03_terminology.md)* diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md new file mode 100644 index 00000000..91cb7571 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -0,0 +1,211 @@ +# The Lmod module system + +*[[back: What is EasyBuild?]](1_01_what_is_easybuild.md)* + +--- + +On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated +as Cray PE). The environment provides several tools, including compilers, communication libraries, +optmized math libraries and various other libraries, analyzers and debuggers. + +The Cray PE is made available through *environment modules* tha allow to select particular versions of +tools and to configure the environment in a flexible way. + +--- + +## Modules + +*Module* is a massively overloaded term in (scientific) software and IT in general +(kernel modules, Python modules, and so on). +In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. + +[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept +on HPC systems: it is a way to specify changes that should be made to one or more +[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a +[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file +is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or +[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, +and specifies which environment variables should be updated, and how (append, +prepend, (re)define, undefine, etc.) upon loading the environment module. +Unloading the environment module will restore the shell environment to its previous state. + +Environment module files are processed via a **modules tool**, of which there +are several conceptually similar yet slightly different implementations. +The oldest module tool still in use today is Environment Modules 3.2, implemented in C and +supporting module files written in Tcl. After a gap in developement, Xavier Delaruelle of CEA +developed [Environment Modules 4 and 5](https://sourceforge.net/projects/modules/) which is +fully implemented on Tcl. An alternative module tool is [Lmod](https://lmod.readthedocs.io), +developed by Robert McLay at TACC and implemented in LUA. This tool supports natively LUA +module files but also offers a high degree of compatibility with Tcl-based module files +developed for Environment Modules fia a translation layer and some API translation. + +The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no +packages or official support for Environment Modules 4 or 5. On LUMI, Lmod was selected as +the module tool. At the user level, Lmod and the various versions of Emvironment Modules +have very simmilar commands for managing the environment, but with different options. +The commands for searching for modules are very different though so if you are not familiar +with Lmod and its commands for users, it is worthwile to read the +[LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). + +--- + +## Lmod hierarchy + +### User view + +Lmod supports a module hierarchy. In a hierarchy, there is a distinction between the *installed +modules* and the *available modules*. Available modules are those that can be loaded directly +without first loading any other module, while the installed modules is the complete set of +modules that one could load one way or another. A typical use case +is a hierarchy to deal with different compilers on a system and different MPI implementations. +After all, it is a common practice to only link libraries and application code compiled with the +same compiler to avoid compatibility problems between compilers (and to be able to use advanced +features such as link time optimization). This is even more important for MPI, as Open MPI and +MPCIH-derived MPI implementations have incompatible Application Binary Interfaces. This would lead +to a hierarchy with 3 levels: + +1. The ``Core`` level containing the modules for the compilers themselves, e.g., one or more versions + of the GNU compiler suite and one or more versions of LLVM-based compilers. + + Loading a compiler module would then make the next level available: + +2. The ``Compiler`` level, containing modules for libraries and packages that only rely on the compilers + but do not use MPI, as well as the MPI modules, e.g., a version of Open MPI and a version of MPICH. + + Loading one of the MPI modules would then make the next level available: + +3. The ``MPI`` level, containing libraries and applications that depend on the compiler used and the MPI + implementation. + +Now assume that we have two compilers in the hierarchy, Compiler_A and Compiler_B. Their modules would reside +at the ``Core`` level. Both copilers provide the same MPI implementation, MPI_C. So there would be two modules +for ``MPI_C`` in two different subdirectories at the ``Compiler`` level. And further assume that we have an +application, Appl_E, compiled with both Compiler_A and Compiler_B and using MPI_C. For that application there would +also be two module files at the ``MPI`` level, one in a subdirectory corresponding ao Compiler_A and MPI_C and one +in a subdirectory corresponding to Compiler_B and MPI_C. To be able to load the module for Appl_E, a user should +first load Compiler_A, then load MPI_C and only then is it possible to load the module for Appl_E: + +```bash +module load Compiler_A MPI_C Appl_E +``` + +What is interesting is what happens if the user now loads Compiler_B: + +```bash +module load Compiler_B +``` + +In a properly designed and implemented hierarchy, Lmod will unload Compiler_A which will also trigger the unloading/deactivation +of MPI_C and Appl_E. It will then load the module for Compiler_B and proceed with looking if it can find another module for +MPI_C. That will then be loaded which now makes a different module for Appl_E available, which Lmod will proceed to load. If it +cannot find an exact match for the version, Lmod will even try to locate a different version. Hence the situation after loading +Compiler_B is that now modules are loaded for Compiler_B, MPI_C for Compiler_B and Appl_E for Compiler_A with MPI_C. +All this requires very little effort from the module file programmer and very little logic in the module files. E.g., rather +then implementing a single module file for Appl_E that would require logic to see which compiler and MPI implementation is loaded +and depending on those adapt the path to the binaries, several very simple modules need to be written with very little +logic, and one could add an Appl_E module for a different compiler or MPI implementation without touching any of the already +existing module files for that application. + + +### Building blocks + +Some mechanisms in Lmod make implementing a hierarchy fairly easy (though there are a lot of hidden pitfalls) + +- The *MODULEPATH* environment variable determines which modules are available. MODULEPATH is different from any other + path-style variable in Lmod in that any change will immediately trigger a re-evaluation of which modules are available + and trigger deactivating modules that are no longer available when a directory is removed from the MODULEPATH or + looking for alternatives for deactivated modules when a directory is added to the MODULEPATH. + +- The *"one name rule"*: Lmod cannot have two modules loaded with the same name (but a different version). By default, when loading + a module with the name of an already loaded module, Lmod will automatically swap the old one with the new one, i.e., unload the + already loaded module and load the new one. + +- The *family* concept: It is possible to declare a module to be part of a family using a command in the module file. No two modules + of the same family can be loaded at the same time, and Lmod will again by default auto-swap the already loaded one with the one + being loaded. The procedure is different though as Lmod now first has to read the new module file to discover the family, and this + may lead to more side effects. But that discussion is outside the scope of this tutorial. + + The family concept was for a long time a unique feature of Lmod, but it has been added now also to Environment Modules version 5.1. + + +### Implementation details + +The above example could be implemented using 6 module files: One for each compiler, two for the MPI module and +two for the application module. + +``` +moduleroot +├── Core +│ ├── Compiler_A +│ │ └── version_A.lua +│ └── Compiler_B +│ └── version_B.lua +├── Compiler +│ ├── Compiler_A +│ │ └── version_A +│ │ └── MPI_C +│ │ └── version_C.lua +│ └── Compiler_B +│ └── version_B +│ └── MPI_C +│ └── version_C.lua +└── MPI + ├── Compiler_A + │ └── version_A + │ └── MPI_C + │ └── version_C + │ └── Appl_E + │ └── version_E.lua + └── Compiler_B + └── version_B + └── MPI_C + └── version_C + └── Appl_E + └── version_E.lua +``` + +Besides the module functions needed to create the environment needed to run the compiler, the module file for +Compiler_A would need only two lines to implement the hierarchy: + +```Lua +family('Compiler') +prepend_path('MODULEPATH', 'moduleroot/Compiler/Compiler_A/version_A') +``` + +There are now two different ``version_C.lua`` files. One contains the necessary calls to module functions to +initialise the environment to use the version compiled with Compiler_A/version_A while the other contains the +necessary functions to do that for Compiler_B/version_B. Again, two more lines are needed to implement the hierarchy. +E.g., for ``moduleroot/Compiler/Compiler_A/version_A/MPI_C/version_C.lua``: + +```Lua +family('MPI') +prepend_path('MODULEPATH', 'moduleroot/MPI/Compiler_A/version_A/MPI_C/version_C') +``` + +Finally two vesions of the ``version_E.lua`` file are needed, one to prepare the environment for using the +package with Compiler_A anmd MPI_C and one for using the package with Compiler_B and MPI_C. However, these +are just regular modules and no additions are needed to work for the hierarchy. + +Both EasyBuild and Spack support Lmod hierarchies and with these tools it is also fairly automatic to create +different versions of the module files for each compiler and MPI library used to build the application. When +hand-writing modules it may be more interesting to have a generic module which would work for all those cases +and that is also possible with Lmod. Lmod does have a range of *introspection functions* that a module can use +to figure out its name, version and place in the module tree. All that would be needed is that the various +instances of the module file are at the correct location in the module tree and link to the generic file which +can be outside the module tree. In fact, this feature is used on LUMI to implement the modules that load a +particular version of the hardware for a particular section of LUMI. + +--- + +## Finding modules + +TODO: module spider, module help, module whatis, module keyword and where do they get their information. + +--- + +## Further reading + + +--- + +*[[next: The Cray Programming Environment]](1_03_CPE.md)* diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md new file mode 100644 index 00000000..ff054066 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -0,0 +1,30 @@ +# The HPE Cray Programming Environment + +*[[back: The Lmod module system]](1_02_Lmod.md)* + +--- + +On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated +as Cray PE). The environment provides several tools, including compilers, communication libraries, +optmized math libraries and various other libraries, analyzers and debuggers. + +The Cray PE is made available through *environment modules* tha allow to select particular versions of +tools and to configure the environment in a flexible way. + +--- + +## Cray PE components + + +--- + +## Configuring the Cray PE through modules + +--- + +## Further reading + + +--- + +*[[next: Terminology]](1_04_terminology.md)* diff --git a/docs/2022-CSC_and_LO/1_03_terminology.md b/docs/2022-CSC_and_LO/1_04_terminology.md similarity index 99% rename from docs/2022-CSC_and_LO/1_03_terminology.md rename to docs/2022-CSC_and_LO/1_04_terminology.md index ec085b0c..3450c8f0 100644 --- a/docs/2022-CSC_and_LO/1_03_terminology.md +++ b/docs/2022-CSC_and_LO/1_04_terminology.md @@ -1,6 +1,6 @@ # Terminology -*[[back: The HPE Cray Programming Environment and Lmod]](1_02_CPE_and_Lmod.md)* +*[[back: The HPE Cray Programming Environment]](1_03_CPE.md)* --- From 861690ed7ae92185ff109b0c4f3ef7477d670e11 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 10 Mar 2022 14:42:32 +0100 Subject: [PATCH 090/159] Corrections to the overview for part I. --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md index e381f45d..bdfddfc5 100644 --- a/docs/2022-CSC_and_LO/1_00_part1_intro.md +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -2,10 +2,10 @@ *[[back to start page]](index.md)* -*(Tue March 4th 2021, 9am-12 CET)* - - [What is EasyBuild?](1_01_what_is_easybuild.md) -- [Terminology](1_02_terminology.md) +- [The Lmod module system](1_02_Lmod.md) +- [The HPE Cray Programming Environment and Lmod](1_03_CPE.md) +- [Terminology](1_04_terminology.md) - [Installation](installation.md) *(hands-on)* - [Configuration](configuration.md) *(hands-on)* From cd69082123a483b4b5d1d3b9bfa8a4b304b9b81f Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 11 Mar 2022 14:16:34 +0100 Subject: [PATCH 091/159] Further work on the Lmod section. --- docs/2022-CSC_and_LO/1_02_Lmod.md | 314 ++++++++++++++++++++++++++++-- 1 file changed, 299 insertions(+), 15 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md index 91cb7571..04ccff8e 100644 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -4,15 +4,6 @@ --- -On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated -as Cray PE). The environment provides several tools, including compilers, communication libraries, -optmized math libraries and various other libraries, analyzers and debuggers. - -The Cray PE is made available through *environment modules* tha allow to select particular versions of -tools and to configure the environment in a flexible way. - ---- - ## Modules *Module* is a massively overloaded term in (scientific) software and IT in general @@ -40,12 +31,16 @@ module files but also offers a high degree of compatibility with Tcl-based modul developed for Environment Modules fia a translation layer and some API translation. The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no -packages or official support for Environment Modules 4 or 5. On LUMI, Lmod was selected as -the module tool. At the user level, Lmod and the various versions of Emvironment Modules -have very simmilar commands for managing the environment, but with different options. -The commands for searching for modules are very different though so if you are not familiar -with Lmod and its commands for users, it is worthwile to read the -[LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). +packages or official support for Environment Modules 4 or 5. At the user level, +Environment Modules 3.2 and Lmod have many commands in common, but with different options. +Lmod also has some powerful features that are lacking in Environment Modules 3.2. + +!!! Note "The Cray PE on LUMI" + On LUMI, Lmod was selected as the module tool. One area where there are significant + differences between Environment Modules 3.2 (and also the newer versions) and Lmod is + in the commands for discovering modules on the system. If you are not familiar with Lmod + and its commands for users, it is worthwile to read the + [LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). --- @@ -199,8 +194,297 @@ particular version of the hardware for a particular section of LUMI. ## Finding modules +In a hierarchical setup, not all modules are available at login. This implies that a user cannot use +``module avail`` to discover which software is available on the system. To this end Lmod has powerful +search commands. It is important to understand how these commands work to ensure that the proper information +is included in the module files to improve discoverability of software. + +!!! Note "Documentation in the LUMI documentation" + Extensive information on search commands with examples of how to use them on LUMI can be found + in the [LUMI documentation]([https://)](https://docs.lumi-supercomputer.eu/), in + [the computing section, "Module environment page", "Finding modules" section](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/#finding-modules). + TODO: module spider, module help, module whatis, module keyword and where do they get their information. +### module spider command + +The available modules at any point in time are often only a subset of all installed modules on a +system. However, Lmod provides the ``module spider`` command to search for a module with a given name +among all installed modules and to tell you how this module can be loaded (i.e., which other modules +need to be loaded to make the module available). + +The ``module spider`` command has three levels, producing different outputs: + + 1. ``module spider`` without further arguments will produce a list of all + installed software and show some basic information about those packages. + Some packages may have an ``(E)`` behind their name and will appear in blue + (in the default colour scheme) which means that they are part of a different + package. These are called *extensions* of packages or modules. + This is explained a little further in this page. + + Note that ``module spider`` will also search in packages that are hidden from + being displayed. These packages can be loaded and used. However administrators + may have decided to hide them + either because they are not useful to regular users or because they think that + they will rarely or never be directly loaded by a user and want to avoid + overloading the module display. + + 2. ``module spider `` will search for the specific package. This + can be the name of a module, but it will also search some other information + that can be included in the modules. The search is also case-insensitive. + E.g., on LUMI + ```bash + module spider GNUplot + ``` + will show something along the lines of + ``` + ------------------------------------------------------------------ + gnuplot: + ------------------------------------------------------------------ + Description: + Gnuplot is a portable command-line driven graphing utility + + Versions: + gnuplot/5.4.2-cpeCray-21.08 + gnuplot/5.4.2-cpeGNU-21.08 + ``` + so even though the capitalisation of the name was wrong, it can tell us that + there are two versions of gnuplot. The ``cpeGNU-21.08`` and ``cpeCray-21.08`` + tell that the difference is the compiler that was used to install gnuplot, + being the GNU compiler (PrgEnv-gnu) and the Cray compiler (PrgEnv-cray) + respectively. + + In some cases, if there is no ambiguity, `module spider` will actually + already produce help about the package, which is the next level. + + 3. `module spider /` will show more help information + about the package, including information on which other modules need to be + loaded to be able to load the package. E.g., + ```bash + module spider git/2.35.1 + ``` + will return something along the lines of + ``` + ------------------------------------------------------------------- + git: git/2.35.1 + ------------------------------------------------------------------- + Description: + Git is a free and open source distributed version control + system + + You will need to load all module(s) on any one of the lines below + before the "git/2.35.1" module is available to load. + + CrayEnv + LUMI/21.12 partition/C + LUMI/21.12 partition/D + LUMI/21.12 partition/G + LUMI/21.12 partition/L + + Help: + ``` + (abbreviated output). Note that it also tells you which other modules need + to be loaded. You need to choose the line which is appropriate for you and + load all modules on that line, not the whole list of in this case 9 + modules. + +!!! failure "Known issue" + The Cray PE uses Lmod in an unconventional manner with the hierarchy not + build fully in the way Lmod expects. As a consequence Lmod is not always + able to generate the correct list of modules that need to be loaded to make + a package available, and the list of ways to make a module available may + also be incomplete. + + The problem is somewhat aggrevated on LUMI because the Cray PE hierarchy sits + next to the hierarchy of the software stack as the Cray PE is installed + separately and hence cannot be integrated in the way the Lmod developer had + in mind. + + +#### Module extensions + +Certain packages, e.g., Python, Perl or R, get a lot of their functionality through +other packages that are installed together with them and extend the functionity, +e.g., NumPy and SciPy for Python. Installing all those packages as separate modules +to make it easy to see if they are installed or not on a system would lead to an +overload of modules on the system. + +Similary, admins of a software stack may chose to bundle several libraries or tools +that are often used together in a single module (and single installation directory), +e.g., to reduce module clutter but also to reduce the length of the search paths for +binaries, libraries or manual pages to speed up loading of applications. + +Lmod offers a way to make those individual packages installed in a module discoverable +by declaring them as *extensions* of the module. The ``module spider`` command will +search for those too. + + 1. ``module spider`` without further arguments: The output may contain lines similar + to + ``` + ----------------------------------------------------------------------- + The following is a list of the modules and extensions currently available: + ----------------------------------------------------------------------- + Autoconf: Autoconf/2.71 (E) + + CMake: CMake/3.21.2 (E), CMake/3.22.2 (E) + ``` + which tells that ``Autoconf`` and ``CMake`` are not available as modules themselves + but as extensions of another module, and it also tells the versions that are available, + though that list may not be complete (and is not always complete for modules either + as it is limited to one line of output). + +2. ``module spider `` will search for extensions also. E.g., + ``` + module spider CMake + ``` + on LUMI will return something along the lines of + ``` + ----------------------------------------------------------------------- + CMake: + ----------------------------------------------------------------------- + Versions: + CMake/3.21.2 (E) + CMake/3.22.2 (E) + ``` + (output abbreviated). + This tells that there is no ``CMake`` module on the system but that two versions + of ``CMake`` are provided in another module. + + 3. `module spider /` will show more information on the + extension, including which module provides the extension and which other modules + have to be loaded to make that module available. E.g., on LUMI, + ``` + module spider CMake/3.22.2 + ``` + will output something along the lines of + ``` + ----------------------------------------------------------------------- + CMake: CMake/3.22.2 (E) + ----------------------------------------------------------------------- + This extension is provided by the following modules. To access the + extension you must load one of the following modules. Note that any + module names in parentheses show the module location in the software + hierarchy. + + buildtools/21.12 (LUMI/21.12 partition/L) + buildtools/21.12 (LUMI/21.12 partition/G) + buildtools/21.12 (LUMI/21.12 partition/D) + buildtools/21.12 (LUMI/21.12 partition/C) + buildtools/21.12 (CrayEnv) + ``` + (output abbreviated and slightly reformatted for readability). This tells that + ``CMake/3.22.2`` is provided by the ``bvuildtools/21.12`` module and that there + are 5 different ways to make that package available. + +??? Bug "Restrictions with older Lmod versions" + At the time of development of this tutorial, Cray is still using the pretty old + 8.3.1 version of Lmod. Even though extensions were supported since Lmod version 8.2.5, + Lmod 8.3.1 has several problems: + + - It is not possible to hide extensions in the output of ``module avail``, a feature + that only became available in version 8.5. This may be annoying to many users as + the extension list of packages such as Python, R and Perl can be very long (the + default EasyBuild installation of R contains on the order of 600 packages). + + For that reason on LUMI extensions are only used for some modules. + + - ``module avail`` also shows extensions for modules that are not available which + makes no sense. This bug was only corrected in Lmod 8.6.13 and 8.6.14. + + +### module keyword + +Another search command that is sometimes useful is `module keyword`. It really +just searches for the given word in the short descriptions that are included in +most module files and in the name of the module. The output is not always +complete since not all modules may have a complete enough short description. + +Consider we are looking for a library or package that supports MP3 audio +encoding. +```bash +module keyword mp3 +``` +will return something along the lines of +``` +---------------------------------------------------------------- + +The following modules match your search criteria: "mp3" +---------------------------------------------------------------- + + LAME: LAME/3.100-cpeCray-21.08, LAME/3.100-cpeGNU-21.08 + LAME is a high quality MPEG Audio Layer III (mp3) encoder +``` +though the output will depend on the version of Lmod. This may not be the most +useful example on a supercomputer, but the library is in fact needed to be able +to install some other packages even though the sound function is not immediately +useful. + +??? bug "Know issue: Irrelevant output" + At the moment of the development of this tutorial, this command actually + returns a lot more output, referring to completely irrelevant extensions. + This is a bug in the HPE-Cray-provided version of Lmod (8.3.1 at the time + of development of this tutorial) that was only solved in more recent versions. + + +### module avail + +The `module avail` command is used to show only available modules, i.e., modules +that can be loaded directly without first loading other modules. It can be used +in two ways: + + 1. Without a further argument it will show an often lengthy list of all + available modules. Some modules will be marked with `(D)` which means that + they are the default module that would be loaded should you load the module + using only its name. + + 2. With the name of a module (or a part of the name) it will show all modules + that match that (part of) a name. E.g., + ```bash + module avail gnuplot + ``` + will show something along the lines of + ``` + ------ EasyBuild managed software for software stack LUMI/21.08 on LUMI-L ------ + gnuplot/5.4.2-cpeCray-21.08 gnuplot/5.4.2-cpeGNU-21.08 (D) + + Where: + D: Default Module + (output abbreviated). + ``` + but + ```bash + module avail gnu + ``` + will show you an often lengthy list that contains all packages with gnu + (case insensitive) in their name or version. + + +### Getting help + +One way to get help on a particular module has already been discussed on this +page: ``module spider /`` will produce help about the package as +soon as it can unambiguously determine the package. It is the only command that +can produce help for all installed packages. The next two commands can only +produce help about available packages. + +A second command is ``module whatis`` with the name or name and version of a +module. It will show the brief description of the module that is included in +most modules on the system. If the full version of the module is not given, it +will display the information for the default version of that module. + +The third command is ``module help``. Without any further argument it will display +some brief help about the module command. However, when used as +``module help `` or ``module help /`` it will produce help for either the +default version of the package (if the version is not specified) or the +indicated version. + + +### Implementation details + +**TODO** Tell about the extensions, whatis and help functions in module file and maybe a +little bit about how module spider works. + + --- ## Further reading From 0f401f5150cfd46ea4d427e7d5bf3c59958cc9ff Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 14 Mar 2022 14:38:34 +0100 Subject: [PATCH 092/159] Finished the Lmod section. --- docs/2022-CSC_and_LO/1_02_Lmod.md | 74 +++++++++++++++++++++++++++++-- 1 file changed, 71 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md index 04ccff8e..2d1d5121 100644 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -41,6 +41,7 @@ Lmod also has some powerful features that are lacking in Environment Modules 3.2 in the commands for discovering modules on the system. If you are not familiar with Lmod and its commands for users, it is worthwile to read the [LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). + Some of those commands are also discussed on this page. --- @@ -204,7 +205,6 @@ is included in the module files to improve discoverability of software. in the [LUMI documentation]([https://)](https://docs.lumi-supercomputer.eu/), in [the computing section, "Module environment page", "Finding modules" section](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/#finding-modules). -TODO: module spider, module help, module whatis, module keyword and where do they get their information. ### module spider command @@ -481,14 +481,82 @@ indicated version. ### Implementation details -**TODO** Tell about the extensions, whatis and help functions in module file and maybe a -little bit about how module spider works. +Lmod works by executing the module file. However, the actions of all Lmod-defined +functions will depend upon the mode in which Lmod is executing the module function, +and the module file can also detect in which mode it is executing. +Modes include "load", "unload"but also "spider". E.g., when te mode is "load", the +``setenv`` function will set an environment variable to the indicated value while in +"unload" mode that environment variable will be unset, and in "spider" mode the +environment variable is left untouched. The working of ``prepend_path``, a function +that modifies PATH-style variables, depends a bit on how Lmod is configured (as it is +possible to work with reference counts), but in its most basic mode, ``prepend_path`` +will add a given directory to a given PATH-style environment variable (or move it to +the front of the PATH-style variable if the directory is already in there), while in +"unload" mode that specific directory will be removed from the PATH (but no error will +be generated should the directory that is used as the argument not be part of the path +in that PATH-style variable). When the mode is "spider", the function has special behaviour +if it is used to change the ``MODULEPATH``. It will then note the change and add that +directory to the list of directories that has to be searched for module files. +This makes ``module spider`` a very expensive command as it may have to traverse a lot +of directories and has to execute all module files in there. Therefor Lmod will build +a so-called spider cache which can be pre-built in the system for certain directories +and otherwise will be build in the user's home directory (in the ``.lmod.d/.cache`` +subdirectory). Our experience is that this cache tends to be rather fragile, +in particular on Cray systems (and that has been confirmed in discussions with +people with access to some other Cray systems) so from time to time Lmod fails to +note changes to the modules, at least when using commands such as ``module spider``. +The actual loading and unloading of the module is not based on cached information. + +Lmod has several functions that can be used in module files to provide the information +that Lmod needs for the search-related and help commands. + +The ``help`` function defines the long help text used by ``module help`` and by +``module spider`` as soon as there is no ambiguity anymore about which module is +being searched for. + +The ``whatis`` function is used to provide short information about a module. That +information is then used by ``module whatis`` and ``module keyword`` , but also +for brief information shown by ``module spider`` when multiple modules or versions +of modules are found by the command. A module file can contain multiple ``whatis`` +commands and the Lmod manuel suggests to use those lines as a kind of database +record. See, e.g., +[the Lmod manual page with module file examples](https://lmod.readthedocs.io/en/latest/100_modulefile_examples.html?highlight=whatis). +One such example is +```Lua +whatis("Name: valgrind") +whatis("Version: 3.7.0") +whatis("Category: tools") +whatis("URL: http://www.valgrind.org") +whatis("Description: memory usage tester") +``` +It is not all that important to include all those lines in a module file, but some of +those lines get a special treatment from Lmod. The line starting with ``Description`` +is used by ``module spider`` to provide some brief information about the module if it +is not totally resolved. This comes with a limitation though: It is not show for each +version of the module, so ideally all "GROMACS" modules should contain the same +description line and use other lines to provide further information about what +distinguished a particular version. +Likewise the ``Category:`` line is used by the ``spider_decoration`` hook that can be +used to add decoration to the spider level 1 output. +All in all the ``whatis`` function if often overlooked in Lmod-based module functionx +but it is a very useful function to include in the proper way in module files. + +A third function that provides information to the search commands is ``extensions``. +It can be used to list up the extensions supported by the module. The argument list +may seem strange as it takes only a single argument, a string of comma-separated ``extension/version`` +elements, but that is because the number of arguments to a function is limited in +Lua and that limit can actually be met easily by modules for Python, Perl or R packages. --- ## Further reading +- [Lmod documentation](https://lmod.readthedocs.io/en/latest/index.html) +- [Lmod on LUMI in the LUMI documentation](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/) +- [Documentation of Environment Modules 5](https://modules.readthedocs.io/en/latest/), + an alternative to Lmod (though not currently supported by HPE Cray) + --- From 290a375ae0ed18bea8242691529f417304e53705 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 14 Mar 2022 15:36:44 +0100 Subject: [PATCH 093/159] Continued merging of the new CSC tutorial in the structure of the repository. --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 2 +- docs/2022-CSC_and_LO/index.md | 4 ++++ mkdocs.yml | 8 ++++++++ 3 files changed, 13 insertions(+), 1 deletion(-) diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md index bdfddfc5..7b138670 100644 --- a/docs/2022-CSC_and_LO/1_00_part1_intro.md +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -4,7 +4,7 @@ - [What is EasyBuild?](1_01_what_is_easybuild.md) - [The Lmod module system](1_02_Lmod.md) -- [The HPE Cray Programming Environment and Lmod](1_03_CPE.md) +- [The HPE Cray Programming Environment](1_03_CPE.md) - [Terminology](1_04_terminology.md) - [Installation](installation.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md index b45a740b..43fb4518 100644 --- a/docs/2022-CSC_and_LO/index.md +++ b/docs/2022-CSC_and_LO/index.md @@ -9,7 +9,11 @@ See that site for [credits to the contributors of those tutorials](https://easyb This is an introductory tutorial to [EasyBuild](https://easybuild.io), a command line tool for installing (scientific) software on High Performance Computing (HPC) systems. +<<<<<<< HEAD This tutorial is specifically for EasyBuild as implemented on [LUMI](https://lumi-supercomputer.eu) +======= +These tutorials are specifically for EasyBuild as implemented on [LUMI](https://lumi-supercomputer.eu) +>>>>>>> 51e451f1 (Continued merging of the new CSC tutorial in the structure of the repository.) and has been developed for CSC and the Local Organisations within the LUMI consortium. Yet much of the material is useful to a broader community of EasyBuild users on Cray systems or even EasyBuild users in general. diff --git a/mkdocs.yml b/mkdocs.yml index 1e7528db..d0c340f0 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -118,6 +118,14 @@ nav: - Contributing to EasyBuild: 2020-06-isc20/contributing.md - Comparison with other tools: 2020-06-isc20/comparison_other_tools.md - Getting help: 2020-06-isc20/getting_help.md + - "CSC (spring '22)": + - (overview): 2022-CSC_and_LO/index.md + - Introduction to EasyBuild: + - (overview): 2022-CSC_and_LO/1_00_part1_intro.md + - What is EasyBuild?: 2022-CSC_and_LO/1_01_what_is_easybuild.md + - The Lmod module system: 2022-CSC_and_LO/1_02_Lmod.md) + - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_03_CPE.md + - Te0rminology: 2022-CSC_and_LO/1_04_terminology.md plugins: # show revision date at bottom of each page - git-revision-date-localized From 744d0a79f0696f23f3b8eba2d80272801cb6362b Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 14 Mar 2022 18:27:32 +0100 Subject: [PATCH 094/159] Tutorial page about the Cray PE, and correction of a typo. --- docs/2022-CSC_and_LO/1_03_CPE.md | 251 +++++++++++++++++++++++++++++++ mkdocs.yml | 2 +- 2 files changed, 252 insertions(+), 1 deletion(-) diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md index ff054066..8de332a4 100644 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -15,16 +15,267 @@ tools and to configure the environment in a flexible way. ## Cray PE components +- **Cray Compiling Environment (CCE)**: + + CCE consists of Cray compilers performing code analysis during compilation to generate highly optimized code. + Supported languages include Fortran, C and C++, and UPC (Unified Parallel C). + The Cray C/C++ compiler is basically the Clang/LLVM compiler with a back-end configured by HPE Cray that also + adds some Cray-specific options while the Cray Fortran compiler uses an HPE-Cray specific front-end + with a back-end based on LLVM. The Fortran compiler supports most of Fortran 2018 and tends to be + considerably stricter than the GNU or Intel Fortran compilers. + + The CCE compilers will also support OpenMP offload to NVIDIA and AMD GPUs but that is still work in progress + at the time of the development of this tutorial. The Fortran compiler (but not the C/C++ compiler) also supports + OpenACC for offloading to GPU. + + The classic pre-LLVM Cray C/C++ compiler is not available on Cray EX systems. + +- **Cray Scientific and Mathematical Libraries (CSML)**: + + A set of high performance libraries providing portability for scientific applications, sparse and dense linear + algebra (BLAS, LAPACK, ScaLAPACK) and fast Fourier transforms (FFTW). + + There is a special (more limited) version of the library with GPU-optimized routines. + + CSML also includes the HDF5 and netCDF libraries, in versions with and without MPI. + +- **Cray Message Passing Toolkit**: + + A collection of software libraries used to perform data transfers between nodes running in parallel applications. + It includes the Message Passing Interface (MPI) and OpenSHMEM parallel programming models. + + The HPE Cray MPI implementation is based on the MPICH3 sources from ANL. However, some algorithms are tweaked + specifically for HPE Cray hardware and the Cray interconnects. + +- **Cray Environment Setup and Compiling Support**: + + Libraries supporting code compilation and setting up the development environment, including compiler drivers. + +- **Cray Performance and Analysis Tools**: + + Tools to analyze the performance and behavior of programs that are run on Cray systems, and a Performance API (PAPI). + +- **Cray Debugging Support Tools**: + + Debugging tools, including `gdb4hpc` and `Valgrind4hpc`. + +- **Third-party compilers**: + + The Cray PE also supports several third-party compilers. Not all compilers are supported on all systems: + + - The GNU C/C++ and Fortran compilers will be present on most systems with the Cray PE. + + - The AMD AOCC C/C++ and Fortran compilers are supported on systems with AMD CPUs. + + - The AMD ROCm compilers (sometimes known as AOMP) are supported on systems with AMD GPU technology. + + - The Intel C/C++ and Fortran compilers (also the oneAPI versions) are supported on systems with Intel + processors. HPE Cray does not include them in licenses for AMD CPUs. + + - The NVIDIA HPC Toolkit compilers are supported on systems with NVIDIA GPUs. + --- ## Configuring the Cray PE through modules +Compilers in the Cray Programming Environment are almost always used through the Cray compiler wrappers. In fact, +until recently, some compilers could not be used without the wrappers as they failed to even detect their own +include files and some components. The behaviour of the wrapper will depend on the other modules that are +loaded in the environment. E.g., there is no separate wrapper for MPI as the MPI header files and libraries are +automatically activated through the regular wrappers as soon as a Cray MPI module is loaded. + +Multiple releases of the HPE Cray PE can be installed simultaneously on the system and users can mix-and-match +components. However, it is also possible to load only components for a specific release of the PE. +Cray PE versions have version numbers of the form ``yy.mm``, e.g., ``22.02`` for the version released in +February of 2022. However, each of the components have their own version number and it is not easy to see +which version of a component came with which version(s) of the Cray PE. + +Below we only discuss those modules that are important when building software with EasyBuild. Debuggers, +profilers, etc., are not included in the list. + + +### The PrgEnv-* modules + +The usual way to initialise the Cray PE is by loading one of the ``PrgEnv-*`` modules. This module will then +load the appropriate compiler, compiler wrappers and other libraries. Some of the components that will be +loaded are configured through the ``/etc/cray-pe.d/cray-pe-configuration.sh`` file, so the list of modules +may depend on the actual system that you are using. + +The Cray PE supports the following ``PrgEnv-*`` modules. On LUMI, only the first three are currently available: + +| Module | Compiler module | What? | +|:---------------|-----------------|:--------------------------------------------| +| PrgEnv-cray | cce | The Cray Compiling Environment compilers | +| PrgEnv-gnu | gcc | The GNU compilers | +| PrgEnv-aocc | aocc | AMD compilers for CPU-only systems | +| PrgEnv-amd | rocm | AMD ROCm compilers for GPU systems | +| PrgEnv-intel | intel | The Intel compilers | +| PrgEnv-nvidia | nvidia | NVIDIA HPC toolkit compilers (formerly PGI) | + +All ``PrgEnv-*`` modules belong to the same module family. Hence Lmod will automatically unload any +already loaded ``PrgEnv-*`` module when you load a different one. + + +### Selecting the version through the cpe module + +Loading one of the ``cpe/yy.mm`` modules (e.g., ``cpe/22.02``) has the following effects: + +- It sets the default versions of each of the Cray PE modules to the version that comes with that + particular release of the Cray PE. E.g., + + ``` bash + module load cpe/22.02 + module load cce + ``` + + would load that version of the ``cce`` compiler that comes with the 22.02 release of the Cray PE. + +- It will reload all already loaded Cray PE modules and switch them over to the version corresponding + to that particular release of the Cray PE. + +???+ Bug "Limitations and bugs" + Due to the way Lmod works and implementation bugs in the ``cpe`` modules, loading the ``cpe`` module + does not always have the desired effect. + + - The Cray PE sets the default version of each module by adding a file to the list of files in the + ``LMOD_MODULERCFILE`` environment variable. This is because Lmod does not re-evaluate the visibility + of modules and the internal list of default version immediately when the value of ``LMOD_MODULERCFILE`` + is changed but only the next time the ``module`` command is executed. Hence + ``` bash + module load cpe/22.02 ; module load cce + ``` + and + ``` + module load cpe/22.02 cce + ``` + do not have the same effect. In the first version, the version of ``cce`` loaded is the version that + corresponds to the 22.02 release of the Cray PE. In the second case however the default version of the + ``cce`` module is determined by whatever list of default modules was used when calling the ``module`` + command so may or may not be the one of the 22.02 release. + + - Loading the ``cpe`` module after loading the other Cray PE modules also does not always have the + desired effect in many versions of the Cray PE. This is because of a bug in the ``cpe `` module that + reloads the modules in the wrong order which may trigger the reload of a module with whatever version + was the default when the ``module`` command was called rather than the version the the ``cpe`` module + intends to (re-)load. + + +### The compiler wrapper module craype + +The ``craype`` module loads the compiler wrappers. There is only one set of compiler wrappers for all compilers. +Which compiler will be called, which libraries will be included, but also processor and GPU target options will +be used, is all determined by other modules. Hence it is in principle possible to use a single Makefile for +a project and still reconfigure the build by loading certain modules. + + +### Target modules + +The targets for CPU and GPU optimization, the network library for MPI bt also some other compiler options, +can be set through target modules: + +- ``craype-x86-*`` (and similar options can be expected on ARM-based systems) set the target for CPU + optimisations. For LUMI, the ``crape-x86-rome``, ``craype-x86-milan`` and ``craype-x86-trento`` modules + are relevant. + + This can also be used to cross-compile to a different target architecture unless the compiler target + gets overwritten by a compiler flag added to the command line through the Makefile or other means, + something that unfortunately happens more and more often in faulty software installation procedures. + +- ``creype-accel-*`` sets the target for OpenMP offload (and likely other technologies in the future). E.g., + loading ``craype-accel-amd-gfx90a`` tells the compilers to target AMD MI200 family GPUs, while loading + ``craype-accel-host`` tells the compiler to use the CPU instead (according to the documentation, the latter + is for ``PrgEnv-cray`` only). + +- ``craype-network-*`` selects the communication library to be used by Cray MPICH. On Slingshot 11 EX systems, + only ``craype-network-ofi`` is supported, but Slingshot 10 EX systems also offer support for UCX through the + ``craype-network-ucx`` module. + +- The ``craype-hugepages*`` modules enable Cray Huge Pages support. To fully enable this support they have to + be used at link-time and at run-time. At link time, support is compiled into the binary wile at run-time they + are used to set the actual size of the huge pages. + + The ``craype-hugepages*`` modules are not supported by all compilers. E.g., the AOCC compiler does not support + huge pages at the moment, and loading the module at link time will cause an error message from the linker. + + +### The compiler modules + +The compiler modules have already been discussed with the ``PrgEnv-*`` modules above. The different regular +compiler modules also all belong to the same family so no two different compilers can be loaded simultaneously +and Lmod will automatically unload the other compiler when a new one is loaded. + + +### The MPI modules + +To load the Cray MPI libraries, both one of the ``craype-network-*`` modules and a compiler module has to be +loaded as the MPI libraries are both network- and compiler specific. + +For some unknown reason, the MPI module for the libfabric (``craype-network-ofi``) transport is called ``cray-mpich`` +while the library for the UCX transport (``craype-network-ucx``) is called ``craype-network-ucx``. As a result, +the MPI module fails to reload automatically when switching between both transports, but it does reload automatically +when switching compilers. + +Loading an MPI module will also automatically configure the regular compiler wrappers to compile with support for +that MPI module. However, the libfabric and UCX versions of the MPI library have compatible interfaces, so it is +always possible to swap between those versions at runtime. + + +### The Cray Scientific libraries + +The Cray Scientific Libraries are loaded through the ``cray-libsci`` module (or ``cray-libsci_acc`` for the GPU +versions). Loading this module makes the BLAS, LAPACK, and ScaLAPACK libraries available, and also the Cray IRT +(Iterative Refinement Toolkit), a Cray-specific library. It will also configure the compiler wrappers to link +with these libraries, so no additional include or link options are needed. + + +### The ``fftw`` module + +As FFTW is a third party library, it gets its own module (``cray-fftw``) and is not included in the ``cray-libsci`` module. + + +--- + +## Some unexpected behaviour of the Cray PE + +On Cray EX systems, dynamic linking is the preferred way of linking applications. Whith that comes some +unexpected behaviour of the Cray modules. EasyBuild users expect that at run time the versions of the +libraries that are used, are the ones from the modules that are loaded. This is not always the case +for the runtime libraries of the Cray PE. By default the Cray PE will use a default set of libraries +that is determined by the system default version of the Cray PE (which is set by the sysadmins, not +determined by the ``cpe`` module). This can only be avoided by either using\ +rpath-linking (which is also special in the Cray PE as the wrappers activate rpath linking if the +environment variable ``CRAY_ADD_RPATH`` is defined and set to ``yes``) or by manually adjusting +the library path after loading the modules: +``` bash +export LD_LIBRARY_PATH=${CRAY_LD_LIBRARY_PATH}:$LD_LIBRARY_PATH +``` +The latter cannot be easily automated in modulefiles. Any technique that can be used (without actually +reworking the Cray PE modules) has nasty side effects in some scenarios. + +The net result of this feature is that some reproducibility of the results is lost. Programs will +react differently if the system default version of the Cray PE is changed as that will change the +set of default libraries that will be used at runtime unless rpath-linking is used or users +redfine ``LD_LIBRARY_PATH``. + + --- ## Further reading +- [LUMI documentation: "Developing" section](https://docs.lumi-supercomputer.eu/development/) +- The Cray PE is mostly documented through man pages. There used to be some documentation on the + Cray web site also but the documentation system got reworked after the merger with HPE. + The documentation is now in the [HPE Support Centre](https://support.hpe.com/) where it + is very difficult to find the right version of the documents. +- The [PE-Cray](https://github.com/PE-Cray) GitHub project also provides some additional + documentation, including + - [some whitepapers](https://github.com/PE-Cray/whitepapers), + - [documentation for Cray OpenSMHEMX](https://github.com/PE-Cray/cray-openshmemx) and + - [information about Cray DSMML](https://github.com/PE-Cray/cray-dsmml) + though much of that information ia also not updated regularly anymore. + --- *[[next: Terminology]](1_04_terminology.md)* diff --git a/mkdocs.yml b/mkdocs.yml index d0c340f0..3a572eb9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -125,7 +125,7 @@ nav: - What is EasyBuild?: 2022-CSC_and_LO/1_01_what_is_easybuild.md - The Lmod module system: 2022-CSC_and_LO/1_02_Lmod.md) - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_03_CPE.md - - Te0rminology: 2022-CSC_and_LO/1_04_terminology.md + - Terminology: 2022-CSC_and_LO/1_04_terminology.md plugins: # show revision date at bottom of each page - git-revision-date-localized From 8496f458b8295a22b98f218b6f3b8dfaecccb1f1 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 16 Mar 2022 15:23:45 +0100 Subject: [PATCH 095/159] Corrected a typo in mkdocs.yml --- mkdocs.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index 3a572eb9..29b837cb 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -123,7 +123,7 @@ nav: - Introduction to EasyBuild: - (overview): 2022-CSC_and_LO/1_00_part1_intro.md - What is EasyBuild?: 2022-CSC_and_LO/1_01_what_is_easybuild.md - - The Lmod module system: 2022-CSC_and_LO/1_02_Lmod.md) + - The Lmod module system: 2022-CSC_and_LO/1_02_Lmod.md - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_03_CPE.md - Terminology: 2022-CSC_and_LO/1_04_terminology.md plugins: From d5b2647a732ae734f7f3ec05aa994d969ff88e7b Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 16 Mar 2022 15:24:08 +0100 Subject: [PATCH 096/159] Adding an example of implementing a hierarchy to the Lmod section of the tutorial. --- docs/2022-CSC_and_LO/1_02_Lmod.md | 191 ++++++++++++++++++++++++++++++ 1 file changed, 191 insertions(+) diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md index 2d1d5121..0caa8a47 100644 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -548,6 +548,197 @@ elements, but that is because the number of arguments to a function is limited i Lua and that limit can actually be met easily by modules for Python, Perl or R packages. +--- + +## Some warnings about writing modulefiles + +***This section is very technical and only useful if you want to manually implement +modules that depend on each other one way or another.*** + +Lmod cannot guarantee that the order of unloading the modules will be the inverse of +the order in which they were loaded. Moreover, unloading a module is not done by reverting +stored actions done when loading the module, but by executing the modulefile again +in a mode that reverts certain actions. This can lead to subtle problems when modulefiles +communicate with each other through environment variables or by detecting which other +modules are loaded. These problems are usually solved by using a proper hierarchy +and basing actions of modulefiles on their position in the hierarchy. + +One case where passing information between modules through environment variables will +go wrong is when that environment variable is subsequently used to compute a directory +name that should be added to a PATH-like variable. Assume we have two versions of +a ``MyPython`` module, e.g., ``MyPython/2.7.18`` and ``MyPython/3.6.10``. That module then +sets an environment variable ``PYTHON_API_VERSION`` to either ``2.7`` or ``3.6``. +Next we have a module ``MyPythonPackage`` that makes a number of Python packages available +for both Python modules. However, as some Python packages have to be installed separately +for each Python version, it does so by adding a directory to the environment variable +``PYTHON_PATH`` that contains the version which it gets by using the Lua function +``os.getenv`` to request the value of ``PYTHON_API_VERSION``. + +One problem becomes clear in the following scenario: +``` bash +module load MyPython/2.7.18 +module load MyPythonPackage/1.0 +module load MyPython/3.6.10 +``` +The ``module load MyPythonPackage`` will find the environment variable ``PYTHON_PACKAGE_API`` +with the value ``2.7`` as set by ``module load MyPython/2.7.18`` and hence add the directory +for the packages for version 2.7 to ``PYTHON_PATH``. The ``module load MyPython/3.6.10`` +command will trigger two operations because of the *"one name rule"*: First it will +automatically unload ``MyPython/2.7.18`` (which will unset ``PYTHON_API_VERSIUON``) and +next it will load ``MyPython/3.6.10`` which will set ``PYTHON_API_VERSION`` to ``3.6``. +However, ``MyPythonPackage`` is not reloaded so the ``PYTHON_PATH`` variable will now point +to the wrong directory. One would be tempted to think that the easy fix for the user would +be to reload ``MyPythonPackage/1.0``: +``` bash +module load MyPythonPackage/1.0 +``` +Because of the *"one name rule"* this will again trigger an unload followed by a load +of the module. The problem is in the unload. One would expect that first unloading +``MyPythonPackage`` would remove the 2.7 directory from the ``PYTHON_PATH`` but it +will not. Lmod does not remeber that last time it loaded ``MyPythonPackage`` it added +the 2.7 directory to ``PythonPath``. Instead it will execute the commands in the +modulefile and reverse certain commands. Since ``PYTHON_API_VERSION`` has now the value +``3.6``, it will try to remove the directory for version ``3.6`` which is not in the +``PYTHON_PATH``. The subsequent load will then add the 3.6 directory to ``PYTHON_PATH`` +so the environment variable now contains both directories. + +In this simple case, a ``module purge`` after the first two ``module load`` commands would +still work as Lmod is able to figure out the right order to unload modules, but in more +complicated examples this may also go wrong. However, a ``module purge`` command after +the load of ``MyPython/3.6.10`` would also fail to clean up the environment as it would +still fail to remove the 2.7 directory from ``PYTHONPATH``. + +??? Note "Running the example" + To test this example for yourself, create a directory and add that directory to + the ``MODULEPATH`` using ``module use``. In that directory, create the following + subdirectories and files: + 1. ``MyPython/2.7.18.lua`` with content: + ``` lua + LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) + setenv( 'PYTHON_API_VERSION', '2.7' ) + ``` + 2. ``MyPython/3.6.10.lua`` with content: + ``` lua + LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) + setenv( 'PYTHON_API_VERSION', '3.6' ) + ``` + 3. ``MyPythonPackage/1.0.lua`` with content: + ``` lua + LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) + LmodMessage( 'PYTHON_API_VERSION = ' .. ( os.getenv( 'PYTHON_API_VERSION' ) or '') ) + prepend_path( 'PYTHON_PATH', 'someroot/python' .. + ( os.getenv( 'PYTHON_API_VERSION' ) or 'TTT' ) .. '/packages' ) + LmodMessage( 'PYTHON_PATH = ' .. ( os.getenv( 'PYTHON_PATH' ) or '') ) + ``` +??? Note "Solution with a hierarchy" + The better way in Lmod to implement the above scenario would be in a module hierarchy. + + Just to show the power of Lmod introspection functions combined with a proper hierarchy + we present a solution using only one version of the code for ``MyPython`` and one version + of the code for ``MyPythonPackages``. + + It is best to start from a clean directory. In that directory, create: + + 1. The files ``level1/MyPython/2.7.18.lua`` and ``level1/MyPython/3.6.10.lua``, + both with the same contents: + ``` lua + LmodMessage( '\nIn ' .. myModuleFullName() .. ' in mode ' .. mode() ) + + local api_version = myModuleVersion():match( '(%d+%.%d+)%..*' ) + + -- Set the variable PYTHON_API_VERSION but not for internal use in the modules. + setenv( 'PYTHON_API_VERSION', api_version ) + + local module_root = myFileName():match( '(.*)/level1/' .. myModuleFullName() ) + prepend_path( 'MODULEPATH', pathJoin( module_root, 'level2/PythonAPI', api_version ) ) + LmodMessage( 'MODULEPATH is now\n ' .. + os.getenv( 'MODULEPATH' ):gsub( ':', '\n ' ) ) + ``` + + 2. The files ``level2/PythonAPI/2.7/MyPythonPackage/1.0.lua`` and + ``level2/PythonAPI/3.6/MyPythonPackage/1.0.lua``, both with the contents: + ``` lua + LmodMessage( '\nIn ' .. myFileName() .. ' in mode ' .. mode() ) + + local python_api_version = myFileName():match( '.*/level2/PythonAPI/([^/]+)/.*' ) + LmodMessage( 'Detected Python API version from hierarchy: ' .. python_api_version ) + LmodMessage( 'Detected Python API version from environment: ' .. + ( os.getenv( 'PYTHON_API_VERSION' ) or '' ) ) + + prepend_path( 'PYTHON_PATH', 'someroot/python' .. python_api_version .. '/packages' ) + + LmodMessage( 'PYTHON_PATH = ' .. (os.getenv( 'PYTHON_PATH' ) or '') ) + ``` + + Now add the ``level1`` subdirectory to ``MODULEPATH``, e.g., if you're in the directory + containing the ``level1`` and ``level2`` subdirectories: + ``` bash + module use $PWD/level1 + ``` + and then try the following commands: + ``` bash + module avail + module load MyPython/2.7.18 + module avail + module load MyPythonPackage/1.0 + module load MyPython/3.6.10 + ``` + and pay attention to the output. + + Initially ``module avail`` will show none of the ``MyPythonPackage`` modules. These are + installed modules but not available modules. ``module load MyPython/2.7.18`` will set the + environment variable ``PYTHON_API_VERSION`` to ``2.7`` and also add a directory to the front + of the ``MODULEPATH`` with the directory name ending on ``level2/PythonAPI/2.7``. Now + ``module avail`` will show the ``MyPythonPackage/1.0`` module. + + The ``MyPythonPackage`` shows two ways to get the version of the Python API to use for + determining the right directory to add to ``PYTHON_PATH``. The fragile way is to enquire + the value of the environment variable ``PYTHON_API_VERSION`` set by loading ``MyPython/2.7.18``. + The more robust way is to use the Lmod introspection function ``myFileName()`` which returns + the full path and file name of the module file that is executing, and extracting the version + from the path with a pattern matching function. In this particular situation both computed + values are the same so both would have worked to correctly add + ``somedir/python2.7/packages`` to the front of ``PYTHON_PATH``. + + The next command, ``module load MyPython/3.6.10`` triggers a chain of events. + + First, Lmod notices that there is already a module loaded with the same name, so it will + unload ``MyPython/2.7.18``. This will unset the environment variable ``PYTHON_API_VERSION`` + (the inverse operation of ``setenv``) and will remove the ``.../level2/PythonAPI/2.7`` + subdirectory from the ``MODULEPATH`` (the inverse action of ``prepend_path``). + + Now due to + the change of the ``MODULEPATH`` the ``MyPythonPackage/1.0`` module which was loaded from + ``.../level2/PythonAPI/2.7`` is no longer available so Lmod will continue with unloading + that module. The interesting bit now is that ``PYTHON_API_VERSION`` is unset. So had we + computed the name of the directory to add to ``PYTHON_PATH`` using the value of that + environment variable, the module would have failed to compute the correct directory name + to remove so ``prepend_path`` would have left the ``PYTHON_PATH`` environment variable + untouched. However, by computing that value from the directory of the modulefile, we get + the right value and can correctly remove ``somedir/python2.7/packages`` from ``PYTHON_PATH``. + Lmod will also remember that the module was only unloaded due to a change in the + ``MODULEPATH`` and not because a user explicitly unloaded the module. I.e., it considers + the module as deactivated but not as unloaded. + + Lmod proceeds with loading the ``MyPython/3.6.10`` module. This will now set + ``PTHON_API_VERSION`` to ``3.6`` and add a directory with name ending on + ``level2/PythonAPI/3.6`` to ``MODULEPATH``. + + Things are not done yet though. As the ``MODULEPATH`` has changed, Lmod looks at its list + of deactivated modules and notices that a different version of ``MyPythonPackage/1.0`` is + now available. Hence it will now automatically load that module from the + ``.../level2/PythonAPI/3.6`` subdirectory so that that module now correctly detects + that ``somedir/python3.6/package`` should be added to ``PYTHON_PATH``. + + Hence at the end of the cycle we have again a correctly configured environment with no + trace of the ``2.7`` version that was loaded initially and with no action required from + the user to ensure that ``MyPythonPackage`` is unloaded and reloaded to ensure the + correct configuration. + + This idea is used on LUMI to implement the various versions of the software stack with + for each software stack also optimised binaries for each of the node types. + + --- ## Further reading From 7eda98dc196ce4eacf459821807f98b8e68af1df Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 10:08:23 +0100 Subject: [PATCH 097/159] More information about hierarchy in the Cray PE, and definition of toolchain in Terminology. --- docs/2022-CSC_and_LO/1_03_CPE.md | 3 +++ docs/2022-CSC_and_LO/1_04_terminology.md | 21 ++++++++++++++++++--- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md index 8de332a4..42eb197e 100644 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -233,6 +233,9 @@ with these libraries, so no additional include or link options are needed. As FFTW is a third party library, it gets its own module (``cray-fftw``) and is not included in the ``cray-libsci`` module. +The ``cray-fftw`` module can only be loaded if one of the processor target modules (the ``craype-x86-*`` modules) +is loaded first. + --- diff --git a/docs/2022-CSC_and_LO/1_04_terminology.md b/docs/2022-CSC_and_LO/1_04_terminology.md index 3450c8f0..19f1e01f 100644 --- a/docs/2022-CSC_and_LO/1_04_terminology.md +++ b/docs/2022-CSC_and_LO/1_04_terminology.md @@ -28,22 +28,37 @@ can also be included. Additional toolchain components are usually special-purpose libraries: -- an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/)); +- an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/) + or vendor-specific MPI implementations like Intel MPI in a toolchain based on the Intel compilers); - libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), - [LAPACK](http://performance.netlib.org/lapack/)); + [LAPACK](http://performance.netlib.org/lapack/), [ScaLAPACK](http://www.netlib.org/scalapack/)); These libraries are included because they tend to provide a common API, so mixing them in a single executable would cause name conflicts when linking. - a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); +These libraries are included because they are some of the most popular libraries in scientific software, +but also because they provide an API that is available through different implementations. Mixing +implementations in a single build is a recipe for problems as the various implementations might +define the same symbols. + A toolchain that includes all of these libraries is referred to as a **full toolchain**, while a **subtoolchain** is a toolchain that is missing one or more of these libraries. A **compiler-only toolchain** only consists of compilers (no additional libraries). +The toolchain concept also maps nicely on the Programming Environment concept in the Cray +Programming Environment. Each ``PrgEnv-*`` module in fact provides a full toolchain in a +typical Cray PE installation, except for the FFTW library. Note that systems could configure +the contents of a ``PrgEnv-*`` module differently as the MPI library and scientific library are +optional. In the Cray PE, the MPI library is provided by the ``cray-mpic`` module, +the BLAS, LAPACK and ScaLAPACK libraries by the ``cray-libsci`` module (for CPU-only nodes) +and the FFTW library through the ``cray-fftw`` module. + Many EasBuild toolchains are organised in a hierarchy. Each toolchain can use components compiled -with itself or with one of the subtoolchains. +with itself or with one of the subtoolchains. The Cray toolchains however are not currently +organised in a hierarchy. ### System toolchain From e1bde8091a59fab6e3264a49c370cfbb9aef5cb8 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 10:33:37 +0100 Subject: [PATCH 098/159] Add a mermaid.js diagram with the toolchain hierarchy. --- docs/2022-CSC_and_LO/1_04_terminology.md | 42 +++++++++++++++++------- 1 file changed, 30 insertions(+), 12 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_04_terminology.md b/docs/2022-CSC_and_LO/1_04_terminology.md index 19f1e01f..ebb99963 100644 --- a/docs/2022-CSC_and_LO/1_04_terminology.md +++ b/docs/2022-CSC_and_LO/1_04_terminology.md @@ -12,7 +12,7 @@ It is important to be familiar with these terms, so we'll briefly cover them one --- -## *Toolchains* +## Toolchains A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. @@ -82,6 +82,8 @@ because they are widely adopted by the EasyBuild community. The `foss` toolchain consists of all open source components (hence the name: "FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), [ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). +In recent versions (since 2021a), [FlexiBLAS](https://www.mpi-magdeburg.mpg.de/projects/flexiblas) +is used as the BLAS library, with OpenBLAS and LAPACK as the backend. The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. @@ -89,11 +91,27 @@ controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.c Roughly every 6 months, a new version of these common toolchains is agreed upon in the EasyBuild community, after extensive testing. +The common toolchains are also organised in a toolchain hierarchy: + +```mermaid +graph TD; +A[SYSTEM] --> B[GCCcore]; +B --> C[GCC]; +B --> D[iccifort]; +C --> E[gompi: Adds OpenMPI]; +D --> F[iimpi: Adds Intel MPI]; +D --> G[imkl: Adds Intel MKL]; +E --> H[foss: Adds OpenBLAS, LAPACK, ScaLAPACK, FFTW]; +F --> I[intel]; +G --> I; +``` + More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). + --- -## *Modules* +## Modules *Module* is a massively overloaded term in (scientific) software and IT in general (kernel modules, Python modules, and so on). @@ -121,7 +139,7 @@ session such that the corresponding software installation can be used. --- -## *EasyBuild framework* +## EasyBuild framework The EasyBuild *framework* consists of a set of Python modules organised in packages (``easybuild.framework``, ``easybuild.toolchains``, ``easybuild.tools``, etc.) that collectively form **the core of EasyBuild**, @@ -137,7 +155,7 @@ the necessary functionality to facilitate this. --- -## *Easyblocks* +## Easyblocks An *easyblock* is **a Python module that implements a specific software installation procedure**, and can be viewed as a plugin to the EasyBuild framework. Easyblocks can be either *generic* or *software-specific*. @@ -158,7 +176,7 @@ in the [``easybuild-easyblocks`` repository on GitHub](https://github.com/easybu --- -## *Easyconfig parameters* +## Easyconfig parameters An **easyconfig parameter** specifies a particular aspect of a software installation that should be performed by EasyBuild. @@ -188,7 +206,7 @@ only supported by certain easyblocks, and only make sense for particular (types --- -## *Easyconfig files* +## Easyconfig files *Easyconfig files* (or *easyconfigs* for short), are **simple text files written in Python syntax that specify what EasyBuild should install**. @@ -211,7 +229,7 @@ the EasyBuild community maintains a large (and growing) collection of easyconfig --- -## *Easystack files* +## Easystack files [**Easystack files**](https://docs.easybuild.io/en/latest/Easystack-files.html) are a new concept in EasyBuild, providing a way to define a *software stack* @@ -227,7 +245,7 @@ which means it is subject to change in future EasyBuild releases, and may be pro --- -## *Extensions* +## Extensions *Extensions* is the collective term we use for **additional software packages that can be installed on top of another software package**. Common examples are *Python packages*, *R libraries*, and *Perl modules*. @@ -244,7 +262,7 @@ Extensions can be installed in different ways: --- -## *Dependencies* +## Dependencies A *dependency* is a common term in the context of software. It refers to **a software package that is either strictly required by other software, or that can be leveraged to @@ -266,7 +284,7 @@ but we will discriminate build-only dependencies. --- -## *Toolchains* +## Toolchains A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. @@ -319,7 +337,7 @@ More information on these toolchains is available [in the EasyBuild documentatio --- -## *Modules* +## Modules *Module* is a massively overloaded term in (scientific) software and IT in general (kernel modules, Python modules, and so on). @@ -347,7 +365,7 @@ session such that the corresponding software installation can be used. --- -## *Bringing it all together* +## Bringing it all together The EasyBuild **framework** leverages **easyblocks** to automatically build and install (scientific) software, potentially including additional **extensions**, using a particular compiler **toolchain**, From a0f4141468043f2770a1acaffc921eebad8a4caa Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 15:02:25 +0100 Subject: [PATCH 099/159] Finished terminology session, extended the installation section with LUMI-specific material. --- docs/2022-CSC_and_LO/1_04_terminology.md | 141 ++---- docs/2022-CSC_and_LO/1_05_installation.md | 543 ++++++++++++++++++++++ docs/2022-CSC_and_LO/index.md | 4 - mkdocs.yml | 1 + 4 files changed, 571 insertions(+), 118 deletions(-) create mode 100644 docs/2022-CSC_and_LO/1_05_installation.md diff --git a/docs/2022-CSC_and_LO/1_04_terminology.md b/docs/2022-CSC_and_LO/1_04_terminology.md index ebb99963..9c643579 100644 --- a/docs/2022-CSC_and_LO/1_04_terminology.md +++ b/docs/2022-CSC_and_LO/1_04_terminology.md @@ -109,38 +109,13 @@ G --> I; More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). ---- - -## Modules - -*Module* is a massively overloaded term in (scientific) software and IT in general -(kernel modules, Python modules, and so on). -In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. - -[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept -on HPC systems: it is a way to specify changes that should be made to one or more -[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a -[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file -is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or -[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, -and specifies which environment variables should be updated, and how (append, -prepend, (re)define, undefine, etc.) upon loading the environment module. -Unloading the environment module will restore the shell environment to its previous state. - -Environment module files are processed via a **modules tool**, of which there -are several conceptually similar yet slightly different implementations. -The Tcl-based [Environment Modules](https://sourceforge.net/projects/modules/) implementation, and -[Lmod](https://lmod.readthedocs.io), a more recent Lua-based implementation (which also supports module -files written in Tcl syntax), are the most commonly used ones. - -**Environment module files are automatically generated for each software installation** by EasyBuild, -and *loading* a module results in changes being made to the environment of the current shell -session such that the corresponding software installation can be used. - --- ## EasyBuild framework +EasyBuild is written in Python and organised in three layers, represented by the three blocks in the logo: +the EasyBuild framework, easyblocks and Easyconfig files. + The EasyBuild *framework* consists of a set of Python modules organised in packages (``easybuild.framework``, ``easybuild.toolchains``, ``easybuild.tools``, etc.) that collectively form **the core of EasyBuild**, and is developed in the [``easybuild-framework`` repository on GitHub](https://github.com/easybuilders/easybuild-framework). @@ -165,12 +140,20 @@ multiple different software packages. Commonly used examples include the ``Confi which implements the ubiquitous ``configure``-``make``-``make install`` procedure, and the ``PythonPackage`` easyblock that can be used to install a Python package. +*In Spack this functionality is provided in the core of the package.* + A **software-specific easyblock** implements an installation procedure that is specific to a particular software packages. Infamous examples include the easyblocks we have for ``GCC``, ``OpenFOAM``, ``TensorFlow``, ``WRF``, ... The installation procedure performed by an easyblock can be controlled by defining **easyconfig parameters** in an easyconfig file. +*In Spack, this functionality is provided by the ``package.py`` files for the various packages. However, EasyBuild +does not need a software-specific easyblock for every package as for many packages the combination of a +generic easyblock with the much simpler description in the easyconfig file is sufficient. +The software-specific easyblocks can be a bit troublesome on Cray PE systems. Some easyblocks contain code that +is specific for certain compilers or toolchains and do not support all compilers or all toolchains.* + A collection of (generic and software-specific) easyblocks is developed by the EasyBuild community in the [``easybuild-easyblocks`` repository on GitHub](https://github.com/easybuilders/easybuild-easyblocks). @@ -227,6 +210,17 @@ In the [``easybuild-easyconfigs`` repository on GitHub](https://github.com/easyb the EasyBuild community maintains a large (and growing) collection of easyconfig files, for a wide range of (scientific) software. +*EasyBuild does not have an equivalent of the concretiser of Spack. EasyBuild always starts from a complete, +precise configuration of a package to install, determined by the easyconfig parameters and possibly a +custom easyblock. Both approaches have their pros and cons. The concretiser in Spack combined with the +configuration of a software package via command line parameters makes it easy to quickly generate a +number of different configurations of a package to test performance or to check which combinations work +well. But one also has to be careful to not cause an explosion of modules and software installations, and +take care of using the appropriate features to ensure that one can generate the same configurations again. +The precise configurations of EayBuild can sometimes slow down things as they may need a lot of file editing, +but when it comes to building a reproducible software stack and sharing precise building instructions, this +approach may be preferred.* + --- ## Easystack files @@ -273,95 +267,14 @@ There are three main types of dependencies for computer software: * a **build dependency** is only required when building/installing a software package; once the software package is installed, it is no longer needed to *use* that software (examples: `CMake`, `pkg-config`); -* a **run-time dependency** (often referred to simply as *dependency*) is a software package that is - required to *use* (or *run*) another software package (example: `Python`); * a **link-time dependency** is somewhere in between a build and runtime dependency: it is only needed when *linking* a software package; it can become either a build or runtime dependency, depending on exactly how the software is installed (example: `OpenBLAS`); +* a **run-time dependency** (often referred to simply as *dependency*) is a software package that is + required to *use* (or *run*) another software package (example: `Python`); -The distinction between link-time and run-time dependencies is mostly irrelevant for this tutorial, -but we will discriminate build-only dependencies. - ---- - -## Toolchains - -A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, -which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. - -We refer to the different parts of a toolchain as **toolchain components**. - -The *compiler component* typically consists of [C](https://en.wikipedia.org/wiki/C_(programming_language)), -[C++](https://en.wikipedia.org/wiki/C%2B%2B), and [Fortran](https://en.wikipedia.org/wiki/Fortran) -compilers in the context of HPC, but additional compilers (for example, -a [CUDA](https://developer.nvidia.com/cuda-zone) compiler for -[GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units) software) -can also be included. - -Additional toolchain components are usually special-purpose libraries: - -* an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/)); -* libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), - [LAPACK](http://performance.netlib.org/lapack/)); -* a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); - -A toolchain that includes all of these libraries is referred to as a **full toolchain**, while -a **subtoolchain** is a toolchain that is missing one or more of these libraries. -A **compiler-only toolchain** only consists of compilers (no additional libraries). - -### System toolchain - -The **`system` toolchain** is a special case which corresponds to using the compilers and libraries -*provided by the operating system*, rather than using toolchain components that were installed using EasyBuild. - -It used sparingly, mostly to install software where no actual compilation is done or -to build a set of toolchain compilers and its dependencies, since the versions of the system compilers -and libraries are beyond the control of EasyBuild, which could affect the reproducibility of the installation. - -### Common toolchains - -The `foss` and `intel` toolchains are also known as the *common toolchains*, -because they are widely adopted by the EasyBuild community. - -The `foss` toolchain consists of all open source components (hence the name: -"FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), -[ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). - -The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version -controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. - -Roughly every 6 months, a new version of these common toolchains is agreed upon -in the EasyBuild community, after extensive testing. - -More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). - ---- - -## Modules - -*Module* is a massively overloaded term in (scientific) software and IT in general -(kernel modules, Python modules, and so on). -In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. - -[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept -on HPC systems: it is a way to specify changes that should be made to one or more -[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a -[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file -is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or -[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, -and specifies which environment variables should be updated, and how (append, -prepend, (re)define, undefine, etc.) upon loading the environment module. -Unloading the environment module will restore the shell environment to its previous state. - -Environment module files are processed via a **modules tool**, of which there -are several conceptually similar yet slightly different implementations. -The Tcl-based [Environment Modules](https://sourceforge.net/projects/modules/) implementation, and -[Lmod](https://lmod.readthedocs.io), a more recent Lua-based implementation (which also supports module -files written in Tcl syntax), are the most commonly used ones. - -**Environment module files are automatically generated for each software installation** by EasyBuild, -and *loading* a module results in changes being made to the environment of the current shell -session such that the corresponding software installation can be used. +EasyBuild currently doesn't treat link-time dependencies differently from run-time dependencies +and hence has no specific easyconfig parameter for them. --- @@ -378,4 +291,4 @@ An **easystack file** can be used to specify a collection of software to install --- -*[[next: Installation]](installation.md)* +*[[next: Installation]](1_05_installation.md)* diff --git a/docs/2022-CSC_and_LO/1_05_installation.md b/docs/2022-CSC_and_LO/1_05_installation.md new file mode 100644 index 00000000..163262ea --- /dev/null +++ b/docs/2022-CSC_and_LO/1_05_installation.md @@ -0,0 +1,543 @@ +# Installing EasyBuild + +*[[back: Terminology]](1_04_terminology.md)* + +--- + +Before deep diving into the wonderful world of EasyBuild and getting your hands dirty with +the hands on exercises coming up in this tutorial, you will need to install EasyBuild. + +In this section we outline a couple of different ways of doing this, and also the things you should pay attention to. +By the end, you will have a functional EasyBuild installation that you can use +for the remainder of this tutorial. + +!!! Remark "EasyBuild on LUMI" + + EasyBuild is already installed for each of the LUMI software stacks. The LUMI software + stacks are based on the Cray PE (using the Cray-specific toolchains). + Moreover, in case you want to install a full shadow stack rather than build on top + of one of the LUMI software stacks, scripts are provided to initialise a new version + of the LUMI software stack based on a specific version of the Cray PE, and those + scripts also take care of the installation of EasyBuild. + + Hence this section of the tutorial is only useful for people who want more background + knowledge or who want to experiment with other EasyBuild toolchains. + + +!!! summary + + * *Requirements*: Linux, Python 2.7 or 3.5+, environment modules tool (Lmod is recommended). + * *Installation methods*: + * ``pip install easybuild`` + * installing EasyBuild with EasyBuild + * development setup + * Verify EasyBuild installation using `eb --version` and `eb --help`. + + +--- + +## Requirements + +### Linux + +The main target platform for EasyBuild is *Linux*, since it is heavily focused on installing software +on HPC system where Linux is dominant operating system (to the point where 100% of the current Top500 list of supercomputers are running Linux). + +EasyBuild is also compatible with *macOS*, but the included easyconfig files are heavily focused +on Linux so most software installations supported by EasyBuild won't work out-of-the-box on macOS. +You can still use the EasyBuild command line interface on macOS for other tasks though, +like development or contributing back to the project. + +### Python + +EasyBuild is implemented in Python, and is **compatible with both Python 2.7 and Python 3.5+** +(that is, Python 3.5 or a newer version of Python 3). + +To check which Python version you have, use: + +```shell +python -V +``` + +*No additional Python packages are required by EasyBuild*, the ones that come with the standard +Python distribution are sufficient. Some additional Python packages *can* be leveraged for specific features. +More on that later. + +### Environment modules tool + +**An *environment modules tool* is required for using EasyBuild.** + +We strongly recommend using [Lmod](https://lmod.readthedocs.io), a Lua-based modern environment +modules implementation and the most commonly used modules tool in the EasyBuild community. +Other implementations, like the original Tcl-based one, are also supported. + +To check if you have a modules tool installed, use: + +``` +module --version +``` + +If this produces output that starts with something like "`Modules based on Lua: Version 8.3.1`" you have Lmod installed, +which is the default modules tool used by EasyBuild, and you are all set for installing and using EasyBuild. +Any sufficiently recent Lmod version (8.x or even 7.x) should be fine. + +If you see output that starts with a line like "`VERSION=3.2.10`" or "`Modules Release 4.5.0`", +you have the original Tcl-based environment modules tool installed, either in its original +C-based imnplementation (``VERSION=3.2.10``) or the new Tcl-based implementation developed at CEA +(``Modules Release 4.5.0``) +and *EasyBuild will need to be configured to use it after installation*. + +If the `module` function is not defined either you do not have a modules tool installed +or your environment is not properly set up to use it. In this case, +please refer to the EasyBuild documentation here for more information. + +HPE Cray supports both the old-style Environments Modules (based on version 3.2.10) and Lmod. On LUMI, +we use Lmod as the default module system for users, and EasyBuild is configured to use that one. + +--- + +## EasyBuild as a Python package + +EasyBuild consists of a number of interdependent Python packages, +and is available via both GitHub at https://github.com/easybuilders, +as well as via the standard Python Package Index (PyPI) at +https://pypi.org/project/easybuild. + +As you may be aware the Python packaging ecosystem is bit convoluted, +which is reflected in the many different ways in which you can install a Python package. +In addition, EasyBuild is packaged in 3 components (framework, easyblocks, easyconfigs) +which slightly complicates the installation. + +
+ +Nevertheless, you don't need to be a rocket scientist to install EasyBuild (and even if you are, +that's OK too), so don't worry. + +You can install EasyBuild just like you can install any other Python software that is released +via the standard *Python Package Index* (PyPI), through one of the standard Python installation tools +(like `pip`, `virtualenv`, `pipenv`, ...). +And since EasyBuild is a software installation tool in its own right, we actually have a couple +of additional tricks up our sleeve! + +### Python 2 or Python 3? + +For EasyBuild it does not matter much whether you install it on top of Python 2 or Python 3. +Since version 4.5, Python 3 does offer some optional features (requiring additional packages) +that are not available with Python 2. +Since Python 2 is end-of-life, +we strongly recommend using Python 3 if you have the choice. + +By default EasyBuild will use the `python` command to run, +but you can control this if needed via ``$EB_PYTHON``. + + +## Installing EasyBuild + +We present three methods for installing EasyBuild. +It is up to you which one you prefer, both result a fully functional EasyBuild installation. + +Time to get your hands dirty! + +* Method 1: Using ``pip`` +* Method 2: installing EasyBuild with EasyBuild +* Method 3: Development setup + +--- + +### Method 1: Using `pip` + +Since EasyBuild is released as a [Python package on PyPI](https://pypi.org/project/easybuild) +you can install it using `pip`, the most commonly used tool for installing Python packages. + +You may need to take additional steps after the installation, depending on the exact installation command. + +!!! note + There are various other ways of installing Python packages, which we won't cover here. + If you are familiar with other tools like `virtualenv` or `pipenv`, feel free to use those + instead to install EasyBuild. + +#### Running `pip install` + +Installing EasyBuild with `pip` is as simple as running the following command: + +```shell +pip install easybuild +``` + +However, you may need to slightly change this command depending on the context and your personal preferences: + +* To install EasyBuild *system-wide*, you can use `sudo` (if you have admin privileges): + ```shell + sudo pip install easybuild + ``` + +* To install EasyBuild *in your personal home directory*, you can use the `--user` option: + ```shell + pip install --user easybuild + ``` + This will result in an EasyBuild installation in `$HOME/.local/`. + +* To install EasyBuild in a *specific directory* you can use the `--prefix` option: + ```shell + pip install --prefix _PREFIX_ easybuild + ``` + In this command, you should replace '`_PREFIX_`' with the location where you want to have EasyBuild installed + (for example, `$HOME/tools` or `/tmp/$USER`). + +#### `pip` vs `pip3` + +On systems where both Python 2 and Python 3 are installed you may also have different `pip` commands +available. Or maybe `pip` is not available at all, and only "versioned" `pip` commands like `pip3` are +available. + +If you (only) have `pip3` available, you can replace `pip` with `pip3` in any of the `pip install` commands +above. + +If you want to ensure that you are using the ``pip`` installation that corresponds to the Python 3 installation +that you intend to use, you can use ``python3 -m pip`` rather than ``pip3``. + +#### Updating your environment + +If you used the `--user` or `--prefix` option in the `pip install` command, +or if you installed EasyBuild with a `pip` version that does not correspond +to your default Python installation, +you will need to update your environment to make EasyBuild ready for use. +This is not required if you did a system-wide installation in a standard location with the default Python version. + +!!! note + Keep in mind that you will have to make these environment changes again if you start a new shell session. + To avoid this, you can update one of the shell startup scripts in your home directory (`.bashrc` for example). + +#### Updating ``$PATH`` + +Update the `$PATH` environment variable to make sure the `eb` command is available: +```shell + +export PATH=_PREFIX_/bin:$PATH +``` +**Replace '`_PREFIX_`' in this command** with the directory path where EasyBuild was installed into +(use `$HOME/.local` if you used `pip install --user`). + +This is not required if you installing EasyBuild in a standard system location. + +You can check with the `which eb` command to determine whether or not you need to update the ``$PATH`` environment variable. + +#### Updating ``$PYTHONPATH`` + +If you installed EasyBuild to a non-standard location using `pip install --prefix`, +you also need to update the Python search path environment variable +[`$PYTHONPATH`](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH) to instruct Python where +it can find the EasyBuild Python packages. + +This is not required if you used the `--user` option, since Python will automatically consider +`$HOME/.local` when searching for installed Python packages, or if you installed EasyBuild in a standard +system-wide location. + +Update `$PYTHONPATH` by running a command like: + +```shell + +export PYTHONPATH=_PREFIX_/lib/pythonX.Y/site-packages:$PYTHONPATH +``` + +Here, you need to replace the `X` and `Y` with the major and minor version of your Python installation, +which you can determine by running `python -V`. +For example, if you are using Python 3.6, make sure you are using `/python3.6/` in the command to update `$PYTHONPATH`. + +And of course, you again need to **replace '`_PREFIX_`'** with the installation prefix where EasyBuild was installed +into. + +For example: + +```shell +# update $PYTHONPATH if EasyBuild was installed in $HOME/tools with Python 3.6 +export PYTHONPATH=$HOME/tools/lib/python3.6/site-packages:$PYTHONPATH +``` + +#### Setting ``$EB_PYTHON`` + +If you want to control which Python version is used to run EasyBuild, +you can specify the name or the full path to the `python` command that should be used by the `eb` command +via the `$EB_PYTHON` environment variable. + +This may be required when you installing EasyBuild with a version of `pip` that does not correspond +with the default Python version. + +For example, to ensure that `eb` uses `python3.6`: + +```shell +export EB_PYTHON=python3.6 +``` + +#### Setting ``$EB_VERBOSE`` + +To determine which `python` commands are being considered by the `eb` command, +you can define the `$EB_VERBOSE` environment variable. For example: + +```shell +$ EB_VERBOSE=1 eb --version +>> Considering 'python3.6'... +>> 'python3' version: 3.6.8, which matches Python 3 version requirement (>= 3.5) +>> Selected Python command: python3 (/usr/bin/python3.6) +>> python3.6 -m easybuild.main --version +This is EasyBuild 4.3.3 (framework: 4.3.3, easyblocks: 4.3.3) on host example +``` + +!!! Tip "Debugging startup problems" + + Using `ÈB_VERBOSE` is useful if EasyBuild fails to start up and complains it + cannot find a suitable Python executable. Rather often the error message is + caused by a failure elsewhere in EasyBuild. + + +--- + +### Method 2: Installing EasyBuild with EasyBuild + +!!! note + This section covers an alternative method for installing EasyBuild. + + If you already have EasyBuild installed through ``pip``, + you can skip ahead to the next section. + +If you prefer having EasyBuild available through an environment module file, +you can consider installing EasyBuild with EasyBuild. This can be done in 3 steps: + +* Step 1: Installing EasyBuild with ``pip`` into a temporary location (only needed if EasyBuild is not installed yet) +* Step 2: Using EasyBuild to install EasyBuild as a module +* Step 3: Loading the EasyBuild module + +!!! note + A [bootstrap script](https://docs.easybuild.io/en/latest/Installation.html#bootstrapping-easybuild) + is available that automates this procedure, + but is known to be problematic in some contexts, and is not being actively + maintained anymore. + + As a result, we do not recommend using the bootstrap script anymore. + + +#### Step 1: Installing EasyBuild into a temporary location + +If you don't have EasyBuild installed yet, you need to install it in a temporary location first. +The recommended way of doing this is using ``pip``. + +For example, to install EasyBuild into a subdirectory `/tmp/$USER` using the default Python 3 version: + +```shell +# pick installation prefix, and install EasyBuild into it +export EB_TMPDIR=/tmp/$USER/eb_tmp +python3 -m pip install --ignore-installed --prefix $EB_TMPDIR easybuild +``` + +```shell +# update environment to use this temporary EasyBuild installation +export PATH=$EB_TMPDIR/bin:$PATH +export PYTHONPATH=$(/bin/ls -rtd -1 $EB_TMPDIR/lib*/python*/site-packages | tail -1):$PYTHONPATH +export EB_PYTHON=python3 +``` + +#### Step 2: Using EasyBuild to install EasyBuild + +Once you have a working (recent) temporary EasyBuild installation, you can use it to +install EasyBuild as a module. Usually this is done in the location where you would +like to install other software too. + +You can use the ``eb --install-latest-eb-release`` command for this, +combined with the ``--prefix`` option to control which directories are used by EasyBuild for the installation. + +For example, to install the latest version of EasyBuild as a module into ``$HOME/easybuild``: + +```shell +eb --install-latest-eb-release --prefix $HOME/easybuild +``` + +!!! note + You may see a harmless deprecation warning popping up when performing this installation, just ignore it. + +#### Step 3: Loading the EasyBuild module + +Once step 2 is completed, you should be able to load the module that was generated alongside +the EasyBuild installation. You will need to do this every time you start a new shell session. + +First, make the module available by running the following command (which will update the module search path +environment variable `$MODULEPATH`): + +```shell + +module use _PREFIX_/modules/all +``` + +**Replace '`_PREFIX_`'** with the path to the directory that you used when running step 2 +(for example, ``$HOME/easybuild``). + +Then, load the `EasyBuild` module to update your environment and make EasyBuild available for use: + +```shell + +module load EasyBuild +``` + +Note that in this case, we don't need to make any changes to our environment for EasyBuild to work correctly. +The environment module file that was generated by EasyBuild specifies all changes that need to be made. + +### Method 3: Development setup + +If you are planning to make changes to EasyBuild, or if you prefer using the latest *bleeding edge* +version of EasyBuild that is being developed, you can consider *cloning* the 3 main EasyBuild repositories +from GitHub, and updating your environment to run EasyBuild from there. + +This can be done as follows (into ``$HOME/easybuild``): + +```shell +mkdir -p $HOME/easybuild +cd $HOME/easybuild +``` + +```shell +# clone EasyBuild repositories from GitHub +git clone https://github.com/easybuilders/easybuild-framework.git +git clone https://github.com/easybuilders/easybuild-easyblocks.git +git clone https://github.com/easybuilders/easybuild-easyconfigs.git +``` +```shell +# update environment for running EasyBuild from there +export PATH=$HOME/easybuild/easybuild-framework:$PATH +export PYTHONPATH=$HOME/easybuild/easybuild-framework:$HOME/easybuild/easybuild-easyblocks:$HOME/easybuild/easybuild-easyconfigs:$PYTHONPATH + +# control which Python command is used to run EasyBuild +export EB_PYTHON=python3 +``` + + +### Approach on LUMI + +*Documentation on the inner workings of the LUMI software stack can be found in the +[LUMI-SoftwareStack GitHub, docs subdirectory](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/docs).* + +To keep the different versions of the LUMI software stack as independent from one another as possible, +EasyBuild is bootstrapped for each software stack by the ``prepare_LUMI_stack.sh`` script. We use the +system Python 3 for running EasyBuild. That Python version currently doesn't have ``pip`` installed. +EasyBuild is first installed in a temporary work directory from files downloaded from PyPi. These +are each untarred and then installed into their location by running +``python3 setup.py install --prefix=...`` in their unpacked directory. +We do so only for the framework and easyblocks files as the easconfig files are not used to +install EasyBuild. Instead we create our own EasyConfig file for EasyBuild which contains +some additional packages that enable extra features in EasyBuild and also provide more information +to Lmod. Next the configuration module for EasyBuild (see the next section, +["Configuring EasyBuild"](1_06_configuring.md), for more information) and use the temporary +installation of EasyBuild with our own EasyConfig file to do a proper installation of EasyBuild +with module in the final location. + + +TODO: follow development setup but with downloading rather than cloning, then use that one with the +EasyBuild configuration module to install EasyBuild with itself. + + + +## Verifying the installation + +Regardless of how EasyBuild was installed, you can now run a couple of basic commands to verify the installation: + +#### Checking the version + +To check which EasyBuild version you have installed, run: + +```shell + +eb --version +``` + +The output should match with the latest EasyBuild version. + + +#### Consulting the help output + +You can consult the help output of the `eb` command, which produces a long list of available options +along with a short informative message. + +```shell +eb --help +``` + +#### Showing the default EasyBuild configuration + +To inspect the current EasyBuild configuration, you can use this command: + +```shell +eb --show-config +``` + +This should tell you that EasyBuild (ab)uses `$HOME/.local/easybuild` as a default location. +More on configuring EasyBuild in the [next part of the tutorial](configuration.md). + +#### System information + +You ask EasyBuild to collect and print some information about the +system you are using it on (OS, CPU, Python, etc.) using this command: + +```shell +eb --show-system-info +``` + +## Updating EasyBuild + +Before we wrap up here, a brief word about updating EasyBuild. + +Once you have EasyBuild installed, the easiest way to update to a newer version is by instructing EasyBuild +to install the latest available version as a module: + +``` + +eb --install-latest-eb-release +``` + +This will result in a *new* EasyBuild installation, which is entirely separate from the EasyBuild installation +you are currently using (so it is *not* an in-place update). +The location where this new EasyBuild version will be installed is determined by the active +EasyBuild configuration. + +If you have installed EasyBuild through ``pip``, and you prefer updating that installation, +you can use ``pip install --upgrade easybuild`` (perhaps with additional options like ``--user`` or ``--prefix``). + +!!! Note "EasyBuild upgrade policy on LUMI" + + Even minor version or patch level updates of EasyBuild may break things (and this has happned + in the past on other systems managed previously by LUMI User Support Team members). Hence a very + conservative upgrade policy is used on LUMI. + + In general we fix the version of EasyBuild within a particular version of the LUMI software stack + and only transition to the latest version when starting a new software stack. The reason is that we + want to be able to rebuild an existing software stack with as little unexpected problems as possible. + A move to a newer version of EasyBuild is acceptable should we run into major problems with an existing + version that cannot be solved in an easyconfig file or LUMI-specific custom easyblock, but in those + cases we will first rebuild the whole software stack on a test setup to ensure that no problems are + introduced by upgrading EasyBuild. + + +--- + +## Exercise (optional) + +*This exercise is not needed to proceed the course when working on LUMI as we will +use the installation that is already present.* + +Install EasyBuild in your home directory. + +Make sure that the EasyBuild installation uses the `python3` command to run, +rather than the standard `python` command. + +Choose your own adventure, or try all these installation methods! + +* install EasyBuild with `pip` (or another very similar command...) using either the `--user` or `--prefix` option; +* perform a (manual) "bootstrap" installation into `$HOME/easybuild`, as outlined in installation method 2; +* set up a development installation; + +Check that the installation works by running the verification commands outlined +[above](#verifying-the-installation). + +--- + +**Make sure you have a working EasyBuild installation before proceeding +with the rest of the tutorial!** + +*[[next: Configuring EasyBuild]](1_06_configuration.md)* diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md index 43fb4518..b45a740b 100644 --- a/docs/2022-CSC_and_LO/index.md +++ b/docs/2022-CSC_and_LO/index.md @@ -9,11 +9,7 @@ See that site for [credits to the contributors of those tutorials](https://easyb This is an introductory tutorial to [EasyBuild](https://easybuild.io), a command line tool for installing (scientific) software on High Performance Computing (HPC) systems. -<<<<<<< HEAD This tutorial is specifically for EasyBuild as implemented on [LUMI](https://lumi-supercomputer.eu) -======= -These tutorials are specifically for EasyBuild as implemented on [LUMI](https://lumi-supercomputer.eu) ->>>>>>> 51e451f1 (Continued merging of the new CSC tutorial in the structure of the repository.) and has been developed for CSC and the Local Organisations within the LUMI consortium. Yet much of the material is useful to a broader community of EasyBuild users on Cray systems or even EasyBuild users in general. diff --git a/mkdocs.yml b/mkdocs.yml index 29b837cb..6dda1ac8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -126,6 +126,7 @@ nav: - The Lmod module system: 2022-CSC_and_LO/1_02_Lmod.md - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_03_CPE.md - Terminology: 2022-CSC_and_LO/1_04_terminology.md + - Installation: 2022-CSC_and_LO/1_05_installation.md plugins: # show revision date at bottom of each page - git-revision-date-localized From 6338bb1a83d9fae4220b7c3a9db5ee07831b1faf Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 17:31:43 +0100 Subject: [PATCH 100/159] Tutorial on EasyBuild configuration added, and some restructuring. --- .../1_04_LUMI_software_stack.md | 17 + ..._04_terminology.md => 1_05_terminology.md} | 0 ...5_installation.md => 1_06_installation.md} | 0 docs/2022-CSC_and_LO/1_07_configuration.md | 610 +++++++++++ docs/2022-CSC_and_LO/1_08_basic_usage.md | 976 ++++++++++++++++++ 5 files changed, 1603 insertions(+) create mode 100644 docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md rename docs/2022-CSC_and_LO/{1_04_terminology.md => 1_05_terminology.md} (100%) rename docs/2022-CSC_and_LO/{1_05_installation.md => 1_06_installation.md} (100%) create mode 100644 docs/2022-CSC_and_LO/1_07_configuration.md create mode 100644 docs/2022-CSC_and_LO/1_08_basic_usage.md diff --git a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md new file mode 100644 index 00000000..0e578a59 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md @@ -0,0 +1,17 @@ +# LUMI software stacks + +*[[back: The Cray Programming Environment]](1_03_CPE.md)* + +--- + + + + + + + + +--- + +*[[next: Terminology]](1_05_terminology.md)* + diff --git a/docs/2022-CSC_and_LO/1_04_terminology.md b/docs/2022-CSC_and_LO/1_05_terminology.md similarity index 100% rename from docs/2022-CSC_and_LO/1_04_terminology.md rename to docs/2022-CSC_and_LO/1_05_terminology.md diff --git a/docs/2022-CSC_and_LO/1_05_installation.md b/docs/2022-CSC_and_LO/1_06_installation.md similarity index 100% rename from docs/2022-CSC_and_LO/1_05_installation.md rename to docs/2022-CSC_and_LO/1_06_installation.md diff --git a/docs/2022-CSC_and_LO/1_07_configuration.md b/docs/2022-CSC_and_LO/1_07_configuration.md new file mode 100644 index 00000000..17e60afe --- /dev/null +++ b/docs/2022-CSC_and_LO/1_07_configuration.md @@ -0,0 +1,610 @@ +# Configuring EasyBuild + +*[[back: Installation]](1_06_installation.md)* + +--- + +After installing EasyBuild, you should **configure** it. + +EasyBuild should work fine out-of-the-box if you use Lmod as your modules tool. +If you are not using Lmod, please see here for more information. + +Nevertheless, we strongly recommend you to inspect the default configuration, +and to configure EasyBuild according to your preferences and the system on which you will use it. + +!!! Note "EasyBuild configuration on LUMI" + + On LUMI serveral configurations of EasyBuild are already available. + + The ``EasyBuild-user`` module is the most important one. It will configure EasyBuild + to install software in either a default location in the user's home directory + (``$HOME/EasyBuild``) or the location pointed to by the environment variable + ``EBU_USER_PREFIX``. The latter can be used to install software in the project directory + so that it is available to all users of the project. It enables the creation of a custom + software environment build on top of the LUMI software stack. Software in this environment is + build in exactly the same way as it would if it were installed in the central repository, but + one can now also easily create custom configurations without confusing other projects with + different or non-standard configurations of software. + + There are two more admin-only configuration modules for EasyBuild. The main one is + ``EasyBuild-production`` which is used for software installations in the central software stack. + The other one is ``EasyBuild-infrastructure`` which is used to install infrastructure modules + that need to be installed in a pure Lmod hierarchy, e.g., the modules that load the toolchains. + + +--- + +## Available configuration settings + +One of the central policies in the EasyBuild project is to avoid hardcoded settings in the codebase. +While this significantly increases the ability to configure EasyBuild to your liking, it also results +in a large amount of available configuration settings. + +The full list of configuration settings can be consulted via `eb --help`, +which shows the corresponding command line option accompanied by a short description. +At the time of writing, *over 240 different configuration settings* are supported by EasyBuild. + +For the sake of this tutorial we will focus on a specific subset of configuration settings, +and cover only the most prominent and important ones. + +We will refer to EasyBuild configuration settings using the names as they appears in the output of `eb --help`, +and omit the leading dashes (`--`) for the sake of clarity. + +Keep in mind that *every* configuration setting can be defined in 3 different ways, +see below for more details. + +--- + +### Overall prefix + +*(default: `HOME/.local/easybuild`)* + +The `prefix` configuration setting specifies the overall prefix that EasyBuild should use, +which **determines the default value for various other configuration settings**: + +* `installpath`: `` +* `buildpath`: `/build` +* `sourcepath`: `/sources` +* `repositorypath` (easyconfigs archive): `/ebfiles_repo` +* `containerpath`: `/containers` + +Here, `` represents the value of the `prefix` configuration setting. + +If one of the configuration settings affected by `prefix` is defined specifically, +the `prefix` value becomes irrelevant for that specific configuration setting. + +--- + +### Install path + +*(default: ``)* + +The **location for both the software installation directories and generated module files** +can be controlled via the `installpath` configuration setting. +Software installation directories will be placed in `/software`, while +`/modules/all` will be used for generated module files. + +The `installpath` location is usually set to a directory on a *shared filesystem* when installing +software for an HPC cluster. Of course, software can also be installed on a local filesystem, +which is particularly useful to test and evaluate software installations. + +Separate configuration settings are available for both software and modules locations, +as well as for controlling the name of the `software` and `modules/all` subdirectories. + +*The EasyBuild community recommends to only change the `installpath` configuration setting to control the location +of software installations and accompanying module files, +such that the software and modules directories are located in the same parent directory, +and the default `software` and `modules/all` names for the subdirectories are used.* + +!!! Note "Path for software and for modules on LUMI" + + On LUMI we do not follow that recommendation. There is a three for the software installations + themselves with subdirectories based on the version of the software stack and LUMI hardware + partition, and a separate tree for the modules organised in a similar way. + + This makes it slightly easier to organise the module tree with user-friendly labeling, but above + all also makes the synchronisation process of the 4 instances of the software directory more robust + as it is now easy to synchonise all modules in the last step, which is a much quicker process than + syncrhonising the software installations. + + We also use short paths for software installations (to avoid overrunning the maximum length of a + shebang line in scripts) while we use longer, more descriptive names for subdirectories in the + module tree. + + +--- + +### Build path + +*(default: `/build`)* + +For each installation it performs, EasyBuild creates a **separate build directory** where software will be compiled +before installing it. This directory is cleaned up automatically when the installation is successfully completed. +To control the location where these build directories are created, you can use the `buildpath` configuration setting. + +Keep in mind that build directories may grow out to several GBs in size during an installation, +and that the commands that run in there can be fairly I/O-intensive since they may involve +manipulating lots of small files. In addition, a build directory that corresponds to a failing installation +is *not* cleaned up automatically, but it will be cleaned up and recycled when the same installation is re-attempted. +Running out of disk space in the location where build directories are created will result in failing +installations. + +It is strongly recommend to use the path to a directory on a *local filesystem* for the value of the +`buildpath` configuration setting, since using a shared filesystem like Lustre or GPFS is known to cause +problems when building certain software packages. Using an in-memory location (like `/dev/shm/$USER`) can +significantly speed up the build process, but may also lead to problems (due to space limitations, +or specific mount options like `noexec`). + +!!! Note "buildpath on LUMI" + + The configuration modules on LUMI will use a RAM disk for the build path. On the login nodes, + ``$XDG_RUNTIME_DIR`` is used as that space is automatically cleared when the last session of a user + ends. However, on the compute nodes a job- or user-specific subdirectory of ``/dev/shm`` is currently used + as ``$XDG_RUNTIME_DIR`` does not exist. + + +--- + +### Source path + +*(default: `/sources`)* + +For most supported software, EasyBuild can **automatically download the source files** required for the installation. +Before trying to download a source file, EasyBuild will first check if it is already present in the source path. + +The locations considered by EasyBuild when checking for available source files, as well as the location to +store downloaded source files, can be controlled via the ``sourcepath`` configuration setting. + +The `sourcepath` value is a colon (`:`) separated list of directory paths. +Each of these paths will be considered in turn when checking for available source files, +until one of them provides the desired source file. Searching for source files is done +based on filename, and a couple of subdirectories are considered. +For example, for a software package named '`Example'`, EasyBuild will consider locations +like `/e/Example/`, `/Example/`, and so on. + +The first path listed in `sourcepath` is the location where EasyBuild will store downloaded source files, +organised by software name through subdirectories, so EasyBuild expects to have write permissions to this path. +For the other paths listed in `sourcepath` only read permissions are required. + +*Make sure you have write permissions to the first path listed in `sourcepath`, so EasyBuild is able +to store downloaded files there. Feel free to list additional paths if you already have a cache of downloaded +files available somewhere.* + +*Storing the downloaded files not only reduces the amount of downloads while developing new easyconfig files +but also greatly helps when recompiling a software stack, as it is not that uncommon that download sites change +of files become unavailable.* + + +--- + +### Easyconfigs archive + +*(default: `/ebfiles_repo`)* + +EasyBuild keeps track of the easyconfig files that were used for installations in the easyconfigs +archive, the location of which is specified by the `repositorypath` configuration setting. + +By default the specified path is assumed to be a regular directory, but using a Git repository +as easyconfigs archive is also supported (for more details, see +[the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Configuration.html#easyconfigs-repository-repository-repositorypath)). + + +--- + +### Modules tool & module syntax + +*(default: `Lmod` as modules tool, and `Lua` as module syntax)* + +By default, EasyBuild assumes you are using **[Lmod](https://lmod.readthedocs.io) as modules tool**. +In addition, it will generate module files in [Lua](https://www.lua.org/) syntax, as supported by Lmod +(next to `Tcl` syntax). + +To diverge from this, you can define the `modules-tool` configuration setting to indicate you +are using a different modules tool; see the output of `eb --avail-modules-tools` for a list of supported +modules tools. +Note that for anything other than Lmod, you *must* make +sure that the actual modules tool binary command is available through `$PATH` (more information +on this [in the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Configuration.html#modules-tool-modules-tool)). + +If you prefer that EasyBuild generates module files in [Tcl](https://tcl.tk/) syntax, you can instruct +it to do so via the `module-syntax` configuration setting. Note that this is *required* if you are using +a modules tool other than Lmod, since only Lmod supports module files in `Lua` syntax. + +*We recommend using EasyBuild with the defaults settings: Lmod as modules tool, and Lua as module syntax.* + +--- + +### Robot search path + +*(default: `robot-paths` specifies the location to the easyconfig files included with EasyBuild installation; +dependency resolution is* ***not*** *enabled)* + +When EasyBuild needs to locate one or more easyconfig files, it will do so via the **robot search path**. +This applies to both easyconfig files that were specified using only their filename as an argument to the +`eb` command, as well as to easyconfigs required to resolve dependencies (more on that later). + +To control the robot search path, two configuration settings are available: `robot` and `robot-paths`. +Both accept a colon-separated list of locations to consider when looking for easyconfig files, +with `robot` having a higher priority than `robot-paths` (i.e., the directories specified with `robot` +appear before those specified by `robot-paths` in the robot search path). + +The key difference between these two configuration settings is that defining `robot` also enables dependency resolution, +and hence makes EasyBuild install missing dependencies, alongside +specifying a list of paths to consider when searching for easyconfig files. +On the other hand, defining `robot-paths` does not have the side effect of enabling dependency resolution. + +In addition, you can use the `--robot` command line option without specifying any paths to it, +to only enable dependency resolution. ***This option is needed on LUMI if you want automatic +dependency resolution as it is disabled on LUMI to prevent accidental mistakes when installing +software.*** + +!!! note "Easyconfig files included with EasyBuild and the robot search path" + Keep in mind that when either of the `robot` or `robot-paths` configuration settings are + defined, the default value corresponding to the location of easyconfigs included with the + EasyBuild is *no longer considered*. + + There are ways around this however, which are outside of the scope of this tutorial. + + For more information, + [see the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Using_the_EasyBuild_command_line.html#controlling-robot-search-path). + + *On LUMI the included easyconfig files cannot be used without changes as they are for the + common toolchains which are currently not supported in the LUMI software stacks.* + + +!!! Note "Easyconfigs archive on LUMI and the robot search path" + + On LUMI we also include the easyconfigs archive at the front of the search path for easyconfig + files. This has the advantage that EasyBuild when checking dependencies will always find the + configuration that is actually installed on the system, even if the easyconfig file in one of + the regular repositories has changed. It also has the disadvantage that one may accidentally + be re-installing with the already installed configuration while one really wants to re-install + the module in a different configuration. + + + +--- + +### Module naming scheme + +*(default: `EasyBuildMNS`)* + +EasyBuild will use a specific naming scheme for the module files it generates for each of the software installations. +This ensures consistency, regardless of who uses EasyBuild to perform the installation. + +Different types of module naming schemes are supported (flat, hierarchical, ...) and you can provide an implementation of +your own custom module naming scheme if desired. A number of different naming schemes are included +with EasyBuild, which you can consult via `eb --avail-module-naming-schemes`. + +The default `EasyBuildMNS` module naming scheme roughly corresponds to the filename of easyconfig files, +and consists of the software name followed by a combination of the software version, toolchain and +an optional label (which corresponds to the value of the `versionsuffix` easyconfig parameter): +`/<-toolchain>`. Just like with [names of easyconfig +files](terminology.md#easyconfig-files), the `<-toolchain>` part is omitted when the `system` toolchain is used, +and the `` value is empty by default. + +!!! Note "Module naming scheme on LUMI" + + On LUMI we use a custom variant of the standard EasyBuild flat naming scheme. The EasyBuild default + naming scheme creates an ``all`` subdirectory in the module directory containing all modules, but also + a directory per category, with the categories determined from the ``moduleclass`` easyconfig parameter. + However, that choice is often rather arbitrary as modules can belong to different categories depending + on who you ask. E.g., GROMACS is used in physics, chemistry and life sciences while EasyBuild can only + assign a single category. + + Hence we do not generate the categories and drop the ``all`` level in the module subdirectory. + + +--- + +## Configuration levels + +Configuring EasyBuild can be done in different ways: + +* through one or more configuration files; +* via `$EASYBUILD_*` environment variables; +* using `eb` command line options; + +Each of the methods corresponds to a *configuration level*. + +*Every* configuration setting can be defined via one of these +mechanisms, without exception! + +--- + +### Configuration level hierarchy + +There is a strict **hierarchy between the different configuration levels** supported by EasyBuild. + +Settings defined via a *configuration file* only override default values. + +A configuration setting that is defined via the corresponding *`$EASYBUILD_*` environment variable* +takes precedence over the value specified in a configuration file (if any). + +Finally, values specified through `eb` command line options **always** win, +regardless of whether the corresponding +configuration setting was already defined some other way, be it via a configuration file or +an environment variable. + +For example, let us consider a fictional configuration setting named `magic`: + +* If a value for `magic` is specified in an EasyBuild configuration file, + then this value will only be used if the corresponding environment variable + (`$EASYBUILD_MAGIC`) is *not* defined **and** if the `--magic` command line option is *not* used; +* If the `$EASYBUILD_MAGIC` environment *is* defined however, then its value + will be used for the `this-is-magic` configuration setting; +* *Unless* the `--magic` command line option is used, + since in that case the value provided as an argument there will be used instead. + +--- + +### Configuration files + +*Configuration files* are the most basic way of configuring EasyBuild. +Two types of are supported by EasyBuild: *user-level* and *system-level* +configuration files. The output of `eb --show-default-configfiles` tells you which locations +are considered, and whether any configuration files were found. + +EasyBuild configuration files are written in the standard [INI format](https://en.wikipedia.org/wiki/INI_file), +and the configuration settings are grouped into different *sections*. + +To create an EasyBuild configuration file, the output of `eb --confighelp` is very useful: +it produces the list of all supported configuration settings which are readily grouped in sections +and with every entry commented out, along with accompanying descriptive comments mentioning the default values, +and in the correct syntax. + +*Configuration files are the recommended mechanism to define configuration settings +that should always be in place, regardless of the software you are installing.* + +#### EasyBuild configuration files vs easyconfig files + +EasyBuild configuration files are sometimes confused with easyconfig files, +due to the similar name. However, these are two entirely different concepts! + +EasyBuild configuration files (usually `*.cfg`) are a way of *configuring the general behaviour of EasyBuild* +across different software installations. They define *configuration settings*, +such as the location where software should be installed, or the syntax that should +be used when generating module files. + +An [easyconfig file](terminology.md#easyconfig-files) (`*.eb`) on the other hand *specifies the details for one particular software installation*. +It does this by defining a set of *easyconfig parameters*, which tell EasyBuild the name and version +of the software to install, which [toolchain](terminology.md#toolchains) and [easyblock](terminology.md#easyblocks) to use, etc. + +For each software installation performed by EasyBuild, there is a corresponding easyconfig file. +There typically are only a handful of configuration files used however, for example a system-level +configuration file, perhaps combined with a user-level one. Or there may be no configuration files involved +at all, since EasyBuild can also be configured through other mechanisms: environment variables and command line +options. + +--- + +### `$EASYBUILD_*` environment variables + +A particularly easy way to configure EasyBuild is through *environment variables*. + +At startup, EasyBuild will pick up any environment variable of which the name starts with '`EASYBUILD_`'. +For each of these, it will determine the corresponding configuration setting (or exit with an error if +none was found). + +Mapping the name of a configuration setting to the name of the corresponding environment variable is straightforward: +use capital letters, replace dashes (`-`) with underscores (`_`), and prefix with `EASYBUILD_`. + +For example: the `module-syntax` configuration setting can be specified by defining +the `$EASYBUILD_MODULE_SYNTAX` environment variable: + +```shell +export EASYBUILD_MODULE_SYNTAX=Tcl +``` + +*Configuring via environment variables is especially practical for controlling the EasyBuild configuration +in a more dynamic way. For example, you can implement a simple shell script that defines `$EASYBUILD_*` +environment variables based on the current context (user, hostname, other environment variables), and +configure EasyBuild through [sourcing](https://bash.cyberciti.biz/guide/Source_command) it.* + +!!! note + Keep in mind that environment variables are only defined for the shell session you are currently working in. + If you want to configure EasyBuild through environment variables in a more persistent way, + you can leverage one of the [shell startup scripts](https://bash.cyberciti.biz/guide/Startup_scripts) (for example `$HOME/.bash_profile` or `$HOME/.bashrc`). + +--- + +### `eb` command line options + +Finally, you can also configure EasyBuild by specifying one or options to the `eb` command. + +As mentioned earlier, the values for configuration settings defined this way override the value that +is specified through any other means. So if you want to be sure that a particular configuration setting +is defined the way you want it to be, you can use the corresponding command line option. + +There are various configuration settings for which it only makes sense to use the command line option. +An example of this is letting the `eb` command print the EasyBuild version (via `eb --version`). +Although you could configure EasyBuild to always print its version and then exit whenever the `eb` command is +run, that would not be very useful... + +*Command line options are typically used to define configuration settings that are only relevant to +that particular EasyBuild session. One example is doing a test installation into a temporary directory:* + +```shell +eb --installpath /tmp/$USER example.eb +``` + +## Inspecting the current configuration (`--show-config`) + +Given the large amount of available configuration settings in EasyBuild and the different configuration levels, +you can easily lose track of exactly how EasyBuild is configured. + +Through the `--show-config` command line option you can +easily inspect the currently active EasyBuild configuration. + +The output of `--show-config` includes a sorted list of all configuration settings that are defined to a +*non-default* value, +along with a couple of important ones that are always shown (like `buildpath`, `installpath`, `sourcepath`, and so on). +In addition, it also indicates at which configuration level each setting was defined, +so you can trace down *where* it was defined if needed. + +This is the output produced by `eb --show-config` for the default EasyBuild configuration, +where EasyBuild was installed via `pip install --user` (which results in the value shown for the +`robot-paths` configuration setting): + +```shell +# +# Current EasyBuild configuration +# (C: command line argument, D: default value, E: environment variable, F: configuration file) +# +buildpath (D) = /home/example/.local/easybuild/build +containerpath (D) = /home/example/.local/easybuild/containers +installpath (D) = /home/example/.local/easybuild +repositorypath (D) = /home/example/.local/easybuild/ebfiles_repo +robot-paths (D) = /home/example/.local/easybuild/easyconfigs +sourcepath (D) = /home/example/.local/easybuild/sources +``` + +As shown here, all configuration settings shown follow the default `prefix` value (`$HOME/.local/easybuild`), +and none of the values diverge from the default value, since all entries are marked with `(D)` for "default value"). + +**Example** + +Now let us do some basic configuring and inspect the resulting output of `--show-config`. + +First, create a user-level EasyBuild configuration file to define the `prefix` configuration setting: + +```shell +mkdir -p $HOME/.config/easybuild +echo '[config]' > $HOME/.config/easybuild/config.cfg +echo 'prefix=/apps' >> $HOME/.config/easybuild/config.cfg +``` + +In addition, define the `buildpath` configuration setting using the corresponding +environment variable: + +```shell +export EASYBUILD_BUILDPATH=/tmp/$USER +``` + +Then run `--show-config` while you specify that the `installpath` configuration +setting should be defined as `/tmp/$USER`: + +```shell +$ eb --installpath=/tmp/$USER --show-config +# +# Current EasyBuild configuration +# (C: command line argument, D: default value, E: environment variable, F: configuration file) +# +buildpath (E) = /tmp/easybuild +containerpath (F) = /apps/containers +installpath (C) = /tmp/easybuild +packagepath (F) = /apps/packages +prefix (F) = /apps +repositorypath (F) = /apps/ebfiles_repo +robot-paths (D) = /home/example/.local/easybuild/easyconfigs +sourcepath (F) = /apps/sources +``` + +The output indicates that the `installpath` setting was specified through a command line option (indicated +with `(C)`), that the `buildpath` setting was defined via an environment +variable (indicated with `(E)`), that the `robot-paths` setting still has the default value (indicated with `(D)`), and that all other configuration +settings were specified via a configuration file, some of which indirectly through the `prefix` value (indicated with +`(F)`). + +--- + +## Exercises (optional) + +*These exercises are not very relevant for LUMI as LUMI already offers a complete configuration +also for user installations of software. However, if you are a very advanced user, you may still +want to make changes to that configuration, and all three options (configuration files, +environment variables and command line parameters) are available to users in the LUMI setup.* + +***Exercise C.1* - Configure EasyBuild** + +Configure EasyBuild to use the `easybuild` subdirectory in your home directory for everything, except for: + +* The location of the build directories: use `/tmp/$USER` for this; +* The locations that should be considered when searching for source files: + include both `$HOME/easybuild/sources` and `/easybuild/sources`, but make + sure that source files that are downloaded by EasyBuild are stored in + `$HOME/easybuild/sources`. + +Leave other configuration settings set to their default value. + +??? success "(click to show solution)" + + This is pretty straightforward. + + Here we just define the corresponding environment variables: + + ```shell + export EASYBUILD_PREFIX=$HOME/easybuild + export EASYBUILD_BUILDPATH=/tmp/$USER + export EASYBUILD_SOURCEPATH=$HOME/easybuild/sources:/easybuild/sources + ``` + + The location where EasyBuild should download source files to + must be listed first in the `sourcepath` configuration setting. + + The output of `--show-config` should look like this: + + ```shell + buildpath (E) = /tmp/example + containerpath (E) = /home/example/easybuild/containers + installpath (E) = /home/example/easybuild + packagepath (E) = /home/example/easybuild/packages + prefix (E) = /home/example/easybuild + repositorypath (E) = /home/example/easybuild/ebfiles_repo + robot-paths (D) = /home/example/easybuild/easyconfigs + sourcepath (E) = /home/example/easybuild/sources:/easybuild/sources + ``` + +--- + +***Exercise C.2* - Install a trivial software package with EasyBuild** + +Try running the following command: + +```shell +eb bzip2-1.0.6.eb +``` + +Where do you expect to find the installation? + +??? success "(click to show solution)" + + The software was installed in `$HOME/easybuild`, + since that's how we configured EasyBuild in *Exercise 3.1*: + + ```shell + $ ls $HOME/easybuild + ebfiles_repo modules software sources + ``` + + The actual installation is in `$HOME/easybuild/software`, + while the module file was generated in `$HOME/easybuild/modules/all`: + + ```shell + $ ls $HOME/easybuild/software + bzip2 + $ ls $HOME/easybuild/software/bzip2 + 1.0.6 + $ ls $HOME/easybuild/software/bzip2/1.0.6 + bin easybuild include lib man + ``` + + ```shell + $ ls $HOME/easybuild/modules/all + bzip2 + $ ls $HOME/easybuild/modules/all/bzip2 + 1.0.6.lua + ``` + + The source file for bzip2 1.0.6 was downloaded to `$HOME/easybuild/sources`: + + ```shell + $ ls $HOME/easybuild/sources/b/bzip2 + bzip2-1.0.6.tar.gz + ``` + + We will discuss this in more detail in the next part of the tutorial. + +--- + +*[[next: Basic usage]](1_08_basic_usage.md)* diff --git a/docs/2022-CSC_and_LO/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_08_basic_usage.md new file mode 100644 index 00000000..5566d376 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_08_basic_usage.md @@ -0,0 +1,976 @@ +# Basic usage + +*[[back: Configuring EasyBuild]](1_07_configuration.md)* + +--- + +Now that we have installed and configured EasyBuild, we can start using it for what it is intended for: +getting scientific software installed without breaking a sweat, or having to resist the urge to +shout out four-letter words. + +We will look at the high-level workflow first, and then cover each aspect in more detail. + +A couple of exercises are available at the end to help to make you more familiar with the +EasyBuild command line interface, so pay attention! + +--- + +## Overall workflow + +Installing software with EasyBuild is as easy as specifying to the **`eb` command** what we +want to install, and then sitting back to enjoy a coffee or tea (or whatever beverage you prefer). + +This is typically done by **specifying the name of one or more easyconfig files**, often combined +with the `--robot` option to let EasyBuild also install missing dependencies. + +It is recommended to first assess the current situation before letting EasyBuild install the software, +and to check which **dependencies** are already installed and which are still missing. In addition, +you may want to inspect the specifics of the **installation procedure** that will be performed by EasyBuild, +and ensure that the active EasyBuild configuration is what it should be. + +--- + +## Specifying easyconfigs + +Letting EasyBuild know what should be installed can be done by specifying one or more easyconfig files, +which is also the most common way. Alternative methods like using the `--software-name` option won't be +covered in this tutorial, since they are not commonly used. + +Arguments passed to the `eb` command, being anything that is *not* an option (which starts with `-` or `--`) or +is a value for a preceding configuration option, are assumed to refer to easyconfig files (with some exceptions). +These could be: + +* the *(absolute or relative) path* to an easyconfig file; +* the *name* of an easyconfig file; +* the path to a *directory* containing easyconfig files; + +Specified paths to files must of course point to existing files; if not, EasyBuild will print an appropriate error message: + +```shell +$ eb /tmp/does_not_exist.eb +ERROR: Can't find path /tmp/does_not_exist.eb +``` + +When only the *name* of an easyconfig file is specified, EasyBuild will automatically try and locate it. +First, it will consider the *current directory*. If no file with the specified name is found there, +EasyBuild will search for the easyconfig file in the [robot search path](../configuration/#robot-search-path). + +If the path to an existing *directory* is provided, EasyBuild will walk through the entire directory +(including all subdirectories), retain all files of which the name ends with `.eb`, and (try to) use these +as easyconfig files. + + +#### Example command + +Suppose we have the current situation in our home directory: + +* two (easyconfig) files named `example1.eb` and `example2.eb`; +* a subdirectory named `some_deps`, which has two easyconfig files `dep1.eb` and `dep2.eb` + alongside a text file named `list.txt`; +* a subdirectory named `more_deps` located *in* the `some_deps` subdirectory, + which contains another easyconfig file `dep3.eb`; + +Or, visually represented: + +```shell +example1.eb +example2.eb +some_deps/ +|-- dep1.eb +|-- dep2.eb +|-- list.txt +|-- more_deps/ + |-- dep3.eb +``` + +In this context, we run the following EasyBuild command from our home directory: + +```shell +eb example1.eb bzip2-1.0.6.eb $HOME/example2.eb some_deps +``` + +EasyBuild will interpret each of these arguments as follows: + +* `example1.eb` is the name of a file in the current directory, so it can be used directly; +* `bzip2-1.0.6.eb` is the name of an easyconfig file to locate via the robot search path + (since it does not exist in the current directory); +* `$HOME/example2.eb` specifies the full path to an existing file, which can be used directly; +* `some_deps` is the relative path to an existing directory, so EasyBuild will scan it and find three + easyconfig files: `some_deps/dep1.eb`, `some_deps/dep2.eb` and `some_deps/more_deps/dep3.eb`, + The `list.txt` file will be ignored since its name does not end with `.eb`. + +--- + +### Easyconfig filenames + +Note that the `eb` command does not care how easyconfig files are named, at least to some extent: the `.eb` +file extension *does* matter when easyconfig files are being picked up in subdirectories. + +File names for easyconfigs being mostly irrelevant is only correct with respect to the arguments passed to +the `eb` command however. As we will learn soon, the name of easyconfig files *does* matter (a lot) +when EasyBuild needs to locate easyconfigs that can be used to resolve a specified dependency +(see [here](#enabling-dependency-resolution)). + +This explains why easyconfig files usually adher to a very specific naming scheme, +corresponding to `--.eb`, where: + +* `` represents the software name; +* `` represents the software version; +* `` represents the toolchain used in the easyconfig file, which consists of the toolchain name + and version separated with a dash (`-`), and which is omitted (including the preceding `-`) when the + [`system` toolchain](../introduction#system-toolchain) is used; +* `` represents the value of the `versionsuffix` easyconfig parameter, + which is sometimes used to distinguish multiple variants of particular software installations + (and is empty by default); + +--- + +### Searching for easyconfigs + +You will frequently need to determine the exact name of an easyconfig file you want to install, +or just check which easyconfigs are available for a given software package. +This can be done by searching for easyconfigs using **`eb --search`** or **`eb -S`**. + +By default all directories listed in the [robot search path](../configuration#robot-search-path) will be +searched. If you want to search in additional directories without changing the robot search path, +you can use the `search-paths` configuration setting, or you can change the robot search path via either +the `robot` or `robot-paths` configuration options. + +Both the `--search` and `-S` options trigger the same search operation, but yield different output: +`eb --search` will print the full path to each easyconfig file that matches the specified search pattern, +while `eb -S` produces a more concise output. + +For example, let's check which easyconfig files are available for OpenFOAM 8 with a `foss` toolchain: + +```shell +$ eb --search openfoam-8-foss + * /home/example/.local/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-8-foss-2020a.eb + * /home/example/.local/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-8-foss-2020b.eb +``` + +The output is a bit more condensed when using `eb -S`: + +```shell +$ eb -S openfoam-8-foss +CFGS1=/home/example/.local/easybuild/easyconfigs/o/OpenFOAM + * $CFGS1/OpenFOAM-8-foss-2020a.eb + * $CFGS1/OpenFOAM-8-foss-2020b.eb +``` + +Note that the search is performed *case-insensitive*. + +The search pattern can include wildcards like `.*` and/or character groups like `[0-9]`, +or other special characters like `^` or `$` to mark the start/end of the filename, +but you need to be careful that `bash` does not expand these before the `eb` command is started, +so it is recommended to wrap the search pattern in single quotes (`'...'`) when using wildcards. + +For example, to check which easyconfigs are available to install TensorFlow 2.4.1 with the `2020b` version of a toolchain: + +```shell +$ eb -S '^tensorflow-2.4.1.*2020b' +CFGS1=/home/example/.local/easybuild/easyconfigs/t/TensorFlow + * $CFGS1/TensorFlow-2.4.1-foss-2020b.eb + * $CFGS1/TensorFlow-2.4.1-fosscuda-2020b.eb +``` + +--- + +#### Search index + +When searching for easyconfig files, you may see a message like this pop up: + +``` +== found valid index for , so using it... +``` + +This indicates that a **search index** was used for this particular directory which significantly speeds +up the search procedure, especially when the easyconfig files are located on a shared filesystem where metadata +operations involving lots of (small) files can be slow. + +For the easyconfig files included with an EasyBuild release, a search index is readily provided. +For other directories, you can create a search index using `eb --create-index `. +[See the EasyBuild documentation for more information](https://easybuild.readthedocs.io/en/latest/Easyconfigs_index.html). + +--- + +## Inspecting easyconfigs + +Once you have determined the name of the easyconfig file that corresponds to the software you want to install, +you may want to take a closer look at its contents before employing it. + +Since easyconfig files are simple text files (in Python syntax), you could use the ubiquitous `cat` command +or your favorite text editor (`vim`, what else). To avoid that you need to locate the easyconfig file first +and copy-paste the full path to it, you can use **`eb --show-ec`**. + +For example, let's inspect the contents of the `bzip2-1.0.6.eb` easyconfig file: + +```shell +$ eb --show-ec bzip2-1.0.6.eb +== temporary log file in case of crash /tmp/eb-jnpzclhl/easybuild-e37cbrj1.log +== Contents of /home/example/.local/easybuild/easyconfigs/b/bzip2/bzip2-1.0.6.eb: +name = 'bzip2' +version = '1.0.6' + +homepage = 'https://sourceware.org/bzip2' +description = """bzip2 is a freely available, patent free, high-quality data compressor. It typically +compresses files to within 10% to 15% of the best available techniques (the PPM family of statistical +compressors), whilst being around twice as fast at compression and six times faster at decompression.""" + +toolchain = SYSTEM +toolchainopts = {'pic': True} + +source_urls = ['https://sourceware.org/pub/bzip2/'] +sources = [SOURCE_TAR_GZ] +patches = ['bzip2-%(version)s-pkgconfig.patch'] +checksums = [ + 'a2848f34fcd5d6cf47def00461fcb528a0484d8edef8208d6d2e2909dc61d9cd', # bzip2-1.0.6.tar.gz + '5a823e820b332eca3684416894f58edc125ac3dace9f46e62f98e45362aa8a6d', # bzip2-1.0.6-pkgconfig.patch +] + +buildopts = "CC=gcc CFLAGS='-Wall -Winline -O3 -fPIC -g $(BIGFILES)'" + +# building of shared libraries doesn't work on OS X (where 'gcc' is actually Clang...) +with_shared_libs = OS_TYPE == 'Linux' + +moduleclass = 'tools' + +== Temporary log file(s) /tmp/eb-jnpzclhl/easybuild-e37cbrj1.log* have been removed. +== Temporary directory /tmp/eb-jnpzclhl has been removed. +``` + +We'll get back to what all of this means later... + +--- + +## Checking dependencies + +!!! note + In some of the examples below, we assume that some software is already installed with EasyBuild. + + If you are following hands-on in a prepared environment, make sure these installations are visible + in the output of "`module avail`". + + +Before kicking off an installation, it is good practice to check which of the required dependencies +are already installed, and which ones are still missing. + +This can be helpful to ensure that your EasyBuild configuration is set up correctly, +and to prevent from accidentally installing an entirely new software stack from scratch +in an unintended location (like `$HOME/.local/easybuild`, the +[default installation path](../configuration/#install-path)). + +--- + +### Dry run + +To get a complete overview of all required dependencies, and see which ones are already installed +and which ones aren't yet, you can use **`eb --dry-run`**. + +Since `--dry-run` produces rather verbose output including the full path to each easyconfig file, +there is a more concise equivalent available as well: `eb --dry-run-short`, which is equivalent with **`eb -D`**. + +For example, to check which of the required dependencies for `SAMtools-1.11-GCC-10.2.0.eb` are already installed: + +```shell +$ eb SAMtools-1.11-GCC-10.2.0.eb -D +== temporary log file in case of crash /tmp/eb-x4qofiph/easybuild-ehhi9fb1.log +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +Dry run: printing build status of easyconfigs and dependencies +CFGS=/home/example/.local/easybuild/easyconfigs + ... + * [x] $CFGS/b/bzip2/bzip2-1.0.8-GCCcore-10.2.0.eb (module: bzip2/1.0.8-GCCcore-10.2.0) + * [x] $CFGS/x/XZ/XZ-5.2.5-GCCcore-10.2.0.eb (module: XZ/5.2.5-GCCcore-10.2.0) + * [x] $CFGS/c/cURL/cURL-7.72.0-GCCcore-10.2.0.eb (module: cURL/7.72.0-GCCcore-10.2.0) + * [x] $CFGS/g/GCC/GCC-10.2.0.eb (module: GCC/10.2.0) + * [x] $CFGS/n/ncurses/ncurses-6.2-GCCcore-10.2.0.eb (module: ncurses/6.2-GCCcore-10.2.0) + * [ ] $CFGS/s/SAMtools/SAMtools-1.11-GCC-10.2.0.eb (module: SAMtools/1.11-GCC-10.2.0) +``` + +(We've trimmed the output a bit here, for the sake of brevity.) + +This output tells us that all dependencies required by `SAMtools-1.11-GCC-10.2.0.eb` are already installed, +since they are all marked with `[x]`, whereas the easyconfig for `SAMtools` itself is not installed yet, +as indicated by lack of an `x` in `[ ]`. + +--- + +### Missing dependencies + +If you are only interested in which dependencies are still *missing*, +you can consult the output of **`eb --missing`**, or the equivalent **`eb -M`**. + +For example, let's see which dependencies are missing in order to get version 3.1.0 of the h5py +Python package installed using the `2020b` version of the `foss` toolchain: + +```shell +$ eb h5py-3.1.0-foss-2020b.eb -M + +2 out of 61 required modules missing: + +* pkgconfig/1.5.1-GCCcore-10.2.0-python (pkgconfig-1.5.1-GCCcore-10.2.0-python.eb) +* h5py/3.1.0-foss-2020b (h5py-3.1.0-foss-2020b.eb) +``` + +That should be pretty self-explanatory: out of the 63 required dependencies (which includes the `foss` toolchain +and everything needed to install it), only 2 dependencies are missing. Great! + +--- + +## Inspecting install procedures + +Finally, before actually installing something you may want to assess *how* exactly EasyBuild is going +to install the software. + +Perhaps you don't trust EasyBuild yet (you will eventually though, hopefully), +or maybe you just want to double check that you have made the right choice before going through with +the actual installation. + +Using **`eb --extended-dry-run`**, or just **`eb -x`** for short, +you can get a **detailed overview of the installation procedure that would be performed by EasyBuild**, +**in a matter of seconds**. + +By means of example, let's inspect some parts of the installation procedure for `Boost-1.74.0-GCC-10.2.0.eb`: + +```shell +$ eb Boost-1.74.0-GCC-10.2.0.eb -x +... + +preparing... [DRY RUN] + +[prepare_step method] +Defining build environment, based on toolchain (options) and specified dependencies... + +Loading toolchain module... + +module load GCC/10.2.0 + +Loading modules for dependencies... + +module load bzip2/1.0.8-GCCcore-10.2.0 +module load zlib/1.2.11-GCCcore-10.2.0 +module load XZ/5.2.5-GCCcore-10.2.0 + +... + +Defining build environment... + + ... + export CXX='mpicxx' + export CXXFLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno -fPIC' + ... + +configuring... [DRY RUN] + +[configure_step method] + running command "./bootstrap.sh --with-toolset=gcc --prefix=/tmp/example/Boost/1.74.0/GCC-10.2.0/obj --without-libraries=python,mpi" + (in /tmp/example/build/Boost/1.74.0/GCC-10.2.0/Boost-1.74.0) + +... + +[sanity_check_step method] +Sanity check paths - file ['files'] + * lib/libboost_system.so + * lib/libboost_thread-mt-x64.so +Sanity check paths - (non-empty) directory ['dirs'] + * include/boost +Sanity check commands + (none) + +... +``` + +We've obviously trimmed the generated output a bit, but it should be sufficient. + +An overview of the installation procedure is shown, following the installation steps as they would be +performed by EasyBuild. The output above shows: + +* how the build environment will be set up during the `prepare` step, by loading the module for both the + toolchains and the dependencies, and defining a set of environment variables like `$CXX`, `$CXXFLAGS`, etc. +* which command will be executed during the configuration step, and in which directory; +* the list of files and directories that will be checked during the sanity check step; + +If you were concerned about EasyBuild being too much of a black box, that is hopefully resolved now. + +!!! note + It is important to highlight here that the reported installation procedure *may* not be 100% correct, + since the [easyblock](../introduction/#easyblocks) can change its mind based on the output of shell commands + that were executed, or based on the contents of a file that was generated during the installation. + Since all "actions" that would be performed during the installation are actually skipped when using `eb -x`, + the reported installation procedure could be partially incorrect. + + In addition, the easyblock may trip over the fact that the installation procedure is not actually being + executed, which sometimes leads to an unexpected error. These situations are handled gracefully by `eb -x` + though, which will report any errors that occurred and then continue anyway with inspecting the remainder of the + installation procedure. Although this obviously limits the value of the generated output, + it doesn't make it completely useless. + +--- + +## Installing software + +You should now be able to make an informed decision on which easyconfig file you should use to +make EasyBuild install the software you require. + +As mentioned before, installing an easyconfig is as simple as passing it to the `eb` command. + +So, let's try to install SAMtools version 1.11: + +```shell +$ eb SAMtools-1.11-GCC-10.2.0.eb +== temporary log file in case of crash /tmp/eb-zh7_fyre/easybuild-4q_lo57b.log +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +== processing EasyBuild easyconfig /home/example/.local/easybuild/easyconfigs/s/SAMtools/SAMtools-1.11-GCC-10.2.0.eb +== building and installing SAMtools/1.11-GCC-10.2.0... +== fetching files... +== creating build dir, resetting environment... +== unpacking... +== patching... +== preparing... +== configuring... +== building... +== testing... +== installing... +== taking care of extensions... +== restore after iterating... +== postprocessing... +== sanity checking... +== cleaning up... +== creating module... +== permissions... +== packaging... +== COMPLETED: Installation ended successfully (took 17 sec) +== Results of the build can be found in the log file(s) /home/example/easybuild/software/SAMtools/1.11-GCC-10.2.0/easybuild/easybuild-SAMtools-1.11-20210309.105601.log +== Build succeeded for 1 out of 1 +== Temporary log file(s) /tmp/eb-zh7_fyre/easybuild-4q_lo57b.log* have been removed. +== Temporary directory /tmp/eb-zh7_fyre has been removed. +``` + +That was... easy. Is that really all there is to it? Well, almost... + +### Enabling dependency resolution + +The SAMtools installation worked like a charm, but remember that all required dependencies were already +available (see [above](#dry-run)). + +If we try this with the `BCFtools-1.11-GCC-10.2.0.eb`, for which the required `GSL` and `HTSlib` dependencies are not available yet, it's less successful: + +```shell +$ eb BCFtools-1.11-GCC-10.2.0.eb -M + +3 out of 23 required modules missing: + +* GSL/2.6-GCC-10.2.0 (GSL-2.6-GCC-10.2.0.eb) +* HTSlib/1.11-GCC-10.2.0 (HTSlib-1.11-GCC-10.2.0.eb) +* BCFtools/1.11-GCC-10.2.0 (BCFtools-1.11-GCC-10.2.0.eb) +``` + +```shell +$ eb BCFtools-1.11-GCC-10.2.0.eb +... +== preparing... +== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/build/BCFtools/1.11/GCC-10.2.0): build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0 (took 2 sec) +== Results of the build can be found in the log file(s) /tmp/eb-3v1dfvnk/easybuild-BCFtools-1.11-20210308.195024.FlxkH.log +ERROR: Build of /home/example/.local/easybuild/easyconfigs/b/BCFtools/BCFtools-1.11-GCC-10.2.0.eb failed (err: 'build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0') +``` + +Oh my, what's this all about? + +If we filter the output a bit and focus on the actual error, the problem is clear: + +``` +Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0 +``` + +The required dependencies `HTSlib/1.11-GCC-10.2.0` and `GSL/2.6-GCC-10.2.0` are not installed yet, +and EasyBuild does not automatically install missing dependencies unless it is told to do so. + +It helpfully suggests to use the `--robot` command line option, so let's try that: + +```shell +$ eb BCFtools-1.11-GCC-10.2.0.eb --robot +... +== resolving dependencies ... +... +== building and installing HTSlib/1.11-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 13 sec) +... +== building and installing GSL/2.6-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 1 min 10 sec) +... +== building and installing BCFtools/1.11-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 8 sec) +... +== Build succeeded for 3 out of 3 +``` + +With dependency resolution enabled the `HTSlib` and `GSL` modules get installed first, +before EasyBuild proceeds with installing `BCFtools`. Great! + +--- + +### Trace output + +As you may have noticed if you tried the previous example hands-on, +some installations take a while. +An installation can be spending quite a bit of time during the build step, but what is actually going on there? + +To provide some more feedback as the installation progresses, you can enable the "`trace`" configuration setting. +Let's do this by defining the `$EASYBUILD_TRACE` environment variable, just to avoid having to type `--trace` +over and over again. + +We will redo the installation of `BCFtools-1.11-GCC-10.2.0.eb` by passing the `--rebuild` +option to the `eb` command (try yourself what happens if you don't use the `--rebuild` option!): + +```shell +$ export EASYBUILD_TRACE=1 +$ eb BCFtools-1.11-GCC-10.2.0.eb --rebuild +... +== configuring... + >> running command: + [started at: 2021-03-08 19:54:53] + [working dir: /tmp/example/build/BCFtools/1.11/GCC-10.2.0/bcftools-1.11] + [output logged in /tmp/eb-9u_ac0nv/easybuild-run_cmd-17m_he2x.log] + ./configure --prefix=/home/example/easybuild/software/BCFtools/1.11-GCC-10.2.0 --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --with-htslib=$EBROOTHTSLIB --enable-libgsl +== building... + >> running command: + [started at: 2021-03-08 19:54:54] + [working dir: /tmp/example/BCFtools/1.11/GCC-10.2.0/bcftools-1.11] + [output logged in /tmp/example/eb-9u_ac0nv/easybuild-run_cmd-bhkgjxi7.log] + make -j 8 + >> command completed: exit 0, ran in 00h00m03s +``` + +That's a bit more comforting to stare at... + +During the *configure* step, the `./configure` command is run with option to +enable support for leveraging `HTSlib` and `GSL`. + +During the *build* step, the software is actually being compiled +by running the `make` command. EasyBuild automatically uses the available cores on the system (in this case 8). + +We even get a pointer to a log file that contains the output of the command being run, +so we can use `tail -f` to see in detail how it progresses. + +Once the `make` command completes, we get a message that the command completed with a exit code 0 +(implying success), and that it took 3 seconds to run. That's good to know. + +Later during the installation, we now also see this output during the sanity check step: + +``` +== sanity checking... + >> file 'bin/bcftools' found: OK + >> file 'bin/plot-vcfstats' found: OK + >> file 'bin/vcfutils.pl' found: OK + >> (non-empty) directory 'libexec/bcftools' found: OK +``` + +Thanks to enabling trace mode, EasyBuild tells us which files & directories it is checking for +in the installation, before declaring it a success. Nice! + +The extra output you get when trace mode is enabled is concise and hence not overwhelming, +while it gives a better insight into what is going on during the installation. +It may also help to spot unexpected actions being taken during the installation early on, +so you can interrupt the installation before it completes, if deemed necessary. + +--- + +## Using installed software + +So far, we have already installed 4 different software packages (SAMtools, HTSlib, GSL, and BCFtools); +we even installed BCFtools twice! + +A lot was going on underneath the covers: locating and unpacking +the source tarballs, setting up the build environment, configuring the build, compiling, +creating and populating the installation directory, performing a quick sanity check on the installation, +cleaning things up, and finally generated the environment module file corresponding to the installation. + +That's great, but how do we now actually *use* these installations? + +This is where the generated module files come into play: they form the access portal to the software +installations, and we'll use the ubiquitous `module` command to digest them. + +First, we need to make sure that the modules tool is aware of where the module files for +our installations are located. If you're unsure where EasyBuild is installing stuff at this point, +check the output of `eb --show-config`; the value of the `installpath` configuration setting is what we are interested in now: + +```shell +$ eb --show-config +... +installpath (E) = /home/example/easybuild +... +repositorypath (E) = /home/example/easybuild/ebfiles_repo +... +sourcepath (E) = /home/example/easybuild/sources +... +``` + +So, what's in this directory? + +```shell +$ ls -l $HOME/easybuild +total 16 +drwxrwxr-x 5 example example 4096 Jun 10 20:11 ebfiles_repo +drwxrwxr-x 5 example example 4096 Jun 10 20:10 modules +drwxrwxr-x 6 example example 4096 Jun 10 20:10 software +drwxrwxr-x 6 example example 4096 Jun 10 20:10 sources +``` + +The `ebfiles_repo` and `sources` directories correspond to the `repositorypath` and `sourcepath` configuration +settings, respectively. The `modules` and `software` directories are what we need now. + +The `modules` subdirectory consists of multiple subdirectories: + +```shell +$ ls $HOME/easybuild/modules +all bio devel numlib tools +``` + +Directories like `bio` and `numlib` correspond to different software categories, +and contain symbolic links to the module files in the `all` directory, +which contains all actual module files for software installed in this EasyBuild installation path. +We'll ignore these separate category directories for now. + +Let's inform the modules tool about the existence of these module files using `"module use"`: + +```shell +module use $HOME/easybuild/modules/all +``` + +This command does little more that updating the `$MODULEPATH` environment variable, +which contains a list of paths that the modules tool should consider when looking for module files. + +Now the modules tool should be aware of our brand new installations: + +```shell +$ module avail + +---------------------- /home/example/easybuild/modules/all ----------------------- + BCFtools/1.11-GCC-10.2.0 GSL/2.6-GCC-10.2.0 SAMtools/1.11-GCC-10.2.0 + EasyBuild/4.3.3 HTSlib/1.11-GCC-10.2.0 bzip2/1.0.6 + +---------------------------- /easybuild/modules/all ----------------------------- + ... +``` + +This output shows both the modules for our own installations as well as the "central" installations in `/easybuild` (which we omitted above for brevity). + +Now we can load these modules and start using these software installations. + +Let's test this for BCFtools. In our current environment, the `bcftools` command is not available yet: + +```shell +$ module list +No modules loaded + +$ bcftools +-bash: bcftools: command not found +``` + +Loading the module for BCFtools changes that: + +```shell +$ module load BCFtools/1.11-GCC-10.2.0 + +$ module list +Currently Loaded Modules: + 1) GCCcore/10.2.0 6) XZ/5.2.5-GCCcore-10.2.0 + 2) zlib/1.2.11-GCCcore-10.2.0 7) cURL/7.72.0-GCCcore-10.2.0 + 3) binutils/2.35-GCCcore-10.2.0 8) HTSlib/1.11-GCC-10.2.0 + 4) GCC/10.2.0 9) GSL/2.6-GCC-10.2.0 + 5) bzip2/1.0.8-GCCcore-10.2.0 10) BCFtools/1.11-GCC-10.2.0 + +$ bcftools --version +bcftools 1.11 +Using htslib 1.11 +... +``` + +Note that the modules for the required dependencies, including the compiler toolchain (which provides runtime libraries +like `libstdc++.so`), are loaded automatically. The "`module load`" command changes the active environment, +by updating environment variables like `$PATH` for example, to make the software available for use. + +##### Resetting your environment + +To restore your environment to a pristine state in which no modules are loaded, you can either +unload the loaded modules one by one using "`module unload`", or you can unload all of them at once using +"`module purge`". + +**If you are using an EasyBuild installation provided by a module, +don't forget to load the `EasyBuild` module again after running "`module purge`".** + +--- + +## Stacking software + +Maybe you have overlooked how the software we are playing around with was not only installed across multiple +different installation directories per software, we are also "stacking" our own installations (in `$HOME/easybuild`) +on top of installations that are provided in a totally different location (`/easybuild`). + +**EasyBuild doesn't care *where* software is installed: as long as the module file that provides access to it +is available, it is happy to pick it up and use it when required.** + +This implies that end users of an HPC system can easily install their +own small software stack on top of what is provided centrally by the HPC support team, +for example. They can even +"replace" a central software installation for their purposes if they need to, since the modules tool will +load the first module file that matches the request being made (there are some caveats with this, but we +won't go into those here). + +--- + +## Exercises + +***Guidelines*** + +Do yourself a favor: don't peek at the solution until you have made an attempt to solve the exercise yourself! + +Please do not spoil solutions to others before they have been discussed by the tutorial organisers. + +The exercises are based on the easyconfig files included with EasyBuild 4.3.3. + +--- + +***Exercise U.0**** - Making installed software available* + +Before working on the exercises for this part of the tutorial, +make sure that the software that is already installed in the prepared environment is available. + +We will assume that you have a small software stack installed using the `2020b` version of the `foss` toolchain. + +**Tip:** execute a "`module use`" command, and verify with "`module avail`" that a bunch of software modules +are available for loading. + +??? success "(click to show solution)" + + Use the following command to make the modules for the software available that is pre-installed + in the prepared environment: + ```shell + module use /easybuild/modules/all + ``` + + If software is installed in a different location than `/easybuild/` in your environment, + you should adjust the command accordingly. + + +--- + +***Exercise U.1**** - Searching easyconfigs* + +See if EasyBuild provides any easyconfig files for installing GROMACS version 2020/5. + +??? success "(click to show solution)" + To check for available easyconfig files, we can use `eb --search` or `eb -S`: + ```shell + $ eb -S gromacs-2020.5 + == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + CFGS1=/home/example/.local/easybuild/easyconfigs/g/GROMACS + * $CFGS1/GROMACS-2020.5-fosscuda-2020a-Python-3.8.2.eb + * $CFGS1/GROMACS-2020.5_fix_threads_gpu_Gmxapitests.patch + ``` + This actually shows one easyconfig file but also a patch file. We can also search specifically + for GROMACS 2020.5 in the `foss` and `fosscuda` toolchains using + ```shell + $ eb -S gromacs-2020.5-foss + == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + CFGS1=/home/example/.local/easybuild/easyconfigs/g/GROMACS + * $CFGS1/GROMACS-2020.5-fosscuda-2020a-Python-3.8.2.eb + ``` + and now we find a single easyconfig file. + +--- + +***Exercise U.2**** - Checking dependencies* + +Check which dependencies are missing to install QuantumESPRESSO version 6.6 with the `2020b` version of the `foss` toolchain. + +??? success "(click to show solution)" + First, we need to determine the name of the easyconfig file for QuantumESPRESSO version 6.6: + ```shell + $ eb -S 'QuantumESPRESSO-6.6.*foss-2020b' + == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + CFGS1=/home/example/.local/easybuild/easyconfigs/q/QuantumESPRESSO + * $CFGS1/QuantumESPRESSO-6.6-foss-2020b.eb + ``` + To determine which dependencies are missing to install this QuantumESPRESSO easyconfig file, we can use `--missing`: + ```shell + $ eb QuantumESPRESSO-6.6-foss-2020b.eb --missing + + 3 out of 58 required modules missing: + + * libxc/4.3.4-GCC-10.2.0 (libxc-4.3.4-GCC-10.2.0.eb) + * ELPA/2020.11.001-foss-2020b (ELPA-2020.11.001-foss-2020b.eb) + * QuantumESPRESSO/6.6-foss-2020b (QuantumESPRESSO-6.6-foss-2020b.eb) + ``` + (some nonessential output removed). + +--- + +***Exercise U.3**** - Performing a dry run* + +Figure out which command EasyBuild would use to compile +the software provided by the `Bowtie2-2.4.2-GCC-9.3.0.eb` easyconfig file, +without actually installing `Bowtie2`. + +Also, which binaries will EasyBuild check for to sanity check the installation? + +??? success "(click to show solution)" + To inspect the installation procedure, we can use `eb -x Bowtie2-2.4.2-GCC-9.3.0.eb`. + + The output for the build step shows the actual compilation command that would be performed (`make ...`): + + ```shell + [build_step method] + >> running command: + [started at: 2021-03-08 20:15:08] + [working dir: /local_scratch/hkenneth/eb-1wodfohg/__ROOT__/local_scratch/hkenneth/Bowtie2/2.4.2/GCC-9.3.0/Bowtie2-2.4.2] + [output logged in /local_scratch/hkenneth/eb-1wodfohg/easybuild-run_cmd-haojzisn.log] + make -j 48 CC="gcc" CPP="g++" CXX="g++" RELEASE_FLAGS="-O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -std=gnu++98" + (in /local_scratch/hkenneth/Bowtie2/2.4.2/GCC-9.3.0/Bowtie2-2.4.2) + ``` + + If the output you get is less detailed, you may not have set `export EASYBUILD_TRACE=1`. + + The output for the sanity check step shows which binaries are expected to be installed: + ``` + [sanity_check_step method] + Sanity check paths - file ['files'] + * bin/bowtie2 + * bin/bowtie2-align-l + * bin/bowtie2-align-s + * bin/bowtie2-build + * bin/bowtie2-build-l + * bin/bowtie2-build-s + * bin/bowtie2-inspect + * bin/bowtie2-inspect-l + * bin/bowtie2-inspect-s + ``` + +--- + +***Exercise U.4**** - Installing software* + +Install version 3.1.0 of the `h5py` Python package and all missing dependencies, +using the `foss/2020b` toolchain, into `/tmp/$USER/easybuild`, +while leveraging the already installed software available from `/easybuild`. + +Enable trace output so you can see which parts of the installation take a while. + +??? success "(click to show solution)" + First, determine the easyconfig file we can use for this: + ```shell + $ eb -S 'h5py-3.1.0.*foss-2020b' + CFGS1=/home/example/.local/easybuild/easyconfigs/h/h5py + * $CFGS1/h5py-3.1.0-foss-2020b.eb + ``` + + Make sure the pre-install software in `/easybuild/` is available: + ```shell + module use /easybuild/modules/all + ``` + + Check which dependencies are missing to install this `h5py` easyconfig: + ```shell + $ eb h5py-3.1.0-foss-2020b.eb --missing + + 2 out of 63 required modules missing: + + * pkgconfig/1.5.1-GCCcore-10.2.0-python (pkgconfig-1.5.1-GCCcore-10.2.0-python.eb) + * h5py/3.1.0-foss-2020b (h5py-3.1.0-foss-2020b.eb) + ``` + + Install `h5py` by specifying the easyconfig file and enabling dependency resolution via `--robot`, + while indicating that we want to install the software into `/tmp/$USER/easybuild` using the `--installpath` + option. Also make sure that trace mode is enabled by defining the `$EASYBUILD_TRACE` environment variable. + ```shell + $ export EASYBUILD_TRACE=1 + $ eb h5py-3.1.0-foss-2020b.eb --robot --installpath /tmp/$USER/easybuild + ... + == building and installing pkgconfig/1.5.1-GCCcore-10.2.0-python... + ... + == building and installing h5py/3.1.0-foss-2020b... + ... + == building... + >> running command: + [started at: 2020-06-10 21:47:32] + [working dir: /tmp/example/h5py/3.1.0/foss-2020b/h5py-3.1.0] + [output logged in /tmp/eb-rjjkbqe1/easybuild-run_cmd-d_dkc4iz.log] + python setup.py configure --mpi --hdf5=$EBROOTHDF5 && /easybuild/software/Python/3.8.6-GCCcore-10.2.0/bin/python setup.py build + >> command completed: exit 0, ran in 00h01m27s + ... + == COMPLETED: Installation ended successfully (took 2 min 46 sec) + ... + == Build succeeded for 2 out of 2 + ``` + + The trace output shows that most time is spent in the build command, + which runs both `python setup.py configure` and `python setup.py build`. + +--- + +***Exercise U.5**** - Using installed software* + +Using the `h5py` installation from the previous exercise to create an empty HDF5 file, +using the following Python statements: + +```python +import h5py +f = h5py.File("empty.hdf5", "w") +f.close() +``` + +Check the resulting file using the `h5stat` command. + +??? success "(click to show solution)" + First, we need to make the modules tool aware of the module files that were installed into `/tmp/$USER/easybuild`: + ```shell + module use /tmp/$USER/easybuild/modules/all + ``` + + Then we can check the `h5py` module is available, and load it: + ```shell + $ module avail h5py + ------------ /tmp/example/easybuild/modules/all ------------ + h5py/3.1.0-foss-2020b + ``` + + ```shell + module load h5py/3.1.0-foss-2020b + ``` + + The Python code snippet can be run directly on the command line using "`python -c '...'`", since it's tiny: + ```shell + python -c 'import h5py; f = h5py.File("empty.hdf5", "w"); f.close()' + ``` + Of course you can also copy the Python code snippet in a file named `test_h5py.py`, + and then run it with `python test_h5py.py`. + + Checking with the `h5stat` command shows that the resulting `empty.hdf5` is indeed a valid HDF5 file: + ```shell + $ ls -l empty.hdf5 + -rw-rw-r-- 1 example example 800 Jun 10 21:54 empty.hdf5 + + $ h5stat empty.hdf5 + Filename: empty.hdf5 + File information + # of unique groups: 1 + # of unique datasets: 0 + ... + ``` + +--- + +If you've made it through the hands-on exercises, congratulations! + +If not, don't worry too much about it. We covered a lot of ground here, +and it's a lot to take in at once, take your time... + +Feel free to ask question in the `#tutorial` channel in the [EasyBuild +Slack](https://docs.easybuild.io/en/latest/#getting-help), +we're happy to help! + +--- + +*[[next: Part 2: Using EasyBuild]](2_00_part2_using.md)* From 42d40a456afd5258524efcbaa822b4afc1663b48 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 17:33:23 +0100 Subject: [PATCH 101/159] Restructuring continued. --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 10 +++++----- docs/2022-CSC_and_LO/1_03_CPE.md | 2 +- docs/2022-CSC_and_LO/1_05_terminology.md | 4 ++-- docs/2022-CSC_and_LO/1_06_installation.md | 10 ++++++---- mkdocs.yml | 7 +++++-- 5 files changed, 19 insertions(+), 14 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md index 7b138670..0674be92 100644 --- a/docs/2022-CSC_and_LO/1_00_part1_intro.md +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -5,8 +5,8 @@ - [What is EasyBuild?](1_01_what_is_easybuild.md) - [The Lmod module system](1_02_Lmod.md) - [The HPE Cray Programming Environment](1_03_CPE.md) -- [Terminology](1_04_terminology.md) - -- [Installation](installation.md) *(hands-on)* -- [Configuration](configuration.md) *(hands-on)* -- [Basic usage](basic_usage.md) *(hands-on)* +- [LUMI software stacks](1_04_LUMI_software_stack.md) +- [Terminology](1_05_terminology.md) +- [Installation](1_06_installation.md) +- [Configuration](1_07_configuration.md) +- [Basic usage](1_08_basic_usage.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md index 42eb197e..077fbc36 100644 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -281,4 +281,4 @@ redfine ``LD_LIBRARY_PATH``. --- -*[[next: Terminology]](1_04_terminology.md)* +*[[next: LUMI software stacks]](1_04_LUMI_software_stack.md)* diff --git a/docs/2022-CSC_and_LO/1_05_terminology.md b/docs/2022-CSC_and_LO/1_05_terminology.md index 9c643579..8e05012f 100644 --- a/docs/2022-CSC_and_LO/1_05_terminology.md +++ b/docs/2022-CSC_and_LO/1_05_terminology.md @@ -1,6 +1,6 @@ # Terminology -*[[back: The HPE Cray Programming Environment]](1_03_CPE.md)* +*[[back: LUMI software stacks]](1_04_LUMI_software_stack.md)* --- @@ -291,4 +291,4 @@ An **easystack file** can be used to specify a collection of software to install --- -*[[next: Installation]](1_05_installation.md)* +*[[next: Installation]](1_06_installation.md)* diff --git a/docs/2022-CSC_and_LO/1_06_installation.md b/docs/2022-CSC_and_LO/1_06_installation.md index 163262ea..6e68b500 100644 --- a/docs/2022-CSC_and_LO/1_06_installation.md +++ b/docs/2022-CSC_and_LO/1_06_installation.md @@ -1,6 +1,6 @@ # Installing EasyBuild -*[[back: Terminology]](1_04_terminology.md)* +*[[back: Terminology]](1_05_terminology.md)* --- @@ -537,7 +537,9 @@ Check that the installation works by running the verification commands outlined --- -**Make sure you have a working EasyBuild installation before proceeding -with the rest of the tutorial!** +**You'll need a working custom EasyBuild installation for some of the +optional generic EasyBuild exercises, but in general you can use the +installation provided by LUMI.** -*[[next: Configuring EasyBuild]](1_06_configuration.md)* + +*[[next: Configuring EasyBuild]](1_07_configuration.md)* diff --git a/mkdocs.yml b/mkdocs.yml index 6dda1ac8..4dff4fbe 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -125,8 +125,11 @@ nav: - What is EasyBuild?: 2022-CSC_and_LO/1_01_what_is_easybuild.md - The Lmod module system: 2022-CSC_and_LO/1_02_Lmod.md - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_03_CPE.md - - Terminology: 2022-CSC_and_LO/1_04_terminology.md - - Installation: 2022-CSC_and_LO/1_05_installation.md + - LUMI software stacks: 2022-CSC_and_LO/1_04_LUMI_software_stack.md + - Terminology: 2022-CSC_and_LO/1_05_terminology.md + - Installation: 2022-CSC_and_LO/1_06_installation.md + - Configuration: 2022-CSC_and_LO/1_07_configuration.md + - Basic usage: 2022-CSC_and_LO/1_08_basic_usage.md plugins: # show revision date at bottom of each page - git-revision-date-localized From 863b933d56a4b179c5453dd1a06191858eb27931 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 30 Mar 2022 12:07:24 +0200 Subject: [PATCH 102/159] Finished reworking the basic usage section, except for the exercises. --- docs/2022-CSC_and_LO/1_08_basic_usage.md | 644 ++++++++++++++++------- 1 file changed, 442 insertions(+), 202 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_08_basic_usage.md index 5566d376..f5031e48 100644 --- a/docs/2022-CSC_and_LO/1_08_basic_usage.md +++ b/docs/2022-CSC_and_LO/1_08_basic_usage.md @@ -21,7 +21,7 @@ Installing software with EasyBuild is as easy as specifying to the **`eb` comman want to install, and then sitting back to enjoy a coffee or tea (or whatever beverage you prefer). This is typically done by **specifying the name of one or more easyconfig files**, often combined -with the `--robot` option to let EasyBuild also install missing dependencies. +with the `--robot` (or `-r`) option to let EasyBuild also install missing dependencies. It is recommended to first assess the current situation before letting EasyBuild install the software, and to check which **dependencies** are already installed and which are still missing. In addition, @@ -53,7 +53,7 @@ ERROR: Can't find path /tmp/does_not_exist.eb When only the *name* of an easyconfig file is specified, EasyBuild will automatically try and locate it. First, it will consider the *current directory*. If no file with the specified name is found there, -EasyBuild will search for the easyconfig file in the [robot search path](../configuration/#robot-search-path). +EasyBuild will search for the easyconfig file in the [robot search path](../1_07_configuration/#robot-search-path). If the path to an existing *directory* is provided, EasyBuild will walk through the entire directory (including all subdirectories), retain all files of which the name ends with `.eb`, and (try to) use these @@ -75,12 +75,12 @@ Or, visually represented: ```shell example1.eb example2.eb -some_deps/ -|-- dep1.eb -|-- dep2.eb -|-- list.txt -|-- more_deps/ - |-- dep3.eb +some_deps +├── deb2.eb +├── dep1.eb +├── list.txt +└── more_deps + └── dep3.eb ``` In this context, we run the following EasyBuild command from our home directory: @@ -118,7 +118,7 @@ corresponding to `--.eb`, where: * `` represents the software version; * `` represents the toolchain used in the easyconfig file, which consists of the toolchain name and version separated with a dash (`-`), and which is omitted (including the preceding `-`) when the - [`system` toolchain](../introduction#system-toolchain) is used; + [`system` toolchain](../1_05_terminology#system-toolchain) is used; * `` represents the value of the `versionsuffix` easyconfig parameter, which is sometimes used to distinguish multiple variants of particular software installations (and is empty by default); @@ -131,7 +131,7 @@ You will frequently need to determine the exact name of an easyconfig file you w or just check which easyconfigs are available for a given software package. This can be done by searching for easyconfigs using **`eb --search`** or **`eb -S`**. -By default all directories listed in the [robot search path](../configuration#robot-search-path) will be +By default all directories listed in the [robot search path](../1_07_configuration#robot-search-path) will be searched. If you want to search in additional directories without changing the robot search path, you can use the `search-paths` configuration setting, or you can change the robot search path via either the `robot` or `robot-paths` configuration options. @@ -202,41 +202,91 @@ Since easyconfig files are simple text files (in Python syntax), you could use t or your favorite text editor (`vim`, what else). To avoid that you need to locate the easyconfig file first and copy-paste the full path to it, you can use **`eb --show-ec`**. +!!! Hint + To follow the examples below on LUMI, load ``LUMI/21.12`` and ``EasyBuild-user`` + (though results may differ or the examples not work anymore as the software installation + on LUMI evolves). + For example, let's inspect the contents of the `bzip2-1.0.6.eb` easyconfig file: ```shell -$ eb --show-ec bzip2-1.0.6.eb -== temporary log file in case of crash /tmp/eb-jnpzclhl/easybuild-e37cbrj1.log -== Contents of /home/example/.local/easybuild/easyconfigs/b/bzip2/bzip2-1.0.6.eb: -name = 'bzip2' -version = '1.0.6' - -homepage = 'https://sourceware.org/bzip2' -description = """bzip2 is a freely available, patent free, high-quality data compressor. It typically -compresses files to within 10% to 15% of the best available techniques (the PPM family of statistical -compressors), whilst being around twice as fast at compression and six times faster at decompression.""" - -toolchain = SYSTEM +$ eb --show-ec bzip2-1.0.8-cpeCray-21.12.eb +== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-53o823qb/easybuild-xn6nmt61.log +== Contents of /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeCray-21.12.eb: +# Contributed by Kurt Lust, LUMI project & UAntwerpen + +local_bzip2_version = '1.0.8' # http://www.bzip.org/downloads.html + +name = 'bzip2' +version = local_bzip2_version + +homepage = 'https://www.sourceware.org/bzip2/' + +whatis = [ + 'Description: bzip2 is a freely available, patent free, high-quality data compressor.', + 'The module contains both executables and libraries.' + 'Keywords: BZ2', +] + +description = """ +bzip2 is a freely available, patent free, high-quality data compressor. It +typically compresses files to within 10% to 15% of the best available techniques +(the PPM family of statistical compressors), whilst being around twice as fast +at compression and six times faster at decompression. It is based on the +Burrows-Wheeler block-sorting text compression algorithm and Huffman coding. +""" + +usage = """ +Check the man pages for the available commands or the web-based documentation for the +library functions. +""" + +docurls = [ + 'Web-based documentation: http://www.bzip.org/docs.html', + 'Man pages available for bzcmp, bzdiff, bzegrep, bzfgrep, bzgrep, bzip2, bunzip2, bzless and bzmore', +] + +toolchain = {'name': 'cpeCray', 'version': '21.12'} toolchainopts = {'pic': True} -source_urls = ['https://sourceware.org/pub/bzip2/'] -sources = [SOURCE_TAR_GZ] -patches = ['bzip2-%(version)s-pkgconfig.patch'] +source_urls = ['https://sourceware.org/pub/%(name)s/'] +sources = [SOURCE_TAR_GZ] +patches = ['bzip2-%(version)s-pkgconfig-manpath.patch'] checksums = [ - 'a2848f34fcd5d6cf47def00461fcb528a0484d8edef8208d6d2e2909dc61d9cd', # bzip2-1.0.6.tar.gz - '5a823e820b332eca3684416894f58edc125ac3dace9f46e62f98e45362aa8a6d', # bzip2-1.0.6-pkgconfig.patch + 'ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269', # bzip2-1.0.8.tar.gz + 'de11269dc6e4917023af4cee9ff83b204953ad9cde561dbc9d6fc70d9b9578e3', # bzip2-1.0.8-pkgconfig-manpath.patch ] -buildopts = "CC=gcc CFLAGS='-Wall -Winline -O3 -fPIC -g $(BIGFILES)'" +builddependencies = [ # Create a reproducible build environment. + ('buildtools', '%(toolchain_version)s', '', True), +] -# building of shared libraries doesn't work on OS X (where 'gcc' is actually Clang...) -with_shared_libs = OS_TYPE == 'Linux' +local_bzip2_major_minor = '.'.join(local_bzip2_version.split('.')[:2]) + +sanity_check_paths = { + 'files': [ 'lib/pkgconfig/bzip2.pc', 'lib/libbz2.a', 'lib/libbz2.%s' % SHLIB_EXT, + 'lib/libbz2.%s.%s' % (SHLIB_EXT, local_bzip2_major_minor), + 'lib/libbz2.%s.%s' % (SHLIB_EXT, local_bzip2_version), + 'include/bzlib.h' ] + + [ 'bin/b%s' % x for x in ['unzip2', 'zcat', 'zdiff', 'zgrep', 'zip2', 'zip2recover', 'zmore'] ] + + [ 'share/man/man1/bz%s.1' % x for x in ['cmp', 'diff', 'egrep', 'fgrep', 'grep', 'ip2', 'less', 'more'] ], + 'dirs': [] +} + +sanity_check_commands = [ + 'bzip2 --help', + 'pkg-config --libs bzip2', +] moduleclass = 'tools' -== Temporary log file(s) /tmp/eb-jnpzclhl/easybuild-e37cbrj1.log* have been removed. -== Temporary directory /tmp/eb-jnpzclhl has been removed. +== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-53o823qb/easybuild-xn6nmt61.log* have been removed. +== Temporary directory /run/user/10012026/easybuild/tmp/eb-53o823qb has been removed. ``` +The output may actually be longer for an easyconfig file that is already installed on the system +as a new easyconfig file is generated in the repository with some information about the installation +added to it and as on LUMI these are at the front of the robot search path to ensure that the system +finds the right easyconfig file matching with a module on the system. We'll get back to what all of this means later... @@ -272,25 +322,30 @@ there is a more concise equivalent available as well: `eb --dry-run-short`, whic For example, to check which of the required dependencies for `SAMtools-1.11-GCC-10.2.0.eb` are already installed: ```shell -$ eb SAMtools-1.11-GCC-10.2.0.eb -D -== temporary log file in case of crash /tmp/eb-x4qofiph/easybuild-ehhi9fb1.log -== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +$ eb SAMtools-1.14-cpeGNU-21.12.eb -D +== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-oo0lj9lq/easybuild-2cyomy8v.log Dry run: printing build status of easyconfigs and dependencies -CFGS=/home/example/.local/easybuild/easyconfigs - ... - * [x] $CFGS/b/bzip2/bzip2-1.0.8-GCCcore-10.2.0.eb (module: bzip2/1.0.8-GCCcore-10.2.0) - * [x] $CFGS/x/XZ/XZ-5.2.5-GCCcore-10.2.0.eb (module: XZ/5.2.5-GCCcore-10.2.0) - * [x] $CFGS/c/cURL/cURL-7.72.0-GCCcore-10.2.0.eb (module: cURL/7.72.0-GCCcore-10.2.0) - * [x] $CFGS/g/GCC/GCC-10.2.0.eb (module: GCC/10.2.0) - * [x] $CFGS/n/ncurses/ncurses-6.2-GCCcore-10.2.0.eb (module: ncurses/6.2-GCCcore-10.2.0) - * [ ] $CFGS/s/SAMtools/SAMtools-1.11-GCC-10.2.0.eb (module: SAMtools/1.11-GCC-10.2.0) +CFGS=/appl/lumi + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeGNU/cpeGNU-21.12.eb (module: cpeGNU/21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ncurses/ncurses-6.2-cpeGNU-21.12.eb (module: ncurses/6.2-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/zlib/zlib-1.2.11-cpeGNU-21.12.eb (module: zlib/1.2.11-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeGNU-21.12.eb (module: bzip2/1.0.8-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12-minimal.eb (module: gettext/0.21-cpeGNU-21.12-minimal) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/XZ/XZ-5.2.5-cpeGNU-21.12.eb (module: XZ/5.2.5-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/Brotli/Brotli-1.0.9-cpeGNU-21.12.eb (module: Brotli/1.0.9-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cURL/cURL-7.78.0-cpeGNU-21.12.eb (module: cURL/7.78.0-cpeGNU-21.12) + * [ ] $CFGS/LUMI-EasyBuild-contrib/easybuild/easyconfigs/h/HTSlib/HTSlib-1.14-cpeGNU-21.12.eb (module: HTSlib/1.14-cpeGNU-21.12) + * [ ] $CFGS/LUMI-EasyBuild-contrib/easybuild/easyconfigs/s/SAMtools/SAMtools-1.14-cpeGNU-21.12.eb (module: SAMtools/1.14-cpeGNU-21.12) +== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-oo0lj9lq/easybuild-2cyomy8v.log* have been removed. +== Temporary directory /run/user/10012026/easybuild/tmp/eb-oo0lj9lq has been removed. ``` -(We've trimmed the output a bit here, for the sake of brevity.) +This output tells you that most of the dependencies required by ``SAMtools-1.14-cpeGNU-21.12.eb`` are +already installed, since they are marked with ``[x]``. However, the easyconfig files for +``HTSLib-1.14-cpeGNU-21.12.eb`` and SAMtools itself are not installed yet, denoted by the +lack of an ``x`` in ``[ ]``. -This output tells us that all dependencies required by `SAMtools-1.11-GCC-10.2.0.eb` are already installed, -since they are all marked with `[x]`, whereas the easyconfig for `SAMtools` itself is not installed yet, -as indicated by lack of an `x` in `[ ]`. --- @@ -299,20 +354,19 @@ as indicated by lack of an `x` in `[ ]`. If you are only interested in which dependencies are still *missing*, you can consult the output of **`eb --missing`**, or the equivalent **`eb -M`**. -For example, let's see which dependencies are missing in order to get version 3.1.0 of the h5py -Python package installed using the `2020b` version of the `foss` toolchain: +For example, for the SAMtools easyconfig file used in the previous example we get (with +some lines removed from the output): ```shell -$ eb h5py-3.1.0-foss-2020b.eb -M - -2 out of 61 required modules missing: +$ eb SAMtools-1.14-cpeGNU-21.12.eb -M +2 out of 11 required modules missing: -* pkgconfig/1.5.1-GCCcore-10.2.0-python (pkgconfig-1.5.1-GCCcore-10.2.0-python.eb) -* h5py/3.1.0-foss-2020b (h5py-3.1.0-foss-2020b.eb) +* HTSlib/1.14-cpeGNU-21.12 (HTSlib-1.14-cpeGNU-21.12.eb) +* SAMtools/1.14-cpeGNU-21.12 (SAMtools-1.14-cpeGNU-21.12.eb) ``` -That should be pretty self-explanatory: out of the 63 required dependencies (which includes the `foss` toolchain -and everything needed to install it), only 2 dependencies are missing. Great! +That should be pretty self-explanatory: out of the 113 required dependencies (which includes the `cpeGNU` toolchain +and everything needed to install it), only 2 dependencies (including SAMtools itself) are missing. Great! --- @@ -329,52 +383,78 @@ Using **`eb --extended-dry-run`**, or just **`eb -x`** for short, you can get a **detailed overview of the installation procedure that would be performed by EasyBuild**, **in a matter of seconds**. -By means of example, let's inspect some parts of the installation procedure for `Boost-1.74.0-GCC-10.2.0.eb`: +By means of example, let's inspect some parts of the installation procedure for ``HTSLib-1.14-cpeGNU-21.12.eb``: ```shell -$ eb Boost-1.74.0-GCC-10.2.0.eb -x +$ eb HTSlib-1.14-cpeGNU-21.12.eb -x ... -preparing... [DRY RUN] +*** DRY RUN using 'ConfigureMake' easyblock (easybuild.easyblocks.generic.configuremake @ /appl/lumi/SW/LUMI-21.12/common/EB/EasyBuild/4.5.3/lib/python3.6/site-packages/easybuild/easyblocks/generic/configuremake.py) *** + +== building and installing HTSlib/1.14-cpeGNU-21.12... +fetching files... [DRY RUN] + +... [prepare_step method] Defining build environment, based on toolchain (options) and specified dependencies... Loading toolchain module... -module load GCC/10.2.0 +module load cpeGNU/21.12 Loading modules for dependencies... -module load bzip2/1.0.8-GCCcore-10.2.0 -module load zlib/1.2.11-GCCcore-10.2.0 -module load XZ/5.2.5-GCCcore-10.2.0 +module load buildtools/21.12 +module load zlib/1.2.11-cpeGNU-21.12 +module load bzip2/1.0.8-cpeGNU-21.12 +module load XZ/5.2.5-cpeGNU-21.12 +module load cURL/7.78.0-cpeGNU-21.12 ... Defining build environment... - ... - export CXX='mpicxx' - export CXXFLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno -fPIC' - ... +... + + export CC='cc' + export CFLAGS='-O2 -ftree-vectorize -fno-math-errno' + +... configuring... [DRY RUN] [configure_step method] - running command "./bootstrap.sh --with-toolset=gcc --prefix=/tmp/example/Boost/1.74.0/GCC-10.2.0/obj --without-libraries=python,mpi" - (in /tmp/example/build/Boost/1.74.0/GCC-10.2.0/Boost-1.74.0) + running command "./configure --prefix=/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/HTSlib/1.14-cpeGNU-21.12" + (in /run/user/10012026/easybuild/build/HTSlib/1.14/cpeGNU-21.12/HTSlib-1.14) + +building... [DRY RUN] + +[build_step method] + running command "make -j 256" + (in /run/user/10012026/easybuild/build/HTSlib/1.14/cpeGNU-21.12/HTSlib-1.14) + +testing... [DRY RUN] + +[test_step method] + +installing... [DRY RUN] ... +sanity checking... [DRY RUN] + [sanity_check_step method] Sanity check paths - file ['files'] - * lib/libboost_system.so - * lib/libboost_thread-mt-x64.so + * bin/bgzip + * bin/tabix + * lib/libhts.so Sanity check paths - (non-empty) directory ['dirs'] - * include/boost + * include Sanity check commands - (none) + * bgzip --version + * htsfile --version + * tabix --version ... ``` @@ -385,7 +465,7 @@ An overview of the installation procedure is shown, following the installation s performed by EasyBuild. The output above shows: * how the build environment will be set up during the `prepare` step, by loading the module for both the - toolchains and the dependencies, and defining a set of environment variables like `$CXX`, `$CXXFLAGS`, etc. + toolchains and the dependencies, and defining a set of environment variables like `$CC`, `$CFLAGS`, etc. * which command will be executed during the configuration step, and in which directory; * the list of files and directories that will be checked during the sanity check step; @@ -393,7 +473,7 @@ If you were concerned about EasyBuild being too much of a black box, that is hop !!! note It is important to highlight here that the reported installation procedure *may* not be 100% correct, - since the [easyblock](../introduction/#easyblocks) can change its mind based on the output of shell commands + since the [easyblock](1_05_terminology.md#easyblocks) can change its mind based on the output of shell commands that were executed, or based on the contents of a file that was generated during the installation. Since all "actions" that would be performed during the installation are actually skipped when using `eb -x`, the reported installation procedure could be partially incorrect. @@ -413,64 +493,105 @@ make EasyBuild install the software you require. As mentioned before, installing an easyconfig is as simple as passing it to the `eb` command. -So, let's try to install SAMtools version 1.11: +So, let's try to install libdap version 3.20.9 with the cpeGNU/21.12 toolchain. + +Let's first check if it has any dependencies that still need to be installed: ```shell -$ eb SAMtools-1.11-GCC-10.2.0.eb -== temporary log file in case of crash /tmp/eb-zh7_fyre/easybuild-4q_lo57b.log -== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... -== processing EasyBuild easyconfig /home/example/.local/easybuild/easyconfigs/s/SAMtools/SAMtools-1.11-GCC-10.2.0.eb -== building and installing SAMtools/1.11-GCC-10.2.0... +$ eb libdap-3.20.9-cpeGNU-21.12.eb -D +== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-wm_bk3j6/easybuild-puyu_559.log +Dry run: printing build status of easyconfigs and dependencies + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeGNU/cpeGNU-21.12.eb (module: cpeGNU/21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/zlib/zlib-1.2.11-cpeGNU-21.12.eb (module: zlib/1.2.11-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libtirpc/libtirpc-1.3.2-cpeGNU-21.12.eb (module: libtirpc/1.3.2-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/Brotli/Brotli-1.0.9-cpeGNU-21.12.eb (module: Brotli/1.0.9-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ICU/ICU-69.1-cpeGNU-21.12.eb (module: ICU/69.1-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cURL/cURL-7.78.0-cpeGNU-21.12.eb (module: cURL/7.78.0-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeGNU-21.12.eb (module: bzip2/1.0.8-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ncurses/ncurses-6.2-cpeGNU-21.12.eb (module: ncurses/6.2-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libreadline/libreadline-8.1-cpeGNU-21.12.eb (module: libreadline/8.1-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12-minimal.eb (module: gettext/0.21-cpeGNU-21.12-minimal) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/PCRE2/PCRE2-10.37-cpeGNU-21.12.eb (module: PCRE2/10.37-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/XZ/XZ-5.2.5-cpeGNU-21.12.eb (module: XZ/5.2.5-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libxml2/libxml2-2.9.12-cpeGNU-21.12.eb (module: libxml2/2.9.12-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libxslt/libxslt-1.1.34-cpeGNU-21.12.eb (module: libxslt/1.1.34-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12.eb (module: gettext/0.21-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/file/file-5.41-cpeGNU-21.12.eb (module: file/5.41-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/util-linux/util-linux-2.37.1-cpeGNU-21.12.eb (module: util-linux/2.37.1-cpeGNU-21.12) + * [ ] /pfs/lustrep3/users/kurtlust/LUMI/LUMI-EasyBuild-contrib/easybuild/easyconfigs/l/libdap/libdap-3.20.9-cpeGNU-21.12.eb (module: libdap/3.20.9-cpeGNU-21.12) +== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-wm_bk3j6/easybuild-puyu_559.log* have been removed. +== Temporary directory /run/user/10012026/easybuild/tmp/eb-wm_bk3j6 has been removed. +``` + +and now instal the library: + +```shell +$ eb libdap-3.20.9-cpeGNU-21.12.eb +== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-kfphjoi8/easybuild-kcs00ai5.log +== processing EasyBuild easyconfig +/pfs/lustrep3/users/kurtlust/LUMI/LUMI-EasyBuild-contrib/easybuild/easyconfigs/l/libdap/libdap-3.20.9-cpeGNU-21.12.eb +== building and installing libdap/3.20.9-cpeGNU-21.12... == fetching files... +== ... (took 2 secs) == creating build dir, resetting environment... == unpacking... == patching... == preparing... +== ... (took 6 secs) == configuring... +== ... (took 1 min 6 secs) == building... +== ... (took 53 secs) == testing... == installing... +== ... (took 5 secs) == taking care of extensions... == restore after iterating... == postprocessing... == sanity checking... +== ... (took 3 secs) == cleaning up... == creating module... +== ... (took 2 secs) == permissions... == packaging... -== COMPLETED: Installation ended successfully (took 17 sec) -== Results of the build can be found in the log file(s) /home/example/easybuild/software/SAMtools/1.11-GCC-10.2.0/easybuild/easybuild-SAMtools-1.11-20210309.105601.log +== COMPLETED: Installation ended successfully (took 2 mins 20 secs) +== Results of the build can be found in the log file(s) +/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/libdap/3.20.9-cpeGNU-21.12/easybuild/easybuild-libdap-3.20.9-20220329.154535.log == Build succeeded for 1 out of 1 -== Temporary log file(s) /tmp/eb-zh7_fyre/easybuild-4q_lo57b.log* have been removed. -== Temporary directory /tmp/eb-zh7_fyre has been removed. +== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-kfphjoi8/easybuild-kcs00ai5.log* have been removed. +== Temporary directory /run/user/10012026/easybuild/tmp/eb-kfphjoi8 has been removed. ``` That was... easy. Is that really all there is to it? Well, almost... ### Enabling dependency resolution -The SAMtools installation worked like a charm, but remember that all required dependencies were already +The libdap installation worked like a charm, but remember that all required dependencies were already available (see [above](#dry-run)). -If we try this with the `BCFtools-1.11-GCC-10.2.0.eb`, for which the required `GSL` and `HTSlib` dependencies are not available yet, it's less successful: +If we try this with the `SAMtools-1.14-cpeGNU-21.12.eb`, for which the required `HTSlib` dependencies is not available yet, it's less successful: ```shell -$ eb BCFtools-1.11-GCC-10.2.0.eb -M +$ eb SAMtools-1.14-cpeGNU-21.12.eb -M -3 out of 23 required modules missing: +2 out of 11 required modules missing: -* GSL/2.6-GCC-10.2.0 (GSL-2.6-GCC-10.2.0.eb) -* HTSlib/1.11-GCC-10.2.0 (HTSlib-1.11-GCC-10.2.0.eb) -* BCFtools/1.11-GCC-10.2.0 (BCFtools-1.11-GCC-10.2.0.eb) +* HTSlib/1.14-cpeGNU-21.12 (HTSlib-1.14-cpeGNU-21.12.eb) +* SAMtools/1.14-cpeGNU-21.12 (SAMtools-1.14-cpeGNU-21.12.eb) ``` ```shell -$ eb BCFtools-1.11-GCC-10.2.0.eb +$ eb SAMtools-1.14-cpeGNU-21.12.eb ... == preparing... -== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/build/BCFtools/1.11/GCC-10.2.0): build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0 (took 2 sec) -== Results of the build can be found in the log file(s) /tmp/eb-3v1dfvnk/easybuild-BCFtools-1.11-20210308.195024.FlxkH.log -ERROR: Build of /home/example/.local/easybuild/easyconfigs/b/BCFtools/BCFtools-1.11-GCC-10.2.0.eb failed (err: 'build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0') +== FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12): build failed +(first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12 (took 3 secs) +== Results of the build can be found in the log file(s) +/run/user/10012026/easybuild/tmp/eb-rgj1v43y/easybuild-SAMtools-1.14-20220329.155911.ZtDcX.log + +ERROR: Build of /appl/lumi/LUMI-EasyBuild-contrib/easybuild/easyconfigs/s/SAMtools/SAMtools-1.14-cpeGNU-21.12.eb failed (err: 'build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12') ``` Oh my, what's this all about? @@ -478,36 +599,33 @@ Oh my, what's this all about? If we filter the output a bit and focus on the actual error, the problem is clear: ``` -Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0 +Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12') ``` -The required dependencies `HTSlib/1.11-GCC-10.2.0` and `GSL/2.6-GCC-10.2.0` are not installed yet, -and EasyBuild does not automatically install missing dependencies unless it is told to do so. +The required dependency `HTSlib/1.14-cpeGNU-21.12` is not installed yet, +and EasyBuild does not automatically install missing dependencies unless it is told to do so +(which we didn't do in the configuration for LUMI). -It helpfully suggests to use the `--robot` command line option, so let's try that: +It helpfully suggests to use the `--robot` (or '-r') command line option, so let's try that: ```shell -$ eb BCFtools-1.11-GCC-10.2.0.eb --robot +$ eb SAMtools-1.14-cpeGNU-21.12.eb --robot ... == resolving dependencies ... ... -== building and installing HTSlib/1.11-GCC-10.2.0... +== building and installing HTSlib/1.14-cpeGNU-21.12... ... == COMPLETED: Installation ended successfully (took 13 sec) ... -== building and installing GSL/2.6-GCC-10.2.0... -... -== COMPLETED: Installation ended successfully (took 1 min 10 sec) -... -== building and installing BCFtools/1.11-GCC-10.2.0... +== building and installing SAMtools/1.14-cpeGNU-21.12... ... == COMPLETED: Installation ended successfully (took 8 sec) ... -== Build succeeded for 3 out of 3 +== Build succeeded for 2 out of 2 ``` -With dependency resolution enabled the `HTSlib` and `GSL` modules get installed first, -before EasyBuild proceeds with installing `BCFtools`. Great! +With dependency resolution enabled the `HTSlib` module gets installed first, +before EasyBuild proceeds with installing `SAMtools`. Great! --- @@ -521,35 +639,50 @@ To provide some more feedback as the installation progresses, you can enable the Let's do this by defining the `$EASYBUILD_TRACE` environment variable, just to avoid having to type `--trace` over and over again. -We will redo the installation of `BCFtools-1.11-GCC-10.2.0.eb` by passing the `--rebuild` +We will redo the installation of `SAMtools-1.14-cpeGNU-21.12.eb` by passing the `--rebuild` option to the `eb` command (try yourself what happens if you don't use the `--rebuild` option!): ```shell $ export EASYBUILD_TRACE=1 -$ eb BCFtools-1.11-GCC-10.2.0.eb --rebuild +$ eb SAMtools-1.14-cpeGNU-21.12.eb --rebuild ... == configuring... >> running command: - [started at: 2021-03-08 19:54:53] - [working dir: /tmp/example/build/BCFtools/1.11/GCC-10.2.0/bcftools-1.11] - [output logged in /tmp/eb-9u_ac0nv/easybuild-run_cmd-17m_he2x.log] - ./configure --prefix=/home/example/easybuild/software/BCFtools/1.11-GCC-10.2.0 --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --with-htslib=$EBROOTHTSLIB --enable-libgsl -== building... + [started at: 2022-03-29 18:46:31] + [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] + [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-g7vd83qv.log] + /users/kurtlust/LUMI-user-appl/sources/generic/eb_v4.5.3/ConfigureMake/config.guess + >> command completed: exit 0, ran in < 1s >> running command: - [started at: 2021-03-08 19:54:54] - [working dir: /tmp/example/BCFtools/1.11/GCC-10.2.0/bcftools-1.11] - [output logged in /tmp/example/eb-9u_ac0nv/easybuild-run_cmd-bhkgjxi7.log] - make -j 8 + [started at: 2022-03-29 18:46:31] + [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] + [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-k0etfv8i.log] + ./configure --prefix=/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/SAMtools/1.14-cpeGNU-21.12 --build=x86_64-pc-linux-gnu +--host=x86_64-pc-linux-gnu --with-htslib=$EBROOTHTSLIB >> command completed: exit 0, ran in 00h00m03s +== ... (took 3 secs) +== building... + >> running command: + [started at: 2022-03-29 18:46:34] + [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] + [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-svcps0yj.log] + make -j 256 CC="cc" CXX="CC" CFLAGS="-O2 -ftree-vectorize -fno-math-errno -fPIC" CXXFLAGS="-O2 -ftree-vectorize -fno-math-errno +-fPIC" + >> command completed: exit 0, ran in 00h00m06s +== ... (took 6 secs) ``` That's a bit more comforting to stare at... -During the *configure* step, the `./configure` command is run with option to -enable support for leveraging `HTSlib` and `GSL`. +SAMtools uses a custom easyblock that is derived from the generic `ConfigureMake` easyblock. +During the *configure* step, the `./configure` command is run with `--build` and `--host` +options added by the generic ConfigureMake easyblock (and the other command, `config.guess` plays +a role in determining the value of those flags). The `--with-htslib=$EBROOTHTSLIB` flag is added +via our easyconfig file to tell SAMtools to use an already available version of HTSlib rather +than the built-in one. During the *build* step, the software is actually being compiled -by running the `make` command. EasyBuild automatically uses the available cores on the system (in this case 8). +by running the `make` command. EasyBuild automatically uses the available cores on the system (in this case 256). We even get a pointer to a log file that contains the output of the command being run, so we can use `tail -f` to see in detail how it progresses. @@ -561,14 +694,19 @@ Later during the installation, we now also see this output during the sanity che ``` == sanity checking... - >> file 'bin/bcftools' found: OK - >> file 'bin/plot-vcfstats' found: OK - >> file 'bin/vcfutils.pl' found: OK - >> (non-empty) directory 'libexec/bcftools' found: OK + >> file 'bin/blast2sam.pl' found: OK + >> file 'bin/bowtie2sam.pl' found: OK + >> file 'bin/export2sam.pl' found: OK + >> file 'bin/interpolate_sam.pl' found: OK +... + >> loading modules: SAMtools/1.14-cpeGNU-21.12... + >> running command 'samtools version' ... + >> result for command 'samtools version': OK ``` -Thanks to enabling trace mode, EasyBuild tells us which files & directories it is checking for -in the installation, before declaring it a success. Nice! +Thanks to enabling trace mode, EasyBuild tells us which files (& directories, but there are non in this case) +it is checking for +in the installation, and which command it is trying to run before declaring it a success. Nice! The extra output you get when trace mode is enabled is concise and hence not overwhelming, while it gives a better insight into what is going on during the installation. @@ -579,13 +717,13 @@ so you can interrupt the installation before it completes, if deemed necessary. ## Using installed software -So far, we have already installed 4 different software packages (SAMtools, HTSlib, GSL, and BCFtools); -we even installed BCFtools twice! +So far, we have already installed 4 different software packages (SAMtools, HTSlib, libdap); +we even installed SAMtools twice! A lot was going on underneath the covers: locating and unpacking the source tarballs, setting up the build environment, configuring the build, compiling, creating and populating the installation directory, performing a quick sanity check on the installation, -cleaning things up, and finally generated the environment module file corresponding to the installation. +cleaning things up, and finally generating the environment module file corresponding to the installation. That's great, but how do we now actually *use* these installations? @@ -593,98 +731,165 @@ This is where the generated module files come into play: they form the access po installations, and we'll use the ubiquitous `module` command to digest them. First, we need to make sure that the modules tool is aware of where the module files for -our installations are located. If you're unsure where EasyBuild is installing stuff at this point, -check the output of `eb --show-config`; the value of the `installpath` configuration setting is what we are interested in now: +our installations are located. On LUMI, when using the EasyBuild-user module to configure EasyBuild, +everything is taken care of for you and the LUMI modules will also automatically add the +suitable module directories for user-installed software to the search path for modules. +By default, EasyBuild-config will install in `$HOME/EasyBuild`, but it is possible to +build the installation in a different directory by pointing to it with the environment +variable `EBU_USER_PREFIX`. Of course this variable needs to be set before loading the `LUMI` +module. (Note that one reason why we don't load a software stack by default is that in +the current setup of LUMI this module would be loaded before the user gets the chance to set +that environment variable in `.bash_profile` or `.bashrc`.) + +When loading the `EasyBuild-user` module, the module command will show you were EasyBuild +will install the software and put the modules, and also put its repository of +processed easyconfig file. ```shell -$ eb --show-config -... -installpath (E) = /home/example/easybuild -... -repositorypath (E) = /home/example/easybuild/ebfiles_repo -... -sourcepath (E) = /home/example/easybuild/sources -... +ml EasyBuild-user + +EasyBuild configured to install software from the LUMI/21.12 software stack for the LUMI/L +partition in the user tree at /users/kurtlust/LUMI-user-appl. + * Software installation directory: /users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L + * Modules installation directory: /users/kurtlust/LUMI-user-appl/modules/LUMI/21.12/partition/L + * Repository: /users/kurtlust/LUMI-user-appl/ebrepo_files/LUMI-21.12/LUMI-L + * Work directory for builds and logs: /run/user/10012026/easybuild + Clear work directory with clear-eb ``` -So, what's in this directory? - -```shell -$ ls -l $HOME/easybuild -total 16 -drwxrwxr-x 5 example example 4096 Jun 10 20:11 ebfiles_repo -drwxrwxr-x 5 example example 4096 Jun 10 20:10 modules -drwxrwxr-x 6 example example 4096 Jun 10 20:10 software -drwxrwxr-x 6 example example 4096 Jun 10 20:10 sources -``` +EasyBuild will copy each easyconfig file it installs to +the repository and add some lines to it with information about the installation. It +also has some options that may edit the source easyconfig, e.g., when asking EasyBuild +to try to build with another toolchain. -The `ebfiles_repo` and `sources` directories correspond to the `repositorypath` and `sourcepath` configuration -settings, respectively. The `modules` and `software` directories are what we need now. +You can always check where EasyBuild is installing stuff by checking the output of +`eb --show-config`. -The `modules` subdirectory consists of multiple subdirectories: +If you're unsure where EasyBuild is installing stuff at this point, +check the output of `eb --show-config`; the value of the `installpath` configuration setting is what we are interested in now: ```shell -$ ls $HOME/easybuild/modules -all bio devel numlib tools +$ eb --show-config +... +buildpath (E) = /run/user/XXXXXXXX/easybuild/build +... +installpath-modules (E) = /users/XXXXXXXX/LUMI-user/modules/LUMI/21.12/partition/L +installpath-software (E) = /users/XXXXXXXX/LUMI-user/SW/LUMI-21.12/L... +... +repositorypath (E) = /users/XXXXXXXX/LUMI-user/ebrepo_files/LUMI-21.12/LUMI-L +... +sourcepath (E) = /users/XXXXXXXX/LUMI-user/sources:/appl/lumi/sources/easybuild +... ``` -Directories like `bio` and `numlib` correspond to different software categories, -and contain symbolic links to the module files in the `all` directory, -which contains all actual module files for software installed in this EasyBuild installation path. -We'll ignore these separate category directories for now. +This is sligthly different from the default EasyBuild setup, where the modules, software, +repository and sources would be installed in respectively the subdirectories `modules`, +`software`, `ebfiles_repo` and `sources` of the directory pointed to by the `installpath` +line. -Let's inform the modules tool about the existence of these module files using `"module use"`: +The modules directory is also a simplified one from the standard EasyBuild one as that also +provides a module categorisation besides a directory containing all modules. As this categorisation +is largely arbitrary and hard to use in the module system, we decided simply not to use it in +our installation and use a custom naming scheme. -```shell -module use $HOME/easybuild/modules/all -``` - -This command does little more that updating the `$MODULEPATH` environment variable, -which contains a list of paths that the modules tool should consider when looking for module files. +However, if you would be using EasyBuild on another system with its default configuration, the +above setup would be used. For more information, we refer to the generic EasyBuild tutorials on the +[EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). Now the modules tool should be aware of our brand new installations: ```shell $ module avail +... +-- EasyBuild managed user software for software stack LUMI/21.12 on LUMI-L --- + HTSlib/1.14-cpeGNU-21.12 libdap/3.20.9-cpeGNU-21.12 + SAMtools/1.14-cpeGNU-21.12 ----------------------- /home/example/easybuild/modules/all ----------------------- - BCFtools/1.11-GCC-10.2.0 GSL/2.6-GCC-10.2.0 SAMtools/1.11-GCC-10.2.0 - EasyBuild/4.3.3 HTSlib/1.11-GCC-10.2.0 bzip2/1.0.6 - ----------------------------- /easybuild/modules/all ----------------------------- - ... +----- EasyBuild managed software for software stack LUMI/21.12 on LUMI-L ----- +... ``` -This output shows both the modules for our own installations as well as the "central" installations in `/easybuild` (which we omitted above for brevity). +This output shows both the modules for our own installations as well as the "central" installations +(which we omitted for brevity). Now we can load these modules and start using these software installations. -Let's test this for BCFtools. In our current environment, the `bcftools` command is not available yet: +Let's test this for SAMtools. In our current environment, the `samtools` command is not available yet: ```shell $ module list -No modules loaded -$ bcftools --bash: bcftools: command not found +Currently Loaded Modules: + 1) perftools-base/21.12.0 + 2) cce/13.0.0 + 3) craype/2.7.13 + 4) cray-dsmml/0.2.2 + 5) cray-mpich/8.1.12 + 6) cray-libsci/21.08.1.2 + 7) PrgEnv-cray/8.2.0 + 8) ModuleLabel/label (S) + 9) init-lumi/0.1 (S) + 10) craype-x86-rome + 11) craype-accel-host + 12) libfabric/1.11.0.4.106 + 13) craype-network-ofi + 14) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta + 15) partition/L (S) + 16) LUMI/21.12 (S) + + Where: + S: Module is Sticky, requires --force to unload or purge +$ samtools +-bash: samtools: command not found ``` -Loading the module for BCFtools changes that: +Loading the module for SAMtools changes that: ```shell -$ module load BCFtools/1.11-GCC-10.2.0 +$ module load SAMtools/1.14-cpeGNU-21.12 + +Lmod is automatically replacing "cce/13.0.0" with "gcc/11.2.0". +Lmod is automatically replacing "PrgEnv-cray/8.2.0" with "cpeGNU/21.12". + +Due to MODULEPATH changes, the following have been reloaded: + 1) cray-mpich/8.1.12 $ module list + Currently Loaded Modules: - 1) GCCcore/10.2.0 6) XZ/5.2.5-GCCcore-10.2.0 - 2) zlib/1.2.11-GCCcore-10.2.0 7) cURL/7.72.0-GCCcore-10.2.0 - 3) binutils/2.35-GCCcore-10.2.0 8) HTSlib/1.11-GCC-10.2.0 - 4) GCC/10.2.0 9) GSL/2.6-GCC-10.2.0 - 5) bzip2/1.0.8-GCCcore-10.2.0 10) BCFtools/1.11-GCC-10.2.0 - -$ bcftools --version -bcftools 1.11 -Using htslib 1.11 + 1) perftools-base/21.12.0 + 2) ModuleLabel/label (S) + 3) init-lumi/0.1 (S) + 4) craype-x86-rome + 5) craype-accel-host + 6) libfabric/1.11.0.4.106 + 7) craype-network-ofi + 8) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta + 9) partition/L (S) + 10) LUMI/21.12 (S) + 11) gcc/11.2.0 + 12) craype/2.7.13 + 13) cray-mpich/8.1.12 + 14) cray-libsci/21.08.1.2 + 15) cray-dsmml/0.2.2 + 16) cpeGNU/21.12 + 17) ncurses/6.2-cpeGNU-21.12 + 18) zlib/1.2.11-cpeGNU-21.12 + 19) bzip2/1.0.8-cpeGNU-21.12 + 20) gettext/0.21-cpeGNU-21.12-minimal + 21) XZ/5.2.5-cpeGNU-21.12 + 22) Brotli/1.0.9-cpeGNU-21.12 + 23) cURL/7.78.0-cpeGNU-21.12 + 24) HTSlib/1.14-cpeGNU-21.12 + 25) SAMtools/1.14-cpeGNU-21.12 + + Where: + S: Module is Sticky, requires --force to unload or purge + +$ samtools --version +samtools 1.14 +Using htslib 1.14 +Copyright (C) 2021 Genome Research Ltd.... ... ``` @@ -696,24 +901,61 @@ by updating environment variables like `$PATH` for example, to make the software To restore your environment to a pristine state in which no modules are loaded, you can either unload the loaded modules one by one using "`module unload`", or you can unload all of them at once using -"`module purge`". +"`module purge`". On LUMI, `module purge` will unload all application modules but will not undo the selection +of the software stack. It will reset the software stack though to use those modules that fit best +with the hardware of the current node (i.e., you may find a different `partition` module). -**If you are using an EasyBuild installation provided by a module, -don't forget to load the `EasyBuild` module again after running "`module purge`".** +```shell +$ module purge +The following modules were not unloaded: + (Use "module --force purge" to unload all): + + 1) LUMI/21.12 6) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta + 2) craype-x86-rome 7) partition/L + 3) craype-accel-host 8) init-lumi/0.1 + 4) libfabric/1.11.0.4.106 9) ModuleLabel/label + 5) craype-network-ofi +$ module list + +Currently Loaded Modules: + 1) LUMI/21.12 (S) + 2) craype-x86-rome + 3) craype-accel-host + 4) libfabric/1.11.0.4.106 + 5) craype-network-ofi + 6) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta + 7) partition/L (S) + 8) init-lumi/0.1 (S) + 9) ModuleLabel/label (S) + + Where: + S: Module is Sticky, requires --force to unload or purge +``` + +Runnin `module --force purge` instead will remove all modules, including the `init-lumi` +module which does part of the initialisation. You will not be able to use the software +stacks completely as before without first loading `init-lumi` in its most recent (or default) +version again! + +```shell +$ module --force purge +$ module list +No modules loaded +``` --- ## Stacking software Maybe you have overlooked how the software we are playing around with was not only installed across multiple -different installation directories per software, we are also "stacking" our own installations (in `$HOME/easybuild`) -on top of installations that are provided in a totally different location (`/easybuild`). +different installation directories per software, we are also "stacking" our own installations (in `$HOME/EasyBuild` +or `$EBU_USER_PREFIX`) on top of installations that are provided in a totally different location (`/appl/lumi`). **EasyBuild doesn't care *where* software is installed: as long as the module file that provides access to it is available, it is happy to pick it up and use it when required.** -This implies that end users of an HPC system can easily install their -own small software stack on top of what is provided centrally by the HPC support team, +This implies that end users of LUMI can easily install their +own small software stack on top of what is provided centrally by the LUMI User Support, for example. They can even "replace" a central software installation for their purposes if they need to, since the modules tool will load the first module file that matches the request being made (there are some caveats with this, but we @@ -723,6 +965,8 @@ won't go into those here). ## Exercises +TODO: Rework for LUMI and provide the necessary EasyConfig files. + ***Guidelines*** Do yourself a favor: don't peek at the solution until you have made an attempt to solve the exercise yourself! @@ -967,10 +1211,6 @@ If you've made it through the hands-on exercises, congratulations! If not, don't worry too much about it. We covered a lot of ground here, and it's a lot to take in at once, take your time... -Feel free to ask question in the `#tutorial` channel in the [EasyBuild -Slack](https://docs.easybuild.io/en/latest/#getting-help), -we're happy to help! - --- *[[next: Part 2: Using EasyBuild]](2_00_part2_using.md)* From 4a5142443128af169cf3007dfe20f4c6f32256fb Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 30 Mar 2022 12:27:23 +0200 Subject: [PATCH 103/159] Additions to the LUMI software stack page of the tutorial. --- .../1_04_LUMI_software_stack.md | 86 ++++++++++++++++++- 1 file changed, 85 insertions(+), 1 deletion(-) diff --git a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md index 0e578a59..3bfae70c 100644 --- a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md +++ b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md @@ -1,9 +1,93 @@ -# LUMI software stacks +# LUMI software stacks (technical) *[[back: The Cray Programming Environment]](1_03_CPE.md)* --- +The user-facing documentation on how to use the LUMI software stacks is +avialable in [the LUMI documentation](https://docs.lumi-supercomputer.eu/computing/softwarestacks/). +On this page we focus more on the technical implementation behind it. + +--- + +# An overview of LUMI + +LUMI has different node types providing compute resources: + +- LUMI has 16 login nodes, though many of those are reserved for special purposes and not + available to all users. TODO + + +--- + +## CrayEnv and LUMI modules + +On LUMI, two types of software stacks are currently offered: + + - ``CrayEnv`` (module name) offers the Cray PE and enables one to use + it completely in the way intended by HPE-Cray. The environment also offers a + limited selection of additional tools, often in updated versions compared to + what SUSE Linux, the basis of the Cray Linux environment, offers. Those tools + are installed and managed via EasyBuild. However, EasyBuild is not available + in that partition. + + It also rectifies a problem caused by the fact that there is only one + configuration file for the Cray PE on LUMI, so that starting a login shell + will not produce an optimal set of target modules for all node types. + The ``CrayEnv`` module recognizes on which node type it is running and + (re-)loading it will trigger a reload of the recommended set of target + modules for that node. + + - ``LUMI`` is an extensible software stack that is mostly managed through + [EasyBuild][easybuild]. Each version of the LUMI software stack is based on + the version of the Cray Programming Environment with the same version + number. + + A deliberate choice was made to only offer a limited number of software + packages in the globally installed stack as the setup of redundancy on LUMI + makes it difficult to update the stack in a way that is guaranteed to not + affect running jobs and as a large central stack is also hard to manage. + However, the EasyBuild setup is such that users can easily install + additional software in their home or project directory using EasyBuild build + recipes that we provide or they develop, and that software will fully + integrate in the central stack (even the corresponding modules will be made + available automatically). + + Each ``LUMI`` module will also automatically activate a set of application + modules tuned to the architecture on which the module load is executed. To + that purpose, the ``LUMI`` module will automatically load the ``partition`` + module that is the best fit for the node. After loading a version of the + ``LUMI`` module, users can always load a different version of the ``partition`` + module. + +Note that the ``partition`` modules are only used by the ``LUMI`` module. In the +``CrayEnv`` environment, users should overwrite the configuration by loading their +set of target modules after loading the ``CrayEnv`` module. + + +--- + +## The ``partition`` module + +The ``LUMI`` module currently supports four partition modules, but that number may +be reduced in the future: + +| Partition | CPU target | Accelerator | +|:----------------|-----------------------|:----------------------------| +| ``partition/L`` | ``craype-x86-rome`` | ``craype-accel-host`` | +| ``partition/C`` | ``craype-x86-milan`` | ``craype-accel-host`` | +| ``partition/G`` | ``craype-x86-trento`` | ``craype-accel-amd-gfx90a`` | +| ``partition/D`` | ``craype-x86-rome`` | ``craype-accel-nvidia80`` | + +All ``partition`` modules also load `craype-network-ofi``. + +``pattition/D`` may be dropped in the future as it seems we have no working CUDA setup +and can only use the GPU nodes in the LUMI-D partition for visualisation and not with CUDA. + +Furthermore if it would turn out that there is no advantage in optimizing for Milan +specifically, or that there are no problems at all in running Milan binaries on Rome +generation CPUs, ``partition/L`` and ``partition/C`` might also be united in a single +partition. From 20f05ed8e876886196819e029de96ea121d8b168 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 30 Mar 2022 12:28:17 +0200 Subject: [PATCH 104/159] Integrating parts of the old tutorial into the new one, texts still need to be rewritten. --- docs/2022-CSC_and_LO/2_00_part2_using.md | 7 + docs/2022-CSC_and_LO/2_01_troubleshooting.md | 488 +++++++++ .../2_02_creating_easyconfig_files.md | 998 ++++++++++++++++++ .../2_03_implementing_easyblocks.md | 665 ++++++++++++ docs/2022-CSC_and_LO/3_00_part3_advanced.md | 9 + .../2022-CSC_and_LO/3_01_easybuild_library.md | 293 +++++ docs/2022-CSC_and_LO/3_02_hooks.md | 190 ++++ docs/2022-CSC_and_LO/3_03_slurm_jobs.md | 304 ++++++ .../3_04_module_naming_schemes.md | 617 +++++++++++ .../3_05_github_integration.md | 442 ++++++++ mkdocs.yml | 14 + 11 files changed, 4027 insertions(+) create mode 100644 docs/2022-CSC_and_LO/2_00_part2_using.md create mode 100644 docs/2022-CSC_and_LO/2_01_troubleshooting.md create mode 100644 docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md create mode 100644 docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md create mode 100644 docs/2022-CSC_and_LO/3_00_part3_advanced.md create mode 100644 docs/2022-CSC_and_LO/3_01_easybuild_library.md create mode 100644 docs/2022-CSC_and_LO/3_02_hooks.md create mode 100644 docs/2022-CSC_and_LO/3_03_slurm_jobs.md create mode 100644 docs/2022-CSC_and_LO/3_04_module_naming_schemes.md create mode 100644 docs/2022-CSC_and_LO/3_05_github_integration.md diff --git a/docs/2022-CSC_and_LO/2_00_part2_using.md b/docs/2022-CSC_and_LO/2_00_part2_using.md new file mode 100644 index 00000000..e033499c --- /dev/null +++ b/docs/2022-CSC_and_LO/2_00_part2_using.md @@ -0,0 +1,7 @@ +# Part II: Using EasyBuild + +*[[back to start page]](index.md)* + +* [Troubleshooting](2_01_troubleshooting.md) *(hands-on)* +* [Creating easyconfig files](2_02_creating_easyconfig_files.md) *(hands-on)* +* [Implementing easyblocks](2_03_implementing_easyblocks.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/2_01_troubleshooting.md b/docs/2022-CSC_and_LO/2_01_troubleshooting.md new file mode 100644 index 00000000..86a2f1d3 --- /dev/null +++ b/docs/2022-CSC_and_LO/2_01_troubleshooting.md @@ -0,0 +1,488 @@ +# Troubleshooting + +*[[back: Overview]](index.md)* + +--- + +When installing scientific software you are bound to run into problems +that make the installation fail sooner or later, even when using EasyBuild. + +In this part we take a look at how you can **troubleshoot a failing installation**, +and focus on a couple of EasyBuild aspects that can be helpful in that context, +like the **error messages produced by EasyBuild**, the detailed **log file** that is +available for the installation and the **build directory** where most software is +being compiled before it actually gets installed. + +At the end of this part an exercise is available in which you get +to fix a failing installation, so pay attention! + +## EasyBuild error messages + +When EasyBuild detects that something went wrong, it will usually produce a +short (and hopefully helpful) error message. + +Things that could go wrong during an installation include: + +* missing source or patch files; +* a checksum error on a downloaded source or patch file; +* required dependencies that are not specified in the easyconfig file; +* failing shell commands; +* running out of available memory or disk space; +* a segmentation fault caused by a flipped bit triggered by a cosmic ray ([really, it happens!](https://blogs.oracle.com/linux/attack-of-the-cosmic-rays-v2)); + +Unfortunately this is not an exhaustive list, there are plenty of other +potential problems that could result in a failing installation... + +For each of the shell commands that EasyBuild executes during an +installation, it will check the exit status. +If the exit status is zero, EasyBuild will usually assume that the shell command +ran correctly, and it will continue with the rest of the installation procedure. +If the exit status is anything but zero, EasyBuild will assume that a problem +has occurred, and the installation will be interrupted. + +### Example + +Here is an example of an EasyBuild error message (slightly reformatted for clarity): + +``` +$ eb example.eb +... +== building... +== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): +build failed (first 300 chars): cmd "make" exited with exit code 2 and output: +/usr/bin/g++ -O2 -ftree-vectorize -march=native -fno-math-errno -std=c++14 -c -o core.o core.cpp +g++: error: unrecognized command line option '-std=c++14' (took 1 sec) +== Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log +ERROR: Build of /home/easybuild/subread.eb failed (err: ...) +``` + +Let's break this down a bit: during the `build` step of the installation +procedure EasyBuild was running `make` as a shell command, which +failed (exit code 2, so not zero). +The `make` command tripped over the compilation of `core.cpp` that failed because +`-std=c++14` is not a known option to the `g++` command. + +OK fine, but now what? Can you spot something suspicious here? +Wait a minute... Why is `make` using `/usr/bin/g++` for the compilation?! +That's not where our toolchain compiler is installed, +that's somewhere under `/easybuild/software`. + +Let's see what `/usr/bin/g++` is: + +```shell +$ /usr/bin/g++ --version +g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) +``` + +Oh my that's pretty ancient (GCC 4.8.5 was released in June 2015). +That could definitely explain why it doesn't know about the C++14 standard yet... + +Your next step in this case should probably be figuring +out why `/usr/bin/g++` is being used rather than just `g++`, which would +result in using the right compiler version because EasyBuild sets up the build +environment carefully. + +This is a fictitious example of course, but hopefully it gives you a feeling +of how errors that occur during the installation are handled. + +## EasyBuild log files + +Finding the cause of a problem that made the installation fail is, unfortunately, not always that straightforward... + +EasyBuild includes the first 300 characters of the output produced by a failing +shell command in the error message, which is a simple way to try include +actionable information in the error message while not flooding your terminal +with the full output of the command that failed. + +In some cases there won't be any useful information in there however, +since the actual error message(s) could only appear way later, perhaps even after +the command was already running for several minutes. + +In that case, you will have the dive into the log file that is created by EasyBuild for +every installation, which is located in the unique temporary directory for the EasyBuild session. + +See for example this output line from our earlier example error message: + +``` +== Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log +``` + +You can open this file with your favorite text editor or a tool like `less` +to take a look at the information collected in the log file, which includes +things like: + +* informative messages produced by both the EasyBuild framework and the easyblock + describing how the installation is progressing; +* how the build environment was set up: which modules were loaded, which environment variables were set; +* the exact shell commands that were executed, and in which directory they were run; +* the full output produced by these commands, and their exit code; + +Note that the installation log is also copied into each software installation +directory for successful installation, into the `easybuild` subdirectory. +For example: + +``` +/easybuild/software/HDF5/1.10.7-gompi-2020b/easybuild/easybuild-HDF5-1.10.7-20210308.214453.log +``` + +### Last log + +The `eb` command supports a handy little option that prints the location +to the most recently updated build log. You can leverage this to quickly +open the build log of the last failed EasyBuild session in an editor: + +``` +vim $(eb --last-log) +``` + +### Navigating log files + +Usually you want to go to the end of the log file and then work your way up, +either by scrolling or by searching for specific patterns. Here are a couple +of suggestions of patterns you can use to locate errors: + +* `ERROR` +* `Error 1` +* `error: ` +* `failure` +* `not found` +* `No such file or directory` +* `bazel` +* `Segmentation fault` + +Using "`error`" as a search pattern is not very useful: you will hit a lot of log lines +that are not actually errors at all (like the compilation of an `error.c` file), +and you'll miss others that do include errors but mention `ERROR` or `Error` +rather than `error`. + +When using `less` to view a log file, you can navigate it by: + +* hitting '`$`' followed by '`G`' to go to the end of the log file; +* using your arrow keys to scroll up/down; +* typing '`?`' followed by some text and Enter to search backwards for a particular +pattern ('`/`' to search forwards, '`n`' for next match); +* hitting '`q`' to exit; + +It can also be helpful to zoom in on a specific step of the installation procedure, +which you can do by looking for step markers like these: + +``` +== 2020-06-13 01:34:48,816 example INFO configuring... +== 2020-06-13 01:34:48,817 example INFO Starting configure step +... +== 2020-06-13 01:34:48,823 main.EB_HPL INFO Running method configure_step part of step configure +``` + +If you want to look at the start of the output produced by a particular command, +you can look for the log message that looks like this (this is from the installation +log for `HDF5`): + +``` +== 2020-06-09 13:11:19,968 run.py:222 INFO running cmd: make install +== 2020-06-09 13:11:25,186 run.py:538 INFO cmd " make install " exited with exit code 0 and output: +Making install in src +make[1]: Entering directory `/tmp/example/HDF5/1.10.7/gompi-2020b/hdf5-1.10.7/src' +... +``` + +It can be useful to look for the *first* error that occurred in the output of a command, since subsequent errors are +often fallout from earlier errors. You can do this by first navigating +to the start of the output for a command using "`INFO running cmd`" as a search pattern, and then looking for patterns +like "`error:`" from there. + +## Inspecting the build directory + +When an installation fails the corresponding build directory is *not* cleaned up +automatically, that is only done for successful installations. +This allows you to dive in and check for clues in the files that are stored there. + +The location of the build directory is mentioned in the EasyBuild error message: + +``` +== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): ... +``` + +For software using a classic `configure` script, you may have to locate +and inspect the `config.log` file in the build directory to determine the underlying cause of an error. For software using CMake as a configuration tool you often have to check in +`CMakeOutput.log` or `CMakeError.log` for clues, which are sneakily hidden by CMake in a `CMakeFiles` subdirectory of the build directory. + +As a side note here: as EasyBuild does not clean out old and failed builds you will need to eventually manually remove these build directories from the `buildpath` directory. + +## Exercise + +Let's work our way through a less smooth software installation, +using the easyconfig file that is provided below. + +Don't worry if most of this is still unclear to you, we'll get +to writing easyconfig files from scratch [later in this tutorial](../creating_easyconfig_files). + +```python +easyblock = 'MakeCp' + +name = 'Subread' +version = '2.0.1' + +homepage = 'http://subread.sourceforge.net' +description = "High performance read alignment, quantification and mutation discovery" + +toolchain = {'name': 'GCC', 'version': '8.5.0'} + +# download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz +sources = ['subread-%(version)s-source.tar.gz'] +checksums = ['d808eb5b1823c572cb45a97c95a3c5acb3d8e29aa47ec74e3ca1eb345787c17b'] + +start_dir = 'src' + +# -fcommon is required to compile Subread 2.0.1 with GCC 10, +# which uses -fno-common by default (see https://www.gnu.org/software/gcc/gcc-10/porting_to.html) +buildopts = '-f Makefile.Linux CFLAGS="-fast -fcommon"' + +files_to_copy = ['bin'] + +sanity_check_paths = { + 'files': ['bin/featureCounts', 'bin/subread-align'], + 'dirs': ['bin/utilities'], +} + +sanity_check_commands = ["featureCounts --version"] + +moduleclass = 'bio' +``` + +Do you spot any potential problems yet with this easyconfig file? + +***Preparation*** + +Start by copying the text above in a file named `subread.eb`, +so you can gradually fix the problem you'll encounter. + +Also make sure that the pre-installed software stack is available, +and that the EasyBuild module is loaded (unless you installed EasyBuild +yourself): + +``` +module use /easybuild/modules/all +module load EasyBuild +``` + +For this exercise, make sure EasyBuild is configured to +use `$HOME/easybuild` as `prefix`, and to use `/tmp/$USER` as `buildpath`: + +```shell +export EASYBUILD_PREFIX=$HOME/easybuild +export EASYBUILD_BUILDPATH=/tmp/$USER +``` + +Check your configuration via `eb --show-config`. + +Strictly speaking the configuration doesn't matter much for the sake of this +exercise, but it may help with the step-wise approach we'll take and +grasping the solutions. + +Remember though: *no peeking* before you tried to solve each step yourself! + +--- + +***Exercise T.1**** - Sources* + +Try to install the `subread.eb` easyconfig file, see what happens. + +Can you fix the problem you run into, perhaps without even changing +the easyconfig file? + +??? success "(click to show solution)" + + The installation fails because the source file `subread-2.0.1-source.tar.gz` + is not found: + ``` + $ eb subread.eb + ... + == FAILED: Installation ended unsuccessfully (build directory: /tmp/example/Subread/2.0.1/GCC-8.5.0): build failed (first 300 chars): + Couldn't find file subread-2.0.1-source.tar.gz anywhere, and downloading it didn't work either... + Paths attempted (in order): ... + ``` + + In this case, the problem is that the easyconfig file does not specify + where the sources can be downloaded from. Not automatically at least, + but there is a helpful comment included: + ```python + # download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz + sources = ['subread-%(version)s-source.tar.gz'] + ``` + + We can download the source tarball ourselves, + and move it to the location where EasyBuild expects to find it + (in the `sourcepath` directory): + ``` + curl -OL https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz + mv subread-2.0.1-source.tar.gz $HOME/easybuild/sources/s/Subread/ + ``` + + If downloading is problematic for some reason, the source tarball is also available + in `/easybuild/sources/s/Subread`. + + Or, we can change the easyconfig file to specify the location where + the easyconfig file can be downloaded from: + ```python + source_urls = ['https://download.sourceforge.net/subread/'] + sources = ['subread-%(version)s-source.tar.gz'] + ``` + Note that the `source_urls` value is a *list* of candidate download URLs, + *without* the filename of the source file itself. + + This way, EasyBuild will download the source file when running `eb subread.eb`. + + The source tarball is fairly large (23MB), so don't be alarmed if the download takes a little while. + + ```shell + $ ls -lh $HOME/easybuild/sources/s/Subread + total 23M + -rw-rw-r-- 1 easybuild easybuild 23M Jun 13 17:42 subread-2.0.1-source.tar.gz + ``` + +--- + +***Exercise T.2**** - Toolchain* + +After fixing the problem with missing source file, try the installation again. + +What's wrong now? How can you fix it quickly? + +Take into account that we just want to get this software package installed, +we don't care too much about details like the version of the dependencies or +the *toolchain* here... + + +??? success "(click to show solution)" + + The installation fails because the easyconfig specifies that GCC 8.5.0 + should be used as toolchain: + ``` + $ eb subread.eb + ... + == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-8.5.0): build failed (first 300 chars): + No module found for toolchain: GCC/8.5.0 (took 1 sec) + ``` + + We don't have this GCC version installed, but we do have GCC 10.2.0: + + ```shell + $ module avail GCC/ + ----------------- /easybuild/modules/all ------------------ + GCC/10.2.0 + ``` + + So let's try using that instead. + + Edit the easyconfig file so it contains this: + + ```python + toolchain = {'name': 'GCC', 'version': '10.2.0'} + ``` +--- + +***Exercise T.3**** - Build step* + +With the first two problems fixed, now we can actually try to build the software. + +Can you fix the next problem you run into? + +??? success "(click to show solution)" + + The compilation fails, but the error message we see is incomplete due to + EasyBuild truncating the command output (only the 300 first characters of the output are shown): + ``` + == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-10.2.0): build failed (first 300 chars): + cmd " make -j 1 -f Makefile.Linux CFLAGS="-fast -fcommon"" exited with exit code 2 and output: + gcc -mtune=core2 -O3 -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\""2.0.1"\" -D_FILE_OFFSET_BITS=64 -fmessage-length=0 -ggdb -fast -fcommon -c -o core.o core.c + gcc: error: unrecognized command line opti (took 1 sec) + ``` + + If you open the log file and scroll to the end, + the error is pretty obvious: + ``` + gcc: error: unrecognized command line option -fast; did you mean -Ofast? + make: *** [core.o] Error 1 + ``` + + The easyconfig file hard specifies the `-fast` compiler flag via the `CFLAGS` argument to the build command: + ```python + # -fcommon is required to compile Subread 2.0.1 with GCC 10, + # which uses -fno-common by default (see https://www.gnu.org/software/gcc/gcc-10/porting_to.html) + buildopts = '-f Makefile.Linux CFLAGS="-fast -fcommon"' + ``` + + EasyBuild sets up the build environment, so there should be no need + to hard specify compiler flags (certainly not incorrect ones). + The comment above the `buildopts` definition makes it clear that the `-fcommon` + flag *is* required though, because GCC 10 became a bit stricter by + using `-fno-common` by default. Note that we are using `-fcommon` + as an escape mechanism here: it would be better to fix the source code + and create a patch file instead. + + An easy way to fix this problem is to replace the `-fast` with `-Ofast`, + as the compiler error suggests. + + In this case it is advised to change the `CFLAGS` argument that is added + to be build command to replace the `-fast` with `$CFLAGS`, + which is defined in the build environment by EasyBuild. + + ```python + buildopts = '-f Makefile.Linux CFLAGS="$CFLAGS -fcommon"' + ``` + + Note that we need to be careful with quotes here: we use inner double quotes + to ensure that `$CFLAGS` will be expanded to its value when the build command is run. + +--- + +***Exercise T.4**** - Sanity check* + +After fixing the compilation issue, you're really close to getting the installation working, we promise! + +Don't give up now, try one last time and fix the last problem that occurs... + +??? success "(click to show solution)" + + Now the installation itself works but the sanity check fails, + and hence the module file does not get generated: + ``` + $ eb subread.eb + ... + == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-10.2.0): build failed (first 300 chars): + Sanity check failed: sanity check command featureCounts --version exited with code 255 + (output: featureCounts: unrecognized option '--version' + ... + ``` + + If you look at the full output in the log file you can see + that the correct option to check the version of the `featureCounts` command + is "`-v`" rather than "`--version`", so we need to fix this in the easyconfig file. + + Make the following change in the easyconfig file: + ```python + sanity_check_commands = ["featureCounts -v"] + ``` + + After doing so, **you don't have to redo the installation + from scratch**, you can use the `--module-only` option to only run the + sanity check and generate the module file again: + ``` + eb subread.eb --module-only + ``` + +--- + +In the end, you should be able to install Subread 2.0.1 with the GCC 10.2.0 toolchain by fixing the problems with the `subread.eb` easyconfig file. + +Check your work by manually loading the module and checking the version +via the `featureCounts` command, which should look like this: + +```shell +$ featureCounts -v +featureCounts v2.0.1 +``` + +--- + +*[[next: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md new file mode 100644 index 00000000..d46e003a --- /dev/null +++ b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md @@ -0,0 +1,998 @@ +# Creating easyconfig files + +*[[back: Troubleshooting]](2_02_troubleshooting.md)* + +--- + +We already know how to install easyconfig files that are included with EasyBuild, +but what about installing software for which no easyconfig is available yet? + +To do this we will need to **create additional easyconfig files**, +since every software installation performed by EasyBuild is done based on an easyconfig file. + +In this part of the tutorial we will look at the guts of easyconfig files and even create some ourselves! + +## Easyconfigs vs easyblocks + +Before we dive into writing [easyconfig files](../introduction/#easyconfig-files), +let us take a brief look at how they relate to [easyblocks](../introduction/#easyblocks). + +As we discussed [earlier](../introduction/#terminology), an easyconfig file (`*.eb`) is required +for each installation +performed by EasyBuild which specifies the details of the installation (which software +version, toolchain, etc.), while the installation procedure is implemented +in an easyblock (a Python module). + +When can we leverage a *generic easyblock*, perhaps via a "fat" easyconfig file that includes +a lot of carefully defined easyconfig parameters, and when should we use a minimal easyconfig file +together with a custom *software-specific* easyblock? + +This is not an easy question to answer in a general sense, since it depends on several factors: +the complexity of the software you want to get installed, how much flexibility you want, +how "intelligent" the installation procedure should be with respect to the compiler toolchain and dependencies +that are used for the installation, etc. + +In a nutshell, custom software-specific easyblocks are "do once and forget": they are central solution to peculiarities in the installation procedure of a particular software package. + +Reasons to consider implementing a software-specific easyblock rather than using a generic easyblock include: + +* 'critical' values for easyconfig parameters required to make installation succeed; +* toolchain-specific aspects of the build and installation procedure (e.g., configure options); +* interactive commands that need to be run; +* custom (configure) options for dependencies; +* having to create or adjust specific (configuration) files; +* 'hackish' usage of a generic easyblock; +* complex or very non-standard installation procedure; + +Implementing easyblocks is out of scope for this basic tutorial, for more information please consult +the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). + +## Writing easyconfig files + +Writing an easyconfig file boils down to defining a set of easyconfig parameters in a text file, +which we give a '`.eb`' extension by convention. The name of an easyconfig file doesn't matter +when using it directly to install software, but it does matter when EasyBuild needs to find it +to resolve a dependency for example (as we [discussed earlier](../basic_usage/#easyconfig-filenames)). + +The syntax for easyconfig files is *Python syntax*: you are basically defining a bunch of Python variables +that correspond to easyconfig parameters. + +The order in which the easyconfig parameters are defined doesn't matter, but we generally try to strick to a particular +order which roughly corresponds to the order in which the easyconfig parameters are used during the installation. +That is mostly helpful for humans staring at easyconfig files or comparing them. + +### Mandatory parameters + +A limited number of easyconfig parameters are *mandatory*, they must be defined in every easyconfig file: + +* `name`: the name of the software to install; +* `version`: the version of the software to install; +* `homepage`: a URL to the website of the software; +* `description`: a short description of the software; +* `toolchain`: the compiler toolchain to use for the installation; + +**`name`, `version`** + +It should be no surprise that specifying the name and version of the software you want to install is mandatory. +This information may influence the value of several other easyconfig parameters (like the name of the source file), and is also used to the determine the name of the module file to install. + +```python +name = 'example' +version = '1.0' +``` + +**`homepage`, `description`** + +The homepage and description are included in the generated module file for the installation. +That way the "`module show`" command provides some useful high-level information about the installation. + +```python +homepage = 'https://example.org' +description = "This is just an example." +``` + +Usually it does not matter whether you use single or double quotes to specify string values, +but you will often see that single quotes are used for values that don't have spaces (words) +and double quotes for values that do have spaces (sentences). There is no technical reason for +this, it just feels more natural to some people. There are cases where it is important to use +the right type of quotes however, we will get back to that later (keep it in mind for the exercises!). + +For multi-line descriptions, you will need to use "triple quoting" (which is standard Python syntax): + +```python +description = """This is an example + of a multi-line description. + It is spread across multiple lines.""" +``` + +**`toolchain`** + +EasyBuild also requires that the [compiler toolchain](../introduction/#toolchains) is specified, via the `toolchain` +easyconfig parameter. + +This can either be the [`system` toolchain](../introduction/#system-toolchain), for which a constant named `SYSTEM` is available: + +```python +toolchain = SYSTEM +``` + +Usually we specify a 'proper' toolchain like the compiler-only toolchain GCC 10.2.0 which we used before, +or the full toolchain `foss` 2020b. The name and version of the toolchain can be specified using a small Python dictionary, +for example: + +```python +toolchain = {'name': 'GCC', 'version': '10.2.0'} +``` + +### Commonly used parameters + +You will often need to specify additional easyconfig parameters to get something useful done. +We will cover the most commonly used ones here, but keep in mind that these are *not* mandatory. + +A full overview of all known easyconfig parameters can be obtained via "`eb --avail-easyconfig-params`" +or just "`eb -a`" for short, or can be consulted in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_parameters.html). + +#### Sources, patches, and checksums + +In most easyconfig files you will see that a list of source files is specified via the `sources` +easyconfig parameter, usually combined +with one or more URLs where these sources can be downloaded specified via `source_urls`. +There also may be patch files listed (specified via `patches`), +and checksums for both the source files and patches (specified via `checksums`). + +The `sources` easyconfig parameter is commonly defined but it is *not* mandatory, +because some easyconfig files only specify bundles of software packages and hence only +serve to generate a module file. + +Here is an example of how these easyconfig parameters can be specified: + +```python +source_urls = [ + 'https://example.org/download/', + 'https://example.org/download/archive/', +] +sources = ['example-1.0-src.tar.gz'] +patches = ['example-fix.patch'] +checksums = [ + '9febae18533d035ac688d977cb2ca050e6ca8379311d7a14490ad1ef948d45fa', + '864395d648ad9a5b75d1a745c8ef82b78421d571584037560a22a581ed7a261c', +] +``` + +Each of these require a *list* of values, so even if there is only a single source file or download URL +you must use square brackets as shown in the example. The default value for each of these is an empty list (`[]`). + +Some things worth pointing out here: + +* The download URLs specified via `source_urls` do *not* include the name of the file, that is added + automatically by EasyBuild when it tries to download the file (only if it's not available already.) +* If multiple download URLs are specified, they are each tried once in order until the download of the source file was + successful. This can be useful to include backup locations where source files can be downloaded from. +* Names of source files and patches should not include hardcoded software versions, they usually use a + template value like `%(version)s` instead: + ```python + sources = ['example-%(version)s-src.tar.gz'] + ``` + EasyBuild will use the value of the `version` easyconfig parameter to determine the actual name of the source + file. This way the software version is only specified in one place and the easyconfig file is easier to + update to other software versions. A list of template values can be consulted via the EasyBuild command + line via the `--avail-easyconfig-templates` option, or in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_templates.html). +* Source files can also be specified in ways other than just using a filename, see the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-param-sources-alt) for more information. +* Specified checksums are usually SHA256 checksum values, but [other types are also supported](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html?highlight=checksums#checksums). + + +#### Easyblock + +The easyblock that should be used for the installation can be specified via the `easyblock` easyconfig parameter. + +This is not mandatory however, because by default EasyBuild will determine the easyblock to use based on the +name of the software. If '`example`' is specified as software name, EasyBuild will try to locate a +software-specific easyblock named `EB_example` (in a Python module named `example.py`). Software-specific +easyblocks follow the convention that the class name starts with `'EB_`', followed by the software name +(where some characters are replaced, like '`-`' with '`_minus_`'). + +**Generic easyblocks** + +Usually the `easyblock` value is the name of a *generic* easyblock, if it is specified. The name of +a generic easyblock does *not* start with '`EB_`', so you can easily distinguish it from a software-specific +easyblock. + +Here are a couple of commonly used generic easyblocks: + +* `ConfigureMake`: implements the standard `./configure`, `make`, `make install` installation procedure; +* `CMakeMake`: same as `ConfigureMake`, but with `./configure` replaced with `cmake` for the configuration step; +* `PythonPackage`: implements the installation procedure for a single Python package, by default using + "`python setup.py install`" but other methods like using "`pip install`" are also supported; +* `Bundle`: a simple generic easyblock to bundle a set of software packages together in a single installation directory; +* `PythonBundle`: a customized version of the `Bundle` generic easyblock to install a bundle of Python packages + in a single installation directory; + +A full overview of the available generic easyblocks is available in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/generic_easyblocks.html). You can also consult the output of +`eb --list-easyblocks`, which gives an overview of *all* known easyblocks, and how they relate to each other. + +**Custom easyconfig parameters** + +Most generic easyblocks provide additional easyconfig parameters to steer their behaviour. +You can consult these via "`eb -a --easyblock`" or just "`eb -a -e`", which results in an +additional "`EASYBLOCK-SPECIFIC`" section to be added. See the (partial) output of this command for example: + +```shell +$ eb -a -e ConfigureMake +Available easyconfig parameters (* indicates specific to the ConfigureMake easyblock): +... +EASYBLOCK-SPECIFIC +------------------ +build_cmd* Build command to use [default: "make"] +build_type* Value to provide to --build option of configure script, e.g., x86_64-pc-linux-gnu (determined by config.guess shipped with EasyBuild if None, False implies to leave it up to the configure script) [default: None] +configure_cmd* Configure command to use [default: "./configure"] +configure_cmd_prefix* Prefix to be glued before ./configure [default: ""] +host_type* Value to provide to --host option of configure script, e.g., x86_64-pc-linux-gnu (determined by config.guess shipped with EasyBuild if None, False implies to leave it up to the configure script) [default: None] +install_cmd* Build command to use [default: "make install"] +prefix_opt* Prefix command line option for configure script ('--prefix=' if None) [default: None] +tar_config_opts* Override tar settings as determined by configure. [default: False] +``` + +#### Dependencies + +You will often need to list one or more [dependencies](../introduction/#dependencies) that are required +to install or run the software. +We distinguish between two main different types of dependencies: runtime dependencies and build dependencies. + +*Runtime dependencies* are required for using the installed software, and may also have to be available +during the installation. These dependencies can be specified via the `dependencies` easyconfig parameter. +EasyBuild will load the modules for these dependencies when setting up the build environment, +and will include load statements for them in the generated module file. + +*Build dependencies* are only required during the installation of the software, not for using the +software once it is installed. The modules for these dependencies will be loaded in the build environment +set up by EasyBuild during the installation, but they will *not* be loaded by the generated module file. +You can specify build dependencies via the `builddependencies` easyconfig parameter. +One typical example of a build dependency is `CMake`, which is only needed for configuring +the build. + +Here is a simple example of specifying dependencies: + +```python +builddependencies = [('CMake', '3.18.4')] + +dependencies = [ + ('Python', '3.8.2'), + ('HDF5', '1.10.6'), + ('SciPy-bundle', '2020.03', '-Python-%(pyver)s'), +] +``` + +Both `builddependencies` and `dependencies` require a list of tuples, +each of which specifying one dependency. +The name and version of a dependency is specified with a 2-tuple (a tuple with two string values). + +In some cases additional information may have to be provided, as is shown in the example above for the `SciPy-bundle` +dependency where a 3rd value is specified corresponding to the `versionsuffix` value of this dependency. +If this is not specified, it is assumed to be the empty string (`''`). + +Note how we use the '`%(pyver)s'` template value in the `SciPy-bundle` dependency +specification, to avoid hardcoding the Python version in different places. + +See also the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#dependencies) for additional options on specifying dependencies. + + +#### Version suffix + +In some cases you may want to build a particular software package in different configurations, or include a label in the module name to highlight a particular aspect +of the installation. + +The `versionsuffix` easyconfig parameter can be used for this purpose. The name of this parameter implies that this label will be added after the +software version (and toolchain label) in the standard module naming scheme. + +If you are configuring the software to build with a particular non-default value, +you can indicate this as follows for example: + +```python +versionsuffix = '-example-label' +``` + +This mechanism is used frequently to indicate that a software installation depends +on a particular version of Python, where the `%(pyver)s` template comes in useful again: + +```python +versionsuffix = '-Python-%(pyver)s' +... +dependencies = [('Python', '2.7.18')] +``` + +Even though Python 2 is officially dead and +buried some scientific software still requires it, and mixing modules where +some use Python 2 and other use Python 3 doesn't work well. +The `versionsuffix` label is helpful to inform the user that a particular Python version is required by the installation. + +#### Customizing configure, build, test, and install commands + +When using a generic easyblock like `ConfigureMake` or `CMakeMake`, you will often +find yourself having to specify options to the configure, build, test, or install commands, or to inject additional commands right before them. + +For this the following standard easyconfig parameters are available: + +* `preconfigopts`: string value that is glued *before* the configure command; +* `configopts`: string value that is added *after* the configure command, which can be used to specify configuration options; + +Equivalent easyconfig parameters are available for the `build`, `test` and `install` steps: `prebuildopts`, `buildopts`, `pretestopts`, `testopts`, `preinstallopts`, and `installopts`. + +Here is a fictitious example of how they can be used: + +```python +easyblock = 'ConfigureMake' +... +dependencies = [('HDF5', '1.10.6')] +... +configopts = '--enable-hdf5-support' + +prebuildopts = 'export HDF5_PREFIX="$EBROOTHDF5" && ' + +installopts = "PREFIX='%(installdir)s'" +``` + +Here we are: + +* Adding the `--enable-hdf5-support` configure option, to convince the `ConfigureMake` easyblock to run the following + command during the configure step: + ```shell + ./configure --prefix ... --enable-hdf5-support + ``` + (where the '`...`' represents the path to installation directory where the software should be installed). + +* Specifying that an additional command has to be run before running `make` in the build step. We use '`&&`' to glue the + command to the `make` command, so `make` will only be run if the command we specified ran correctly. So, the build step will run something like: + ```shell + export HDF5_PREFIX="$EBROOTHDF5" && make -j 4 + ``` + The '`4`' value passed to the `-j` option shown here, which specifies how many commands `make` can run in parallel, is automatically determined by EasyBuild based on the number of available cores (taking into account `ulimit` settings, and cpuset and cgroup restrictions). + +* Passing the location where the software should be installed via the `PREFIX` argument to the `make install` command during the installation step. This results in the + following command being run: + ```shell + make install PREFIX=... + ``` + (where the '`...`' again represents the path to installation directory). + Even though the + installation directory is already specified in the configure command, it is + apparently blatantly ignored by the software we are installing here, and we are expected to specify it + this way instead. How rude! + +The `$EBROOTHDF5` environment variable that we are using in `prebuildopts` corresponds to the path of +the installation directory of the HDF5 dependency. EasyBuild includes a statement +to define an `$EBROOT*` environment variable +like this in every environment module file it generates (see the output of "`module show HDF5`"). + +#### Sanity check + +One seemingly trivial yet important aspect of the installation procedure that EasyBuild performs +is the sanity check step. + +By default EasyBuild does a simple sanity check that verifies whether there is a non-empty `bin` subdirectory +in the installation, next to a non-empty `lib` or `lib64` directory (either is sufficient). + +It is recommended to customize the sanity check and check for something more specific, like a particular +binary or directory, or making sure that a trivial command (like `example -V` or `example --help`) +runs correctly. + +To specify a custom set of files and/or directories to check, +you can use the `sanity_check_paths` easyconfig parameter. The expected value is Python dictionary +with two keys: `files` and `dirs`. For example: + +```python +sanity_check_paths = { + 'files': ['bin/example'], + 'dirs': ['examples/one', 'examples/two'], +} +``` + +In addition, you can specify one or more commands that should be working without a problem (that is, have a zero exit status) via the `sanity_check_commands` easyconfig parameter. +These commands will be run just like a user would: after loading the module that was generated for this installation. +Here is an example: + +```python +sanity_check_commands = [ + "example --version", + "example --help", +] +``` + + +#### Module class + +Finally, you will usually see the `moduleclass` easyconfig parameter to be defined as well, for example: + +```python +moduleclass = 'lib' +``` + +This is done to categorize software, and it is used to group the generated module files into smaller sets ([remember what we saw when installing software earlier](../basic_usage/#using-installed-software)). + +## Generating tweaked easyconfigs + +Sometimes you may want to install software that differs only slightly from an +existing easyconfig file, like a newer software version or using a different +compiler toolchain. Do we need to create an easyconfig file for this too? + +We do, but EasyBuild does provide some help so you don't need to *manually* +create the easyconfig file. You can use one of the `--try-*` options provided +by the `eb` command to make EasyBuild *generate* a new easyconfig file based on +an existing one. + +For example, to try installing a different software version you can use the `--try-software-version` option: + +```shell +eb example-1.2.3.eb --try-software-version 1.2.4 +``` + +Or, to try using a different compiler toolchain you can use `--try-toolchain`: + +```shell +eb example-1.2.3-foss-2020b.eb --try-toolchain intel,2020b +``` + +It is important to keep in mind the *"try"* aspect here: while easyconfigs that +are generated by EasyBuild via a `--try-*` option often do work fine, there is +no strong guarantee they will. Newer software versions may come with changes to +the installation procedure, additional dependencies that are required, etc. +Using a different compiler toolchain may be as simple as just switching one for +another, but it may require additional changes to be made to configure options, for example. + +## Copying easyconfigs + +One additional handy command line option we want to highlight is `--copy-ec`, which can be used to +copy easyconfig files to a specific location. That may sound trivial, but +keep in mind that you can specify easyconfigs to the `eb` command using only +the filename, and letting the robot search mechanism locate them. + +So to copy an easyconfig file, we would have to use `eb --search` first to +get the full location to it, copy-paste that, and then use the `cp` command. + +It is a lot easier with `--copy-ec`: + +```shell +$ eb --copy-ec SAMtools-1.11-GCC-10.2.0.eb SAMtools.eb +... +SAMtools-1.10-GCC-10.2.0.eb copied to SAMtools.eb +``` + +If you omit the target location, the easyconfig file will simply be copied +to the current working directory, retaining the original filename. + +You can copy multiple easyconfig files at once, as long as the target location +is an existing directory. + +## Example + +By means of example, we are going to puzzle together an easyconfig file to install the +example software package `eb-tutorial`. + +The sources for `eb-tutorial` version 1.0.1 are available at: + +``` +https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.0.1.tar.gz +``` + +You can consult the unpacked sources at [https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1). + +### Preparation + +Make sure EasyBuild is properly configured before you start: + +```shell +export EASYBUILD_PREFIX=$HOME/easybuild +export EASYBUILD_BUILDPATH=/tmp/$USER +``` + +and that the installed software in `/easybuild` is available: + +```shell +module use /easybuild/modules/all +``` + +### Mandatory parameters + +Let's start by getting the mandatory easyconfig parameters defined in the easyconfig file: + +```python +name = 'eb-tutorial' +version = '1.0.1' + +homepage = 'https://easybuilders.github.io/easybuild-tutorial' +description = "EasyBuild tutorial example" +``` + +If we try using this (very) minimal easyconfig file, EasyBuild will inform us that we failed to specify one of the mandatory easyconfig parameters: `toolchain`: + +``` +$ eb example.eb +== Temporary log file in case of crash /tmp/eb-90j723rl/easybuild-q21plqvx.log +== found valid index for /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs, so using it... +ERROR: Failed to process easyconfig /home/example/example.eb: mandatory parameters not provided in pyheader: toolchain +``` + +We will use `GCC/10.2.0` as toolchain, since we know it is already installed in `/easybuild` in the prepared environment, so we also define the `toolchain` easyconfig parameter: + +```python +toolchain = {'name': 'GCC', 'version': '10.2.0'} +``` + +In addition, we'll also specify the `moduleclass`. +This is not required, but it is usually set to a sensible value: + +```python +moduleclass = 'tools' +``` + +The default value is '`base`', at least '`tools`' has *some* meaning. + +### Easyblock + +Let us see what happens if we take our current easyconfig file for a spin: + +```shell +$ eb example.eb +== temporary log file in case of crash /tmp/eb-8_vxjfn7/easybuild-k3aaoan2.log +ERROR: Failed to process easyconfig /home/example/example.eb: +No software-specific easyblock 'EB_eb_minus_tutorial' found for eb-tutorial +``` + +That didn't get us very far... + +The error shows that there is no software-specific easyblock available for installing the software with the name '`eb-tutorial`'. +Does that mean we have to implement an easyblock? + +In this simple case it doesn't, since we can leverage one of the available *generic easyblocks*. +But, which one? + +Build instructions are usually included in a `README` file, or in the documentation. +In this case, there's indeed a minimal [`README` +file](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1/README) available, +which tells us that we should use the `cmake` command to configure the installation, followed by `make` and `make install`. + +[We briefly discussed](#easyblock) a generic easyblock that does exactly this: `CMakeMake`. + +```python +easyblock = 'CMakeMake' +``` + +The "`easyblock =`" line is usually at the top of the easyconfig file, but strictly speaking +the order of the parameter definitions doesn't matter (unless one is defined in terms of another one). + +### CMake build dependency + +Does using the `CMakeMake` generic easyblock help at all? + +``` +$ eb example.eb +== temporary log file in case of crash /tmp/eb-yutbor1p/easybuild-4jc9v1u9.log +== found valid index for /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs, so using it... +== processing EasyBuild easyconfig /home/example/example.eb +== building and installing eb-tutorial/1.0.1-GCC-10.2.0... +== fetching files... +== creating build dir, resetting environment... +== unpacking... +== patching... +== preparing... +== configuring... +== FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/ebtutorial/1.0.1/GCC-10.2.0): +build failed (first 300 chars): cmd " cmake -DCMAKE_INSTALL_PREFIX=/home/example/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER='gcc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno' -DCMAKE_CXX_COMPILER='g++' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=native (took 0 sec) +``` + +It did help: EasyBuild made an attempt to configure the build using the `cmake` command, but that failed almost +instantly. We need to dive into the log file to see the actual reason. By starting at the end of the log file and +scrolling up, you should be able to locate the following error message: + +``` +/bin/bash: cmake: command not found +``` + +Ah, that explains it, `cmake` isn't even installed on this system. Or is it? + +```shell +$ module avail CMake + +--------------------------- /easybuild/modules/all ---------------------------- + CMake/3.18.4-GCCcore-10.2.0 +``` + +Since a module is available for `CMake` that is compatible with the toolchain we are using (GCC 10.2.0), +we can use it as a dependency for the installation. +It is only needed for building the software, not for running it, so it's only a *build* dependency: + +```python +builddependencies = [('CMake', '3.18.4')] +``` + +There is usually no need to specify toolchain for (build) dependencies, EasyBuild will automatically consider +[subtoolchains](../terminology/#toolchains) compatible with the specified toolchain to locate module for the dependencies. + +You can verify this via `eb -D` (equivalent with `eb --dry-run`): + +``` +$ eb example.eb -D + ... + * [x] /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs/g/GCC/GCC-10.2.0.eb (module: GCC/10.2.0) + * [x] /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs/c/CMake/CMake-3.18.4-GCCcore-10.2.0.eb (module: CMake/3.18.4-GCCcore-10.2.0) + * [ ] /home/example/example.eb (module: eb-tutorial/1.0.1-GCC-10.2.0) +``` + +### Sources + +If you try again after adding `CMake` as a build dependency, you will see the installation fail again in the +configuration step. Inspecting the log file reveals this: + +``` +CMake Error: The source directory "/tmp/example/ebtutorial/1.0.1/GCC-10.2.0" does not appear to contain CMakeLists.txt. +``` + +Wait, but there *is* a `CMakeLists.txt`, we can see it in the [unpacked sources](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1)! + +Let's inspect the build directory: + +``` +$ ls /tmp/$USER/ebtutorial/1.0.1/GCC-10.2.0 +easybuild_obj +$ ls /tmp/$USER/ebtutorial/1.0.1/GCC-10.2.0/easybuild_obj +$ +``` + +There's nothing there at all! And that's not strange because we didn't actually +specify any sources in our easyconfig file... + +The `sources` easyconfig parameter is commonly defined but it is *not* mandatory, +because some easyconfig files only specify bundles of software packages and hence only +serve to generate a module file. + +So we need to specify one or more source files that should be used, +via the `sources` easyconfig parameter which specifies a *list* of +names of source files: + +```python +sources = ['eb-tutorial-1.0.1.tar.gz'] +``` + +We can avoid hardcoding the version number here by using a *template value*: + +```python +sources = ['eb-tutorial-%(version)s.tar.gz'] +``` + +And since this is a standard way of naming software files, there's +even a constant available that we can use: + +```python +sources = [SOURCE_TAR_GZ] +``` + +That way, we only have the software version specified *once* in the easyconfig file, +via the `version` easyconfig parameter. That will come in useful later (see [Exercise 7.2](#exercises))... + +If now we try installing the easyconfig file again, EasyBuild complains +that it can't find the specified source file anywhere: + +``` +Couldn't find file eb-tutorial-1.0.1.tar.gz anywhere, and downloading it didn't work either... +``` + +To let EasyBuild automatically download the source file if it is not available yet, +we have to specify *where* it can be downloaded. This is done via `source_urls`: + +```python +source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] +sources = [SOURCE_TAR_GZ] +``` + +### Required configure option + +With `sources` and `source_urls` defined, we can try again. Yet again we see the configure step fail. +Is this a ruse to make you hate CMake with a passion? Maybe... + +Here's what we find in the log file: + +``` +CMake Error at CMakeLists.txt:7 (message): + EBTUTORIAL_MSG is not set! +``` + +Apparently the `eb-tutorial` software has a required configure option. It's almost as if that +was done on purpose, how silly! + +Options to the configure command can be specified by the `configopts` easyconfig parameter. +To define the value of a CMake option, we need to use `-DNAME_OF_OPTION`, so: + +```python +configopts = "-DEBTUTORIAL_MSG='Hello from the EasyBuild tutorial!' " +``` + +We need to be a little bit careful with quotes here. If we use outer double quotes, +we have to use single quotes to specify the actual value for the `EBTUTORIAL_MSG` configure option. +That works fine here, but that's not always the case! +In some cases we will have to use inner doubles quotes, for example to get environment variables +expanded when the configure command is run (see [Exercise 7.1](#exercises)). + +### Sanity check + +Hopefully that brings us closer to getting the installation to work... + +``` +$ eb example.eb +.... +== sanity checking... +== FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/ebtutorial/1.0.1/GCC-10.2.0): build failed (first 300 chars): +Sanity check failed: no (non-empty) directory found at 'lib' or 'lib64' in /home/easybuild/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 (took 2 sec) +``` + +It got all the way to the sanity check step, that's great! + +The sanity check failed because no '`lib`' or `'lib64'` directory was found. +Indeed: + +``` +$ ls $HOME/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 +bin +$ ls $HOME/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0/bin +eb-tutorial +``` + +There is only a binary named `eb-tutorial` in the `bin` subdirectory. +So we need to customize the standard sanity check: + +```python +sanity_check_paths = { + 'files': ['bin/eb-tutorial'], + 'dirs': [], +} +``` + +Since we want to obtain a *working* installation, we might as well try to run this `eb-tutorial` command as well: + +```python +sanity_check_commands = ['eb-tutorial'] +``` + +Let us now retry, but use `--module-only` rather than redoing the whole installation. +`--module-only` still sanity checks the installation, so if it creates +a module, we know it will work as expected. +By enabling trace mode via `--trace` we can get some more information too: + +```shell +$ eb example.eb --module-only --trace +... +== sanity checking... + >> file 'bin/eb-tutorial' found: OK + >> running command 'eb-tutorial' ... + >> result for command 'eb-tutorial': OK +... +== COMPLETED: Installation ended successfully (took 4 sec) +``` + +Yes, great success! + +To convince yourself that the installation works as intended, try to load the `eb-tutorial` module and +run the `eb-tutorial` command yourself: + +``` +$ module use $HOME/easybuild/modules/all +$ module load eb-tutorial +$ eb-tutorial +Hello from the EasyBuild tutorial! +``` + +### Complete easyconfig + +Here is the complete easyconfig we puzzled together for this example: + +```python +easyblock = 'CMakeMake' + +name = 'eb-tutorial' +version = '1.0.1' + +homepage = 'https://easybuilders.github.io/easybuild-tutorial' +description = "EasyBuild tutorial example" + +toolchain = {'name': 'GCC', 'version': '10.2.0'} + +source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] +sources = [SOURCE_TAR_GZ] +checksums = ['d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9'] + +builddependencies = [('CMake', '3.18.4')] + +configopts = "-DEBTUTORIAL_MSG='Hello from the EasyBuild tutorial!' " + +sanity_check_paths = { + 'files': ['bin/eb-tutorial'], + 'dirs': [], +} + +sanity_check_commands = ['eb-tutorial'] + +moduleclass = 'tools' +``` + +One additional easyconfig parameter we defined here is `checksums`, +which specifies SHA256 checksums for source (and patch) files, +so EasyBuild can verify them before performing an installation. + +You can let EasyBuild determine *and* inject these SHA256 checksums +automatically via `eb --inject-checksums`: + +``` +$ eb example.eb --inject-checksums +... +== injecting sha256 checksums for sources & patches in example.eb... +== * eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 +``` + +--- + +## Exercises + +--- + +***Exercise E.1**** - Making `eb-tutorial` a bit more personal* + +Change the easyconfig file for `eb-tutorial` to make the message printed by the `eb-tutorial` command +a bit more personal: include the username of the account that was used to install the software in it +(using the `$USER` environment variable). + +??? success "(click to show solution)" + For this we need to change the value that is passed to the `EBTUTORIAL_MSG` configure option: + ```python + configopts = '-DEBTUTORIAL_MSG="Hello from the EasyBuild tutorial! I was installed by $USER." ' + ``` + Here we have to use inner double quotes, to ensure that the `$USER` environment variable is expanded + by the shell when running the `cmake` configure command. + + When you run the `eb-tutorial` command yourself, you should get output like this (not a message that + includes a literal '`$USER`' string): + + ```shell + Hello from the EasyBuild tutorial! I was installed by example. + ``` + + To re-install the `example.eb` easyconfig, you will need to use `eb --rebuild` or `eb --force`. + +--- + +***Exercise E.2**** - Installing eb-tutorial version 1.1.0* + +Install version 1.1.0 of the `eb-tutorial` example software, +which is a trivial version bump compared to version 1.0.1. + +The sources are available via: + +``` +https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.1.0.tar.gz +``` + +You can leverage the `eb-tutorial` easyconfig file we have composed in the example above, +**but you should not make any manual changes to it!** + +??? success "(click to show solution)" + You can use the `--try-software-version` option for this: + ```shell + $ eb example.eb --try-software-version 1.1.0 + ... + == building and installing eb-tutorial/1.1.0-GCC-10.2.0... + ... + == COMPLETED: Installation ended successfully (took 4 sec) + ``` + To test: + ``` + $ module use $HOME/easybuild/modules/all + $ module load eb-tutorial/1.1.0-GCC-10.2.0 + $ eb-tutorial + I have a message for you: + Hello from the EasyBuild tutorial! + ``` + (`eb-tutorial` version 1.0.1 doesn't print "`I have a message for you:`") + +--- + +***Exercise E.3**** - Installing py-eb-tutorial 1.0.0* + +Try composing an easyconfig file for the `py-eb-tutorial` example software, which is a tiny Python package. +The source tarball can be downloaded from this link: [py-eb-tutorial-1.0.0.tar.gz](https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/py-eb-tutorial-1.0.0.tar.gz). + +A couple of tips: + +* There is a [generic easyblock](#easyblock) available for installing Python packages, which will come in useful here. + +* By default EasyBuild performs an `import` check when install Python packages, using a Python module name that is derived from the software name by default, which will be incorrect in this case. You can specify the correct name to use in the import check by specifying it via the `options` +easyconfig parameter in your easyconfig file: + ```python + options = {'modulename': 'example'} + ``` + (you will need to change '`example`' here, of course) + +* Leverage the software that is already pre-installed in `/easybuild` in the prepared environment. + Remember that some already installed modules may be a *bundle* of a couple of other software packages. + +Please also take this into account: + +* Unfortunately this software doesn't come with documentation. That is done to make it an example that + is representative for software that you may run into in the wild (it's *not* because + we were lazy when preparing the exercises, really!). + You can inspect the sources of this software [here](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/py-eb-tutorial-1.0.0). Definitely take a look at the `setup.py` file, it includes some clues + about the requirements to get this software installed. + +* Make sure the installation actually *works*, by checking that the `py-eb-tutorial` command runs correctly. + Maybe you will need to make sure other required software is available as well, for it to work correctly... + + +??? success "(click to show solution)" + + Here is a complete working easyconfig file for `py-eb-tutorial`: + ```python + + easyblock = 'PythonPackage' + + name = 'py-eb-tutorial' + version = '1.0.0' + versionsuffix = '-Python-%(pyver)s' + + homepage = 'https://easybuilders.github.io/easybuild-tutorial' + description = "EasyBuild tutorial Python example" + + source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] + sources = [SOURCE_TAR_GZ] + checksums = ['fcf73a9efc65527a210b993e8889d41ebf05977eef1f6a65ebac3188152cd496'] + + toolchain = {'name': 'foss', 'version': '2020b'} + + dependencies = [ + ('Python', '3.8.6'), + ('SciPy-bundle', '2020.11'), + ('eb-tutorial', '1.0.1'), + ] + + use_pip = True + + options = {'modulename': 'eb_tutorial'} + + sanity_check_paths = { + 'files': ['bin/py-eb-tutorial'], + 'dirs': ['lib/python%(pyshortver)s/site-packages'], + } + + sanity_check_commands = ["py-eb-tutorial"] + + moduleclass = 'tools' + ``` + + Some remarks: + + * We used the `PythonPackage` generic easyblock. There is also a `PythonBundle` easyblock for installing + bundles of Python packages, which is used for `SciPy-bundle` for example. But we don't need that here, + since we are only dealing with a single Python package. + + * The `versionsuffix` is not strictly needed, but it's common to tag Python packages with the Python version + for which they were installed. + * The SHA256 checksum for the source tarball was added automatically via `eb py-eb-tutorial.eb --inject-checksums`. + + * `py-eb-tutorial` only wants to be installed with `pip install`, so we had to set `use_pip = True`. + You can consult the custom easyconfig parameters supported by the `PythonPackage` easyblock via + "`eb -a -e PythonPackage`", see the `EASYBLOCK-SPECIFIC` part of the output. + Even when the default installation mechanism used by `PythonPackage` + (which consists of running `python setup.py install`) works fine, + it is recommended to instruct EasyBuild to use `pip install` instead. + + * By default EasyBuild will try to import `py_eb_tutorial`, while the actual name of the Python package + provided by `py-eb-tutorial` is just `eb_tutorial`. We fixed this by specifying the correct Python module name to + use via `options`. + + * Strictly speaking we don't need to specify a custom `sanity_check_paths`, since the default used + by Python package is already pretty decent (it will check for a non-empty `lib/python3.8/site-packages` + directory in the installation). We also want to make sure the `py-eb-tutorial` command is available in + the `bin` subdirectory however. Hardcoding to `python3.8` can be avoided using the `%(pyshortver)s` + template value. + + * A good way to check whether the `py-eb-tutorial` command works correctly is by running it as a sanity check + command. If the `eb-tutorial` command is not available the `py-eb-tutorial` command will fail, + since it basically just runs the `eb-tutorial` command. So we need to include `eb-tutorial` as a (runtime) + dependency in the `py-eb-tutorial` easyconfig file. + +*[[next: Implementing easyblocks]](2_03_implementing_easyblocks.md)* diff --git a/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md new file mode 100644 index 00000000..5355548e --- /dev/null +++ b/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md @@ -0,0 +1,665 @@ +# Implementing easyblocks + +*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* + +--- + +## The basics + +An *easyblock* is a Python module that implements a software build and installation procedure. + +This concept is essentially implemented as a Python script that plugs into the EasyBuild framework. + +EasyBuild will leverage easyblocks as needed, depending on which software packages it needs to install. Which easyblock is required is determined by the ``easyblock`` easyconfig parameter, if it is present, or by the software name. + + +## Generic vs software-specific easyblocks + +Easyblocks can either be *generic* or *software-specific*. + +*Generic* easyblocks implement a "standard" software build and installation procedure that is used by multiple different +software packages. +A commonly used example is the +[``ConfigureMake``](https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/generic/configuremake.py) +generic easyblock, which implements the standard ``configure`` - ``make`` - ``make install`` installation procedure used +by most GNU software packages. + +*Software-specific* easyblocks implement the build and installation procedure for a particular software package. +Typically this involves highly customised steps, for example specifying dedicated configuration options, creating +or adjusting specific files, executing non-standard shell commands, etc. Usually a custom implementation of the +sanity check is also included. + +Using a generic easyblock requires specifying the ``easyblock`` parameter in the easyconfig file. +If it is not specified, EasyBuild will try and find the software-specific easyblock derived from the software name. + +The distinction between generic and software-specific easyblocks can be made based on the naming scheme that is used +for an easyblock (see below). + + +## Naming + +Easyblocks need to follow a strict naming scheme, to ensure that EasyBuild can pick them up as needed. This involves two aspects: + +* the name of the Python class; +* the name and location of the Python module file. + +### Python class name + +The name of the Python class is determined by the *software name* for software-specific easyblocks. +It consists of a prefix '``EB_``', followed by the (encoded) software name. + +Because of limitations in Python on characters allowed in names of Python classes, +only alphanumeric characters and underscores (``_``) are allowed. Any other characters are replaced following an encoding scheme: + +* spaces are replaced by underscores (``_``); +* dashes ``-`` are replaced by ``_minus_``; +* underscores are replaced by ``_underscore_``; + +The ``encode_class_name`` function provided in ``easybuild.tools.filetools`` returns the expected class name +for a given software name; for example: + +```shell +$ python3 -c "from easybuild.tools.filetools import encode_class_name; print(encode_class_name('netCDF-Fortran'))" +EB_netCDF_minus_Fortran +``` + +**Python class name for *generic* easyblocks** + +For *generic* easyblocks, the class name does *not* include an ``EB_`` prefix (since there is no need for an escaping +mechanism) and hence the name is fully free to choose, taking into account the restriction to alphanumeric characters +and underscores. + +For code style reasons, the class name should start with a capital letter and use CamelCasing. + +Examples include ``Bundle``, ``ConfigureMake``, ``CMakePythonPackage``. + +### Python module name and location + +The *filename* of the Python module is directly related to the name of Python class it provides. + +It should: + +* *not* include the ``EB_`` prefix of the class name for software-specific easyblocks; +* consists only of lower-case alphanumeric characters (``[a-z0-9]``) and underscores (``_``); + * dashes (``-``) are replaced by underscores (``_``); + * any other non-alphanumeric characters (incl. spaces) are simply dropped; + +Examples include: + +* ``gcc.py`` (for *GCC*) +* ``netcdf_fortran.py`` (for *netCDF-Fortran*) +* ``gamess_us.py`` (for *GAMESS (US)*) + +The ``get_module_path`` function provided by the EasyBuild framework in the +``easybuild.framework.easyconfig.easyconfig`` module returns the (full) +module location for a particular software name or easyblock class name. For example: + +```python +>>> from easybuild.framework.easyconfig.easyconfig import get_module_path +>>> get_module_path('netCDF-Fortran') +'easybuild.easyblocks.netcdf_fortran' +>>> get_module_path('EB_netCDF_minus_Fortran') +'easybuild.easyblocks.netcdf_fortran' +``` + +The location of the Python module is determined by whether the easyblock is generic or software-specific. +Generic easyblocks are located in the ``easybuid.easyblocks.generic`` namespace, while software-specific easyblocks +live in the ``easybuild.easyblocks`` namespace directly. + +To keep things organised, the actual Python module files +for software-specific easyblocks are kept in 'letter' subdirectories, +rather than in one large '``easyblocks``' directory +(see +[https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/](https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/)), +but this namespace is collapsed transparently by EasyBuild (you don't need to import from letter subpackages). + +To let EasyBuild pick up one or more new or customized easyblocks, you can use the [``--include-easyblocks``](https://docs.easybuild.io/en/latest/Including_additional_Python_modules.html#including-additional-easyblocks-include-easyblocks) +configuration option. As long as both the filename of the Python module and the name of the Python class +are correct, EasyBuild will use these easyblocks when needed. + +## Structure of an easyblock + +The example below shows the overal structure of an easyblock: + +```python +from easybuild.framework.easyblock import EasyBlock +from easybuild.tools.run import run_cmd + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def configure_step(self): + """Custom implementation of configure step for Example""" + + # run configure.sh to configure the build + run_cmd("./configure.sh --install-prefix=%s" % self.installdir) +``` + + +Each easyblock includes an implementation of a ``class`` that (directly or indirectly) derives from the abstract +``EasyBlock`` class. + +Typically some useful functions provided by the EasyBuild framework are imported at the top of the Python module. + +In the class definition, one or more '``*_step``' methods (and perhaps a couple of others) are redefined, +to implement the corresponding step in the build and installation procedure. + +Each easyblock *must* implement the ``configure``, ``build`` and ``install`` steps, since these are not implemented +in the abstract ``EasyBlock`` class. This could be done explicitly by redefining the corresponding ``*_step`` methods, +or implicitly by deriving from existing (generic) easyblocks. + +The full list of methods that can be redefined in an easyblock can be consulted in +the [API documentation](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyblock.html). + +## Deriving from existing easyblocks + +When implementing an easyblock, it is common to derive from an existing (usually generic) easyblock, +and to leverage the functionality provided by it. This approach is typically used when only a specific part +of the build and installation procedure needs to be customised. + +In the (fictitious) example below, we derive from the generic ``ConfigureMake`` easyblock to redefine the ``configure`` +step. In this case, we are *extending* the ``configure`` step as implemented by ``ConfigureMake`` rather than +redefining it entirely, since we call out to the original ``configure_step`` method at the end. + +```python +from easybuild.easyblocks.generic.configuremake import ConfigureMake +from easybuild.tools.filetools import copy_file + + +class EB_Example(ConfigureMake): + """Custom easyblock for Example""" + + def configure_step(self): + """Custom implementation of configure step for Example""" + + # use example make.cfg for x86-64 + copy_file('make.cfg.x86', 'make.cfg') + + # call out to original configure_step implementation of ConfigureMake easyblock + super(EB_Example, self).configure_step() +``` + +## Easyconfig parameters + +All of the easyconfig parameters that are defined in an easyconfig file +are available via the ``EasyConfig`` instance that can be accessed through ``self.cfg`` in an easyblock. +For instance, if the easyconfig file specifies + +```python +name = 'example' +version = '2.5.3' +versionsuffix = '-Python-3.7.4' +``` + +then these three parameters are accessible within an easyblock via ``self.cfg['name']``, ``self.cfg['version']`` +and ``self.cfg['versionsuffix']``. + +A few of the most commonly used parameters can be referenced directly: + +* **``self.name``** is equivalent with ``self.cfg['name']``; +* **``self.version``** is equivalent with ``self.cfg['version']``; +* **``self.toolchain``** is equivalent with ``self.cfg['toolchain']``; + + +### Updating parameters + +You will often find that you need to *update* some easyconfig parameters in an easyblock, +for example ``configopts`` which specifies options for the configure command. + +Because of implementation details (related to +how template values like ``%(version)s`` are handled), you need to be a bit careful here... + +To completely redefine the value of an easyconfig parameter, you can use simple assignment. For example: + +```python +self.cfg['example'] = "A new value for the example easyconfig parameter." +``` + +If want to *add* to the existing value however, you *must* use the ``self.cfg.update`` method. For example: + +```python +self.cfg.update('some_list', 'example') +``` + +This will ***not*** work (because ``self.cfg['some_list']`` does not return a reference to the original value, +but to a *temporary copy* thereof): + +```python +# anti-pattern, this does NOT work as expected! +self.cfg['some_list'].append('example') +``` + +### Custom parameters + +Additional custom easyconfig parameters can be defined in an easyblock to allowing steering its behaviour. +This is done via the ``extra_options`` *static* method. Custom parameters can be specified to be mandatory. + +The example below shows how this can be implemented: + +```python +from easybuild.easyblocks.generic.configuremake import ConfigureMake +from easybuild.framework.easyconfig import CUSTOM, MANDATORY + + +class EB_Example(ConfigureMake): + """Custom easyblock for Example""" + + @staticmethod + def extra_options(): + """Custom easyconfig parameters for Example""" + extra_vars = { + 'required_example_param': [None, "Example required custom parameter", MANDATORY], + 'optional_example_param': [None, "Example optional custom parameter", CUSTOM], + } + return ConfigureMake.extra_options(extra_vars) +``` + +The first element in the list of a defined custom parameter corresponds to the default value for that parameter +(both ``None`` in the example above). The second element provides some informative help text, and the last element +indicates whether the parameter is mandatory (``MANDATORY``) or just an optional custom parameter (``CUSTOM``). + +## Easyblock constructor + +In the ``class`` constructor of the easyblock, i.e. the ``__init__`` method, one or more class variables +can be initialised. These can be used for sharing information between different ``*_step`` methods in the easyblock. + +For example: + +```python +from easybuild.framework.easyblock import EasyBlock + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def __init__(self, *args, **kwargs): + """Constructor for Example easyblock, initialises class variables.""" + + # call out to original constructor first, so 'self' (i.e. the class instance) is initialised + super(EB_Example, self).__init__(*args, **kwargs) + + # initialise class variables + self.example_value = None + self.example_list = [] +``` + +## File operations + +File operations is a common use case for implementing easyblocks, hence the EasyBuild framework provides a +number of useful functions related to this, including: + +* ``read_file()``: read file at a specified location and returns its contents; + +* ``write_file(, )`` at a specified location with provided contents; + to append to an existing file, use ``append=True`` as an extra argument; + +* ``copy_file(, )`` to copy an existing file; + +* ``apply_regex_substitutions(, )`` to patch an existing file; + +All of these functions are provided by the [``easybuild.tools.filetools``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html) module. + +## Executing shell commands + +For executing shell commands two functions are provided by the +[``easybuild.tools.run``](https://docs.easybuild.io/en/latest/api/easybuild.tools.run.html) module: + +* ``run_cmd()`` to run a non-interactive shell command; + +* ``run_cmd_qa(, )`` to run an interactive shell command; + +Both of these accept a number of optional arguments: + +* ``simple=True`` to just return ``True`` or ``False`` to indicate a successful execution, + rather than the default return value, i.e., a tuple that provides the command output and the exit code (in that + order); + +* ``path=`` to run the command in a specific subdirectory; + +The ``run_cmd_qa`` function takes two additional specific arguments: + +* ``no_qa=`` to specify a list of patterns to recognize non-questions; + +* ``std_qa=`` to specify regular expression patterns for common questions, and the matching answer; + +## Manipulating environment variables + +To (re)define environment variables, the ``setvar`` function provided by the +[``easybuild.tools.environment``](https://docs.easybuild.io/en/latest/api/easybuild.tools.environment.html) +module should be used. + +This makes sure that the changes being made to the specified environment variable are kept track of, +and that they are handled correctly under ``--extended-dry-run``. + +## Logging and errors + +It is good practice to include meaningful log messages in the ``*_step`` methods being customised in the easyblock, +to enrich the EasyBuild log with useful information for later debugging or diagnostics. + +For logging, the provided ``self.log`` logger class should be used. +You can use the ``self.log.info`` method to log an informative message. +Similar methods are available for logging debug messages (``self.log.debug``), which are +only emitted when ``eb`` is run with debugging mode enabled (``--debug`` or ``-d``), +and for logging warning messages (``self.log.warning``). + +If something goes wrong, you can raise an ``EasyBuildError`` instance to report the error. + +For example: + +```python +from easybuild.framework.easyblock import EasyBlock +from easybuild.tools.build_log import EasyBuildError +from easybuild.tools.run import run_cmd + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def configure_step(self): + """Custom implementation of configure step for Example""" + + cmd = "./configure --prefix %s" % self.installdir) + out, ec = run_cmd(cmd) + + success = 'SUCCESS' + if success in out: + self.log.info("Configuration command '%s' completed with success." % cmd) + else: + raise EasyBuildError("Pattern '%s' was not found in output of '%s'." % (success, cmd)) +``` + +## Custom sanity check + +For software-specific easyblocks, a custom sanity check is usually included to verify that the installation was +successful or not. + +This is done by redefining the ``sanity_check_step`` method in the easyblock. For example: + +```python +from easybuild.framework.easyblock import EasyBlock + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def sanity_check_step(self): + """Custom sanity check for Example.""" + + custom_paths = { + 'files': ['bin/example'], + 'dirs': ['lib/examples/'], + } + custom_commands = ['example --version'] + + # call out to parent to do the actual sanity checking, pass through custom paths and commands + super(EB_Example, self).sanity_check_step(custom_paths=custom_paths, custom_commands=custom_commands) +``` + +You can both specify file paths and subdirectories to check for, which are specified relative to the installation directory, +as well as simple commands that should execute successfully after completing the installation and loading the generated module file. + +It is up to you how extensive you make the sanity check, but it is recommended to make the check as complete +as possible to catch any potential build or installation problems that may occur, while ensuring that it can +run relatively quickly (in seconds, or at most a couple of minutes). + +## Version-specific parts + +In some cases version-specific actions or checks need to be included in an easyblock. +For this, it is recommended to use ``LooseVersion`` rather than directly comparing version numbers using string values. + +For example: + +```python +from distutils.version import LooseVersion +from easybuild.framework.easyblock import EasyBlock + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def sanity_check_step(self): + """Custom sanity check for Example.""" + + custom_paths = { + 'files': [], + 'dirs': [], + } + + # in older versions, the binary used to be named 'EXAMPLE' rather than 'example' + if LooseVersion(self.version) < LooseVersion('1.0'): + custom_paths['files'].append('bin/EXAMPLE') + else: + custom_paths['files'].append('bin/example') + + super(EB_Example, self).sanity_check_step(custom_paths=custom_paths) +``` + +## Compatibility with ``--extended-dry-run`` and ``--module-only`` + +Some special care must be taken to ensure that an easyblock is fully compatible with ``--extended-dry-run`` / ``-x`` +(see [Inspecting install procedures](../basic_usage/#inspecting-install-procedures)) and ``--module-only``. + +For compatibility with ``--extended-dry-run``, you need to take into account that specified operations +like manipulating files or running shell commands will not actually be executed. You can check +whether an easyblock is being run in dry run mode via ``self.dry_run``. + +For example: + +```python +from easybuild.framework.easyblock import EasyBlock +from easybuild.tools.build_log import EasyBuildError +from easybuild.tools.run import run_cmd + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def configure_step(self): + """Custom implementation of configure step for Example""" + + cmd = "./configure --prefix %s" % self.installdir) + out, ec = run_cmd(cmd) + + success = 'SUCCESS' + if success in out: + self.log.info("Configuration command '%s' completed with success." % cmd) + + # take into account that in dry run mode we won't get any output at all + elif self.dry_run: + self.log.info("Ignoring missing '%s' pattern since we're running in dry run mode." % success) + + else: + raise EasyBuildError("Pattern '%s' was not found in output of '%s'." % (success, cmd)) +``` + +For ``--module-only``, you should make sure that both the ``make_module_step``, including the ``make_module_*`` submethods, +and the ``sanity_check_step`` methods do not make any assumptions about the presence of certain environment variables, or that class variables have been defined already. + +This is required because under ``--module-only`` the large majority of the ``*_step`` functions are +simply *skipped* entirely. So, if the ``configure_step`` method is responsible for defining class variables that are +picked up in ``sanity_check_step``, the latter may run into unexpected initial values like ``None``. +A possible workaround is to define a separate custom method to define the class variables, and to call out to this +method from ``configure_step`` and ``sanity_check_step`` (for the latter, conditionally, i.e., only if the class +variables still have the initial values). + +For example: + +```python +from easybuild.framework.easyblock import EasyBlock + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def __init__(self, *args, **kwargs): + """Easyblock constructor.""" + super(EB_Example, self).__init__(*args, **kwargs) + + self.command = None + + def set_command(self): + """Initialize 'command' class variable.""" + # $CC environment variable set by 'prepare' step determines exact command + self.command = self.name + '-' + os.getenv('CC') + + def configure_step(self): + """Custom configure step for Example.""" + + self.set_command() + self.cfg.update('configopts', "COMMAND=%s" % self.command) + + super(EB_Example, self).configure_step() + + def sanity_check_step(self): + """Custom implementation of configure step for Example""" + + if self.command is None: + self.set_command() + + super(EB_Example, self).sanity_check_step(custom_commands=[self.command]) +``` + +## Exercise + +### Exercise I.1 + +Try implementing a new custom easyblock for ``eb-tutorial``, which derives directly +from the base ``EasyBlock`` class. + +Your easyblock should: + +* define a custom mandatory easyconfig parameter named ``message``; +* run `cmake` to configure the installation, which includes at least: + * specifying the correct installation prefix (using the `-DCMAKE_INSTALL_PREFIX=...` option); + * passing down the value of ``message`` easyconfig parameter via `-DEBTUTORIAL_MSG=...` +* run `make` to build `eb-tutorial`; +* run `make install` to install the generated binary; +* perform a custom sanity check to ensure the installation is correct; +* pick up on commonly used easyconfig parameters like `configopts` and `preinstallopts` where appropriate; + +??? success "(click to show solution)" + + Here's a complete custom easyblock for ``eb-tutorial`` that derives from the base ``EasyBlock`` class, + which should be included in a file named ``eb_tutorial.py``. + + We need to implement the ``configure_step``, ``build_step``, and ``install_step`` methods in + order to have a fully functional easyblock. + + The configure, build, and install steps take into account the corresponding easyconfig + parameters that allow customizing these commands from an easyconfig file. + + ```python + from easybuild.framework.easyblock import EasyBlock + from easybuild.framework.easyconfig import MANDATORY + from easybuild.tools.run import run_cmd + + + class EB_eb_minus_tutorial(EasyBlock): + """Custom easyblock for eb-tutorial.""" + + @staticmethod + def extra_options(): + extra = EasyBlock.extra_options() + extra.update({ + 'message': [None, "Message that eb-tutorial command should print", MANDATORY], + }) + return extra + + def configure_step(self): + """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" + + cmd = ' '.join([ + self.cfg['preconfigopts'], + "cmake", + "-DCMAKE_INSTALL_PREFIX='%s'" % self.installdir, + "-DEBTUTORIAL_MSG='%s'" % self.cfg['message'], + self.cfg['configopts'], + ]) + run_cmd(cmd) + + def build_step(self): + """Build step for eb-tutorial""" + + cmd = ' '.join([ + self.cfg['prebuildopts'], + "make", + self.cfg['buildopts'], + ]) + run_cmd(cmd) + + def install_step(self): + """Install step for eb-tutorial""" + + cmd = ' '.join([ + self.cfg['preinstallopts'], + "make install", + self.cfg['installopts'], + ]) + run_cmd(cmd) + + def sanity_check_step(self): + custom_paths = { + 'files': ['bin/eb-tutorial'], + 'dirs': [], + } + custom_commands = ['eb-tutorial'] + return super(EB_eb_minus_tutorial, self).sanity_check_step(custom_paths=custom_paths, + custom_commands=custom_commands) + ``` + +### Exercise I.2 + +Try implementing another new custom easyblock for ``eb-tutorial``, +which derives from the generic ``CMakeMake`` easyblock. + +Your easyblock should only: + +* define a custom mandatory easyconfig parameter named ``message``; +* pass down the value of ``message`` easyconfig parameter via `-DEBTUTORIAL_MSG=...` +* perform a custom sanity check to ensure the installation is correct; + +??? success "(click to show solution)" + + When deriving from the ``CMakeMake`` generic easyblock, there is a lot less to worry about. + + We only need to customize the ``configure_step`` method to ensure that the ``-DEBTUTORIAL_MSG`` configuration + option is specified; the ``CMakeMake`` easyblock already takes care of specifying the location of + the installation directory (and a bunch of other configuration options, like compiler commands and flags, etc.). + + Implementing the ``build_step`` and ``install_step`` methods is no longer needed, + the standard procedure that is run by the ``CMakeMake`` generic easyblock is fine, + and even goes beyond what we did in the previous exercise (like building in parallel with ``make -j``). + + ```python + from easybuild.easyblocks.generic.cmakemake import CMakeMake + from easybuild.framework.easyconfig import MANDATORY + from easybuild.tools.run import run_cmd + + class EB_eb_minus_tutorial(CMakeMake): + """Custom easyblock for eb-tutorial.""" + + @staticmethod + def extra_options(): + extra = CMakeMake.extra_options() + extra.update({ + 'message': [None, "Message that eb-tutorial command should print", MANDATORY], + }) + return extra + + def configure_step(self): + """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" + self.cfg.update('configopts', "-DEBTUTORIAL_MSG='%s'" % self.cfg['message']) + + super(EB_eb_minus_tutorial, self).configure_step() + + def sanity_check_step(self): + custom_paths = { + 'files': ['bin/eb-tutorial'], + 'dirs': [], + } + custom_commands = ['eb-tutorial'] + return super(EB_eb_minus_tutorial, self).sanity_check_step(custom_paths=custom_paths, + custom_commands=custom_commands) + ``` + +*[[next: PArt 3: OVerview]](3_00_part3_advanced.md)* diff --git a/docs/2022-CSC_and_LO/3_00_part3_advanced.md b/docs/2022-CSC_and_LO/3_00_part3_advanced.md new file mode 100644 index 00000000..708c57f8 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_00_part3_advanced.md @@ -0,0 +1,9 @@ +# Part III: Advanced topics + +*[[back to start page]](index.md)* + +* [Using EasyBuild as a library](3_01_easybuild_library.md) *(hands-on)* +* [Using hooks to customise EasyBuild](3_02_hooks.md) *(hands-on)* +* [Submitting installations as Slurm jobs](3_03_slurm_jobs.md) *(hands-on)* +* [Module naming schemes (incl. hierarchical)](3_04_module_naming_schemes.md) *(hands-on)* +* [GitHub integration to facilitate contributing to EasyBuild](3_05_github_integration.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/3_01_easybuild_library.md b/docs/2022-CSC_and_LO/3_01_easybuild_library.md new file mode 100644 index 00000000..c037f49f --- /dev/null +++ b/docs/2022-CSC_and_LO/3_01_easybuild_library.md @@ -0,0 +1,293 @@ +# Using EasyBuild as a library + +*[[back: Configuring EasyBuild]](1_07_configuration.md)* + +--- + +You can use EasyBuild as a Python library and implement a script to automate a particular task. + +All the functionality provided by the EasyBuild framework can be leveraged, +so you only have to glue things together and add the missing pieces. + + +## Setting up the EasyBuild configuration + +Before you call any of the functions provided by the EasyBuild framework, +you should initialize EasyBuild by **setting up the configuration**. +Some of the EasyBuild framework functions assume that EasyBuild has been initialized properly, +and calling them without doing so will result in errors. + +To set up the EasyBuild configuration, you should call the ``set_up_configuration`` function +that is provided by the ``easybuild.tools.options`` module. + +This function takes a couple of optional arguments that are useful in the context of +a Python script that uses EasyBuild as a library: + +* Via ``args`` you can provide a list of command line arguments that should be passed to the option parser. + By default, the standard command line arguments are picked up via ``sys.args``, which may not be what you want. +* By passing ``silent=True`` you can specify that the ``set_up_configuration`` function should not print + anything when it is called. By default, it will print the location to the temporary EasyBuild log file. + +### Cleaning up the temporary directory + +When EasyBuild is configured, a temporary directory specific to that EasyBuild session will be created automatically. +You should clean up that directory, especially if you will be creating temporary files, or if the script will be run +often. + +Note that cleaning up the temporary directory implies removes the temporary log files, +so you probably only want to do this if no errors occurred. + +### Minimal example script + +Here is a minimal (and pretty useless) example Python script, which sets up the EasyBuild configuration +and cleans up the temporary directory: + +```python +#!/usr/bin/env python3 +from easybuild.tools.filetools import remove_dir +from easybuild.tools.options import set_up_configuration + +opts, _ = set_up_configuration(args=[], silent=True) + +remove_dir(opts.tmpdir) +``` + + +## Example use cases + +Once the EasyBuild configuration has been set up, the functions provided by the EasyBuild framework +can be called from a Python script (or directly from the Python interpreter). + +A full overview of all functions is available via the [EasyBuild API documentation](https://docs.easybuild.io/en/latest/api/easybuild.html). + +We highlight a couple commonly used functions in the sections below. + +### File operations + +The [``easybuild.tools.filetools``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html) +module provides a bunch of functions for performing file operations. + +That includes straightforward things like reading, writing, and copying files +(see [``read_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.read_file), [``write_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.write_file), [``copy_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.copy_file)), but also more specific functionality like applying a patch file ([``apply_patch``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.apply_patch)), +computing different types of checksums for a file ([``compute_checksum``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.compute_checksum)), and downloading files +([``download_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.download_file)). + +Here is a trivial example where we download a file from a specified URL to a temporary directory, +compute the SHA256 checksum, and then copy the downloaded file to the current working directory: + +```python +#!/usr/bin/env python3 +import os +import sys +import tempfile + +from easybuild.tools.filetools import download_file, compute_checksum, copy_file, remove_dir +from easybuild.tools.options import set_up_configuration + +url = sys.argv[1] + +opts, _ = set_up_configuration() + +fn = os.path.basename(url) +tmp_path = os.path.join(opts.tmpdir, fn) +download_file(fn, url, tmp_path) + +sha256 = compute_checksum(tmp_path, checksum_type='sha256') +print("SHA256 checksum for %s: %s" % (fn, sha256)) + +target_dir = os.getcwd() +copy_file(tmp_path, target_dir) +print("%s copied to %s" % (fn, target_dir)) + +# cleanup temp dir +remove_dir(opts.tmpdir) +``` + +Example output: +```shell +$ export TEST_URL='https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.0.1.tar.gz' +$ python3 download_and_copy.py $TEST_URL +== Temporary log file in case of crash /tmp/eb-or_xhaw8/easybuild-ewooa00c.log +SHA256 checksum for eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 +eb-tutorial-1.0.1.tar.gz copied to /home/example +``` +We are deliberately *not* specifying the `args` optional argument to the `set_up_configuration` call here, +so we can pass EasyBuild configuration options as command line arguments to this script. +Of course, only those configuration options that are taken into account by the EasyBuild +framework functions being called have any impact. For example: + +```shell +# wait for max. 1h for the download to complete +python3 download_and_copy.py $TEST_URL --download-timeout 3600 +``` + +### Running shell commands + +To run shell commands, we can leverage the ``run_cmd`` functions from the ``easybuild.tools.run`` module. + +Here is a simple example for running the '`make`' command via a Python script: + +```python +#!/usr/bin/env python3 +import sys + +from easybuild.tools.filetools import remove_dir, which +from easybuild.tools.run import run_cmd +from easybuild.tools.options import set_up_configuration + +opts, _ = set_up_configuration(args=[]) + +cmd = 'make' +cmd_path = which(cmd) +if cmd_path: + print(">>> '%s' command found at %s" % (cmd, cmd_path)) +else: + sys.stderr.write("ERROR: '%s' command not found!\n" % cmd) + sys.exit(1) + +cmd = ' '.join(["make"] + sys.argv[1:]) +out, ec = run_cmd(cmd) + +print("\n>>> Output of '%s' (exit code %s):\n\n%s" % (cmd, ec, out)) + +remove_dir(opts.tmpdir) +``` + +Note that in this case it is important to use `args=[]` when calling `set_up_configuration`, +to avoid that the additional script arguments which are options for the '`make`' command +are picked up by the EasyBuild option parser. + +Example usage: + +```shell +$ python3 run_make.py --version +== Temporary log file in case of crash /tmp/eb-zfvbceg6/easybuild-7tynetaj.log + +>> Output of 'make --version' (exit code 0): + +GNU Make 3.81 +``` + +### Interacting with the modules tool + +You can interact with the environment modules tool in a Python script using the +[``easybuild.tools.modules``](https://docs.easybuild.io/en/latest/api/easybuild.tools.modules.html) +Python module that is a part of the EasyBuild framework. +The modules tool that will be used is determined by the active EasyBuild configuration. + +Here is an example script that checks which modules are available and currently loaded, +loads the default module file for ``bzip2``, and inspects the resulting changes to the environment. + +```python +#!/usr/bin/env python3 +import os +import sys + +from easybuild.tools.filetools import remove_dir +from easybuild.tools.modules import get_software_root_env_var_name, modules_tool +from easybuild.tools.options import set_up_configuration + +opts, _ = set_up_configuration() + +# obtain ModulesTool instance for preferred modules tool (determined by active EasyBuild configuration) +mod_tool = modules_tool() +print("Active modules tool: %s version %s" % (mod_tool.NAME, mod_tool.version)) + +avail_modules = mod_tool.available() +print("Found %d available modules in total" % len(avail_modules)) + +avail_eb_modules = mod_tool.available('EasyBuild') +print("Found %d available modules for EasyBuild: %s" % (len(avail_eb_modules), ', '.join(avail_eb_modules))) + +loaded_modules = mod_tool.loaded_modules() +print("%d modules are currently loaded: %s" % (len(loaded_modules), ', '.join(loaded_modules))) + +# load default module for bzip2, check changes to environment +name = 'bzip2' +env_var_name = get_software_root_env_var_name(name) +if any(m.startswith(name + '/') for m in avail_modules): + + print("Current $%s value: %s" % (env_var_name, os.getenv(env_var_name, '(no set)'))) + print("Loading (default) '%s' module..." % name) + + mod_tool.load([name]) + print("Loaded modules: %s" % ', '.join(mod_tool.loaded_modules())) + + # inspect $_LMFILES_ environment variable to determine path to loaded bzip2 module file + for mod_file_path in os.getenv('_LMFILES_').split(':'): + if name in mod_file_path: + print("Path to loaded %s module: %s" % (name, mod_file_path)) + break + + # $EBROOTBZIP2 should be set now (if the bzip2 module was installed with EasyBuild) + print("Current $%s value: %s" % (env_var_name, os.getenv(env_var_name, '(no set)'))) +else: + sys.stderr.write("No modules available for %s\n" % name) + sys.exit(1) + +remove_dir(opts.tmpdir) +``` + +### Parsing easyconfig files + +Here is another small example Python script, which uses the EasyBuild framework functionality +to locate and parse an easyconfig file, and inspect the value of specific easyconfig parameters. + +We define a small helper function named ``parse_easyconfig``, because the EasyBuild framework API +is a bit awkward to use for this simple use case. + +```python +#!/usr/bin/env python3 +import sys + +from easybuild.framework.easyconfig.tools import det_easyconfig_paths, parse_easyconfigs +from easybuild.tools.options import set_up_configuration + + +def parse_easyconfig(ec_fn): + """ + Helper function: find and parse easyconfig with specified filename, + and return parsed easyconfig file (an EasyConfig instance). + """ + # determine path to easyconfig file + ec_path = det_easyconfig_paths([ec_fn])[0] + + # parse easyconfig file; + # the 'parse_easyconfigs' function expects a list of tuples, + # where the second item indicates whether or not the easyconfig file was automatically generated or not + ec_dicts, _ = parse_easyconfigs([(ec_path, False)]) + + # only retain first parsed easyconfig, ignore any others (which are unlikely anyway) + return ec_path, ec_dicts[0]['ec'] + + +# check whether required arguments are provided +if len(sys.argv) < 3: + sys.stderr.write("ERROR: Usage: %s ") + sys.exit(1) + +ec_fn = sys.argv[1] +keys = sys.argv[2:] + +set_up_configuration(args=[], silent=True) + +ec_path, ec = parse_easyconfig(ec_fn) + +print("Inspecting %s ..." % ec_path) +for key in keys: + print("%s: %s" % (key, ec[key])) +``` + +Example usage: + +```shell +$ ./inspect_easyconfig.py Subread-2.0.0-GCC-8.3.0.eb name version sources sanity_check_paths +name: Subread +version: 2.0.0 +sources: ['subread-2.0.0-source.tar.gz'] +sanity_check_paths: {'files': ['bin/exactSNP', 'bin/featureCounts', 'bin/subindel', 'bin/subjunc', 'bin/sublong', 'bin/subread-align', 'bin/subread-buildindex'], 'dirs': ['bin/utilities']} +``` + +--- + +*[[next: Using hooks to customise EasyBuild]](3_02_hooks.md)* diff --git a/docs/2022-CSC_and_LO/3_02_hooks.md b/docs/2022-CSC_and_LO/3_02_hooks.md new file mode 100644 index 00000000..a2663ff0 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_02_hooks.md @@ -0,0 +1,190 @@ +# Using hooks to customise EasyBuild + +*[[back: Using EasyBuild as a library]](3_01_easybuild_library.md)* + +--- + +Sometimes you may have the need to customise the behaviour of EasyBuild, +while you want to avoid copying easyconfig files in order to make small changes +(like adding an additional configuration option), having to maintain a set +of modified easyblocks, or using a patched version of the EasyBuild framework. + +EasyBuild provides support for implementing so-called *hooks*, +which are small functions that are triggered automatically at specific times. + +Through these hook functions you can take additional actions, or even tweak +internal data structures, to change the software installation procedure that EasyBuild +would normally execute. + +This is a very powerful feature, but it is important to aware of some details in order +to make good use of it. + +Support for hooks is also covered in detail in the EasyBuild documentation, see +[here](https://docs.easybuild.io/en/latest/Hooks.html). + + +## Implementing and using hooks + +Using hooks is done by implementing functions with specific names in a Python module, +and configuring EasyBuild to use them by specifying the path to the file that contains them +via the ``hooks`` configuration option. + +For example, if the file named ``my_eb_hooks.py`` in your home directory contains the implementation +of your hooks, you can configure EasyBuild to use them by defining the ``$EASYBUILD_HOOKS`` environment +variable: + +```shell +export EASYBUILD_HOOKS=$HOME/my_eb_hooks.py +``` + +## Available hooks + +EasyBuild supports various hooks, the full list can be consulted via ``eb --avail-hooks``. + +There are 3 different types of hooks: + +- start/end hooks +- parse hook +- pre/post-step hooks + +These are discussed in more details in the sections below. + +### Start/end hooks + +The first type of hooks are the ``start`` and ``end`` hooks, which are respectively triggered +at the very start of the EasyBuild session (right after setting up the EasyBuild configuration), +and at the very end of the session (right before stopping the logger and doing final cleanup). + +These hooks are only called *once* for every EasyBuild session, no matter how many installations +are performed during that session. + +The ``end`` hook is *not* triggered if an error occurred during one of the installations, +only on successful completion of the EasyBuild session. + +These hooks can be implementing by defining a function named ``start_hook`` and ``end_hook`` +in the Python module file that is provided via the ``hooks`` configuration option. +No arguments are provided when calling these hooks. + +### Parse hook + +The ``parse`` hook is triggered right after an easyconfig file is being parsed, +*before* EasyBuild sets up its internal data structures based on the parsed easyconfig. + +If you want to dynamically change one or more easyconfig parameters without changing the corresponding +easyconfig files, using this hook may be appropriate. + +Note that parsing an easyconfig file can happen for a variety of reasons, +not only when the easyconfig file will actually be installed. EasyBuild will also +parse easyconfig files to check whether they resolve required dependencies, +to check whether the corresponding module file is already installed, etc. + +This hook can be implemented via a function named ``parse_hook``, and exactly one +argument is provided when it is called: the [``EasyConfig``](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyconfig.easyconfig.html#easybuild.framework.easyconfig.easyconfig.EasyConfig) +instance that represents the parsed easyconfig file. + +### Pre/post-step hooks + +The third type of hooks are *pre/post-step* hooks, which are triggered right before or +after a particular installation step is executed. + +EasyBuild performs each software installation by stepping through over a dozen different methods, +and for each of these steps there a pre- and post-hook is triggered, which results in over 30 +additional hooks. + +To use any of these hooks, you need to implement a function that follow a strict naming scheme: +``__hook``. For example, the hook that is triggered right before the ``configure`` +step is run is a function named ``pre_configure_hook``. + +Every time these hooks are called, a single argument is provided: an [``EasyBlock``](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyblock.html#easybuild.framework.easyblock.EasyBlock) +instance that represents the easyblock that is being used to perform the installation. +The parsed easyconfig file can be accessed via the ``cfg`` class variable of the ``EasyBlock`` instance. + +These hooks are useful for influencing the installation procedure at a particular stage. + +## Caveats + +There are a couple of important caveats to take into account when implementing hooks. + +### Breaking EasyBuild with hooks + +Since hooks allow you to inject custom code into EasyBuild at runtime, +it is also easy to break EasyBuild by using hooks... + +Make sure to carefully test your hook implementations, and constrain the actions +you take a much as possible, for example by adding conditions to control for which +software names you will actually modify the installation procedure, etc. + +Any errors that are triggered or raised while a hook function is running +will interrupt the EasyBuild session. + +So don't forget: with great power comes great responsibility! + +### Template values + +Depending on the type of hook, you may observe "raw" values of easyconfig parameters where +template values have not been resolved yet, or values in which template values have been resolved already. + +In the ``parse`` hook, you will always see unresolved template values. + +In the pre/post-step hooks you will see resolved template values, +unless you explicitly disable templating. + +To obtain easyconfig parameter values with unresolved template values in step hooks, +you can use the ``disable_templating`` [context manager](https://docs.python.org/3/reference/compound_stmts.html#with). +For example: + +```python +from easybuild.framework.easyconfig.easyconfig import disable_templating +from easybuild.tools.build_log import print_warning + +def pre_source_hook(eb): + """Print warning when software version was found in 'raw' name of source file.""" + with disable_templating(eb.cfg): + for src in eb.cfg['sources']: + if eb.version in src: + msg = "Software version '%s' found in name of source file (%s), " % (eb.version, src) + msg += "please use %(version)s template value instead!" + print_warning(msg) +``` + + +### Manipulating easyconfig parameters + +If you want update a particular easyconfig parameter without overwriting the existing value, +a bit of care has to be taken: you should use the ``update`` method of the ``EasyConfig`` instance +for this, unless you disable template resolution. This is particularly important when +updating easyconfig parameters that have *mutable* value (like a ``list`` or ``dict``). + +Here's a correct example of a pre-install hook: + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # always copy the README directory too when installing pigz + eb.cfg.update('files_to_copy', 'README') +``` + +This seemingly equivalent implementation will ***not*** work (the value of the `files_to_copy` +easyconfig parameter will *not* be updated): + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # incorrect way of adding 'README' to 'files_to_copy' (DON'T USE THIS!) + eb.cfg['files_to_copy'].append('README') +``` + +To use this coding style successfully, you have to disable the templating mechanism +when updating the easyconfig parameter: + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # this works, but it is better to use the 'update' method instead... + with disable_templating(eb.cfg): + eb.cfg['files_to_copy'].append('README') +``` + +--- + +*[[next: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* diff --git a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md new file mode 100644 index 00000000..0ffa12a9 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md @@ -0,0 +1,304 @@ +# Submitting installations as Slurm jobs + +*[[back: Using hooks to customise EasyBuild]](3_02_hooks.md)* + +--- + +EasyBuild can submit jobs to different backends including Slurm to install software, +to *distribute* the often time-consuming installation of a set of software applications and +the dependencies they require to a cluster. + +This is done via the ``--job`` command line option. + +It is important to be aware of some details before you start using this, which we'll cover here. + +## Configuration + +The EasyBuild configuration that is active at the time that ``eb --job`` is used +will be *passed down* into the submitted job automatically, via command line options to the ``eb`` +command that is run in the job script. + +This includes not only command line options used next to ``--job``, but also configuration settings +that are specified via an [EasyBuild configuration file](configuration.md#configuration-files) or through +[``$EASYBUILD_*`` environment variables](configuration.md#easybuild_-environment-variables). + +This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables +that are in place in the job environment are most likely *irrelevant*, since configuration settings +they specify they will most likely be overruled by the corresponding command line options. + + +## Using ``eb --job`` + +### Job backend + +The default job backend in EasyBuild v4.x is [``GC3Pie``](https://gc3pie.readthedocs.io). +To let EasyBuild submit jobs to Slurm instead, you should set the ``job-backend`` configuration setting +to ``Slurm``, for example by setting the corresponding environment variable: + +```shell +export EASYBUILD_JOB_BACKEND='Slurm' +``` + +### Job resources + +To submit an installation as a job, simply use ``eb --job``: + +```shell +eb example.eb --job +``` + +By default, EasyBuild will submit single-core jobs requesting for 24 hours of walltime. +You can tweak the requested resources via the ``job-cores`` and ``job-max-walltime`` configuration options. +For example: + +```shell +# submit job to install example, using 5 cores and 2 hours of max. walltime +eb example.eb --job --job-cores 5 --job-max-walltime 2 +``` + +Note that not all ``job-*`` configuration settings apply to all job backends, +see the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Submitting_jobs.html) for more details. + +### Controlling Slurm submission options + +When using Slurm as a job backend, EasyBuild will automatically generate job scripts which +use the ``eb`` command to perform a single installation. These scripts will be submitted +using the ``sbatch`` command. + +EasyBuild currently doesn't provide away to customize the Slurm submission options, +for example to submit to a particular partition, or to use a particular account, +build you can set the corresponding ``$SBATCH_*`` environment variables prior to running ``eb --job``. + +For example, to specify a particular account that should be used for the jobs submitted by EasyBuild +(equivalent with using the ``-A`` or ``--account`` command line option for ``sbatch``): + +```shell +export SBATCH_ACCOUNT='example_project' +``` + +Or to submit to a particular Slurm partition (equivalent with the ``-p`` or ``--partition`` option for ``sbatch``): + +```shell +export SBATCH_PARTITION='example_partition' +``` + +For more information about supported ``$SBATCH_*`` environment variables, +see the [Slurm documentation](https://slurm.schedmd.com/sbatch.html#lbAJ). + +## Combining ``--job`` and ``--robot`` + +If one or more dependencies are still missing for the software you want to install, +you can combine ``--job`` and ``--robot`` to get EasyBuild to submit a *separate* job +for each of the installations. These jobs will *not* ``--robot``, they will each only +perform a single installation. + +Dependencies between jobs will be "registered" at submission time, so Slurm will put jobs +on hold until the jobs that install the required (build) dependencies have completed successfully, +and cancel jobs if the job to install a dependency failed for some reason. + +## Attention points + +There are a couple of important things to keep an eye on when submitting installations as jobs... + +### Differences on cluster workernodes + +Sometimes the resources available on the login nodes and cluster workernodes are slightly different, +and you may need to take this into account in your EasyBuild configuration. + +For example, plenty of disk space may be available in the `/tmp` temporary filesystem on a login node, +while the workernodes require you to use a different location for temporary files and directories. +As a result, you may need to slightly change your EasyBuild configuration when submitting installations +as jobs, to avoid that they fail almost instantly due to a lack of disk space. + +Keep in mind that the active EasyBuild configuration is passed down into the submitted jobs, +so any configuration that is present on the workernodes may not have any effect. + +For example, if you commonly use `/tmp/$USER` for build directories on a login node, +you may need to tweak that when submitting jobs to use a different location: + +```shell +# EasByuild is configured to use /tmp/$USER on the login node +login01 $ eb --show-config | grep buildpath +buildpath (E) = /tmp/example + +# use /localdisk/$USER for build directories when submitting installations as jobs +login01 $ eb --job --buildpath /localdisk/$USER example.eb --robot +``` + +### Temporary log files and build directories + +The temporary log file that EasyBuild creates is most likely going to end up on the local disk +of the workernode on which the job was started (by default in `$TMPDIR` or `/tmp`). +If an installation fails, the job will finish and temporary files will likely be cleaned up instantly, +which may leave you wondering about the actual cause of the failing installation... + +To remedy this, there are a couple of EasyBuild configuration options you can use: + +* You can use ``--tmp-logdir`` to specify a different location where EasyBuild should store temporary log files, + for example: + ```shell + $ eb --job example.eb --tmp-logdir $HOME/eb_tmplogs + ``` + +* If you prefer having the entire log file stored in the Slurm job output files, + you can use ``--logtostdout`` when submitting the jobs. This will result in extensive logging + to your terminal window when submitting the jobs, but it will also make EasyBuild + log to ``stdout`` when the installation is running in the job, and hence the log messages will be + captured in the job output files. + +The same remark applies to build directories: they should be on a local filesystem (to avoid problems +that often occur when building software on a parallel filesystem like GPFS or Lustre), +which will probably be cleaned up automatically when a job fails. Here it is less easy to provide +general advice on how to deal with this, but one thing you can consider is retrying the installation +in an interactive job, so you can inspect the build directory after the installation fails. + +### Lock files + +EasyBuild creates [locks](https://docs.easybuild.io/en/latest/Locks.html) +to prevent that the same installation is started multiple times on +different system to the same installation directory. + +If an installation fails or gets interrupted, EasyBuild cleans up those locks automatically. + +However, if a Slurm job that is using EasyBuild to install software gets cancelled (because it +ran out of walltime, tried to consume too much memory, through an ``scancel`` command, etc.), +EasyBuild will not get the chance to clean up the lock file. + +If this occurs you will need to either clean up the lock file (which is located in the `software/.locks` +subdirectory of ``installpath``) manually, or re-submit the job with ``eb --job --ignore-locks``. + +## Example + +As an example, we will let EasyBuild submit jobs to install ``AUGUSTUS`` with the ``foss/2020b`` toolchain. + +### Configuration + +Before using ``--job``, let's make sure that EasyBuild is properly configured: + +```shell +# use $HOME/easybuild for software, modules, sources, etc. +export EASYBUILD_PREFIX=$HOME/easybuild + +# use ramdisk for build directories +export EASYBUILD_BUILDPATH=/dev/shm/$USER + +# use Slurm as job backend +export EASYBUILD_JOB_BACKEND=Slurm +``` + +In addition, add the path to the centrally installed software to ``$MODULEPATH`` via ``module use``: + +```shell +module use /easybuild/modules/all +``` + +Load the EasyBuild module: + +```shell +module load EasyBuild +``` + +Let's assume that we also need to inform Slurm that jobs should be submitted into a particular account: + +```shell +export SBATCH_ACCOUNT=example_project +``` + +This will be picked up by the ``sbatch`` commands that EasyBuild will run to submit the software installation jobs. + + +### Submitting jobs to install AUGUSTUS + +Now we can let EasyBuild submit jobs for AUGUSTUS. + +Let's first check what is still missing: + +```shell +$ eb AUGUSTUS-3.4.0-foss-2020b.eb --missing +... +11 out of 61 required modules missing: + +* HTSlib/1.11-GCC-10.2.0 (HTSlib-1.11-GCC-10.2.0.eb) +* lpsolve/5.5.2.11-GCC-10.2.0 (lpsolve-5.5.2.11-GCC-10.2.0.eb) +* Boost/1.74.0-GCC-10.2.0 (Boost-1.74.0-GCC-10.2.0.eb) +* GSL/2.6-GCC-10.2.0 (GSL-2.6-GCC-10.2.0.eb) +* SAMtools/1.11-GCC-10.2.0 (SAMtools-1.11-GCC-10.2.0.eb) +* BCFtools/1.11-GCC-10.2.0 (BCFtools-1.11-GCC-10.2.0.eb) +* METIS/5.1.0-GCCcore-10.2.0 (METIS-5.1.0-GCCcore-10.2.0.eb) +* BamTools/2.5.1-GCC-10.2.0 (BamTools-2.5.1-GCC-10.2.0.eb) +* MPFR/4.1.0-GCCcore-10.2.0 (MPFR-4.1.0-GCCcore-10.2.0.eb) +* SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0.eb) +* AUGUSTUS/3.4.0-foss-2020b (AUGUSTUS-3.4.0-foss-2020b.eb) +``` + +Several dependencies are not installed yet, so we will need to use ``--robot`` to ensure that +EasyBuild also submits jobs to install these first. + +To speed up the installations a bit, we will request 10 cores for each submitted job (via ``--job-cores``). +That should be sufficient to let each installation finish in (well) under 1 hour, +so we only request 1 hour of walltime per job (via ``--job-max-walltime``). + +In order to have some meaningful job output files, we also enable trace mode (via ``--trace``). + +``` +$ eb AUGUSTUS-3.4.0-foss-2020b.eb --job --job-cores 10 --job-max-walltime 1 --robot --trace +... +== resolving dependencies ... +... +== List of submitted jobs (11): Boost-1.74.0-GCC-10.2.0 (Boost/1.74.0-GCC-10.2.0): 1000011; GSL-2.6-GCC-10.2.0 (GSL/2.6-GCC-10.2.0): 1000004; SAMtools-1.11-GCC-10.2.0 (SAMtools/1.11-GCC-10.2.0): 1000005; HTSlib-1.11-GCC-10.2.0 (HTSlib/1.11-GCC-10.2.0): 1000006; BCFtools-1.11-GCC-10.2.0 (BCFtools/1.11-GCC-10.2.0): 1000001; lpsolve-5.5.2.11-GCC-10.2.0 (lpsolve/5.5.2.11-GCC-10.2.0): 1000007; BamTools-2.5.1-GCC-10.2.0 (BamTools/2.5.1-GCC-10.2.0): 1000008; METIS-5.1.0-GCCcore-10.2.0 (METIS/5.1.0-GCCcore-10.2.0): 1000009; MPFR-4.1.0-GCCcore-10.2.0 (MPFR/4.1.0-GCCcore-10.2.0): 1000010; SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0): 1000002; AUGUSTUS-3.4.0-foss-2020b (AUGUSTUS/3.4.0-foss-2020b): 1000003 +== Submitted parallel build jobs, exiting now +``` + +### Inspecting the submitted jobs + +Once EasyBuild has submitted the jobs, we can inspect them via Slurm's ``squeue`` command: + +``` +$ squeue -u $USER -la + JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON) +1000001 small BCFtools user123 PENDING 0:00 2:00:00 1 (Dependency) +1000002 small SuiteSpa user123 PENDING 0:00 2:00:00 1 (Dependency) +1000003 small AUGUSTUS user123 PENDING 0:00 2:00:00 1 (Dependency) +1000004 small GSL-2.6- user123 RUNNING 0:21 2:00:00 1 node003 +1000005 small SAMtools user123 RUNNING 0:21 2:00:00 1 node007 +1000006 small HTSlib-1 user123 RUNNING 0:21 2:00:00 1 node007 +1000007 small lpsolve- user123 RUNNING 0:21 2:00:00 1 node011 +1000008 small BamTools user123 RUNNING 0:21 2:00:00 1 node011 +1000009 small METIS-5. user123 RUNNING 0:21 2:00:00 1 node013 +1000010 small MPFR-4.1 user123 RUNNING 0:21 2:00:00 1 node029 +1000011 small Boost-1. user123 RUNNING 0:24 2:00:00 1 node029 +``` + +Note that 3 jobs can not be started yet, because those installations require on one or more +missing dependencies. As soon as the jobs for those dependencies (successfully) complete, +these jobs will be able to start. + +### Final result + +After about 20 minutes, AUGUSTUS and all missing dependencies should be installed: + +``` +$ ls -lrt $HOME/easybuild/modules/all/*/*.lua | tail -11 +-rw-rw----. 1 example example 1634 Mar 29 10:13 /users/example/easybuild/modules/all/HTSlib/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1792 Mar 29 10:13 /users/example/easybuild/modules/all/SAMtools/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1147 Mar 29 10:13 /users/example/easybuild/modules/all/BamTools/2.5.1-GCC-10.2.0.lua +-rw-rw----. 1 example example 957 Mar 29 10:13 /users/example/easybuild/modules/all/lpsolve/5.5.2.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1549 Mar 29 10:13 /users/example/easybuild/modules/all/METIS/5.1.0-GCCcore-10.2.0.lua +-rw-rw----. 1 example example 1525 Mar 29 10:14 /users/example/easybuild/modules/all/GSL/2.6-GCC-10.2.0.lua +-rw-rw----. 1 example example 1221 Mar 29 10:15 /users/example/easybuild/modules/all/MPFR/4.1.0-GCCcore-10.2.0.lua +-rw-rw----. 1 example example 1678 Mar 29 10:15 /users/example/easybuild/modules/all/BCFtools/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1292 Mar 29 10:21 /users/example/easybuild/modules/all/Boost/1.74.0-GCC-10.2.0.lua +-rw-rw----. 1 example example 1365 Mar 29 10:28 /users/example/easybuild/modules/all/SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0.lua +-rw-rw----. 1 example example 2233 Mar 29 10:30 /users/example/easybuild/modules/all/AUGUSTUS/3.4.0-foss-2020b.lua + +$ module use $HOME/easybuild/modules/all + +$ module avail AUGUSTUS + +-------- /users/hkenneth/easybuild/modules/all -------- + AUGUSTUS/3.4.0-foss-2020b +``` + +--- + +*[[next: Module naming schemes]](3_04_module_naming_scheme.md)* diff --git a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md new file mode 100644 index 00000000..14229d97 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md @@ -0,0 +1,617 @@ +# Module naming schemes + +*[[back: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* + +--- + +Up until now we have used EasyBuild's default module naming scheme (`EasyBuildMNS`), +which produces module files with names that closely resemble to the names of the +corresponding easyconfig files. +For example, when installing `h5py-3.1.0-foss-2020b.eb` the generated module was named `h5py/3.1.0-foss-2020b`. + +EasyBuild supports several different module naming schemes: + +```shell +$ eb --avail-module-naming-schemes +List of supported module naming schemes: + CategorizedHMNS + CategorizedModuleNamingScheme + EasyBuildMNS + HierarchicalMNS + MigrateFromEBToHMNS +``` + +In this part of the tutorial we will take a closer look at `HierarchicalMNS`, +which is the standard **hierarchical** module naming scheme included with EasyBuild. + +We will also take a quick look at implementing our own custom module naming scheme. + +## Flat vs hierarchical + +The default module naming scheme `EasyBuildMNS` is an example of regular *"flat"* module naming scheme, which is characterized by: + +* all module files are directly available for loading; +* each module name uniquely identifies a particular installation; + +In contrast, a *hierarchical* module naming scheme +consists of a *hierarchy* of module files. + +The typical module hierarchy has 3 levels: + +* a **core** level, where module files for software that was installed using the + [`system` toolchain](../introduction/#system-toolchain) are kept; +* a **compiler** level, where module files for software that was installed using a +*compiler-only toolchain* are stored; +* and an **MPI** level, which houses module files for software that was installed using a toolchain that includes (at least) a compiler and MPI component; + +Here is a simple example of such a 3-level module hierarchy: + +
+ +In this example the core level only includes a single module `GCC/9.3.0`, +while the compiler level includes two modules: `OpenMPI/4.0.3` and `MPICH/3.3.2`. +In the MPI level, three modules are available: one for `FFTW`, one for `ScaLAPACK`, and one for `HDF5`. + +Initially only the modules on the top level of a module hierarchy are available for loading. +If you run "`module avail`" with the example module hierarchy, you will only see the `GCC/9.3.0` module. + +Some modules in the top level of the hierarchy act as a "gateway" to modules in the +next level below. +To make additional modules available for loading one of these gateway modules has to be loaded. In our exampe, loading the `GCC/9.3.0` module results in two additional modules coming into view from the compiler level, as indicated by the arrows: the modules for `OpenMPI` and `MPICH`. These correspond to installations of `OpenMPI` +and `MPICH` that were built using `GCC/9.3.0` as a (compiler-only) toolchain. + +Similarly, the `OpenMPI/4.0.3` module serves as a gateway to the three modules in the MPI level. Only by loading the `OpenMPI` module will these additional three modules become +available for loading. They correspond to software installations built using the `gompi/2020a` toolchain that +consists of the `GCC/9.3.0` compiler module and the `OpenMPI/4.0.3` MPI module. Software installed using +`foss/2020a` (which is a full toolchain that also includes OpenBLAS, FFTW and ScaLAPACK) would also be stored +in this level of the module hierarchy. + +The characteristics of a module hierarchy are: + +* not all module files are directly available for loading; +* some modules serve as a gateway to more modules; +* to access some software installations you will first need to load one or more gateway modules in order + to use them; + +You can probably think of other ways to organize module files in a hierarchical module tree, but here we will stick to the standard core / compiler / MPI hierarchy. + +### Pros & cons + +So why go through all this trouble of organizing modules hierarchically? + +There are a couple of advantages to this approach: + +* shorter module names; +* less overwhelming list of available modules; +* only compatible modules can be loaded together; + +However, the are some minor disadvantages too: + +* not all existing modules are directly visible; +* gateway modules may have little meaning to end users; + +#### Length of module names + +When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For our `HDF5` installation for example, +we have `HDF5/1.10.6-gompi-2020a` as module name. The `-gompi-2020a` part of the name refers to the toolchain that was +used for this installation, but it may be confusing to some people (what kind of Pokémon is a "gompi"?!). + +In the example module hierarchy shown above, the module for `HDF5` is named `HDF5/1.10.6` which is basically the bare +essentials: software name and version. That's way better, nice and clean! + +#### Amount of available modules + +The output of "`module avail`" can be quite overwhelming if lots of module files +are installed and a flat module naming scheme is used, since *all* modules are +*always* available. +EasyBuild makes it very easy to install lots of software, +so the number of installed modules can easily grow into the hundreds or even thousands... + +This often explosive growth of modules is less of an issue when using a hierarchical module naming scheme, since +initially only a small set of modules are available, and relatively limited +groups of additional modules become available as gateway modules are loaded. + +#### Loading compatible modules + +Since all modules are available at once when using a flat module naming scheme, you can easily load +modules together that are not compatible with each other. + +Imagine loading two modules that were built with a different compiler toolchain (different compiler, +different MPI library). That's likely to end in tears, unless you have the necessary technical expertise +to fully understand what is going on, *and* you are being very careful... + +In a module hierarchy this can be prevented, since modules for software that were installed with a +different compiler and/or a different MPI library are located in a different part of the module hierarchy, +and so these modules can not be loaded together easily. + + +#### Visibility of existing modules + +One downside of a module hierarchy is that not all existing modules are directly available for loading +or are even *visible* to the user, since the output of "`module avail`" only shows a subset of all modules. + +[Lmod](https://lmod.readthedocs.io) has a solution for this though: it provides a separate "`module spider`" +command to search for module files throughout the entire module hierarchy. So as long as the end users are +aware of this additional command, it should not be difficult to discover which software installations exist +and how they can be accessed. The "`module spider`" command will inform the user which of the gateway modules +need to be loaded in order to load a specific module file. + +#### Semantics of gateway modules + +An additional potential problem of a module hierarchy is that the semantics of the gateway modules may not be clear +to end users. They may wonder why they need to pick a specific compiler and MPI library, or which of the +available options is the best one. Maybe there are not even be aware what exactly a "compiler" is, or how it is +relevant to the software they need in their bioinformatics pipeline... + +This can be partially resolved by loading a default compiler and MPI module so a particular set of modules +is available right after login, which could be the ones used in the most recent toolchain, or the +recommended versions. More experienced users could then leverage the "`module spider`" command to navigate +the module hierarchy. + +## Using a custom module naming scheme + +Next to the module naming schemes that are included with EasyBuild, +you can also define your own module naming scheme (MNS), and configure EasyBuild to use it. + +### Implementation + +To use a custom module naming scheme, you need to implement a Python module file, +where you define a Python class that derives from the general [``ModuleNamingScheme``](https://docs.easybuild.io/en/latest/api/easybuild.tools.module_naming_scheme.mns.html#easybuild.tools.module_naming_scheme.mns.ModuleNamingScheme) class. + +For a flat module naming scheme, it is sufficient to implement the ``det_full_module_name`` method, +which should return a string value (the full module name). +You may also need to customize the ``is_short_modname_for`` method, which verifies whether +a given (short) module name is for the software with a particular given name (or not). + +The argument provided to ``det_full_module_name`` can be a small Python dictionary which only specifies +the value of a handful of easyconfig parameters (`name`, `version`, `toolchain`, and `versionsuffix`), +or an ``EasyConfig`` instance which represents a parsed easyconfig file and contains values for *all* +known easyconfig parameters. + +For simple module naming schemes, just have ``name``, ``version``, ``toolchain``, and ``versionsuffix`` +available is sufficient. If it is not, you can list which additional easyconfig parameters are required for +your module naming scheme via a class constant ``REQUIRED_KEYS``. + +A fairly straightforward example of a *flat* module name scheme is the default EasyBuild module naming scheme +[``EasyBuildMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/easybuild_mns.py). + +For a *hierarchical* module naming scheme, various additional methods have to be implemented. + +Some of these, like ``det_module_subdir`` and ``det_short_module_name``, determine which part of +the (full) module name determines the location of the module in the module hierarchy, and which part +is the user-facing (or "short") module name. Others, like ``det_modpath_extensions``, +determine which modules are *gateway* modules that open up an additional level of the hierarchy +(by *extending* the ``$MODULEPATH``). + +A typical example of a hierarchical module naming scheme is [``HierarchicalMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/hierarchical_mns.py) that comes with EasyBuild (which is quite complex because it includes quite a bit of special handling for particular compiler +toolchains), which implements the traditional ``Core``-``Compiler``-``MPI`` module hierarchy we discussed +above. + +### Configuring EasyBuild + +To let EasyBuild use a custom module naming scheme, you need to: + +* specify the path to the Python module file implementing it via the ``include-module-naming-schemes`` + EasyBuild configuration option; + +* indicate that you also want to *use* this custom module naming scheme via the ``module-naming-scheme`` + EasyBuild configuration option. + +For example: + +```shell +export EASYBUILD_INCLUDE_MODULE_NAMING_SCHEMES=$HOME/easybuild/example_mns.py +export EASYBUILD_MODULE_NAMING_SCHEME=ExampleMNS +``` + +#### Example custom module naming scheme + +Here is an example of a custom module naming scheme, where: + +* the ``versionsuffix`` goes directly after the ``version``; +* all dashes are replaced by underscores; +* all module names are lowercase; + +Note that we also need to customise the ``is_short_modname_for`` method, +to make sure it returns ``True`` when the EasyBuild framework checks whether +``scipy_bundle/2020.11_foss_2020b`` is a module name for ``SciPy-bundle``. + +```python +import os + +from easybuild.tools.module_naming_scheme.mns import ModuleNamingScheme + + +class ExampleMNS(ModuleNamingScheme): + + REQUIRED_KEYS = ['name', 'version', 'versionsuffix', 'toolchain'] + + def det_full_module_name(self, ec): + """ + Determine full module name: + - all lowercase + - replace all dashes with underscores + """ + parts = [ec['version']] + + # versionsuffix directly after version (but only if it's not empty) + if ec['versionsuffix']: + parts.append(ec['versionsuffix']) + + # only add toolchain name/version for non-system toolchain + tc = ec['toolchain'] + if tc['name'].lower() != 'system': + parts.extend([tc['name'], tc['version']]) + + modname = ec['name'] + '/' + '_'.join(parts) + + modname = modname.replace('-', '_').replace('__', '_') + + return modname.lower() + + def is_short_modname_for(self, short_modname, name): + """Determine whether short module name is a module for the software with specified name.""" + return short_modname.startswith(name.lower().replace('-', '_') + '/') +``` + +We can see what the module names with this module naming scheme would like like via ``eb -D``: + +``` +$ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D + ... + * [ ] $CFGS/g/GCC/GCC-10.2.0.eb (module: gcc/10.2.0) + ... + * [ ] $CFGS/p/Python/Python-2.7.18-GCCcore-10.2.0.eb (module: python/2.7.18_gcccore_10.2.0) + ... + * [ ] $CFGS/o/OpenMPI/OpenMPI-4.0.5-GCC-10.2.0.eb (module: openmpi/4.0.5_gcc_10.2.0) + ... + * [ ] $CFGS/s/SciPy-bundle/SciPy-bundle-2020.03-foss-2020a-Python-2.7.18.eb (module: scipy_bundle/2020.03_python_2.7.18_foss_2020a) +``` + +## Example module hierarchy: HDF5 + +!!! Warning + **This exercise will only work if you have write access to the software installation prefix**, + because EasyBuild will try to copy the installation log file to each installlation directory. + +Now that we know more about hierarchical module naming schemes, +let us see how EasyBuild can help us with generating a hierarchical module tree. + +In this example we will use EasyBuild to generate modules organised in a hierarchy +for some of the software that is already installed in the prepared environment. + +The good news is that the existing installations can be reused. There is absolutely no need +to reinstall the software, we are just creating a different "view" on these software installations. + +### Preparing the environment + +Before running EasyBuild to generate a hierarchical module tree, we have to be a bit careful +with preparing our environment. + +**We must absolutely avoid mixing modules from a flat and hierarchical module naming scheme!** + +Some module files will have the same name in both module trees (like `GCC/10.2.0` for example), +but their contents will be different. +Mixing modules from a flat and hierarchical module tree *will* trigger problems... + +So we have to make sure that the module files we already have in `/easybuild` are *not* visible. +The easiest way to do this is to unload all modules (using "`module purge`") +and resetting the module search path to be empty, which we can do with "`module unuse $MODULEPATH`". + +```shell +module purge +module unuse $MODULEPATH +``` + +In this part of the tutorial, we are assuming you are *not* using an EasyBuild installation provided through +a module. We have just made all modules unavailable, so we would have to first +install EasyBuild again in our hierarchical module tree before we can continue. + +**We strongly recommend using an EasyBuild installation that was [installed via "`pip install`" +or "`pip3 install`"](../installation/#method-1-using-pip-recommended) in this part of the tutorial.** + +An easy way to do this is in the prepared environment is to run: + +```shell +pip3 install --user easybuild +export PATH=$HOME/.local/bin:$PATH +export EB_PYTHON=python3 +``` + +### Configuring EasyBuild + +First of all, we need to make sure that EasyBuild is properly configured. +We can do this by defining this set of environment variables: + +```shell +export EASYBUILD_PREFIX=$HOME/easybuild +export EASYBUILD_BUILDPATH=/tmp/$USER +export EASYBUILD_INSTALLPATH_SOFTWARE=/easybuild/software +export EASYBUILD_MODULE_NAMING_SCHEME=HierarchicalMNS +export EASYBUILD_INSTALLPATH_MODULES=$HOME/hmns/modules +``` + +To make sure we didn't make any silly mistakes, we double check using `eb --show-config`: + + +```shell +$ eb --show-config +# +# Current EasyBuild configuration +# (C: command line argument, D: default value, E: environment variable, F: configuration file) +# +buildpath (E) = /tmp/example +containerpath (E) = /home/example/easybuild/containers +installpath (E) = /home/example/easybuild +installpath-modules (E) = /home/example/hmns/modules +installpath-software (E) = /easybuild/software +module-naming-scheme (E) = HierarchicalMNS +packagepath (E) = /home/example/easybuild/packages +prefix (E) = /home/example/easybuild +repositorypath (E) = /home/example/easybuild/ebfiles_repo +robot-paths (D) = /home/example/.local/easybuild/easyconfigs +sourcepath (E) = /home/example/easybuild/sources +``` + +There are a couple of things worth pointing out here: + +* We have defined the `module-naming-scheme` configuration setting to `HierarchicalMNS`, + which makes EasyBuild use the included standard hierarchical module naming scheme (the classic + core / compiler / MPI one we discussed above). +* We have specified *different* locations for the software (via `installpath-software`) + and the module files (via `installpath-modules`). This is important because we want to + reuse the software that is already installed in `/easybuild/software` while we want to + generate an entirely new module tree for it (in `$HOME/hmns/modules`). + +The other configuration settings are the same as before, and mostly irrelevant for this example. + +### Generating modules for HDF5 + +Let us now generate a hierarchical module tree for `HDF5` and all of its dependencies, +including the toolchain. That sounds complicated, and it sort of is since there are +a lot of details you have to get right for the module hierarchy to works as intended, +but EasyBuild can do all the hard work for us. + +The steps we will have to go through are: + +* Tell EasyBuild we want to "install" the `HDF5-1.10.7-gompi-2020b.eb` easyconfig file; +* Enable dependency resolution via `--robot`; +* Instruct EasyBuild to only generate the module files, not to install the software (since it is + there already in `/easybuild/software`), via the `--module-only` option. + +These steps translate to this single `eb` command: + +``` +$ eb HDF5-1.10.7-gompi-2020b.eb --robot --module-only +... +== building and installing MPI/GCC/10.2.0/OpenMPI/4.0.5/HDF5/1.10.7... +... +== sanity checking... +== cleaning up [skipped] +== creating module... +... +== COMPLETED: Installation ended successfully (took 9 sec) +... +== Build succeeded for 41 out of 41 +``` + +This should take a couple of minutes in total, for generating 41 modules. +Remember that this also includes generating module files for the toolchain and +all of its components. + +In addition, there is a bit more going on one that just generating module files, +since the sanity check step is still being run for each of the installations +when using `--module-only` to ensure the installation is actually functional. +After all, there is no point in generating a module for an obviously broken +installation... + +### Loading the HDF5 module + +After generating the hierarchical module tree for HDF5, how do we access the HDF5 installation through it? + +Here's what the module tree looks like on disk: + +``` +$ ls $HOME/hmns/modules/all +Compiler Core MPI +``` + +Those are basically the 3 levels in the module hierarchy we showed in our example earlier. + +The starting point is the top level of the module hierarchy named `Core`: + +``` +module use $HOME/hmns/modules/all/Core +``` + +Let us see what that gives us in terms of available modules: + +``` +$ module avail + +--------------------- /home/easybuild/hmns/modules/all/Core --------------------- + Bison/3.5.3 GCCcore/10.2.0 flex/2.6.4 help2man/1.47.4 + Bison/3.7.1 (D) M4/1.4.18 gettext/0.21 ncurses/6.2 + GCC/10.2.0 binutils/2.35 gompi/2020b zlib/1.2.11 +``` + + +Nice and short module names, but only a limited set of them. + +We know a module file exists for `HDF5`, but we can't see it yet (and hence +we can't load it either). + +``` +$ module avail HDF5 +No module(s) or extension(s) found! +Use "module spider" to find all possible modules and extensions. +``` + +Let us see if `module spider` is of any help, as "`module avail`" so kindly suggests: + +``` +$ module spider HDF5 +... + + You will need to load all module(s) on any one of the lines below + before the "HDF5/1.10.7" module is available to load. + + GCC/10.2.0 OpenMPI/4.0.5 +``` + +This tells us we need to load two gateway modules before we can load the module +for HDF5. + +Let us start with loading the `GCC` compiler module: + +``` +module load GCC/10.2.0 +``` + +And then check again which modules are available: + +``` +$ module avail + +-------------- /home/easybuild/hmns/modules/all/Compiler/GCC/10.2.0 -------------- + OpenMPI/4.0.5 + +------------ /home/easybuild/hmns/modules/all/Compiler/GCCcore/10.2.0 ------------ + Autoconf/2.69 UCX/1.9.0 libpciaccess/0.16 + ... + Szip/2.1.1 libfabric/1.11.0 zlib/1.2.11 (L,D) + Autoconf/2.69 XZ/5.2.5 libtool/2.4.6 + +--------------------- /home/easybuild/hmns/modules/all/Core --------------------- + Bison/3.5.3 GCCcore/10.2.0 (L) flex/2.6.4 help2man/1.47.4 + Bison/3.7.1 (D) M4/1.4.18 gettext/0.21 ncurses/6.2 + GCC/10.2.0 (L) binutils/2.35 gompi/2020b zlib/1.2.11 +``` + +Good news, we now have additional modules available! + +The compiler level of our hierarchy actually consists of two directories here: `Compiler/GCCcore/10.2.0` +and `Compiler/GCC/10.2.0`. The modules in the `GCCcore` directory are ones we can use in other compiler +toolchains that use GCC 10.2.0 as a base compiler (the details of that are out of scope here). + +The module we are interested in is `OpenMPI/4.0.5`, which is another gateway module. + +Remember that the "`module spider`" output told us that there does indeed exist a module for `HDF5`, but that +we need to load *both* the `GCC/10.2.0` and `OpenMPI/4.0.5` modules first. + +So, let us do exactly that (remember that `GCC/10.2.0` is already loaded): + +``` +module load OpenMPI/4.0.5 +``` + +If you now check the output of "`module avail`" again, you should see the `HDF5/1.10.7` module: + +``` +$ module avail + +-------- /home/easybuild/hmns/modules/all/MPI/GCC/10.2.0/OpenMPI/4.0.5 ------- + HDF5/1.10.7 + +------------ /home/easybuild/hmns/modules/all/Compiler/GCC/10.2.0 ------------ + OpenMPI/4.0.5 (L) + +... +``` + +To use HDF5, we need to load this `HDF5/1.10.7` module. We can verify that the installation works +using one of the commands provided by HDF5, `h5dump` for example: + +``` +module load HDF5/1.10.7 +``` + +``` +$ h5dump --version +h5dump: Version 1.10.7 +``` + +If you now check which modules are loaded via "`module list`", you will notice that all module names +and nice and short now, which is one of the advantages of using a hierarchical module tree: + +```shell +$ module list + +Currently Loaded Modules: + 1) GCCcore/10.2.0 5) numactl/2.0.13 9) hwloc/2.2.0 13) PMIx/3.1.5 + 2) zlib/1.2.11 6) XZ/5.2.5 10) libevent/2.1.12 14) OpenMPI/4.0.5 + 3) binutils/2.35 7) libxml2/2.9.10 11) UCX/1.9.0 15) Szip/2.1.1 + 4) GCC/10.2.0 8) libpciaccess/0.16 12) libfabric/1.11.0 16) HDF5/1.10.7 +``` + +## Exercise + +Now it is your turn! + +Try to get a feeling for how a hierarchical module tree works by: + +* installing the missing modules for the `SciPy-bundle-2020.11-foss-2020b.eb` in the module hierarchy we + generated for HDF5; +* figure out where the `SciPy-bundle` module is located in the hierarchy, and then also load it; + +You can verify your work by running this command (since `pandas` is one of the Python packages included +in the `SciPy-bundle` installation): + +```shell +python -c 'import pandas; print(pandas.__version__)' +``` + +Start from a clean slate, by first running: + +```shell +module purge +module unuse $MODULEPATH +``` + +??? success "(click to show solution)" + + * Step 0: check which modules are still missing, using `--missing` or `-M`: + ```shell + eb SciPy-bundle-2020.11-foss-2020b.eb -M + ``` + The output should tell you that 15 out of 50 required modules are still missing. + + * Install the missing modules in the module hierarchy we have generated in `$HOME/hmns/modules`: + ```shell + eb SciPy-bundle-2020.11-foss-2020b.eb --robot --module-only + ``` + Don't forget to use both `--robot` (to enable dependency resolution) and `--module-only` + (to only run the sanity check and generate module files, not install the software again). + + * Start at the top of the module hierarchy (the `Core` level), + and run module spider to check which gateway modules to load to make `SciPy-bundle` available: + ```shell + module use $HOME/hmns/modules/all/Core + module spider SciPy-bundle/2020.11 + ``` + * Load the gateway modules: + ```shell + module load GCC/10.2.0 OpenMPI/4.0.5 + ``` + * Check that the `SciPy-bundle` module is available, and load it: + ```shell + $ module avail SciPy-bundle + ----- /home/example/hmns/modules/all/MPI/GCC/10.2.0/OpenMPI/4.0.5 ------ + SciPy-bundle/2020.11 + ``` + ``` + module load SciPy-bundle/2020.11 + ``` + * Run the test command: + ``` + $ python -c 'import pandas; print(pandas.__version__)' + 1.1.4 + ``` + +!!! Warning + **This exercise will only work if you have write access to the software installation prefix**, + because EasyBuild will try to copy the installation log file to each installlation directory. + +--- + +*[[next: GitHub integration]](3_05_github_integration.md)* diff --git a/docs/2022-CSC_and_LO/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_05_github_integration.md new file mode 100644 index 00000000..59dcbb67 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_05_github_integration.md @@ -0,0 +1,442 @@ +# GitHub integration to facilitate contributing to EasyBuild + +*[[back: Module naming schemes]](3_04_module_naming_schemes.md)* + +--- + +To contribute changes to the EasyBuild code (framework or easyblocks) or easyconfigs, +you will need to be a bit familiar with Git and GitHub. Or maybe not? + +## Manual contribution procedure + +0) Create and setup a [GitHub account](https://github.com/join) (and register your SSH public key); + +1) Clone and fork the appropriate GitHub repository, for example when contributing an easyconfig file: + +```shell +git clone git@github.com:easybuilders/easybuild-easyconfigs.git +cd easybuild-easyconfigs +git remote add my_fork git@github.com:your_github_account/easybuild-easyconfigs.git +``` + +**You should change '`your_github_account`' in the last line to your own GitHub user name!** + +2) Create and check out a new branch, starting from the (up-to-date) ``develop`` branch: + +``` +git checkout develop +git pull origin develop +git checkout -b example +``` + +3) Stage the changes you want to contribute, after you make sure that your easyconfig file has the +[correct filename](../basic_usage/#easyconfig-filenames), and that it's located in the appropriate directory. + +```shell +mkdir -p easybuild/easyconfigs/e/example/ +mv example.eb easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb +git add easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb +``` + +4) Commit those changes with a sensible commit message: + +```shell +git commit -m "This is just an example" +``` + +5) Push your branch to your fork of the repository on GitHub: + +```shell +git push my_fork example +``` + +6) Open the pull request through the GitHub web interface, making sure that: + +* the target branch is correct (should be `develop`); +* an appropriate title is used; +* a short description of the changes is provided; +* the changes are indeed the ones you want to propose; +* clicking the (correct) green button; + +
Pfft
+ +That didn't exactly motivate you to contribute, did it... + +## Github integration features + +Over the years we noticed that some people were keen on contributing to EasyBuild, +but they were not very familiar with Git or GitHub. That meant they had to overcome a +relatively steep learning curve before they could contribute... + +
Gandalf vs Git
+ +In addition, the contribution workflow can be a bit daunting and time consuming, +even if you're already familiar with the procedure. You will have dozens of +branches flying around in no time, and if you get stuck in a weird corner +with `git` you may quickly end up demotivated. + +This is frustrating not only for the people who wanted to contribute but +also for the EasyBuild maintainers, and it doesn't agree with the philosophy of +a project that aims to *automate* tedious software installation procedures. + +At the end of 2015 efforts were made to tackle this issue by implementing +GitHub integration features in EasyBuild, which automate the contribution +workflow by running `git` commands and interacting with the [GitHub API](https://developer.github.com/v3/). + +We will briefly go over some of these features here, but they are also covered in detail [in the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html). + +### Requirements & configuration + +First of all, the GitHub integration features impose a couple of additional [requirements](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) +and configuration. + + +**Additional dependencies** + +Both the `GitPython` and `keyring` Python packages as well as the `keyrings.cryptfile` add-on package must be installed. +In the prepared environment, you can do this via: + +```shell +pip3 install --user GitPython keyring keyrings.cryptfile +``` + +!!! Note + You may experiences problems installing the ``cryptography`` Python packages, + which is a dependency of keyring. The underlying cause is that you need to have + the [``Rust``](https://www.rust-lang.org/) compiler installed to install the latest version + of ``cryptography`` (see [here](https://github.com/pyca/cryptography/issues/5771)). + + You can work around this issue using: + + ```shell + pip3 install --user 'cryptography<3.4' + ``` + +**SSH public key in GitHub account** + +You need to have a GitHub account that has your SSH public key registered in it +(via [https://github.com/settings/keys](https://github.com/settings/keys)). + +If you need to generate an SSH key pair, you can run the following command: + +```shell +ssh-keygen -t rsa -b 4096 +``` + +You can copy the SSH public key from the output of this command: + +```shell +cat .ssh/id_rsa.pub +``` + + +**Forked repository in GitHub** + +In addition, you must have *forked* the EasyBuild repository you want to contribute to +(for example [https://github.com/easybuilders/easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs)). + +**EasyBuild configuration, incl. GitHub token** + +You also have to configure EasyBuild a bit more, so it knows about your +GitHub user name *and* has a GitHub token available in order to perform actions +in GitHub with your credentials. + +To do this, you should define the `github-user` configuration option and +run the "`eb --install-github-token`" command: + +```shell +# replace 'ebtutorial' with your own GitHub username! +$ export EASYBUILD_GITHUB_USER=ebtutorial +$ eb --install-github-token +``` + +To create a GitHub token: + +* Visit [https://github.com/settings/tokens](https://github.com/settings/tokens). +* Click *"Personal access tokens"*. +* Click followed by *"Generate new token"*. +* Give the token a name (for example *"Token for EasyBuild"*). +* Select both the '`repo`' and '`gist`' scopes. +* Click the green *"Generate token"* button. +* Copy the generated token. +* Paste the token when asked by `--install-github-token` (and hit *Enter*). +* Enter a password to encrypt your GitHub token. + +The output should look something like this: + +```shell +$ eb --install-github-token +== temporary log file in case of crash /tmp/eb-9z0bdve9/easybuild-hfpti62w.log +Token: +Validating token... +Token seems to be valid, installing it. +Please set a password for your new keyring: +Please confirm the password: +Token 'fed..987' installed! +``` + + +**Checking status of GitHub integration** + +You can check the status of the GitHub integration using "`eb --check-github`": + +```shell +$ eb --check-github +== temporary log file in case of crash /tmp/eb-4ckdlyfy/easybuild-gp69ev2w.log + +Checking status of GitHub integration... + +Making sure we're online...OK + +* GitHub user...ebtutorial => OK +Please enter password for encrypted keyring: +* GitHub token...fed..987 (len: 40) => OK (validated) +* git command...OK ("git version 1.8.3.1; ") +* GitPython module...OK (GitPython version 3.1.3) +* push access to ebtutorial/easybuild-easyconfigs repo @ GitHub...OK +* creating gists...OK +* location to Git working dirs... not found (suboptimal) + +All checks PASSed! + +Status of GitHub integration: +* --from-pr: OK +* --new-pr: OK +* --review-pr: OK +* --update-pr: OK +* --upload-test-report: OK +``` + +If you see '`OK`' for each of the status checks, you're all set +to try out the GitHub integration features! + +!!! Note + If your SSH private key is protected with a password, you may need + to enter your password a couple of times when running "`eb --check-github`". + + You can avoid this by [using an SSH agent](https://help.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent). + +!!! Note + If you see the ``push access`` check fail with ``Failed to fetch branch 'main'``, + you will need to rename the ``master`` branch in your fork of the ``easybuild-easyconfigs`` + repository from ``master`` to ``main`` (this is required since EasyBuild v4.3.3). + + This can be done via the *pencil* icon at + [https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches](https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches) + (replace ``YOUR_GITHUB_ACCOUNT`` with the name of your GitHub account in this URL). + +### Creating pull requests + +The GitHub integration in EasyBuild allows you to **create pull requests +using the `eb` command**, without even leaving your shell environment. +How cool is that‽ + +To create a pull request to the `easybuild-easyconfigs` repository, +you can either do it in a single go by +running "`eb --new-pr`" and passing it one or more easyconfig files to add +into the pull request. + +The more detailed option is to first create a branch in your repository fork +in GitHub via "`eb --new-branch-github`" and then later open the pull request +via "`eb --new-pr-from-branch`". This method can be useful when preparing multiple +interdependent pull requests, or to check whether your changes pass the unit tests +(which are run automatically in the GitHub Actions CI environment for +all branches pushed to your fork). + +The `--new-pr` option can also be used to open pull requests to the easyblocks +and framework repositories, and it will even automatically determine the target +repository based on the contents of the files you provide. Whoa! + +You can control the target repository for your pull request using +`--pr-target-account` (default is `easybuilders`) and `--pr-target-repo`. + +If you want you can carefully double check your work before actually +opening the pull request by doing a dry run via "`eb --dry-run --new-pr`" +or "`eb -D --new-pr`". + +Finally, you can use "`eb --preview-pr`" to see how the easyconfig files +for which you plan to create a pull request differ from existing easyconfig +files. + +### Updating pull requests + +To update an existing pull request with additional changes +you can use "`eb --update-pr`" and pass the pull request ID, +alongside the paths to the updated files. + +If you have only created a branch (for example via `eb --new-branch-github`) +you can update it via `--update-branch-github` in the same way, +passing the branch name instead of a pull request ID. + +### Using a pull request + +Next to creating and updating branches and pull requests +you can also *use* easyconfig files and easyblocks from a pull request, +regardless of its status (open, merged, or closed). This is particularly +useful when testing contributions, or to install software for which +support is not yet included in the latest EasyBuild release. + +Using the `--from-pr` option you can install easyconfig files from the +pull request with specified ID. By default all easyconfig files that are +touched by the pull request will be installed, but you can specify +particular ones to use as well. It is generally advised to also use the +`--robot` option to ensure that the easyconfig files are installed in the +correct order with respect to dependencies. + +Similarly, using a new or updated easyblock from a pull request is as simple +as using the `--include-easyblocks-from-pr` option. And of course you can +combine it with `--from-pr`! + +Via `--upload-test-report` you can let EasyBuild submit a comment into the +easyconfig pull request to show that the installation worked on your system. This is +useful for others to know, in particular EasyBuild maintainers, since the comment +will include information about your system (OS, processor, etc.) and your EasyBuild configuration. + +## Demo + +That is a lot to digest, so let us make this a bit more concrete with an example: +we will open a pull request for the [`eb-tutorial` example software](../adding_support_software/#example) to *a fork* of the [`easybuild-easyconfigs` repository](https://github.com/easybuilders/easybuild-easyconfigs) using the `eb` command, +and submit a test report in it. + +!!! Note + Make sure that you have correctly configured the GitHub integration, + [see above](#requirements-configuration). + +### Creating pull request + +We first configure EasyBuild to target the `ebtutorial` GitHub account rather +than the default `easybuilders` GitHub organisation, +by defining the `pr-target-account` configuration setting: + +```shell +export EASYBUILD_PR_TARGET_ACCOUNT=ebtutorial +``` + +In the output of "`eb --show-config`" you should see a line like this: + +``` +pr-target-account (E) = ebtutorial +``` + +We only do this to avoid that lots of pull requests for the `eb-tutorial` +example software are opened in the [central easyconfigs repository](https://github.com/easybuilders/easybuild-easyconfigs). + +Opening a pull request is as simple as running "`eb --new-pr`" and passing +the easyconfig file: + +```shell +$ eb --new-pr example.eb +== temporary log file in case of crash /tmp/eb-ggr6scbq/easybuild-hnk271xj.log +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +== fetching branch 'develop' from https://github.com/ebtutorial/easybuild-easyconfigs.git... +== copying files to /tmp/eb-ggr6scbq/git-working-dirxwk1fzaw/easybuild-easyconfigs... +== pushing branch '20200622095415_new_pr_eb-tutorial100' to remote 'github_ebtutorial_qgtfU' (git@github.com:ebtutorial/easybuild-easyconfigs.git) +Enter passphrase for key '/home/example/.ssh/id_rsa': +Please enter password for encrypted keyring: + +Opening pull request +* target: ebtutorial/easybuild-easyconfigs:develop +* from: ebtutorial/easybuild-easyconfigs:20200622095415_new_pr_eb-tutorial100 +* title: "{tools}[GCC/10.2.0] eb-tutorial v1.0.1" +* labels: new +* description: +""" +(created using `eb --new-pr`) + +""" +* overview of changes: + easybuild/easyconfigs/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb | 26 ++++++++++++++++++++++++++ + 1 file changed, 26 insertions(+) + +Opened pull request: https://github.com/ebtutorial/easybuild-easyconfigs/pull/ +== Temporary log file(s) /tmp/eb-ggr6scbq/easybuild-hnk271xj.log* have been removed. +== Temporary directory /tmp/eb-ggr6scbq has been removed. +``` + +Take a moment to grasp what we did here: we ran **a single `eb` command** which +took care of the **[whole contribution procedure](#contribution-procedure)** for us, including: + +* Cloning the `easybuilders/easybuild-easyconfigs` repository and checking out the `develop` branch (in a temporary + directory); +* Picking a sensible name for a branch and creating it; +* Adding the `eb-tutorial` easyconfig file to the branch, in the correct location + (`easybuild/easyconfigs/e/eb-tutorial/`) and with the correct filename (`eb-tutorial-1.0.1-GCC-10.2.0.eb`); +* Pushing the branch to our fork (`example/easybuild-easyconfigs`); +* Actually opening the pull request, using an informative title. + +That is so... easy! + +This feature not only *significantly* lowers the bar for contributing, +it also saves quite a bit of time since you don't need to double check +various details (like targeting the `develop` branch) or spend time on +coming up with a nice looking title or funny branch name (although you +still can if you really want to). + +There are a couple of nice side effects too, like not having any local branches +to tidy up on once the pull request gets merged (since `--new-pr` created the +branch only in a temporary directory). + +If many contributions are made via `--new-pr` it also simplifies the task +of EasyBuild maintainers, since pull requests opened this way have a particular +structure to them and thus are easier to digest because they look familiar. + +### Uploading test report + +After opening the pull request, we should also upload a test report to show that the installation is working. +This is just as easy as creating the pull request. + +First make sure that the pre-installed software in the prepared environment +is available, since the required dependencies for `eb-tutorial` are already +installed there: + +```shell +module use /easybuild/modules/all +``` + +You can verify which dependencies are still missing using `--from-pr` combined with `--missing`: + +```shell +# change '1' to the ID of your own pull request (see output of --new-pr) +$ eb --from-pr 1 --missing +== temporary log file in case of crash /tmp/eb-ioi9ywm1/easybuild-e3v0xa1b.log +Please enter password for encrypted keyring: +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + +1 out of 20 required modules missing: + +* eb-tutorial/1.0.1-GCC-10.2.0 (eb-tutorial-1.0.1-GCC-10.2.0.eb) +``` + +Uploading a test report boils down to combining `--from-pr` with `--upload-test-report`: + +```shell +# change '1' to the ID of your own pull request (see output of --new-pr) +$ eb --rebuild --from-pr 1 --upload-test-report +Please enter password for encrypted keyring: +... +== processing EasyBuild easyconfig /tmp/eb-bnb1pv3n/files_pr65/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb +== building and installing eb-tutorial/1.0.1-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 2 sec) +... +Adding comment to easybuild-easyconfigs issue #65: 'Test report by @ebtutorial +**SUCCESS** +Build succeeded for 1 out of 1 (1 easyconfigs in this PR) +example - Linux centos linux 7.8.2003, x86_64, Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz (haswell), Python 3.6.8 +See https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 for a full test report.' +== Test report uploaded to https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 and mentioned in a comment in easyconfigs PR#1 +``` + +Note that we may need to use `--rebuild` here since `eb-tutorial` may already be installed. + +This results in a comment being added to the pull request: + +
Test report comment
+ +The gist linked from this comment provides more detailed information: + +
Test report gist
+ +--- + +*[[next: Overview]](index.md)* diff --git a/mkdocs.yml b/mkdocs.yml index 4dff4fbe..15c98d78 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -130,6 +130,20 @@ nav: - Installation: 2022-CSC_and_LO/1_06_installation.md - Configuration: 2022-CSC_and_LO/1_07_configuration.md - Basic usage: 2022-CSC_and_LO/1_08_basic_usage.md + - Using EasyBuild: + - (overview): 2022-CSC_and_LO/2_00_part2_using.md + - Troubleshooting: 2022-CSC_and_LO/2_01_troubleshooting.md + - Creating easyconfig files: 2022-CSC_and_LO/2_02_creating_easyconfig_files.md + - Implementing easyblocks: 2022-CSC_and_LO/2_03_implementing_easyblocks.md + - Advanced topics: + - (overview): 2022-CSC_and_LO/3_00_part3_advanced.md + - Using EasyBuild as a library: 2022-CSC_and_LO/3_01_easybuild_library.md + - Using hooks to customise EasyBuild: 2022-CSC_and_LO/3_02_hooks.md + - Submitting installations as Slurm jobs: 2022-CSC_and_LO/3_03_slurm_jobs.md + - Module naming schemes (incl. hierarchical): 2022-CSC_and_LO/3_04_module_naming_schemes.md + - GitHub integration to facilitate contributing to EasyBuild: 2022-CSC_and_LO/3_05_github_integration.md + + plugins: # show revision date at bottom of each page - git-revision-date-localized From 158c78d13b144e65c1417f7a3698cb5576548ac5 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 30 Mar 2022 18:58:28 +0200 Subject: [PATCH 105/159] Troubleshooting section reworked for LUMI. --- docs/2022-CSC_and_LO/2_01_troubleshooting.md | 159 +++++++++++++------ 1 file changed, 108 insertions(+), 51 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_01_troubleshooting.md b/docs/2022-CSC_and_LO/2_01_troubleshooting.md index 86a2f1d3..30f2f5cc 100644 --- a/docs/2022-CSC_and_LO/2_01_troubleshooting.md +++ b/docs/2022-CSC_and_LO/2_01_troubleshooting.md @@ -4,8 +4,8 @@ --- -When installing scientific software you are bound to run into problems -that make the installation fail sooner or later, even when using EasyBuild. +Whatever tool you use, when installing scientific software you'll +be running into problems rather sooner than later. In this part we take a look at how you can **troubleshoot a failing installation**, and focus on a couple of EasyBuild aspects that can be helpful in that context, @@ -50,8 +50,8 @@ $ eb example.eb == building... == FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): build failed (first 300 chars): cmd "make" exited with exit code 2 and output: -/usr/bin/g++ -O2 -ftree-vectorize -march=native -fno-math-errno -std=c++14 -c -o core.o core.cpp -g++: error: unrecognized command line option '-std=c++14' (took 1 sec) +/usr/bin/g++ -O2 -ftree-vectorize -march=znver2 -fno-math-errno -c -o core.o core.cpp +cc1plus: error: bad value (‘znver2’) for ‘-march=’ switch (took 1 sec) == Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log ERROR: Build of /home/easybuild/subread.eb failed (err: ...) ``` @@ -60,22 +60,24 @@ Let's break this down a bit: during the `build` step of the installation procedure EasyBuild was running `make` as a shell command, which failed (exit code 2, so not zero). The `make` command tripped over the compilation of `core.cpp` that failed because -`-std=c++14` is not a known option to the `g++` command. +`-march=znver2` is not a known option to the `g++` command. OK fine, but now what? Can you spot something suspicious here? Wait a minute... Why is `make` using `/usr/bin/g++` for the compilation?! That's not where our toolchain compiler is installed, -that's somewhere under `/easybuild/software`. +that's somewhere under `/opt/cray/pe/gcc`. Let's see what `/usr/bin/g++` is: -```shell +``` $ /usr/bin/g++ --version -g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) +g++ (SUSE Linux) 7.5.0 ``` -Oh my that's pretty ancient (GCC 4.8.5 was released in June 2015). -That could definitely explain why it doesn't know about the C++14 standard yet... +Oh my that's an ancient compiler (7.5 was released on November 14, 2019, +a few months after the release of the Zen2 architecture, but +the base version, 7.1, is really from May 2, 2017, long before the Zen2 architecture was around) +That could definitely explain why it doesn't know about the Zen2 architecture yet... Your next step in this case should probably be figuring out why `/usr/bin/g++` is being used rather than just `g++`, which would @@ -98,7 +100,7 @@ In some cases there won't be any useful information in there however, since the actual error message(s) could only appear way later, perhaps even after the command was already running for several minutes. -In that case, you will have the dive into the log file that is created by EasyBuild for +In that case, you will have to dive into the log file that is created by EasyBuild for every installation, which is located in the unique temporary directory for the EasyBuild session. See for example this output line from our earlier example error message: @@ -122,14 +124,14 @@ directory for successful installation, into the `easybuild` subdirectory. For example: ``` -/easybuild/software/HDF5/1.10.7-gompi-2020b/easybuild/easybuild-HDF5-1.10.7-20210308.214453.log +/appl/lumi/SW/LUMI-21.12/L/EB/ncurses/6.2-cpeGNU-21.12/easybuild/easybuild-ncurses-6.2-20220302.110244.log ``` ### Last log The `eb` command supports a handy little option that prints the location to the most recently updated build log. You can leverage this to quickly -open the build log of the last failed EasyBuild session in an editor: +open the build log of the last ***failed*** EasyBuild session in an editor: ``` vim $(eb --last-log) @@ -206,7 +208,10 @@ For software using a classic `configure` script, you may have to locate and inspect the `config.log` file in the build directory to determine the underlying cause of an error. For software using CMake as a configuration tool you often have to check in `CMakeOutput.log` or `CMakeError.log` for clues, which are sneakily hidden by CMake in a `CMakeFiles` subdirectory of the build directory. -As a side note here: as EasyBuild does not clean out old and failed builds you will need to eventually manually remove these build directories from the `buildpath` directory. +As a side note here: as EasyBuild does not clean out old and failed builds you will need to eventually manually remove these build directories from the `buildpath` directory. On the login nodes of LUMI this is currently not much of an issue as `$XDG_RUNTIME_DIR` is (ab)used for build and +temporary files and that directory is cleaned automatically. However, when building on compute nodes, where `$XDG_RUNTIME_DIR` is +not available, it is essential to manually clean that directory as the space is not automatically cleaned when your session +ends. The `EasyBuild-user` module does define the bash function `clear-eb` that can be used to clear that space. ## Exercise @@ -214,7 +219,7 @@ Let's work our way through a less smooth software installation, using the easyconfig file that is provided below. Don't worry if most of this is still unclear to you, we'll get -to writing easyconfig files from scratch [later in this tutorial](../creating_easyconfig_files). +to writing easyconfig files from scratch [later in this tutorial](2_02_creating_easyconfig_files). ```python easyblock = 'MakeCp' @@ -225,7 +230,7 @@ version = '2.0.1' homepage = 'http://subread.sourceforge.net' description = "High performance read alignment, quantification and mutation discovery" -toolchain = {'name': 'GCC', 'version': '8.5.0'} +toolchain = {'name': 'PrgEnv-gnu', 'version': '21.10'} # download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz sources = ['subread-%(version)s-source.tar.gz'] @@ -253,25 +258,23 @@ Do you spot any potential problems yet with this easyconfig file? ***Preparation*** -Start by copying the text above in a file named `subread.eb`, +Start by copying the text above in a file named `subread.eb` +(which does not follow the EasyBuild conventions but that is not a problem for this exercise), so you can gradually fix the problem you'll encounter. Also make sure that the pre-installed software stack is available, -and that the EasyBuild module is loaded (unless you installed EasyBuild +and that the EasyBuild-user module is loaded (unless you installed EasyBuild yourself): ``` -module use /easybuild/modules/all -module load EasyBuild +module load LUMI/21.12 +module load EasyBuild-user ``` -For this exercise, make sure EasyBuild is configured to -use `$HOME/easybuild` as `prefix`, and to use `/tmp/$USER` as `buildpath`: - -```shell -export EASYBUILD_PREFIX=$HOME/easybuild -export EASYBUILD_BUILDPATH=/tmp/$USER -``` +This will configure EasyBuild correctly for this exercise, though if you already have +an existing EasyBuild user installation you may want to work in a different one +by pointing `$EBU_USER_PREFIX` to the desired work directory before loading +`LUMI/21.12`. Check your configuration via `eb --show-config`. @@ -297,7 +300,7 @@ the easyconfig file? ``` $ eb subread.eb ... - == FAILED: Installation ended unsuccessfully (build directory: /tmp/example/Subread/2.0.1/GCC-8.5.0): build failed (first 300 chars): + == FAILED: Installation ended unsuccessfully (build directory: /run/user/XXXXXXXX/easybuild/build/Subread/2.0.1/cpeGNU-21.12): build failed (first 300 chars): Couldn't find file subread-2.0.1-source.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order): ... ``` @@ -315,12 +318,10 @@ the easyconfig file? (in the `sourcepath` directory): ``` curl -OL https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz - mv subread-2.0.1-source.tar.gz $HOME/easybuild/sources/s/Subread/ + mkdir -p $EBU_USER_PREFIX/sources/s/Subread + mv subread-2.0.1-source.tar.gz $EBU_USER_PREFIX/sources/s/Subread/ ``` - If downloading is problematic for some reason, the source tarball is also available - in `/easybuild/sources/s/Subread`. - Or, we can change the easyconfig file to specify the location where the easyconfig file can be downloaded from: ```python @@ -335,9 +336,9 @@ the easyconfig file? The source tarball is fairly large (23MB), so don't be alarmed if the download takes a little while. ```shell - $ ls -lh $HOME/easybuild/sources/s/Subread + $ ls -lh $EBU_USER_PREFIX/sources/s/Subread total 23M - -rw-rw-r-- 1 easybuild easybuild 23M Jun 13 17:42 subread-2.0.1-source.tar.gz + -rw-rw-r-- 1 XXXXXXXX XXXXXXXX 23M Mar 30 16:08 subread-2.0.1-source.tar.gz ``` --- @@ -350,26 +351,52 @@ What's wrong now? How can you fix it quickly? Take into account that we just want to get this software package installed, we don't care too much about details like the version of the dependencies or -the *toolchain* here... +the toolchain here... ??? success "(click to show solution)" The installation fails because the easyconfig specifies that GCC 8.5.0 should be used as toolchain: + ``` $ eb subread.eb ... - == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-8.5.0): build failed (first 300 chars): - No module found for toolchain: GCC/8.5.0 (took 1 sec) + ERROR: Failed to process easyconfig /pfs/lustrep3/users/kurtlust/easybuild-tutorial/Troubleshooting/subread.eb: Toolchain PrgEnv-gnu not found, + available toolchains: ... + ... + ``` + + `PrgEnv-gnu` is an HPE Cray PE module that may look like a toolchain - it certainly has + the same function: provide compiler, MPI and basic math libraries - but it is not + recognised as a toolchain by EasyBuild. EasyBuild prefers to manage its own modules so that it knows + well what is in it which is not the case with the `PrgEnv-*` modules from the Cray PE + as the content may differe between systems and as the versions of the compilers etc. that + are loaded differ on other modules that are loaded. Hence we created Cray-specific toolchains. + You'll actually find two series of Cray toolchains in the list of available toolchains. The + `CrayGNU`, `CrayIntel`, `CrayPGI` and `CrayCCE` are included with the EasyBuild distribution + and where developed by CSCS for their systems using Environment Modules. These were not compatible + with the initial releases of the Cray PE with Lmod modules so new ones were developed on which we + also built for the LUMI toolchains. Those are called `cpeCray`, `cpeGNU`, `cpeAOCC` and `cpeAMD` + and are maintained by LUST and available via the LUMI repositories. + + Changing the toolchain name to `cpeGNU` is not enough to solve all problems though: + + ``` + $ eb subread.eb + ... + ERROR: Build of /pfs/lustrep3/users/kurtlust/easybuild-tutorial/Troubleshooting/subread.eb failed (err: 'build failed (first 300 chars): + No module found for toolchain: cpeGNU/21.10') + ... ``` - We don't have this GCC version installed, but we do have GCC 10.2.0: + We don't have this `cpeGNU` version installed, but we do have GCC 21.12: ```shell - $ module avail GCC/ - ----------------- /easybuild/modules/all ------------------ - GCC/10.2.0 + $ module avail cpeGNU/ + ----- Infrastructure modules for the software stack LUMI/21.12 on LUMI-L ----- + cpeGNU/21.12 + ... ``` So let's try using that instead. @@ -377,7 +404,7 @@ the *toolchain* here... Edit the easyconfig file so it contains this: ```python - toolchain = {'name': 'GCC', 'version': '10.2.0'} + toolchain = {'name': 'cpeGNU', 'version': '21.12'} ``` --- @@ -392,13 +419,13 @@ Can you fix the next problem you run into? The compilation fails, but the error message we see is incomplete due to EasyBuild truncating the command output (only the 300 first characters of the output are shown): ``` - == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-10.2.0): build failed (first 300 chars): - cmd " make -j 1 -f Makefile.Linux CFLAGS="-fast -fcommon"" exited with exit code 2 and output: - gcc -mtune=core2 -O3 -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\""2.0.1"\" -D_FILE_OFFSET_BITS=64 -fmessage-length=0 -ggdb -fast -fcommon -c -o core.o core.c - gcc: error: unrecognized command line opti (took 1 sec) + == FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12): build failed + (first 300 chars): cmd " make -j 256 -f Makefile.Linux CFLAGS="-fast -fcommon"" exited with exit code 2 and output: + gcc -mtune=core2 -O3 -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\""2.0.1"\" -D_FILE_OFFSET_BITS=64 -fmessage-length=0 + -ggdb -fast -fcommon -I/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86 (took 4 secs) ``` - If you open the log file and scroll to the end, + If you open the log file (e.g., with `view $(eb --last-log)`) and scroll to the end, the error is pretty obvious: ``` gcc: error: unrecognized command line option -fast; did you mean -Ofast? @@ -416,7 +443,8 @@ Can you fix the next problem you run into? to hard specify compiler flags (certainly not incorrect ones). The comment above the `buildopts` definition makes it clear that the `-fcommon` flag *is* required though, because GCC 10 became a bit stricter by - using `-fno-common` by default. Note that we are using `-fcommon` + using `-fno-common` by default (and we're using GCC 11 in `cpeGNU/21.12`). + Note that we are using `-fcommon` as an escape mechanism here: it would be better to fix the source code and create a patch file instead. @@ -433,6 +461,32 @@ Can you fix the next problem you run into? Note that we need to be careful with quotes here: we use inner double quotes to ensure that `$CFLAGS` will be expanded to its value when the build command is run. + As you can see after the change by doing a dry-run: + + ``` + $ eb subread.eb -x + ... + + Defining build environment... + + export BLAS_INC_DIR='/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/include' + ... + export CC='cc' + export CFLAGS='-O2 -ftree-vectorize -fno-math-errno' + ... + [build_step method] + running command "make -j 256 -f Makefile.Linux CFLAGS="$CFLAGS -fcommon"" + (in /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12/Subread-2.0.1/src) + ... + ``` + + EasyBuild will launch the command + ``` + make -j 256 -f Makefile.Linux CFLAGS="$CFLAGS -fcommon" + ``` + in a shell where `CFLAGS` is defined and set to an appropriate value (determined by + defaults in EasyBuild, settings in the EasyBuild configuration and settings in the + easyconfig file that we shall discuss later). --- @@ -449,10 +503,11 @@ Don't give up now, try one last time and fix the last problem that occurs... ``` $ eb subread.eb ... - == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-10.2.0): build failed (first 300 chars): - Sanity check failed: sanity check command featureCounts --version exited with code 255 + == FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12): + build failed (first 300 chars): Sanity check failed: sanity check command featureCounts --version exited with code 255 (output: featureCounts: unrecognized option '--version' ... + ... ``` If you look at the full output in the log file you can see @@ -473,12 +528,14 @@ Don't give up now, try one last time and fix the last problem that occurs... --- -In the end, you should be able to install Subread 2.0.1 with the GCC 10.2.0 toolchain by fixing the problems with the `subread.eb` easyconfig file. +In the end, you should be able to install Subread 2.0.1 with the cpeGNU 21.12 toolchain by fixing the problems with the `subread.eb` easyconfig file. Check your work by manually loading the module and checking the version via the `featureCounts` command, which should look like this: ```shell +$ module load Subread/2.0.1-cpeGNU-21.12 +... $ featureCounts -v featureCounts v2.0.1 ``` From 202a53b7359023b0d3139c75515bf0369f324ede Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 1 Apr 2022 18:15:00 +0200 Subject: [PATCH 106/159] Section on creating easyconfig files. --- .../2_02_creating_easyconfig_files.md | 527 +++++++++--------- 1 file changed, 278 insertions(+), 249 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md index d46e003a..99ac7d88 100644 --- a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md @@ -4,8 +4,9 @@ --- -We already know how to install easyconfig files that are included with EasyBuild, -but what about installing software for which no easyconfig is available yet? +We already know how to install easyconfig files that are provided by LUMI, +but what about installing software for which no easyconfig is available yet, +or for which we need to adapt an existing easyconfig? To do this we will need to **create additional easyconfig files**, since every software installation performed by EasyBuild is done based on an easyconfig file. @@ -14,10 +15,10 @@ In this part of the tutorial we will look at the guts of easyconfig files and ev ## Easyconfigs vs easyblocks -Before we dive into writing [easyconfig files](../introduction/#easyconfig-files), -let us take a brief look at how they relate to [easyblocks](../introduction/#easyblocks). +Before we dive into writing [easyconfig files](../1_05_terminology/#easyconfig-files), +let us take a brief look at how they relate to [easyblocks](../1_05_terminology/#easyblocks). -As we discussed [earlier](../introduction/#terminology), an easyconfig file (`*.eb`) is required +As we discussed [earlier](../1_05_terminology/#terminology), an easyconfig file (`*.eb`) is required for each installation performed by EasyBuild which specifies the details of the installation (which software version, toolchain, etc.), while the installation procedure is implemented @@ -30,34 +31,47 @@ together with a custom *software-specific* easyblock? This is not an easy question to answer in a general sense, since it depends on several factors: the complexity of the software you want to get installed, how much flexibility you want, how "intelligent" the installation procedure should be with respect to the compiler toolchain and dependencies -that are used for the installation, etc. +that are used for the installation, whether you want an easyconfig that is more user-focused (with easy-to-understand +parameters) or one that is more focused towards people who understand installation commands, etc. -In a nutshell, custom software-specific easyblocks are "do once and forget": they are central solution to peculiarities in the installation procedure of a particular software package. +In a nutshell, custom software-specific easyblocks are "do once and forget": they are central solution to peculiarities in the installation procedure of a particular software package. However, they also hide a lot from direct view, so if things go +wrong, it is often harder to debug the exact problem. And since a single easyblock has to cover +multiple software versions, multiple toolchains and multiple possible configurations of the package, +they are often harder to develop and certainly harder to test. As a result many of the easyblocks +included with EasyBuild work poorly on HPE Cray systems, e.g., because they want to add compiler flags +specific for a certain compiler and don't recognise the Cray compilers. Reasons to consider implementing a software-specific easyblock rather than using a generic easyblock include: -* 'critical' values for easyconfig parameters required to make installation succeed; -* toolchain-specific aspects of the build and installation procedure (e.g., configure options); -* interactive commands that need to be run; -* custom (configure) options for dependencies; -* having to create or adjust specific (configuration) files; -* 'hackish' usage of a generic easyblock; -* complex or very non-standard installation procedure; - -Implementing easyblocks is out of scope for this basic tutorial, for more information please consult -the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). +- 'critical' values for easyconfig parameters required to make installation succeed; +- toolchain-specific aspects of the build and installation procedure (e.g., configure options); +- interactive commands that need to be run; + *For example: The easyblock to [install maple](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/m/maple.py) + responds to a number of interactive questions.* +- custom (configure) options for dependencies; + *For example, the [easyblock for VMD](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/v/vmd.py) + will add configure options depending on the dependency list of the package.* +- having to create or adjust specific (configuration) files; +- 'hackish' usage of a generic easyblock; +- complex or very non-standard installation procedure; + *For example, the [easyblock to install the gcc compilers from source](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/g/gcc.py), + bootstrapping with the system compiler and then re-installing with itself.* + +For implementing easyblocks we refer to the +["Implementing easyblocks" section of this tutorial](2_03_implementing_easyblocks.md) +and the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). ## Writing easyconfig files Writing an easyconfig file boils down to defining a set of easyconfig parameters in a text file, which we give a '`.eb`' extension by convention. The name of an easyconfig file doesn't matter when using it directly to install software, but it does matter when EasyBuild needs to find it -to resolve a dependency for example (as we [discussed earlier](../basic_usage/#easyconfig-filenames)). +to resolve a dependency for example (as we [discussed earlier](../1_08_basic_usage/#easyconfig-filenames)). The syntax for easyconfig files is *Python syntax*: you are basically defining a bunch of Python variables that correspond to easyconfig parameters. -The order in which the easyconfig parameters are defined doesn't matter, but we generally try to strick to a particular +The order in which the easyconfig parameters are defined doesn't matter, but we generally try to stick to a particular order which roughly corresponds to the order in which the easyconfig parameters are used during the installation. That is mostly helpful for humans staring at easyconfig files or comparing them. @@ -65,13 +79,13 @@ That is mostly helpful for humans staring at easyconfig files or comparing them. A limited number of easyconfig parameters are *mandatory*, they must be defined in every easyconfig file: -* `name`: the name of the software to install; -* `version`: the version of the software to install; -* `homepage`: a URL to the website of the software; -* `description`: a short description of the software; -* `toolchain`: the compiler toolchain to use for the installation; +- `name`: the name of the software to install; +- `version`: the version of the software to install; +- `homepage`: a URL to the website of the software; +- `description`: a description of the software; +- `toolchain`: the compiler toolchain to use for the installation; -**`name`, `version`** +#### `name`, `version` It should be no surprise that specifying the name and version of the software you want to install is mandatory. This information may influence the value of several other easyconfig parameters (like the name of the source file), and is also used to the determine the name of the module file to install. @@ -81,7 +95,7 @@ name = 'example' version = '1.0' ``` -**`homepage`, `description`** +#### `homepage`, `description` The homepage and description are included in the generated module file for the installation. That way the "`module show`" command provides some useful high-level information about the installation. @@ -91,10 +105,8 @@ homepage = 'https://example.org' description = "This is just an example." ``` -Usually it does not matter whether you use single or double quotes to specify string values, -but you will often see that single quotes are used for values that don't have spaces (words) -and double quotes for values that do have spaces (sentences). There is no technical reason for -this, it just feels more natural to some people. There are cases where it is important to use +Usually it does not matter whether you use single or double quotes to specify string values. +There are cases where it is important to use the right type of quotes however, we will get back to that later (keep it in mind for the exercises!). For multi-line descriptions, you will need to use "triple quoting" (which is standard Python syntax): @@ -105,23 +117,43 @@ description = """This is an example It is spread across multiple lines.""" ``` -**`toolchain`** +If no homepage is known for the package, the convention in the EasyBuild community is to use +`(none)` as the value. + +The `description` field is used in two different places in the LUA module file generated by +EasyBuild: +- In the help block, for the first section of the help information +- If there is no `whatis` parameter in the easyconfig file, it is also used to generate + a whatis-command with the description in the module file. + +*Note:* We ask people who contribute easyconfigs to LUMI to instead use a decent description +that is useful in the help information, and to specify a short description via the `whatis` +parameter: + +```python +whatis = ['Description: Blosc is an extremely fast, multi-threaded, meta-compressor library'] +``` + +Note that in this case the word "Description:" has to be explicitly specified. -EasyBuild also requires that the [compiler toolchain](../introduction/#toolchains) is specified, via the `toolchain` + +#### `toolchain` + +EasyBuild also requires that the [compiler toolchain](../1_05_terminology/#toolchains) is specified, via the `toolchain` easyconfig parameter. -This can either be the [`system` toolchain](../introduction/#system-toolchain), for which a constant named `SYSTEM` is available: +This can either be the [`system` toolchain](../1_05_terminology/#system-toolchain), for which a constant named `SYSTEM` is available: ```python toolchain = SYSTEM ``` -Usually we specify a 'proper' toolchain like the compiler-only toolchain GCC 10.2.0 which we used before, -or the full toolchain `foss` 2020b. The name and version of the toolchain can be specified using a small Python dictionary, +Usually we specify a 'proper' toolchain like the `cpeGNU/21.12` toolchain we have used bvefore. +The name and version of the toolchain can be specified using a small Python dictionary, for example: ```python -toolchain = {'name': 'GCC', 'version': '10.2.0'} +toolchain = {'name': 'cpeGNU', 'version': '21.12'} ``` ### Commonly used parameters @@ -177,8 +209,11 @@ Some things worth pointing out here: file. This way the software version is only specified in one place and the easyconfig file is easier to update to other software versions. A list of template values can be consulted via the EasyBuild command line via the `--avail-easyconfig-templates` option, or in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_templates.html). -* Source files can also be specified in ways other than just using a filename, see the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-param-sources-alt) for more information. -* Specified checksums are usually SHA256 checksum values, but [other types are also supported](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html?highlight=checksums#checksums). +* Source files can also be specified in ways other than just using a filename, see the + [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-param-sources-alt) for more information. + It is also possible to download a given commit from a GitHub repository. +* Specified checksums are usually SHA256 checksum values, but + [other types are also supported](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html?highlight=checksums#checksums). #### Easyblock @@ -189,7 +224,9 @@ This is not mandatory however, because by default EasyBuild will determine the e name of the software. If '`example`' is specified as software name, EasyBuild will try to locate a software-specific easyblock named `EB_example` (in a Python module named `example.py`). Software-specific easyblocks follow the convention that the class name starts with `'EB_`', followed by the software name -(where some characters are replaced, like '`-`' with '`_minus_`'). +(where some characters are replaced, like '`-`' with '`_minus_`'). It is possible to use different +naming conventions for software-specific easyblocks, but then EasyBuild will not automatically detect +that there is one for the package and it will also need to be specified via the `easyblock` parameter. **Generic easyblocks** @@ -234,7 +271,7 @@ tar_config_opts* Override tar settings as determined by configure. [defau #### Dependencies -You will often need to list one or more [dependencies](../introduction/#dependencies) that are required +You will often need to list one or more [dependencies](../1_05_terminology/#dependencies) that are required to install or run the software. We distinguish between two main different types of dependencies: runtime dependencies and build dependencies. @@ -248,12 +285,17 @@ software once it is installed. The modules for these dependencies will be loaded set up by EasyBuild during the installation, but they will *not* be loaded by the generated module file. You can specify build dependencies via the `builddependencies` easyconfig parameter. One typical example of a build dependency is `CMake`, which is only needed for configuring -the build. +the build. On LUMI we define a `buildtools` module for each version of the `LUMI` software stack +which contains most popular build tools. This ensures that we use a consistent set of build tools +and that they do not change over the life time of a stack, also in case of OS upgrades, to +increase the reproducibility of the build process Here is a simple example of specifying dependencies: ```python -builddependencies = [('CMake', '3.18.4')] +builddependencies = [ + ('buildtools', '%(toolchain_version)s', '', True) +] dependencies = [ ('Python', '3.8.2'), @@ -268,20 +310,38 @@ The name and version of a dependency is specified with a 2-tuple (a tuple with t In some cases additional information may have to be provided, as is shown in the example above for the `SciPy-bundle` dependency where a 3rd value is specified corresponding to the `versionsuffix` value of this dependency. -If this is not specified, it is assumed to be the empty string (`''`). +If this is not specified, it is assumed to be the empty string (`''`). Note how we use the '`%(pyver)s'` template value in the `SciPy-bundle` dependency -specification, to avoid hardcoding the Python version in different places. +specification, to avoid hardcoding the Python version in different places. (Though this +specific parameter is less useful on LUMI as we currently try to build on top of `cray-python`.) + +The `buildtools` build dependency shows that there is a fourth parameter specifying the toolchain +used for that dependency and is needed if that toolchain is different from the one used in the example. +As it is not possible to load several Cray toolchains together (they are not in a hierachical relation) +the only useful value on LUMI is `True` which tells that `buildtools` is build with the `SYSTEM` +toolchain. Here also we use a template, `%(toolchain_version)s` which - as its name suggests - expands +to the version of the toolchain, as we version our `buildtools` modules after the version of the Cray +toolchains for which they are intended. + +See also the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#dependencies) +for additional options on specifying dependencies. That page specifies two more dependency types: -See also the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#dependencies) for additional options on specifying dependencies. +* `hiddendependencies` are currently not used on LUMI, and if we would use them in the future, it will likely + be through a way that does not require this parameter. +* `osdependencies` can be used to let EasyBuild check if certain needed OS packages are installed. + See, e.g., the [easyconfigs for the `buildtools` package](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/easybuild/easyconfigs/b/buildtools) + on LUMI. #### Version suffix -In some cases you may want to build a particular software package in different configurations, or include a label in the module name to highlight a particular aspect +In some cases you may want to build a particular software package in different configurations, +or include a label in the module name to highlight a particular aspect of the installation. -The `versionsuffix` easyconfig parameter can be used for this purpose. The name of this parameter implies that this label will be added after the +The `versionsuffix` easyconfig parameter can be used for this purpose. +The name of this parameter implies that this label will be added after the software version (and toolchain label) in the standard module naming scheme. If you are configuring the software to build with a particular non-default value, @@ -303,7 +363,8 @@ dependencies = [('Python', '2.7.18')] Even though Python 2 is officially dead and buried some scientific software still requires it, and mixing modules where some use Python 2 and other use Python 3 doesn't work well. -The `versionsuffix` label is helpful to inform the user that a particular Python version is required by the installation. +The `versionsuffix` label is helpful to inform the user that a particular Python version is +required by the installation. #### Customizing configure, build, test, and install commands @@ -330,6 +391,7 @@ prebuildopts = 'export HDF5_PREFIX="$EBROOTHDF5" && ' installopts = "PREFIX='%(installdir)s'" ``` +(again an imaginary example as on LUMI we advise to use the Cray-provided HDF5 modules.) Here we are: @@ -406,7 +468,13 @@ Finally, you will usually see the `moduleclass` easyconfig parameter to be defin moduleclass = 'lib' ``` -This is done to categorize software, and it is used to group the generated module files into smaller sets ([remember what we saw when installing software earlier](../basic_usage/#using-installed-software)). +This is done to categorize software, and it is used to group the generated module files into smaller sets +([remember what we saw when installing software earlier](../1_08_basic_usage/#using-installed-software)). + +This is currently not used on LUMI since we feel that (a) it is not easy to explain to users how they can +then only make certain classes that are useful to them visible and more importantly (b) since it is not +always intuitive to decide which moduleclass should be used for a package, or from a user 's perspective, +in which category to look for a package. ## Generating tweaked easyconfigs @@ -428,7 +496,7 @@ eb example-1.2.3.eb --try-software-version 1.2.4 Or, to try using a different compiler toolchain you can use `--try-toolchain`: ```shell -eb example-1.2.3-foss-2020b.eb --try-toolchain intel,2020b +eb example-1.2.3-foss-2020b.eb --try-toolchain cpeCray,21.12 ``` It is important to keep in mind the *"try"* aspect here: while easyconfigs that @@ -477,20 +545,18 @@ You can consult the unpacked sources at [https://github.com/easybuilders/easybui ### Preparation -Make sure EasyBuild is properly configured before you start: +Make sure that a proper version of the `LUMI` software stack is loaded +(we recommend `LUMI/21.12` for the session for which this tutorial was designed) +and also `EasyBuild-user` is loaded to configure EasyBuild to install in +`$HOME/EasyBuild` or `$EBU_USER_PREFIX`. -```shell -export EASYBUILD_PREFIX=$HOME/easybuild -export EASYBUILD_BUILDPATH=/tmp/$USER +``` shell +module load LUMI/21.12 +module load EasyBuild-user` ``` -and that the installed software in `/easybuild` is available: -```shell -module use /easybuild/modules/all -``` - -### Mandatory parameters +### Easyblock Let's start by getting the mandatory easyconfig parameters defined in the easyconfig file: @@ -499,48 +565,26 @@ name = 'eb-tutorial' version = '1.0.1' homepage = 'https://easybuilders.github.io/easybuild-tutorial' -description = "EasyBuild tutorial example" -``` -If we try using this (very) minimal easyconfig file, EasyBuild will inform us that we failed to specify one of the mandatory easyconfig parameters: `toolchain`: +whatis = [ 'Description: EasyBuild tutorial example'] -``` -$ eb example.eb -== Temporary log file in case of crash /tmp/eb-90j723rl/easybuild-q21plqvx.log -== found valid index for /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs, so using it... -ERROR: Failed to process easyconfig /home/example/example.eb: mandatory parameters not provided in pyheader: toolchain +description = """ +This is a short C++ example program that can be buid using CMake. +""" ``` -We will use `GCC/10.2.0` as toolchain, since we know it is already installed in `/easybuild` in the prepared environment, so we also define the `toolchain` easyconfig parameter: - -```python -toolchain = {'name': 'GCC', 'version': '10.2.0'} -``` - -In addition, we'll also specify the `moduleclass`. -This is not required, but it is usually set to a sensible value: - -```python -moduleclass = 'tools' -``` - -The default value is '`base`', at least '`tools`' has *some* meaning. - -### Easyblock - -Let us see what happens if we take our current easyconfig file for a spin: +Let's see what EasyBuild does with this very minimal easyconfig file: ```shell -$ eb example.eb -== temporary log file in case of crash /tmp/eb-8_vxjfn7/easybuild-k3aaoan2.log -ERROR: Failed to process easyconfig /home/example/example.eb: +$ eb eb-tutorial.eb +== Temporary log file in case of crash /run/user/XXXXXXXXX/easybuild/tmp/eb-k_82wotb/easybuild-wg0k_reb.log +ERROR: Failed to process easyconfig /pfs/lustrep4/users/XXXXXXXX/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v1.eb: No software-specific easyblock 'EB_eb_minus_tutorial' found for eb-tutorial ``` -That didn't get us very far... - -The error shows that there is no software-specific easyblock available for installing the software with the name '`eb-tutorial`'. -Does that mean we have to implement an easyblock? +It is not mandatory to specify an easyblock in the easyconfig. However, in the absense of that +specification, EasyBuild goes looking for an application-specific easyblock with the standard name, +in this case `EB_eb_minus_tutorial`, which it does not have. Does that mean we have to implement an easyblock? In this simple case it doesn't, since we can leverage one of the available *generic easyblocks*. But, which one? @@ -559,81 +603,90 @@ easyblock = 'CMakeMake' The "`easyblock =`" line is usually at the top of the easyconfig file, but strictly speaking the order of the parameter definitions doesn't matter (unless one is defined in terms of another one). -### CMake build dependency -Does using the `CMakeMake` generic easyblock help at all? +### Mandatory parameters + +When trying this improved easyconfig file, EasyBuild will inform us that we failed to specify +one of the mandatory easyconfig parameters: `toolchain`: ``` $ eb example.eb -== temporary log file in case of crash /tmp/eb-yutbor1p/easybuild-4jc9v1u9.log -== found valid index for /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs, so using it... -== processing EasyBuild easyconfig /home/example/example.eb -== building and installing eb-tutorial/1.0.1-GCC-10.2.0... -== fetching files... -== creating build dir, resetting environment... -== unpacking... -== patching... -== preparing... -== configuring... -== FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/ebtutorial/1.0.1/GCC-10.2.0): -build failed (first 300 chars): cmd " cmake -DCMAKE_INSTALL_PREFIX=/home/example/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER='gcc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno' -DCMAKE_CXX_COMPILER='g++' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=native (took 0 sec) +== Temporary log file in case of crash /run/user/XXXXXXXXX/easybuild/tmp/eb-22om7hut/easybuild-ipozjipf.log +ERROR: Failed to process easyconfig /pfs/lustrep4/users/XXXXXXXX/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v2.eb: +mandatory parameters not provided in pyheader: toolchain ``` -It did help: EasyBuild made an attempt to configure the build using the `cmake` command, but that failed almost -instantly. We need to dive into the log file to see the actual reason. By starting at the end of the log file and -scrolling up, you should be able to locate the following error message: +We will use `cpeCray/21.12` as toolchain, so we also define the `toolchain` easyconfig parameter: +```python +toolchain = {'name': 'cpeCray', 'version': '21.12'} ``` -/bin/bash: cmake: command not found + +In addition, we'll also specify the `moduleclass`. +This is not required, but it is usually set to a sensible value: + +```python +moduleclass = 'tools' ``` -Ah, that explains it, `cmake` isn't even installed on this system. Or is it? +The default value is '`base`', at least '`tools`' has *some* meaning. -```shell -$ module avail CMake +### CMake build dependency ---------------------------- /easybuild/modules/all ---------------------------- - CMake/3.18.4-GCCcore-10.2.0 -``` +The `CMakeMake` easyblock needs the `cmake` command. On LUMI we are lucky as +`cmake` is already installed in the OS. It may be a somewhat older version, but for this +program is is probably enough. Yet in general it is better to use sufficiently recent +build tools, and `cmake` is one of those tools that is typically entered as a build +dependency. After all, the less you rely on the OS, the more likely it becomes that +your easyconfig is useful for other sites also. -Since a module is available for `CMake` that is compatible with the toolchain we are using (GCC 10.2.0), -we can use it as a dependency for the installation. -It is only needed for building the software, not for running it, so it's only a *build* dependency: +In the [section on Lmod](../1_02_Lmod#module-extensions) we've already seen that on LUMI +the `cmake` command is available through the `buildtools` modules, and as discussed in +other examples on this page, LUMI has one for every `LUMI` software stack with its version +number the same as the stack and corresponding toolchains. It is a good practice to +add this module as a build dependency: ```python -builddependencies = [('CMake', '3.18.4')] +builddependencies = [ + ('buildtools', '%(toolchain_version)s', '', True) +] ``` -There is usually no need to specify toolchain for (build) dependencies, EasyBuild will automatically consider -[subtoolchains](../terminology/#toolchains) compatible with the specified toolchain to locate module for the dependencies. +In a more traditional EasyBuild setup with the common toolchains, there is usually no need to specify +the toolchain for (build) dependencies. EasyBuild will automatically consider +[subtoolchains](../1_05_terminology/#toolchains) compatible with the specified toolchain to locate +modules for the dependencies. However, the Cray PE toolchains on LUMI are currently not part of +such a hierarchy and the `SYSTEM` toolchain we used for `buildtools` is not automatically considered +which is why we need the 4-element version of the dependency specification. -You can verify this via `eb -D` (equivalent with `eb --dry-run`): +You can verify that EasyBuild now locates the dependency via `eb -D` (equivalent with `eb --dry-run`): ``` -$ eb example.eb -D +$ eb eb-tutorial.eb -D ... - * [x] /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs/g/GCC/GCC-10.2.0.eb (module: GCC/10.2.0) - * [x] /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs/c/CMake/CMake-3.18.4-GCCcore-10.2.0.eb (module: CMake/3.18.4-GCCcore-10.2.0) - * [ ] /home/example/example.eb (module: eb-tutorial/1.0.1-GCC-10.2.0) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeCray/cpeCray-21.12.eb (module: cpeCray/21.12) + * [ ] /pfs/lustrep4/users/kulust/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v4.eb (module: eb-tutorial/1.0.-cpeCray-21.12) ``` ### Sources -If you try again after adding `CMake` as a build dependency, you will see the installation fail again in the +If you try again after adding `buildtools` as a build dependency, you will see the installation fail again in the configuration step. Inspecting the log file reveals this: ``` -CMake Error: The source directory "/tmp/example/ebtutorial/1.0.1/GCC-10.2.0" does not appear to contain CMakeLists.txt. +CMake Error: The source directory "/run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12" does not appear to contain CMakeLists.txt. ``` -Wait, but there *is* a `CMakeLists.txt`, we can see it in the [unpacked sources](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1)! +Wait, but there *is* a `CMakeLists.txt`, we can see it in the +[unpacked sources](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1)! Let's inspect the build directory: ``` -$ ls /tmp/$USER/ebtutorial/1.0.1/GCC-10.2.0 +$ ls /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12 easybuild_obj -$ ls /tmp/$USER/ebtutorial/1.0.1/GCC-10.2.0/easybuild_obj +$ ls /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12/easybuild_obj $ ``` @@ -672,7 +725,8 @@ If now we try installing the easyconfig file again, EasyBuild complains that it can't find the specified source file anywhere: ``` -Couldn't find file eb-tutorial-1.0.1.tar.gz anywhere, and downloading it didn't work either... +Couldn't find file eb-tutorial-1.0.1.tar.gz anywhere, and downloading it didn't work either... +Paths attempted (in order):... ``` To let EasyBuild automatically download the source file if it is not available yet, @@ -716,11 +770,13 @@ expanded when the configure command is run (see [Exercise 7.1](#exercises)). Hopefully that brings us closer to getting the installation to work... ``` -$ eb example.eb +$ eb eb-tutorial.eb .... == sanity checking... -== FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/ebtutorial/1.0.1/GCC-10.2.0): build failed (first 300 chars): -Sanity check failed: no (non-empty) directory found at 'lib' or 'lib64' in /home/easybuild/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 (took 2 sec) +== ... (took 2 secs) +== FAILED: Installation ended unsuccessfully (build directory: /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12): +build failed (first 300 chars): Sanity check failed: no (non-empty) directory found at 'lib' or 'lib64' in +/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 (took 7 secs) ``` It got all the way to the sanity check step, that's great! @@ -729,9 +785,9 @@ The sanity check failed because no '`lib`' or `'lib64'` directory was found. Indeed: ``` -$ ls $HOME/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 +$ ls /users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 bin -$ ls $HOME/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0/bin +$ ls /users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 eb-tutorial ``` @@ -757,10 +813,11 @@ a module, we know it will work as expected. By enabling trace mode via `--trace` we can get some more information too: ```shell -$ eb example.eb --module-only --trace +$ eb eb-tutorial.eb --module-only --trace ... == sanity checking... >> file 'bin/eb-tutorial' found: OK + >> loading modules: eb-tutorial/1.0.1-cpeCray-21.12... >> running command 'eb-tutorial' ... >> result for command 'eb-tutorial': OK ... @@ -773,7 +830,6 @@ To convince yourself that the installation works as intended, try to load the `e run the `eb-tutorial` command yourself: ``` -$ module use $HOME/easybuild/modules/all $ module load eb-tutorial $ eb-tutorial Hello from the EasyBuild tutorial! @@ -820,12 +876,73 @@ You can let EasyBuild determine *and* inject these SHA256 checksums automatically via `eb --inject-checksums`: ``` -$ eb example.eb --inject-checksums +$ eb eb-tutorial.eb --inject-checksums ... == injecting sha256 checksums for sources & patches in example.eb... == * eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 ``` +Finally, we should consider changing the name of the easyconfig that we just developed +to align with the EasyBuild conventions as otherwise it would not be found when used as +a dependency of another package. In this case, the name should be +`eb-tutorial-1.0.0-cpeCray-21.12.eb`. In fact, EasyBuild stored a processed version +of our easyconfig with that name in the repository: + +``` +$ ls $EASYBUILD_REPOSITORYPATH +eb-tutorial +$ ls $EASYBUILD_REPOSITORYPATH/eb-tutorial +eb-tutorial-1.0.1-cpeCray-21.12.eb +$ cat $EASYBUILD_REPOSITORYPATH/eb-tutorial/eb-tutorial-1.0.1-cpeCray-21.12.eb +... +``` + +Note that EasyBuild has added an additional parameter to the easyconfig file, `buildstats`, +with a lot of information about how easybuild was called and properties of the node on which +it was run. This file is still a valid easyconfig file though from which we can build the +program again. + +Let's also inspect the installation directory when the whole build process has finished successfully. +This is very easy after loading the module, as EasyBuild-generated modules define a number of environment +variables for each module: + +``` +$ module load eb-tutorial +$ env | grep TUTORIAL +EBVERSIONEBMINTUTORIAL=1.0.1 +EBDEVELEBMINTUTORIAL=/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12/easybuild/eb-tutorial-1.0.1-cpeCray-21.12-easybuild-devel +EBROOTEBMINTUTORIAL=/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 +``` +The most interesting one of those variables is the `EBROOT` variable which points to the installation directory. +As variable names cannot contain minus signs, the minus in the module name is replaced with `MIN` in the name +of the variable (which is not the most consistent thing however as in the name of an easyblock it is replaced +with `minus`). + +Let's have a look in that directory: + +``` +$ ls $EBROOTEBMINTUTORIAL +bin easybuild +$ ls EBROOTEBMINTUTORIAL/easybuild +easybuild-eb-tutorial-1.0.1-20220401.184518.log +easybuild-eb-tutorial-1.0.1-20220401.184518_test_report.md +eb-tutorial-1.0.1-cpeCray-21.12-easybuild-devel +eb-tutorial-1.0.1-cpeCray-21.12.eb +reprod +$ ls EBROOTEBMINTUTORIAL/easybuild/reprod +easyblocks +eb-tutorial-1.0.1-cpeCray-21.12.eb +eb-tutorial-1.0.1-cpeCray-21.12.env +hooks +ls $EBROOTEBMINTUTORIAL/easybuild/reprod/easyblocks +cmakemake.py configuremake.py +``` + +As you can see, EasyBuild has also created the `easybuild` subdirectory (and it actually told us about that +at the end of the installation) which contains a lot of information about the build, also to make it easier +to reproduce a build process afterwards. + + --- ## Exercises @@ -854,6 +971,9 @@ a bit more personal: include the username of the account that was used to instal ``` To re-install the `example.eb` easyconfig, you will need to use `eb --rebuild` or `eb --force`. + EasyBuild will also complain about modules that are loaded already if you tested the program above. + A good way to get rid of all those modules on LUMI is to simply use `module purge`. You don't need + to reload the software stack, but you will need to load `EasyBuild-user` again. --- @@ -874,125 +994,34 @@ You can leverage the `eb-tutorial` easyconfig file we have composed in the examp ??? success "(click to show solution)" You can use the `--try-software-version` option for this: ```shell - $ eb example.eb --try-software-version 1.1.0 + $ eb eb-tutorial.eb --try-software-version 1.1.0 ... - == building and installing eb-tutorial/1.1.0-GCC-10.2.0... + == building and installing eb-tutorial/1.1.0-cpeCray-21.12... ... == COMPLETED: Installation ended successfully (took 4 sec) ``` To test: ``` - $ module use $HOME/easybuild/modules/all - $ module load eb-tutorial/1.1.0-GCC-10.2.0 + $ module load eb-tutorial/1.1.0-cpeCray-21.12 $ eb-tutorial I have a message for you: - Hello from the EasyBuild tutorial! + Hello from the EasyBuild tutorial! I was installed by XXXXXXXX. ``` (`eb-tutorial` version 1.0.1 doesn't print "`I have a message for you:`") ---- - -***Exercise E.3**** - Installing py-eb-tutorial 1.0.0* - -Try composing an easyconfig file for the `py-eb-tutorial` example software, which is a tiny Python package. -The source tarball can be downloaded from this link: [py-eb-tutorial-1.0.0.tar.gz](https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/py-eb-tutorial-1.0.0.tar.gz). - -A couple of tips: - -* There is a [generic easyblock](#easyblock) available for installing Python packages, which will come in useful here. - -* By default EasyBuild performs an `import` check when install Python packages, using a Python module name that is derived from the software name by default, which will be incorrect in this case. You can specify the correct name to use in the import check by specifying it via the `options` -easyconfig parameter in your easyconfig file: - ```python - options = {'modulename': 'example'} - ``` - (you will need to change '`example`' here, of course) - -* Leverage the software that is already pre-installed in `/easybuild` in the prepared environment. - Remember that some already installed modules may be a *bundle* of a couple of other software packages. - -Please also take this into account: - -* Unfortunately this software doesn't come with documentation. That is done to make it an example that - is representative for software that you may run into in the wild (it's *not* because - we were lazy when preparing the exercises, really!). - You can inspect the sources of this software [here](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/py-eb-tutorial-1.0.0). Definitely take a look at the `setup.py` file, it includes some clues - about the requirements to get this software installed. - -* Make sure the installation actually *works*, by checking that the `py-eb-tutorial` command runs correctly. - Maybe you will need to make sure other required software is available as well, for it to work correctly... - - -??? success "(click to show solution)" - - Here is a complete working easyconfig file for `py-eb-tutorial`: - ```python - - easyblock = 'PythonPackage' - - name = 'py-eb-tutorial' - version = '1.0.0' - versionsuffix = '-Python-%(pyver)s' - - homepage = 'https://easybuilders.github.io/easybuild-tutorial' - description = "EasyBuild tutorial Python example" - - source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] - sources = [SOURCE_TAR_GZ] - checksums = ['fcf73a9efc65527a210b993e8889d41ebf05977eef1f6a65ebac3188152cd496'] - - toolchain = {'name': 'foss', 'version': '2020b'} - - dependencies = [ - ('Python', '3.8.6'), - ('SciPy-bundle', '2020.11'), - ('eb-tutorial', '1.0.1'), - ] - - use_pip = True - - options = {'modulename': 'eb_tutorial'} - - sanity_check_paths = { - 'files': ['bin/py-eb-tutorial'], - 'dirs': ['lib/python%(pyshortver)s/site-packages'], - } - - sanity_check_commands = ["py-eb-tutorial"] - - moduleclass = 'tools' + EasyBuild has also created a new easyconfig for this configuration and stored + in the repository and the `easybuild` subdirectory from the installation + directory. As on LUMI the repository is in the search path we can actually copy + the file back to the current directory: + ``` + eb --copy-ec eb-tutorial-1.1.0-cpeCray-21.12. ``` + Some of the formatting is lost though and the checksum is still missing, so you may want + to do some cleaning up. ``` + eb eb-tutorial-1.1.0-cpeCray-21.12.eb --inject-checksum + ``` + - Some remarks: - - * We used the `PythonPackage` generic easyblock. There is also a `PythonBundle` easyblock for installing - bundles of Python packages, which is used for `SciPy-bundle` for example. But we don't need that here, - since we are only dealing with a single Python package. - - * The `versionsuffix` is not strictly needed, but it's common to tag Python packages with the Python version - for which they were installed. - * The SHA256 checksum for the source tarball was added automatically via `eb py-eb-tutorial.eb --inject-checksums`. - - * `py-eb-tutorial` only wants to be installed with `pip install`, so we had to set `use_pip = True`. - You can consult the custom easyconfig parameters supported by the `PythonPackage` easyblock via - "`eb -a -e PythonPackage`", see the `EASYBLOCK-SPECIFIC` part of the output. - Even when the default installation mechanism used by `PythonPackage` - (which consists of running `python setup.py install`) works fine, - it is recommended to instruct EasyBuild to use `pip install` instead. - - * By default EasyBuild will try to import `py_eb_tutorial`, while the actual name of the Python package - provided by `py-eb-tutorial` is just `eb_tutorial`. We fixed this by specifying the correct Python module name to - use via `options`. - - * Strictly speaking we don't need to specify a custom `sanity_check_paths`, since the default used - by Python package is already pretty decent (it will check for a non-empty `lib/python3.8/site-packages` - directory in the installation). We also want to make sure the `py-eb-tutorial` command is available in - the `bin` subdirectory however. Hardcoding to `python3.8` can be avoided using the `%(pyshortver)s` - template value. - - * A good way to check whether the `py-eb-tutorial` command works correctly is by running it as a sanity check - command. If the `eb-tutorial` command is not available the `py-eb-tutorial` command will fail, - since it basically just runs the `eb-tutorial` command. So we need to include `eb-tutorial` as a (runtime) - dependency in the `py-eb-tutorial` easyconfig file. +--- *[[next: Implementing easyblocks]](2_03_implementing_easyblocks.md)* From 71e04e497bf817f36b4c88dfdcc97c79aab9219f Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 1 Apr 2022 19:12:05 +0200 Subject: [PATCH 107/159] Adding in an additional section about external modules. --- docs/2022-CSC_and_LO/2_00_part2_using.md | 3 ++- docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md | 2 +- ...lementing_easyblocks.md => 2_04_implementing_easyblocks.md} | 2 +- mkdocs.yml | 3 ++- 4 files changed, 6 insertions(+), 4 deletions(-) rename docs/2022-CSC_and_LO/{2_03_implementing_easyblocks.md => 2_04_implementing_easyblocks.md} (99%) diff --git a/docs/2022-CSC_and_LO/2_00_part2_using.md b/docs/2022-CSC_and_LO/2_00_part2_using.md index e033499c..c50036be 100644 --- a/docs/2022-CSC_and_LO/2_00_part2_using.md +++ b/docs/2022-CSC_and_LO/2_00_part2_using.md @@ -4,4 +4,5 @@ * [Troubleshooting](2_01_troubleshooting.md) *(hands-on)* * [Creating easyconfig files](2_02_creating_easyconfig_files.md) *(hands-on)* -* [Implementing easyblocks](2_03_implementing_easyblocks.md) *(hands-on)* +* [Using external modules from the Cray PE](2_03_external_modules.md) +* [Implementing easyblocks](2_04_implementing_easyblocks.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md index 99ac7d88..f7d6a646 100644 --- a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md @@ -1024,4 +1024,4 @@ You can leverage the `eb-tutorial` easyconfig file we have composed in the examp --- -*[[next: Implementing easyblocks]](2_03_implementing_easyblocks.md)* +*[[next: Using external modules from the Cray PE]](2_03_external_modules.md)* diff --git a/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md similarity index 99% rename from docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md rename to docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md index 5355548e..e0c07510 100644 --- a/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md +++ b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md @@ -1,6 +1,6 @@ # Implementing easyblocks -*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* +*[[back: Using external modules from the Cray PE]](2_03_external_modules.md)* --- diff --git a/mkdocs.yml b/mkdocs.yml index 15c98d78..9a64f55d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -134,7 +134,8 @@ nav: - (overview): 2022-CSC_and_LO/2_00_part2_using.md - Troubleshooting: 2022-CSC_and_LO/2_01_troubleshooting.md - Creating easyconfig files: 2022-CSC_and_LO/2_02_creating_easyconfig_files.md - - Implementing easyblocks: 2022-CSC_and_LO/2_03_implementing_easyblocks.md + - Using external modules from the Cray PE: 2022-CSC_and_LO/2_03_external_modules.md + - Implementing easyblocks: 2022-CSC_and_LO/2_04_implementing_easyblocks.md - Advanced topics: - (overview): 2022-CSC_and_LO/3_00_part3_advanced.md - Using EasyBuild as a library: 2022-CSC_and_LO/3_01_easybuild_library.md From da4ad40dc2ae495982a116f6bd13ce6609ae0edb Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 1 Apr 2022 19:12:42 +0200 Subject: [PATCH 108/159] Adding in a section taken from a previous tutorial and integrating some of the material in an earlier section on the Cray PE. --- docs/2022-CSC_and_LO/1_03_CPE.md | 222 ++++++++++++++---- docs/2022-CSC_and_LO/2_03_external_modules.md | 81 +++++++ 2 files changed, 259 insertions(+), 44 deletions(-) create mode 100644 docs/2022-CSC_and_LO/2_03_external_modules.md diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md index 077fbc36..d3a2ff84 100644 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -15,76 +15,207 @@ tools and to configure the environment in a flexible way. ## Cray PE components -- **Cray Compiling Environment (CCE)**: +### Cray compiler environments - CCE consists of Cray compilers performing code analysis during compilation to generate highly optimized code. - Supported languages include Fortran, C and C++, and UPC (Unified Parallel C). - The Cray C/C++ compiler is basically the Clang/LLVM compiler with a back-end configured by HPE Cray that also - adds some Cray-specific options while the Cray Fortran compiler uses an HPE-Cray specific front-end - with a back-end based on LLVM. The Fortran compiler supports most of Fortran 2018 and tends to be - considerably stricter than the GNU or Intel Fortran compilers. +Cray PE supports multiple compilers, Cray and third party compilers as well: AOCC, Cray, Intel, GNU. - The CCE compilers will also support OpenMP offload to NVIDIA and AMD GPUs but that is still work in progress - at the time of the development of this tutorial. The Fortran compiler (but not the C/C++ compiler) also supports - OpenACC for offloading to GPU. +Users can access the compilers loading a programming environment module (see below). - The classic pre-LLVM Cray C/C++ compiler is not available on Cray EX systems. +Compilers in the Cray Programming Environment are almost always used through the Cray compiler wrappers. In fact, +until recently, some compilers could not be used without the wrappers as they failed to even detect their own +include files and some components. The behaviour of the wrapper will depend on the other modules that are +loaded in the environment. E.g., there is no separate wrapper for MPI as the MPI header files and libraries are +automatically activated through the regular wrappers as soon as a Cray MPI module is loaded. -- **Cray Scientific and Mathematical Libraries (CSML)**: +The commands to invoke compiler wrappers are `ftn` (Fortran), `cc` (C), `CC` (C++). They wrap automatically +to the right compilers based on the programming environment module that is loaded and the compiler module. - A set of high performance libraries providing portability for scientific applications, sparse and dense linear - algebra (BLAS, LAPACK, ScaLAPACK) and fast Fourier transforms (FFTW). +The online help can be accessed with the `-help` option. E.g.: `cc -help`, `CC -help`. +One of the most frequently used options of the compiler wrappers is `-craype-verbose`: +``` + ftn -help | grep verbose -A 1 + -craype-verbose Print the command which is forwarded + to compiler invocation +``` +More information is available with the `info` or `man` commands. +E.g.: both `info cc` or `man cc` will display the man page of the C compiler wrapper. - There is a special (more limited) version of the library with GPU-optimized routines. +The compiler wrappers call the correct compiler in the currently loaded programming environment, +with appropriate options to build and link applications with relevant libraries, as required by the modules loaded. +Besides a number of generic options, they also will pass all other options to the underlying compile, +so you can still use the regular compiler options also. - CSML also includes the HDF5 and netCDF libraries, in versions with and without MPI. +The compiler wrappers should replace direct calls to compiler drivers in Makefiles and build scripts +to ensure that the proper libraries are used. -- **Cray Message Passing Toolkit**: +!!! Note + For system libraries, only dynamic linking is supported by compiler wrappers on the Cray EX system - A collection of software libraries used to perform data transfers between nodes running in parallel applications. - It includes the Message Passing Interface (MPI) and OpenSHMEM parallel programming models. +--- - The HPE Cray MPI implementation is based on the MPICH3 sources from ANL. However, some algorithms are tweaked - specifically for HPE Cray hardware and the Cray interconnects. +#### Cray Compiling Environment (CCE) -- **Cray Environment Setup and Compiling Support**: - - Libraries supporting code compilation and setting up the development environment, including compiler drivers. +The Cray Compiling Environment is set by the module `PrgEnv-cray`, which is loaded by default at login. +The compiler itself is contained in the `cce` module. -- **Cray Performance and Analysis Tools**: +CCE consists of Cray compilers performing code analysis during compilation to generate highly optimized code. +Supported languages include Fortran, C and C++, and UPC (Unified Parallel C). +The Cray C/C++ compiler is basically the Clang/LLVM compiler with a back-end configured by HPE Cray that also +adds some Cray-specific options while the Cray Fortran compiler uses an HPE-Cray specific front-end +with a back-end based on LLVM. The Fortran compiler supports most of Fortran 2018 and tends to be +considerably stricter than the GNU or Intel Fortran compilers. - Tools to analyze the performance and behavior of programs that are run on Cray systems, and a Performance API (PAPI). +The CCE compilers will also support OpenMP offload to NVIDIA and AMD GPUs but that is still very immature and +very much work in progress at the time of the development of this tutorial. +The Fortran compiler (but not the C/C++ compiler) also supports +OpenACC for offloading to GPU and is being updated to the newest versions of this standard. -- **Cray Debugging Support Tools**: +The classic pre-LLVM Cray C/C++ compiler is not available on Cray EX systems. - Debugging tools, including `gdb4hpc` and `Valgrind4hpc`. +Compiler-specific manpages can be accessed on the system with `man crayftn`, `man craycc` or `man crayCC`. + +More details are given in the +[Cray Fortran Reference Manual](https://support.hpe.com/connect/s/search?language=en_US#q=Cray%20Fortran%20Reference%20manual&t=All&sort=relevancy&numberOfResults=25) + and the + [Cray Compiling Environment Release](https://support.hpe.com/connect/s/search?language=en_US#q=Cray%20Compiling%20Environment%20Release&t=All&sort=relevancy&numberOfResults=25) +that unfortunately are hidden deep in the new [support pages of HPE](https://support.hpe.com/). +The [Clang Compiler User’s Manual](https://clang.llvm.org/docs/UsersManual.html) +is another source of information for the Cray C and C++ Clang compilers. + +For more information about compiler pragmas and directives, see `man intro_directives` on the system. + +--- + +### Third-Party Compilers + +#### GNU + +The GNU C/C++ and Fortran compilers are probably the best supported third party +compiler in the Cray PE. + +Compiler-specific manpages can be accessed on the system with `man gfortran`, `man gcc` or `man g++`. + +More details are provided by the [GCC online documentation](https://gcc.gnu.org/onlinedocs). + + +#### AOCC + +There AOCC compilers, the AMD Optimizing C/C++ Compiler and matching fortran compilers, +AMD's compiler offering for CPU-only systems, have a matching programming environment module +and are a full citizen of the Cray PE. + +Compiler-specific documentation is available in the [AOCC User Guide](https://developer.amd.com/amd-aocc/#userguide). + +Cray provides a bundled package of support libraries to install into the PE environment to enable AOCC, +and Cray PE utilities such as debuggers and performance tools work with AOCC. + + +#### AMD ROCm compilers + +The AMD ROCm compilers (in the AMD world sometimes known as AOMP) are supported on systems +with AMD CPUs. However, at the time of writing of this tutorial they are not yet available in the +LUMI environment and the integration with the Cray environment still seems to be work in progress. + + +#### Intel (not on LUMI) + +The Cray PE also provides a programming environment module to enable +the Intel® oneAPI compiler and tools. + +The documentation is available in the [Intel® oneAPI Programming Guide](https://software.intel.com/content/www/us/en/develop/documentation/oneapi-programming-guide/top/oneapi-development-environment-setup.html) + +Cray provides a bundled package of support libraries to install into the Cray PE to enable the Intel compiler, allowing +utilities such as debuggers and performance tools to work with it. + + +#### NVIDIA HPC toolkit (not on LUMI) -- **Third-party compilers**: +The NVIDIA HPC Toolkit compilers (formerly PGI) are supported on systems with NVIDIA GPUs. - The Cray PE also supports several third-party compilers. Not all compilers are supported on all systems: - - The GNU C/C++ and Fortran compilers will be present on most systems with the Cray PE. +### Cray Scientific and Math Library - - The AMD AOCC C/C++ and Fortran compilers are supported on systems with AMD CPUs. +* Manpages: `intro_libsci`, `intro_fftw3` - - The AMD ROCm compilers (sometimes known as AOMP) are supported on systems with AMD GPU technology. +The Cray Scientific and Math Libraries (CSML, also known as LibSci) are a collection of numerical routines +optimized for best performance on Cray systems. - - The Intel C/C++ and Fortran compilers (also the oneAPI versions) are supported on systems with Intel - processors. HPE Cray does not include them in licenses for AMD CPUs. +These libraries satisfy dependencies for many commonly used applications on Cray systems for a wide variety of domains. - - The NVIDIA HPC Toolkit compilers are supported on systems with NVIDIA GPUs. +When the module for a CSML package (such as `cray-libsci` or `cray-fftw`) is loaded, +all relevant headers and libraries for these packages are added to the compile +and link lines of the `cc`, `ftn`, and `CC` compiler wrappers, so linking with them is +completely transparant (to the extent that users wonder where the libraries are). + +The CSML collection contains the following Scientific Libraries: + +* BLAS (Basic Linear Algebra Subroutines) +* BLACS (Basic Linear Algebra Communication Subprograms) +* CBLAS (Collection of wrappers providing a C interface to the Fortran BLAS library) +* IRT (Iterative Refinement Toolkit) +* LAPACK (Linear Algebra Routines) +* LAPACKE (C interfaces to LAPACK Routines) +* ScaLAPACK (Scalable LAPACK) +* `libsci_acc` (library of Cray-optimized BLAS, LAPACK, and ScaLAPACK routines) +* HDF5 +* NetCDF (Network Common Data Format) +* FFTW3 (the Fastest Fourier Transforms in the West, release 3) + + +### Cray Message Passing Toolkit + +* Manpages: `intro_mpi` + +* Website: [http://www.mpi-forum.org](http://www.mpi-forum.org) + +MPI is a widely used parallel programming model that establishes a practical, portable, efficient, +and flexible standard for passing messages between ranks in parallel processes. + +Cray MPI is derived from Argonne National Laboratory MPICH and implements the MPI-3.1 standard +as documented by the MPI Forum in MPI: A Message Passing Interface Standard, Version 3.1. + +Support for MPI varies depending on system hardware. To see which functions and environment variables the +system supports, please have a look at the corresponding man pages with `man intro_mpi` on the system. + +Note that though on LUMI at the time of the tutorial both the OpenFabric Interface (OFI) based and UCX-based +versions of the library are supported, the finalised LUMI system will only support OFI. + + +### DSMML + +* Modules: `cray-dsmml` +* +* Manpages: `intro_dsmml` +* +* Website: [https://pe-cray.github.io/cray-dsmml](https://pe-cray.github.io/cray-dsmml) + +Distributed Symmetric Memory Management Library (DSMML) is a HPE Cray proprietary memory management library. + +DSMML is a standalone memory management library for maintaining distributed shared symmetric memory +heaps for top-level PGAS languages and libraries like Coarray Fortran, UPC, and OpenSHMEM. + +DSMML allows user libraries to create multiple symmetric heaps and share information with other libraries. + +Through DSMML, interoperability can be extracted between PGAS programming models. + +Further details are available in the man page on the system with `man intro_dsmml`. + + +### Other components + +- **Cray Performance and Analysis Tools**: + + Tools to analyze the performance and behavior of programs that are run on Cray systems, and a Performance API (PAPI). + +- **Cray Debugging Support Tools**: + + Debugging tools, including `gdb4hpc` and `Valgrind4hpc`. --- ## Configuring the Cray PE through modules -Compilers in the Cray Programming Environment are almost always used through the Cray compiler wrappers. In fact, -until recently, some compilers could not be used without the wrappers as they failed to even detect their own -include files and some components. The behaviour of the wrapper will depend on the other modules that are -loaded in the environment. E.g., there is no separate wrapper for MPI as the MPI header files and libraries are -automatically activated through the regular wrappers as soon as a Cray MPI module is loaded. - Multiple releases of the HPE Cray PE can be installed simultaneously on the system and users can mix-and-match components. However, it is also possible to load only components for a specific release of the PE. Cray PE versions have version numbers of the form ``yy.mm``, e.g., ``22.02`` for the version released in @@ -95,7 +226,7 @@ Below we only discuss those modules that are important when building software wi profilers, etc., are not included in the list. -### The PrgEnv-* modules +### The programming environment modules The usual way to initialise the Cray PE is by loading one of the ``PrgEnv-*`` modules. This module will then load the appropriate compiler, compiler wrappers and other libraries. Some of the components that will be @@ -117,7 +248,10 @@ All ``PrgEnv-*`` modules belong to the same module family. Hence Lmod will autom already loaded ``PrgEnv-*`` module when you load a different one. -### Selecting the version through the cpe module +### Selecting the version through the cpe meta-module + +The Cray PE on the EX system provides the meta-module `cpe`: the purpose of the meta-module is +similar to the scope of the `cdt` and `cdt-cuda` meta-modules available on the XC systems. Loading one of the ``cpe/yy.mm`` modules (e.g., ``cpe/22.02``) has the following effects: diff --git a/docs/2022-CSC_and_LO/2_03_external_modules.md b/docs/2022-CSC_and_LO/2_03_external_modules.md new file mode 100644 index 00000000..5d6632c0 --- /dev/null +++ b/docs/2022-CSC_and_LO/2_03_external_modules.md @@ -0,0 +1,81 @@ +# Using external modules from the Cray PE + +*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* + +--- + +## What are external modules? + +EasyBuild supports the use of modules that were not installed via EasyBuild. +We refer to such modules as [external modules](https://docs.easybuild.io/en/latest/Using_external_modules.html). +These modules do not define the `EBROOT*` and `EBVERSION*` environment variables that EasyBuild would define +and uses internally in several easyblocks and some easyconfig files. + +This feature is used extensively on Cray systems to interface with the Cray PE (which comes with its own +modules and cannot be installed via EasyBuild): +[external modules can be used as dependencies](https://docs.easybuild.io/en/latest/Using_external_modules.html#using-external-modules-as-dependencies), +by including the module name in the dependencies list, +along with the `EXTERNAL_MODULE` constant marker. + +For example, to specify the module `cray-fftw` as a dependency, you should write the following in your easyconfig file: +``` python +dependencies = [('cray-fftw', EXTERNAL_MODULE)] +``` + +For such dependencies, EasyBuild will: + +* load the module before initiating the software build and install procedure + +* include a `module load` statement in the generated module file (for runtime dependencies) + +!!! Note + The default version of the external module will be loaded unless a specific version is given as dependency, + and here that version needs to be given as part of the name of the module and not as the second element in the + tuple. + + ```python + dependencies = [('cray-fftw/3.3.8.12', EXTERNAL_MODULE)] + ``` + +If the specified module is not available, EasyBuild will exit with an error message stating that the dependency +can not be resolved because the module could not be found, without searching for a matching easyconfig file +from which it could generate the module. + + +--- + +## EasyBuild Metadata for external modules + +[Metadata](https://docs.easybuild.io/en/latest/Using_external_modules.html#metadata-for-external-modules) + can be supplied to EasyBuild for external modules: using the `--external-modules-metadata` +configuration option, the location of one or more metadata files can be specified. + +The files are expected to be in INI format, with a section per module name +and key-value assignments specific to that module. + +The external modules metadata file can be also defined with the corresponding environment variable: +``` +echo $EASYBUILD_EXTERNAL_MODULES_METADATA +/apps/common/UES/jenkins/production/easybuild/cpe_external_modules_metadata-21.04.cfg +``` + +The following keys are +[supported by EasyBuild](https://docs.easybuild.io/en/latest/Using_external_modules.html#supported-metadata-values): + +* name: software name(s) provided by the module +* version: software version(s) provided by the module +* prefix: installation prefix of the software provided by the module + +For instance, the external module version loaded by the dependency `cray-fftw` can be specified as follows: +```ini +[cray-fftw] +name = FFTW +prefix = FFTW_DIR/.. +version = 3.3.8.10 +``` + +The environment variable `$EBROOTFFTW` will also be defined according to the `prefix` specified in the metadata file. + +--- + +*[[next: Implementing easyblocks]](2_04_implementing_easyblocks.md)* From fa09f7d54bc3ba5ffe4740b3d4351343aa06b313 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 5 Apr 2022 15:54:42 +0200 Subject: [PATCH 109/159] Implementing EasyBlocks adapted for LUMI. --- .../2_04_implementing_easyblocks.md | 113 ++++++++++++++++-- 1 file changed, 101 insertions(+), 12 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md index e0c07510..a91b0dc8 100644 --- a/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md +++ b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md @@ -27,7 +27,10 @@ by most GNU software packages. *Software-specific* easyblocks implement the build and installation procedure for a particular software package. Typically this involves highly customised steps, for example specifying dedicated configuration options, creating or adjusting specific files, executing non-standard shell commands, etc. Usually a custom implementation of the -sanity check is also included. +sanity check is also included. Much of the work done in software-specific easyblocks can often also be done +in generic easyblocks using parameters such as ``confdigopts`` etc., but a software-specific easyblock can +hide some of that complexity from the user. Other software-specific easyblocks implement very specific +installation procedures that do not fit in one of the generic ones. Using a generic easyblock requires specifying the ``easyblock`` parameter in the easyconfig file. If it is not specified, EasyBuild will try and find the software-specific easyblock derived from the software name. @@ -38,7 +41,8 @@ for an easyblock (see below). ## Naming -Easyblocks need to follow a strict naming scheme, to ensure that EasyBuild can pick them up as needed. This involves two aspects: +Easyblocks need to follow a strict naming scheme, to ensure that EasyBuild can pick them up automatically as needed. +This involves two aspects: * the name of the Python class; * the name and location of the Python module file. @@ -52,7 +56,7 @@ Because of limitations in Python on characters allowed in names of Python classe only alphanumeric characters and underscores (``_``) are allowed. Any other characters are replaced following an encoding scheme: * spaces are replaced by underscores (``_``); -* dashes ``-`` are replaced by ``_minus_``; +* dashes ``-`` are replaced by ``_minus_`` (note the inconsistency with the naming of ``EBROOT`` and ``EBVERSION`` variables); * underscores are replaced by ``_underscore_``; The ``encode_class_name`` function provided in ``easybuild.tools.filetools`` returns the expected class name @@ -117,6 +121,11 @@ To let EasyBuild pick up one or more new or customized easyblocks, you can use t configuration option. As long as both the filename of the Python module and the name of the Python class are correct, EasyBuild will use these easyblocks when needed. +On LUMI, the EasyBuild configuration modules take care of setting this parameter (using the corresponding environment +variable), pointing to custom easyblocks in the LUMI software stack itself and a repo (with a fixed name) that users +can create themselves. At this moment it does not yet include possible other easyblock repositories in other repositories. + + ## Structure of an easyblock The example below shows the overal structure of an easyblock: @@ -232,7 +241,7 @@ self.cfg['some_list'].append('example') ### Custom parameters -Additional custom easyconfig parameters can be defined in an easyblock to allowing steering its behaviour. +Additional custom easyconfig parameters can be defined in an easyblock to steer its behaviour. This is done via the ``extra_options`` *static* method. Custom parameters can be specified to be mandatory. The example below shows how this can be implemented: @@ -256,7 +265,9 @@ class EB_Example(ConfigureMake): ``` The first element in the list of a defined custom parameter corresponds to the default value for that parameter -(both ``None`` in the example above). The second element provides some informative help text, and the last element +(both ``None`` in the example above). The second element provides some informative help text +(which can then be displayed with ``eb -a -e ``, eg, ``eb -a -e EB_GCC``), +and the last element indicates whether the parameter is mandatory (``MANDATORY``) or just an optional custom parameter (``CUSTOM``). ## Easyblock constructor @@ -519,6 +530,29 @@ class EB_Example(EasyBlock): super(EB_Example, self).sanity_check_step(custom_commands=[self.command]) ``` + +## Easyblocks in the Cray ecosystem + +The generic easyblocks are usually rather independent of compilers etc. and tend to work well with all toolchains. +However, software-specific easyblocks may contain code that is specific for certain toolchains and are often only +tested with the common toolchains (foss and intel and their subtoolchains). Many of those easyblocks will fail +on Cray systems (or any system that uses other toolchains) as they don't recognise the compiler and rather than +implementing some generic behaviour that may or may not work, produce an error message instead that the compiler +toolchain is not supported. + +Several packages on LUMI therefore use generic easyblocks rather than the software-specific easyblocks that may +exist for those applications. Adapting those software-specific easyblocks for LUMI poses an interesting maintenance +problem. Either one could decide to not contribute back to the community, but this implies then that all modifications +made to the corresponding easyblocks in the EasyBuild distribution should be monitored and implemented in the custom +easyblocks for Cray also. On the other hand, contributing back to the community also poses two problems. First it +would also require to implement the Cray toolchains as used on LUMI in the core of EasyBuild (which already contains +a different set of toolchains targeted more at how the Cray PE works with the regular environment modules), and that +only makes sense if these toolchains are first extended to not only cover the programming environments supported on +LUMI but also the Intel and NVIDIA programming environments. Second, the EasyBuild community has no easy way of testing +any modification made to such an easyblock on a Cray PE system. Hence every update made in the community may break +the Cray PE support again. + + ## Exercise ### Exercise I.1 @@ -570,9 +604,9 @@ Your easyblock should: cmd = ' '.join([ self.cfg['preconfigopts'], - "cmake", - "-DCMAKE_INSTALL_PREFIX='%s'" % self.installdir, - "-DEBTUTORIAL_MSG='%s'" % self.cfg['message'], + 'cmake', + '-DCMAKE_INSTALL_PREFIX=\'%s\'' % self.installdir, + '-DEBTUTORIAL_MSG="%s"' % self.cfg['message'], self.cfg['configopts'], ]) run_cmd(cmd) @@ -582,7 +616,7 @@ Your easyblock should: cmd = ' '.join([ self.cfg['prebuildopts'], - "make", + 'make', self.cfg['buildopts'], ]) run_cmd(cmd) @@ -592,7 +626,7 @@ Your easyblock should: cmd = ' '.join([ self.cfg['preinstallopts'], - "make install", + 'make install', self.cfg['installopts'], ]) run_cmd(cmd) @@ -607,6 +641,58 @@ Your easyblock should: custom_commands=custom_commands) ``` + We also need to adapt our easyconfig file for ``eb-tutorial``: + + - The ``easyblock`` line is no longer needed as we will rely on the automatic selection of the + software-specific easyblock. + - We don't need to define the message through ``configopts`` but via the easyblock-specific + configuration parameter ``message``. In fact, we were so careful when implementing the ``configure_step`` + that even variable expansion will still work so we can still include ``$USER`` in the message. + - The sanity check is also no longer needed as it is done by the software-specific easyblock. + + So the easyconfig file simplifies to: + + ```python + name = 'eb-tutorial' + version = "1.1.0" + + homepage = 'https://easybuilders.github.io/easybuild-tutorial' + + whatis = [ 'Description: EasyBuild tutorial example'] + + description = """ + This is a short C++ example program that can be buid using CMake. + """ + + toolchain = {'name': 'cpeCray', 'version': '21.12'} + + builddependencies = [ + ('buildtools', '%(toolchain_version)s', '', True) + ] + + source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] + sources = [SOURCE_TAR_GZ] + checksums = ['def18b69b11a3ec34ef2a81752603b2118cf1a57e350aee41de9ea13c2e6a7ef'] + + message = 'Hello from the EasyBuild tutorial! I was installed by $USER.' + + moduleclass = 'tools' + + ``` + + Running this example on LUMI is a little tricky as using ``--include-easyblocks`` to point EasyBuild to + our new easyblock interfers with settings already made by the EasyBuild configuration modules (``EasyBuild-user``) + and causes error messages about the toolchains. So either the easyblock needs to be copied to the user location + that can be found by looking at the output of ``eb --show-config`` or we simply need to extend the list of + easyblocks that EasyBuild searches with the easyblocks in the current directory: + + ``` shell + EASYBUILD_INCLUDE_EASYBLOCKS="$EASYBUILD_INCLUDE_EASYBLOCKS,./*.py" + ``` + + + + ### Exercise I.2 Try implementing another new custom easyblock for ``eb-tutorial``, @@ -648,7 +734,7 @@ Your easyblock should only: def configure_step(self): """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" - self.cfg.update('configopts', "-DEBTUTORIAL_MSG='%s'" % self.cfg['message']) + self.cfg.update('configopts', '-DEBTUTORIAL_MSG="%s"'% self.cfg['message']) super(EB_eb_minus_tutorial, self).configure_step() @@ -662,4 +748,7 @@ Your easyblock should only: custom_commands=custom_commands) ``` -*[[next: PArt 3: OVerview]](3_00_part3_advanced.md)* + This is a much simpler easyblock as we already use all the logic that has been written for us to build + with CMake. + +*[[next: Part 3: Overview]](3_00_part3_advanced.md)* From 1c6894aec08424ea74e609e27669ae2e39091062 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 5 Apr 2022 15:54:55 +0200 Subject: [PATCH 110/159] Correction of typos. --- docs/2022-CSC_and_LO/2_03_external_modules.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_03_external_modules.md b/docs/2022-CSC_and_LO/2_03_external_modules.md index 5d6632c0..b19313ff 100644 --- a/docs/2022-CSC_and_LO/2_03_external_modules.md +++ b/docs/2022-CSC_and_LO/2_03_external_modules.md @@ -8,10 +8,10 @@ EasyBuild supports the use of modules that were not installed via EasyBuild. We refer to such modules as [external modules](https://docs.easybuild.io/en/latest/Using_external_modules.html). -These modules do not define the `EBROOT*` and `EBVERSION*` environment variables that EasyBuild would define +External modules do not define the `EBROOT*` and `EBVERSION*` environment variables that EasyBuild would define and uses internally in several easyblocks and some easyconfig files. -This feature is used extensively on Cray systems to interface with the Cray PE (which comes with its own +External modules are used extensively on Cray systems to interface with the Cray PE (which comes with its own modules and cannot be installed via EasyBuild): [external modules can be used as dependencies](https://docs.easybuild.io/en/latest/Using_external_modules.html#using-external-modules-as-dependencies), by including the module name in the dependencies list, @@ -47,7 +47,7 @@ from which it could generate the module. ## EasyBuild Metadata for external modules [Metadata](https://docs.easybuild.io/en/latest/Using_external_modules.html#metadata-for-external-modules) - can be supplied to EasyBuild for external modules: using the `--external-modules-metadata` +can be supplied to EasyBuild for external modules: using the `--external-modules-metadata` configuration option, the location of one or more metadata files can be specified. The files are expected to be in INI format, with a section per module name @@ -76,6 +76,10 @@ version = 3.3.8.10 The environment variable `$EBROOTFFTW` will also be defined according to the `prefix` specified in the metadata file. +On LUMI, users in generally don't need to be too concerned about the metadata file as the EasyBuild-user (and other hidden +EasyBuild configuration modules) take care of pointing to the right metadata file, which is specific for each version of the +Cray PE and hence each version of the LUMI software stack. + --- *[[next: Implementing easyblocks]](2_04_implementing_easyblocks.md)* From 81274e3c3f26e11ee314b79fa9c236e7bd2ead0f Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 5 Apr 2022 15:55:13 +0200 Subject: [PATCH 111/159] Using EasyBuild as a library corrected for LUMI. --- .../2022-CSC_and_LO/3_01_easybuild_library.md | 26 ++++++++++++------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_01_easybuild_library.md b/docs/2022-CSC_and_LO/3_01_easybuild_library.md index c037f49f..f0c255eb 100644 --- a/docs/2022-CSC_and_LO/3_01_easybuild_library.md +++ b/docs/2022-CSC_and_LO/3_01_easybuild_library.md @@ -254,10 +254,12 @@ def parse_easyconfig(ec_fn): # parse easyconfig file; # the 'parse_easyconfigs' function expects a list of tuples, - # where the second item indicates whether or not the easyconfig file was automatically generated or not + # where the second item indicates whether or not the easyconfig file was + # automatically generated or not ec_dicts, _ = parse_easyconfigs([(ec_path, False)]) - # only retain first parsed easyconfig, ignore any others (which are unlikely anyway) + # only retain first parsed easyconfig, ignore any others (which are unlikely + # anyway) return ec_path, ec_dicts[0]['ec'] @@ -278,15 +280,21 @@ for key in keys: print("%s: %s" % (key, ec[key])) ``` -Example usage: +Example usage (ensure that ``EasyBuild-user`` is loaded as EasyBuild should be +configured properly!): -```shell -$ ./inspect_easyconfig.py Subread-2.0.0-GCC-8.3.0.eb name version sources sanity_check_paths -name: Subread -version: 2.0.0 -sources: ['subread-2.0.0-source.tar.gz'] -sanity_check_paths: {'files': ['bin/exactSNP', 'bin/featureCounts', 'bin/subindel', 'bin/subjunc', 'bin/sublong', 'bin/subread-align', 'bin/subread-buildindex'], 'dirs': ['bin/utilities']} ``` +$ ./inspect_easyconfig.py GMP-6.2.1-cpeCray-21.12.eb name version sources sanity_check_paths +Inspecting /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/GMP/GMP-6.2.1-cpeCray-21.12.eb ... +name: GMP +version: 6.2.1 +sources: ['gmp-6.2.1.tar.bz2'] +sanity_check_paths: {'files': ['include/gmp.h', 'include/gmpxx.h', 'lib/libgmp.a', 'lib/libgmp.la', 'lib/libgmp.so', 'lib/libgmpxx.a', 'lib/libgmpxx.la', 'lib/libgmpxx.so', 'lib/pkgconfig/gmp.pc', 'lib/pkgconfig/gmpxx.pc'], 'dirs': ['share/info']} +``` + +This feature can be used, e.g., to extract information from easyconfig files to generate documentation +for the library of easyconfig files. + --- From 4e5ac8fc6ca6da26f9e32907b1b710c524585cda Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 5 Apr 2022 15:55:33 +0200 Subject: [PATCH 112/159] Section about hooks extended with references to additional examples. --- docs/2022-CSC_and_LO/3_02_hooks.md | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_02_hooks.md b/docs/2022-CSC_and_LO/3_02_hooks.md index a2663ff0..f91f7d7c 100644 --- a/docs/2022-CSC_and_LO/3_02_hooks.md +++ b/docs/2022-CSC_and_LO/3_02_hooks.md @@ -19,9 +19,32 @@ would normally execute. This is a very powerful feature, but it is important to aware of some details in order to make good use of it. -Support for hooks is also covered in detail in the EasyBuild documentation, see -[here](https://docs.easybuild.io/en/latest/Hooks.html). - +Support for hooks is also +[covered in detail in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Hooks.html). + +## Why hooks? + +Hooks can be implemented for various reasons. Some examples used by some sites in the EasyBuild +community: + +- Enforce some site policies on easyconfig files. A practical example for LUMI could be imposing + the use of the `whatis` parameter and ensuring that it contains a `Description` line to have a + more consistent format for module files. +- Ensuring that some information is always in the module file. E.g., on LUMI the `site_contacts` + parameter is added automatically when installing in the central stack if it is not already in + the easyconfig file to refer to the LUST support pages. +- Modify the behaviour of standard easyconfig files that come with EasyBuild to adapt them to the system + while allowing users to simply use the standard easyconfig files. E.g., we could consider this option + to offer the common EasyBuild FOSS toolchain and its subtoolchains to users on LUMI-C, while using hooks to + adapt, e.g., the easyconfig files for Open MPI to work on LUMI. + +Some site have really large hook files to implement policies and modify standard EasyBuild build recipes, e.g., +- [The hooks file from Jülich Supercomputing Centre](https://github.com/easybuilders/JSC/blob/2022/Custom_Hooks/eb_hooks.py) +- ComputeCanada [cc_hooks_common.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_common.py) + and [cc_hooks_gento.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_gentoo.py) +- [The hooks file from the EESSI software stack](https://github.com/EESSI/software-layer/blob/main/eb_hooks.py). + [EESSI](https://www.eessi-hpc.org/) is an effort to build a software stack distributed via + [CernVM-FS](https://cernvm.cern.ch/fs/) using EasyBuild to build all software. ## Implementing and using hooks From db5cdec7a27204fc3e19ad227793691d907ee73b Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 13:57:15 +0200 Subject: [PATCH 113/159] Slurm job submission from EasyBuild a bit reworked. --- docs/2022-CSC_and_LO/3_03_slurm_jobs.md | 109 +++++++++++++++--------- 1 file changed, 69 insertions(+), 40 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md index 0ffa12a9..567f8abd 100644 --- a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md +++ b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md @@ -6,12 +6,30 @@ EasyBuild can submit jobs to different backends including Slurm to install software, to *distribute* the often time-consuming installation of a set of software applications and -the dependencies they require to a cluster. +the dependencies they require to a cluster. Each individual package is installed in a separate +job and job dependencies are used to manage the dependencies between package so that no build +is started before the dependencies are in place. This is done via the ``--job`` command line option. It is important to be aware of some details before you start using this, which we'll cover here. +!!! Warning "This section is not supported on LUMI, use at your own risk" + + EasyBuild on LUMI is currently not fully configured to support job submission via Slurm. Several + changes would be needed to the configuration of EasyBuild, including the location of the + temporary files and build directory. Those have to be made by hand. + + Due to the setup of the central software stack, this feature is currently useless to install + the central stack. For user installations, there are also limitations as the enviornment + on the compute nodes is different from the login nodes so, e.g., different locations for + temporary files are being used. These would only be refreshed if the EasyBuild configuration + modules are reloaded on the compute nodes which cannot be done currently in the way Slurm + job submission is set up in EasyBuild. + + Use material in this section with care; it has not been completely tested. + + ## Configuration The EasyBuild configuration that is active at the time that ``eb --job`` is used @@ -25,6 +43,8 @@ that are specified via an [EasyBuild configuration file](configuration.md#config This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables that are in place in the job environment are most likely *irrelevant*, since configuration settings they specify they will most likely be overruled by the corresponding command line options. +It does also imply however that the EasyBuild configuration that is in place when ``eb --job`` is used +does also work on the compute nodes to which the job is submitted. ## Using ``eb --job`` @@ -39,6 +59,9 @@ to ``Slurm``, for example by setting the corresponding environment variable: export EASYBUILD_JOB_BACKEND='Slurm' ``` +On LUMI this is taken care of in the EasyBuild configuration modules such as ``EasyBuild-user``. + + ### Job resources To submit an installation as a job, simply use ``eb --job``: @@ -73,13 +96,13 @@ For example, to specify a particular account that should be used for the jobs su (equivalent with using the ``-A`` or ``--account`` command line option for ``sbatch``): ```shell -export SBATCH_ACCOUNT='example_project' +export SBATCH_ACCOUNT='project_XXXXXXXXX' ``` Or to submit to a particular Slurm partition (equivalent with the ``-p`` or ``--partition`` option for ``sbatch``): ```shell -export SBATCH_PARTITION='example_partition' +export SBATCH_PARTITION='small' ``` For more information about supported ``$SBATCH_*`` environment variables, @@ -113,24 +136,29 @@ as jobs, to avoid that they fail almost instantly due to a lack of disk space. Keep in mind that the active EasyBuild configuration is passed down into the submitted jobs, so any configuration that is present on the workernodes may not have any effect. -For example, if you commonly use `/tmp/$USER` for build directories on a login node, -you may need to tweak that when submitting jobs to use a different location: +For example, on LUMI it is possible to use ``$XDG_RUNTIME_DIR`` on the login nodes which has +the advantage that any leftovers of failed builds will be cleaned up when the user ends their last +login session on that node, but it is not possible to do so on the compute nodes. ```shell # EasByuild is configured to use /tmp/$USER on the login node -login01 $ eb --show-config | grep buildpath -buildpath (E) = /tmp/example +uan01 $ eb --show-config | grep buildpath +buildpath (E) = /run/user/XXXXXXXX/easybuild/build -# use /localdisk/$USER for build directories when submitting installations as jobs -login01 $ eb --job --buildpath /localdisk/$USER example.eb --robot +# use /dev/shm/$USER for build directories when submitting installations as jobs +login01 $ eb --job --buildpath /dev/shm/$USER/easybuild example.eb --robot ``` + ### Temporary log files and build directories -The temporary log file that EasyBuild creates is most likely going to end up on the local disk -of the workernode on which the job was started (by default in `$TMPDIR` or `/tmp`). -If an installation fails, the job will finish and temporary files will likely be cleaned up instantly, -which may leave you wondering about the actual cause of the failing installation... +The problems for the temporary log files are twofold. First, they may end up in a place +that is not available on the compute nodes. E.g., for the same reasons as for the build +path, the LUMI EasyBuild configuration will place the temporary files in a subdirectory of +``$XDG_RUNTIME_DIR`` on the loginnodes but a subdirectory of ``/dev/shm/$USER`` on the +compute nodes. The second problem however is that if an installation fails, those log files are +not even accessible anymore which may leave you wondering about the actual cause of the failing +installation... To remedy this, there are a couple of EasyBuild configuration options you can use: @@ -139,6 +167,7 @@ To remedy this, there are a couple of EasyBuild configuration options you can us ```shell $ eb --job example.eb --tmp-logdir $HOME/eb_tmplogs ``` + This will move at least the log file to a suitable place. * If you prefer having the entire log file stored in the Slurm job output files, you can use ``--logtostdout`` when submitting the jobs. This will result in extensive logging @@ -146,11 +175,13 @@ To remedy this, there are a couple of EasyBuild configuration options you can us log to ``stdout`` when the installation is running in the job, and hence the log messages will be captured in the job output files. -The same remark applies to build directories: they should be on a local filesystem (to avoid problems -that often occur when building software on a parallel filesystem like GPFS or Lustre), -which will probably be cleaned up automatically when a job fails. Here it is less easy to provide -general advice on how to deal with this, but one thing you can consider is retrying the installation -in an interactive job, so you can inspect the build directory after the installation fails. +The build directory of course also suffers from the problem of being no longer accessible if the +installation fails, but there it is not so easy to find a solution. Building on a shared file system +is not only much slower, but in particular on parallel file systems like GPFS/SpectrumScale, Lustre +or BeeGFS buiding sometimes fails in strange ways. One thing you can consider if you cannot do the +build on a login node (e.g., because the code is not suitable for cross-compiling or the configure +system does tests that would fail on the login node), is to rety the installation in an +interactive job, so you can inspect the build directory after the installation fails. ### Lock files @@ -171,37 +202,37 @@ subdirectory of ``installpath``) manually, or re-submit the job with ``eb --job As an example, we will let EasyBuild submit jobs to install ``AUGUSTUS`` with the ``foss/2020b`` toolchain. +!!! Warning "This example does not work on LUMI" + + Note that this is an example using the FOSS common toolchain. For this reason it does not work on + LUMI. + ### Configuration Before using ``--job``, let's make sure that EasyBuild is properly configured: ```shell -# use $HOME/easybuild for software, modules, sources, etc. -export EASYBUILD_PREFIX=$HOME/easybuild +# Load the EasyBuild-user module (central installations will not work at all +# using job submission) +module load LUMI/21.12 +module load partition/C +module load EasyBuild-user # use ramdisk for build directories -export EASYBUILD_BUILDPATH=/dev/shm/$USER +export EASYBUILD_BUILDPATH=/dev/shm/$USER/build +export EASYBUILD_TMPDIR=/dev/shm/$USER/tmp # use Slurm as job backend export EASYBUILD_JOB_BACKEND=Slurm ``` -In addition, add the path to the centrally installed software to ``$MODULEPATH`` via ``module use``: -```shell -module use /easybuild/modules/all -``` - -Load the EasyBuild module: - -```shell -module load EasyBuild -``` - -Let's assume that we also need to inform Slurm that jobs should be submitted into a particular account: +We will also need to inform Slurm that jobs should be submitted into a particular account, and +in a particular partition: ```shell -export SBATCH_ACCOUNT=example_project +export SBATCH_ACCOUNT=project_XXXXXXXXX +export SBATCH_PARTITION='small' ``` This will be picked up by the ``sbatch`` commands that EasyBuild will run to submit the software installation jobs. @@ -234,14 +265,14 @@ $ eb AUGUSTUS-3.4.0-foss-2020b.eb --missing Several dependencies are not installed yet, so we will need to use ``--robot`` to ensure that EasyBuild also submits jobs to install these first. -To speed up the installations a bit, we will request 10 cores for each submitted job (via ``--job-cores``). +To speed up the installations a bit, we will request 8 cores for each submitted job (via ``--job-cores``). That should be sufficient to let each installation finish in (well) under 1 hour, so we only request 1 hour of walltime per job (via ``--job-max-walltime``). In order to have some meaningful job output files, we also enable trace mode (via ``--trace``). ``` -$ eb AUGUSTUS-3.4.0-foss-2020b.eb --job --job-cores 10 --job-max-walltime 1 --robot --trace +$ eb AUGUSTUS-3.4.0-foss-2020b.eb --job --job-cores 8 --job-max-walltime 1 --robot --trace ... == resolving dependencies ... ... @@ -278,7 +309,7 @@ these jobs will be able to start. After about 20 minutes, AUGUSTUS and all missing dependencies should be installed: ``` -$ ls -lrt $HOME/easybuild/modules/all/*/*.lua | tail -11 +$ ls -lrt $HOME/EasyBuild/modules/.../*.lua | tail -11 -rw-rw----. 1 example example 1634 Mar 29 10:13 /users/example/easybuild/modules/all/HTSlib/1.11-GCC-10.2.0.lua -rw-rw----. 1 example example 1792 Mar 29 10:13 /users/example/easybuild/modules/all/SAMtools/1.11-GCC-10.2.0.lua -rw-rw----. 1 example example 1147 Mar 29 10:13 /users/example/easybuild/modules/all/BamTools/2.5.1-GCC-10.2.0.lua @@ -291,11 +322,9 @@ $ ls -lrt $HOME/easybuild/modules/all/*/*.lua | tail -11 -rw-rw----. 1 example example 1365 Mar 29 10:28 /users/example/easybuild/modules/all/SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0.lua -rw-rw----. 1 example example 2233 Mar 29 10:30 /users/example/easybuild/modules/all/AUGUSTUS/3.4.0-foss-2020b.lua -$ module use $HOME/easybuild/modules/all - $ module avail AUGUSTUS --------- /users/hkenneth/easybuild/modules/all -------- +-- EasyBuild managed user software for software stack ... -- AUGUSTUS/3.4.0-foss-2020b ``` From cc30e750e0bf9f492eb235020247b541f04e667a Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 15:42:35 +0200 Subject: [PATCH 114/159] Some explanation in the overview of part 3 --- docs/2022-CSC_and_LO/1_02_Lmod.md | 82 +++++++- docs/2022-CSC_and_LO/3_00_part3_advanced.md | 6 + .../3_04_module_naming_schemes.md | 192 +++++++----------- 3 files changed, 152 insertions(+), 128 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md index 0caa8a47..d6d73a6c 100644 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -73,12 +73,54 @@ to a hierarchy with 3 levels: 3. The ``MPI`` level, containing libraries and applications that depend on the compiler used and the MPI implementation. +??? Example "A simple Lmod hierarchy with a single compiler" + + Here is a simple example of such a 3-level module hierarchy + (that almost could have been generated by EasyBuild): + +
+ + In this example the ``Core`` level only includes a single module `GCC/9.3.0`, + while the ``Compiler`` level includes two modules: `OpenMPI/4.0.3` and `MPICH/3.3.2`. + In the ``MPI `` level, three modules are available: one for `FFTW`, one for `ScaLAPACK`, + and one for `HDF5`. + + Initially only the modules on the top level of a module hierarchy are available for loading. + If you run "`module avail`", the command that is used to view all modules that are available + for loading, with this example module hierarchy, you will only see the `GCC/9.3.0` module. + + Some modules in the top level of the hierarchy act as a "gateway" to modules in the + next level below. + To make additional modules available for loading one of these gateway modules has to be loaded. + In our example, loading the `GCC/9.3.0` module results in two additional modules coming into + view from the ``Compiler`` level, as indicated by the arrows: the modules for `OpenMPI` and `MPICH`. + These correspond to installations of `OpenMPI` + and `MPICH` that were built using `GCC/9.3.0`. + + Similarly, the `OpenMPI/4.0.3` module serves as a gateway to the three modules in the ``MPI`` level. + Only by loading the `OpenMPI` module will these additional three modules become + available for loading. They correspond to software installations built using the ``GCC/9.3.0`` + compiler with ``OpenMPI/4.0.3``. + Now assume that we have two compilers in the hierarchy, Compiler_A and Compiler_B. Their modules would reside -at the ``Core`` level. Both copilers provide the same MPI implementation, MPI_C. So there would be two modules +at the ``Core`` level. Both compilers provide the same MPI implementation, MPI_C. So there would be two modules for ``MPI_C`` in two different subdirectories at the ``Compiler`` level. And further assume that we have an application, Appl_E, compiled with both Compiler_A and Compiler_B and using MPI_C. For that application there would also be two module files at the ``MPI`` level, one in a subdirectory corresponding ao Compiler_A and MPI_C and one -in a subdirectory corresponding to Compiler_B and MPI_C. To be able to load the module for Appl_E, a user should +in a subdirectory corresponding to Compiler_B and MPI_C. + +```mermaid +graph TD; +A[Compiler_A] --> AC[MPI_C]; +A --> AD[MPI_D] +B[Compiler_B] --> BC[MPI_C]; +AC --> ACE[Appl_E]; +AD --> ADE[Appl_E] +BC --> BCE[Appl_E]; +``` + + +To be able to load the module for Appl_E, a user should first load Compiler_A, then load MPI_C and only then is it possible to load the module for Appl_E: ```bash @@ -102,6 +144,23 @@ and depending on those adapt the path to the binaries, several very simple modul logic, and one could add an Appl_E module for a different compiler or MPI implementation without touching any of the already existing module files for that application. +Similarly, if after + +```bash +module load Compiler_A MPI_C Appl_E +``` + +one does + +```bash +module load MPI_D +``` + +thent MPI_C gets unloaded, Lmod notices that it also has to unload/deactivate Appl_E, then will load MPI_D for Compiler_A and +finally will notice that there is an equivalent Appl_E module available again, and Lmod will load that one also. However, +now loading Compiler_B will cause a warning that MPI_D and Appl_E have been deactivated as there is no module name MPI_D in +any version for Compiler_B. + ### Building blocks @@ -126,8 +185,9 @@ Some mechanisms in Lmod make implementing a hierarchy fairly easy (though there ### Implementation details -The above example could be implemented using 6 module files: One for each compiler, two for the MPI module and -two for the application module. +The above example could be implemented using 8 module files: One for each compiler, three for the MPI modules +(two for MPI_C and one for MPI_D) and +three for the application modules. ``` moduleroot @@ -139,8 +199,10 @@ moduleroot ├── Compiler │ ├── Compiler_A │ │ └── version_A -│ │ └── MPI_C -│ │ └── version_C.lua +│ │ ├── MPI_C +│ │ │ └── version_C.lua +│ │ └── MPI_D +│ │ └── version_D.lua │ └── Compiler_B │ └── version_B │ └── MPI_C @@ -148,8 +210,12 @@ moduleroot └── MPI ├── Compiler_A │ └── version_A - │ └── MPI_C - │ └── version_C + │ ├── MPI_C + │ │ └── version_C + │ │ └── Appl_E + │ │ └── version_E.lua + │ └── MPI_D + │ └── version_D │ └── Appl_E │ └── version_E.lua └── Compiler_B diff --git a/docs/2022-CSC_and_LO/3_00_part3_advanced.md b/docs/2022-CSC_and_LO/3_00_part3_advanced.md index 708c57f8..28a24990 100644 --- a/docs/2022-CSC_and_LO/3_00_part3_advanced.md +++ b/docs/2022-CSC_and_LO/3_00_part3_advanced.md @@ -2,6 +2,12 @@ *[[back to start page]](index.md)* +In this section we mostly cover "good to know that they exist" features as they are not used +on LUMI, or not really accessible to regular user installations that are performed with the +LUMI ``EasyBuild-user`` module. Hooks are used on LUMI, but it is not really advised to +overwrite the centrally defined hooks with a local file. And the whole structure of the +EasyBuild integration is also set up to make use of the GitHub integration in the future. + * [Using EasyBuild as a library](3_01_easybuild_library.md) *(hands-on)* * [Using hooks to customise EasyBuild](3_02_hooks.md) *(hands-on)* * [Submitting installations as Slurm jobs](3_03_slurm_jobs.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md index 14229d97..d0b4ebe8 100644 --- a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md +++ b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md @@ -4,21 +4,24 @@ --- -Up until now we have used EasyBuild's default module naming scheme (`EasyBuildMNS`), -which produces module files with names that closely resemble to the names of the +Up until now we have used the naming scheme in use on LUMI which is for many practical purposes +is indistinguishable from the default EasyBuild naming scheme (`EasyBuildMNS`). +It produces module files with names that closely resemble to the names of the corresponding easyconfig files. -For example, when installing `h5py-3.1.0-foss-2020b.eb` the generated module was named `h5py/3.1.0-foss-2020b`. +For example, when installing `zlib-1.2.11-cpeGNU-21.12.eb` the generated module was named +`zlib/1.2.11-cpeGNU-21.12`. EasyBuild supports several different module naming schemes: ```shell $ eb --avail-module-naming-schemes List of supported module naming schemes: - CategorizedHMNS - CategorizedModuleNamingScheme - EasyBuildMNS - HierarchicalMNS - MigrateFromEBToHMNS + EasyBuildMNS + LUMI_FlatMNS + MigrateFromEBToHMNS + HierarchicalMNS + CategorizedHMNS + CategorizedModuleNamingScheme ``` In this part of the tutorial we will take a closer look at `HierarchicalMNS`, @@ -28,52 +31,54 @@ We will also take a quick look at implementing our own custom module naming sche ## Flat vs hierarchical -The default module naming scheme `EasyBuildMNS` is an example of regular *"flat"* module naming scheme, which is characterized by: +!!! Note -* all module files are directly available for loading; -* each module name uniquely identifies a particular installation; + This text is very much based on the generic EasyBuild tutorials as we cannot demonstrate + all aspects on LUMI as it is configured today. -In contrast, a *hierarchical* module naming scheme -consists of a *hierarchy* of module files. - -The typical module hierarchy has 3 levels: + On LUMI, the complete module scheme is partly hierarchical, but EasyBuild uses a flat naming + scheme. The two levels in the hierarchy that are present to deal with different versions of + the software stack and the various types of compute nodes, are not managed by EasyBuild. -* a **core** level, where module files for software that was installed using the - [`system` toolchain](../introduction/#system-toolchain) are kept; -* a **compiler** level, where module files for software that was installed using a -*compiler-only toolchain* are stored; -* and an **MPI** level, which houses module files for software that was installed using a toolchain that includes (at least) a compiler and MPI component; +The default module naming scheme `EasyBuildMNS` +and the custom `LUMI_FlatMNS` naming scheme are both examples of regular *"flat"* +module naming schemes, which are characterized by: -Here is a simple example of such a 3-level module hierarchy: +- all module files are directly available for loading; +- each module name uniquely identifies a particular installation; -
+In contrast, a *hierarchical* module naming scheme +consists of a *hierarchy* of module files. +A fairly typical 3-level schme (``Core``, ``Compiler`` and ``MPI``) has been +discussed in the [section on Lmod](../1_02_Lmod#lmod-hierarchy). +This typical Lmod hierarcny would map very well on the EasyBuild common toolchains. -In this example the core level only includes a single module `GCC/9.3.0`, -while the compiler level includes two modules: `OpenMPI/4.0.3` and `MPICH/3.3.2`. -In the MPI level, three modules are available: one for `FFTW`, one for `ScaLAPACK`, and one for `HDF5`. +In fact, for the example -Initially only the modules on the top level of a module hierarchy are available for loading. -If you run "`module avail`" with the example module hierarchy, you will only see the `GCC/9.3.0` module. +
-Some modules in the top level of the hierarchy act as a "gateway" to modules in the -next level below. -To make additional modules available for loading one of these gateway modules has to be loaded. In our exampe, loading the `GCC/9.3.0` module results in two additional modules coming into view from the compiler level, as indicated by the arrows: the modules for `OpenMPI` and `MPICH`. These correspond to installations of `OpenMPI` -and `MPICH` that were built using `GCC/9.3.0` as a (compiler-only) toolchain. +software at the ``Core`` level would be installed with the ``GCCcore`` and ``SYSTEM`` +toolchains. Software at the ``Compiler`` level would be installed with the ``GCC`` +toolchain, while software at the ``MPI`` level after loading the ``OpenMPI`` module +would be installed using the regular ``foss`` toolchain or the ``gompi`` toolchain +(see the diagram in the +["Common toolchains" section of the page on terminology](../1_05_terminology/#common-toolchains)). -Similarly, the `OpenMPI/4.0.3` module serves as a gateway to the three modules in the MPI level. Only by loading the `OpenMPI` module will these additional three modules become -available for loading. They correspond to software installations built using the `gompi/2020a` toolchain that -consists of the `GCC/9.3.0` compiler module and the `OpenMPI/4.0.3` MPI module. Software installed using -`foss/2020a` (which is a full toolchain that also includes OpenBLAS, FFTW and ScaLAPACK) would also be stored -in this level of the module hierarchy. +On LUMI, where software is installed through the Cray Programming Environment with no real choice of +MPI implementation, a two-level arrangement would still make a lot of sense, with at the ``Core`` level +all software compiled with the SYSTEM toolchain while there could be a ``PrgEnv`` level for software +compiled with a particula programming enviornment aka cepGNU/cpeCray/cpeAOCC toolchain. Such a scheme +is used on the Cray systems at CSCS. -The characteristics of a module hierarchy are: +To recap, the characteristics of a module hierarchy are: * not all module files are directly available for loading; * some modules serve as a gateway to more modules; * to access some software installations you will first need to load one or more gateway modules in order to use them; -You can probably think of other ways to organize module files in a hierarchical module tree, but here we will stick to the standard core / compiler / MPI hierarchy. +You can probably think of other ways to organize module files in a hierarchical module tree, but here +we will stick to the standard core / compiler / MPI hierarchy. ### Pros & cons @@ -81,22 +86,24 @@ So why go through all this trouble of organizing modules hierarchically? There are a couple of advantages to this approach: -* shorter module names; +* shorter module names (or at least for the version part of the name); * less overwhelming list of available modules; * only compatible modules can be loaded together; However, the are some minor disadvantages too: -* not all existing modules are directly visible; +* not all existing modules are directly visible, so users have to learn how to find + modules using ``module spider`` etc.; * gateway modules may have little meaning to end users; #### Length of module names -When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For our `HDF5` installation for example, -we have `HDF5/1.10.6-gompi-2020a` as module name. The `-gompi-2020a` part of the name refers to the toolchain that was +When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For a `HDF5` installation +with the EasyBuild common toolchains for example, +one might have `HDF5/1.10.6-gompi-2020a` as module name. The `-gompi-2020a` part of the name refers to the toolchain that was used for this installation, but it may be confusing to some people (what kind of Pokémon is a "gompi"?!). -In the example module hierarchy shown above, the module for `HDF5` is named `HDF5/1.10.6` which is basically the bare +In the example module hierarchy shown above, the module for `HDF5` could simply be named `HDF5/1.10.6` which is basically the bare essentials: software name and version. That's way better, nice and clean! #### Amount of available modules @@ -130,8 +137,10 @@ and so these modules can not be loaded together easily. One downside of a module hierarchy is that not all existing modules are directly available for loading or are even *visible* to the user, since the output of "`module avail`" only shows a subset of all modules. -[Lmod](https://lmod.readthedocs.io) has a solution for this though: it provides a separate "`module spider`" -command to search for module files throughout the entire module hierarchy. So as long as the end users are +As we have discussed in [the Lmod section](../LMOD). +[Lmod](https://lmod.readthedocs.io) provides a powerful solution to search for modules through the +``module spider`` and to some extent also the ``module keyword`` commands. +So as long as the end users are aware of this additional command, it should not be difficult to discover which software installations exist and how they can be accessed. The "`module spider`" command will inform the user which of the gateway modules need to be loaded in order to load a specific module file. @@ -140,7 +149,7 @@ need to be loaded in order to load a specific module file. An additional potential problem of a module hierarchy is that the semantics of the gateway modules may not be clear to end users. They may wonder why they need to pick a specific compiler and MPI library, or which of the -available options is the best one. Maybe there are not even be aware what exactly a "compiler" is, or how it is +available options is the best one. Maybe they are not even aware what exactly a "compiler" is, or how it is relevant to the software they need in their bioinformatics pipeline... This can be partially resolved by loading a default compiler and MPI module so a particular set of modules @@ -148,10 +157,15 @@ is available right after login, which could be the ones used in the most recent recommended versions. More experienced users could then leverage the "`module spider`" command to navigate the module hierarchy. +On LUMI this would mean loading a default software stack, but due to the the default modules are currently +loaded on LUMI this was not possible to accomplish without losing other functionality of the module tree. + ## Using a custom module naming scheme Next to the module naming schemes that are included with EasyBuild, -you can also define your own module naming scheme (MNS), and configure EasyBuild to use it. +you can also define your own module naming scheme (MNS), and configure EasyBuild to use it +(which is eactly what has been done on LUMI to remove a feature of the default ``EasyBuildMNS`` scheme +that we do not use). ### Implementation @@ -254,7 +268,9 @@ class ExampleMNS(ModuleNamingScheme): return short_modname.startswith(name.lower().replace('-', '_') + '/') ``` -We can see what the module names with this module naming scheme would like like via ``eb -D``: +We can see what the module names with this module naming scheme would like like via ``eb -D``. +E.g., for the common toolchains (the example will not work on LUMI as the default easyconfig +files are not in the robot- and the search path): ``` $ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D @@ -270,9 +286,10 @@ $ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D ## Example module hierarchy: HDF5 -!!! Warning - **This exercise will only work if you have write access to the software installation prefix**, - because EasyBuild will try to copy the installation log file to each installlation directory. +!!! Warning "Example not suitable for LUMI" + **This exercise is meant for a system where the common toolchains can be used and requires an + indpendent EasyBuild installation in your personal file space**, + because EasyBuild will try to copy the installation log file to each installation directory. Now that we know more about hierarchical module naming schemes, let us see how EasyBuild can help us with generating a hierarchical module tree. @@ -308,7 +325,7 @@ a module. We have just made all modules unavailable, so we would have to first install EasyBuild again in our hierarchical module tree before we can continue. **We strongly recommend using an EasyBuild installation that was [installed via "`pip install`" -or "`pip3 install`"](../installation/#method-1-using-pip-recommended) in this part of the tutorial.** +or "`pip3 install`"](../1_06_installation#method-1-using-pip) in this part of the tutorial.** An easy way to do this is in the prepared environment is to run: @@ -376,7 +393,9 @@ The steps we will have to go through are: * Tell EasyBuild we want to "install" the `HDF5-1.10.7-gompi-2020b.eb` easyconfig file; * Enable dependency resolution via `--robot`; -* Instruct EasyBuild to only generate the module files, not to install the software (since it is +* Assuming the software would have been installed already with the default naming scheme + in a different module directory, instruct EasyBuild to only generate the module files, + not to install the software (since it is there already in `/easybuild/software`), via the `--module-only` option. These steps translate to this single `eb` command: @@ -544,73 +563,6 @@ Currently Loaded Modules: 4) GCC/10.2.0 8) libpciaccess/0.16 12) libfabric/1.11.0 16) HDF5/1.10.7 ``` -## Exercise - -Now it is your turn! - -Try to get a feeling for how a hierarchical module tree works by: - -* installing the missing modules for the `SciPy-bundle-2020.11-foss-2020b.eb` in the module hierarchy we - generated for HDF5; -* figure out where the `SciPy-bundle` module is located in the hierarchy, and then also load it; - -You can verify your work by running this command (since `pandas` is one of the Python packages included -in the `SciPy-bundle` installation): - -```shell -python -c 'import pandas; print(pandas.__version__)' -``` - -Start from a clean slate, by first running: - -```shell -module purge -module unuse $MODULEPATH -``` - -??? success "(click to show solution)" - - * Step 0: check which modules are still missing, using `--missing` or `-M`: - ```shell - eb SciPy-bundle-2020.11-foss-2020b.eb -M - ``` - The output should tell you that 15 out of 50 required modules are still missing. - - * Install the missing modules in the module hierarchy we have generated in `$HOME/hmns/modules`: - ```shell - eb SciPy-bundle-2020.11-foss-2020b.eb --robot --module-only - ``` - Don't forget to use both `--robot` (to enable dependency resolution) and `--module-only` - (to only run the sanity check and generate module files, not install the software again). - - * Start at the top of the module hierarchy (the `Core` level), - and run module spider to check which gateway modules to load to make `SciPy-bundle` available: - ```shell - module use $HOME/hmns/modules/all/Core - module spider SciPy-bundle/2020.11 - ``` - * Load the gateway modules: - ```shell - module load GCC/10.2.0 OpenMPI/4.0.5 - ``` - * Check that the `SciPy-bundle` module is available, and load it: - ```shell - $ module avail SciPy-bundle - ----- /home/example/hmns/modules/all/MPI/GCC/10.2.0/OpenMPI/4.0.5 ------ - SciPy-bundle/2020.11 - ``` - ``` - module load SciPy-bundle/2020.11 - ``` - * Run the test command: - ``` - $ python -c 'import pandas; print(pandas.__version__)' - 1.1.4 - ``` - -!!! Warning - **This exercise will only work if you have write access to the software installation prefix**, - because EasyBuild will try to copy the installation log file to each installlation directory. --- From 7a9f3c77f7ae95170e296c6db4a0d974e890fd29 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 16:22:58 +0200 Subject: [PATCH 115/159] GitHub integration section, mostly old text and not all suited for LUMI. --- .../3_05_github_integration.md | 28 +++++++++++++++---- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_05_github_integration.md index 59dcbb67..6d04ce7f 100644 --- a/docs/2022-CSC_and_LO/3_05_github_integration.md +++ b/docs/2022-CSC_and_LO/3_05_github_integration.md @@ -4,6 +4,20 @@ --- +!!! Warning "Preliminary not for LUMI and many other Cray systems" + + Not everything in this section is as useful for EasyBuild installations that build + on the Cray Programming Environment. E.g., the default easyconfig repository does + not contain easyconfigs for the Cray PE. + + Moreover, due to the restricted operating system on the compute nodes of big + Cray systems such as the EX series (COS instead of the full SUSE linux), + the GitHub integration does not work on the compute nodes of a typical setup as + one of the required Python packages fails to work as it needs the DBus daemon. + + Since GitHub integration is not yet really used on LUMI, this section is not fully + tested but left in the tutorial for completeness. + To contribute changes to the EasyBuild code (framework or easyblocks) or easyconfigs, you will need to be a bit familiar with Git and GitHub. Or maybe not? @@ -94,11 +108,6 @@ and configuration. **Additional dependencies** Both the `GitPython` and `keyring` Python packages as well as the `keyrings.cryptfile` add-on package must be installed. -In the prepared environment, you can do this via: - -```shell -pip3 install --user GitPython keyring keyrings.cryptfile -``` !!! Note You may experiences problems installing the ``cryptography`` Python packages, @@ -294,6 +303,15 @@ will include information about your system (OS, processor, etc.) and your EasyBu ## Demo +!!! Warning "Not currently suited for LUMI" + + This is text from a previous (non-Cray) version of the tutorial. The demo does require + a number of files not installed on LUMI after going through this tutorial, + and someone who can master the repository used to clean up again. + However, having a look at the output (which was generated in the spring of 2021 on + the CSC system puhti) still offers some information, so the demo was left in this + version of the tutorial. + That is a lot to digest, so let us make this a bit more concrete with an example: we will open a pull request for the [`eb-tutorial` example software](../adding_support_software/#example) to *a fork* of the [`easybuild-easyconfigs` repository](https://github.com/easybuilders/easybuild-easyconfigs) using the `eb` command, and submit a test report in it. From 9a01408c2c3d19290d34c90c7ad4f27f57d1b2d9 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 16:34:33 +0200 Subject: [PATCH 116/159] Adapt the structure and include an additional reading section. --- docs/2022-CSC_and_LO/3_05_github_integration.md | 2 +- docs/2022-CSC_and_LO/4_00_additional_reading.md | 1 - mkdocs.yml | 1 + 3 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_05_github_integration.md index 6d04ce7f..6e19cb9a 100644 --- a/docs/2022-CSC_and_LO/3_05_github_integration.md +++ b/docs/2022-CSC_and_LO/3_05_github_integration.md @@ -457,4 +457,4 @@ The gist linked from this comment provides more detailed information: --- -*[[next: Overview]](index.md)* +*[[next: Additional reading]](4_00_additional_reading.md)* diff --git a/docs/2022-CSC_and_LO/4_00_additional_reading.md b/docs/2022-CSC_and_LO/4_00_additional_reading.md index f14ba6fb..14f8454c 100644 --- a/docs/2022-CSC_and_LO/4_00_additional_reading.md +++ b/docs/2022-CSC_and_LO/4_00_additional_reading.md @@ -50,6 +50,5 @@ ([video](https://www.youtube.com/watch?v=hZezVG6lJNk&list=PLhnGtSmEGEQgCneeSQvYoIZrbv7wIKlo2)/[slides](https://easybuild.io/eum22/001_eum22_EasyBuild_on_LUMI.pdf)) during the [7th EasyBuild User Meeting in 2022](https://easybuild.io/eum22/) ---- *[[next: Overview]](index.md)* diff --git a/mkdocs.yml b/mkdocs.yml index 9a64f55d..8ce643c9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -143,6 +143,7 @@ nav: - Submitting installations as Slurm jobs: 2022-CSC_and_LO/3_03_slurm_jobs.md - Module naming schemes (incl. hierarchical): 2022-CSC_and_LO/3_04_module_naming_schemes.md - GitHub integration to facilitate contributing to EasyBuild: 2022-CSC_and_LO/3_05_github_integration.md + - Additional reading: 2022-CSC_and_LO/4_00_additional_reading.md plugins: From fe44f8711adaf5ec92e06601cff6a737fe6262a3 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 17:43:28 +0200 Subject: [PATCH 117/159] Correcting a number of links. --- docs/2022-CSC_and_LO/1_06_installation.md | 4 ++-- docs/2022-CSC_and_LO/1_07_configuration.md | 6 +++--- docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md | 6 +++--- docs/2022-CSC_and_LO/2_03_external_modules.md | 2 +- docs/2022-CSC_and_LO/3_03_slurm_jobs.md | 6 +++--- 5 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_06_installation.md b/docs/2022-CSC_and_LO/1_06_installation.md index 6e68b500..24c71d5e 100644 --- a/docs/2022-CSC_and_LO/1_06_installation.md +++ b/docs/2022-CSC_and_LO/1_06_installation.md @@ -424,7 +424,7 @@ We do so only for the framework and easyblocks files as the easconfig files are install EasyBuild. Instead we create our own EasyConfig file for EasyBuild which contains some additional packages that enable extra features in EasyBuild and also provide more information to Lmod. Next the configuration module for EasyBuild (see the next section, -["Configuring EasyBuild"](1_06_configuring.md), for more information) and use the temporary +["Configuring EasyBuild"](1_07_configuration), for more information) and use the temporary installation of EasyBuild with our own EasyConfig file to do a proper installation of EasyBuild with module in the final location. @@ -468,7 +468,7 @@ eb --show-config ``` This should tell you that EasyBuild (ab)uses `$HOME/.local/easybuild` as a default location. -More on configuring EasyBuild in the [next part of the tutorial](configuration.md). +More on configuring EasyBuild in the [next part of the tutorial](configuration). #### System information diff --git a/docs/2022-CSC_and_LO/1_07_configuration.md b/docs/2022-CSC_and_LO/1_07_configuration.md index 17e60afe..a1c2f3de 100644 --- a/docs/2022-CSC_and_LO/1_07_configuration.md +++ b/docs/2022-CSC_and_LO/1_07_configuration.md @@ -280,7 +280,7 @@ The default `EasyBuildMNS` module naming scheme roughly corresponds to the filen and consists of the software name followed by a combination of the software version, toolchain and an optional label (which corresponds to the value of the `versionsuffix` easyconfig parameter): `/<-toolchain>`. Just like with [names of easyconfig -files](terminology.md#easyconfig-files), the `<-toolchain>` part is omitted when the `system` toolchain is used, +files](1_05_terminology#easyconfig-files), the `<-toolchain>` part is omitted when the `system` toolchain is used, and the `` value is empty by default. !!! Note "Module naming scheme on LUMI" @@ -366,9 +366,9 @@ across different software installations. They define *configuration settings*, such as the location where software should be installed, or the syntax that should be used when generating module files. -An [easyconfig file](terminology.md#easyconfig-files) (`*.eb`) on the other hand *specifies the details for one particular software installation*. +An [easyconfig file](1_05_terminology#easyconfig-files) (`*.eb`) on the other hand *specifies the details for one particular software installation*. It does this by defining a set of *easyconfig parameters*, which tell EasyBuild the name and version -of the software to install, which [toolchain](terminology.md#toolchains) and [easyblock](terminology.md#easyblocks) to use, etc. +of the software to install, which [toolchain](1_05_terminology#toolchains) and [easyblock](1_05_terminology#easyblocks) to use, etc. For each software installation performed by EasyBuild, there is a corresponding easyconfig file. There typically are only a handful of configuration files used however, for example a system-level diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md index f7d6a646..f9bec25a 100644 --- a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md @@ -1,6 +1,6 @@ # Creating easyconfig files -*[[back: Troubleshooting]](2_02_troubleshooting.md)* +*[[back: Troubleshooting]](2_02_troubleshooting)* --- @@ -58,7 +58,7 @@ Reasons to consider implementing a software-specific easyblock rather than using bootstrapping with the system compiler and then re-installing with itself.* For implementing easyblocks we refer to the -["Implementing easyblocks" section of this tutorial](2_03_implementing_easyblocks.md) +["Implementing easyblocks" section of this tutorial](2_03_implementing_easyblocks) and the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). ## Writing easyconfig files @@ -1024,4 +1024,4 @@ You can leverage the `eb-tutorial` easyconfig file we have composed in the examp --- -*[[next: Using external modules from the Cray PE]](2_03_external_modules.md)* +*[[next: Using external modules from the Cray PE]](2_03_external_modules)* diff --git a/docs/2022-CSC_and_LO/2_03_external_modules.md b/docs/2022-CSC_and_LO/2_03_external_modules.md index b19313ff..80a29712 100644 --- a/docs/2022-CSC_and_LO/2_03_external_modules.md +++ b/docs/2022-CSC_and_LO/2_03_external_modules.md @@ -1,6 +1,6 @@ # Using external modules from the Cray PE -*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* +*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files)* --- diff --git a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md index 567f8abd..283a53a0 100644 --- a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md +++ b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md @@ -37,8 +37,8 @@ will be *passed down* into the submitted job automatically, via command line opt command that is run in the job script. This includes not only command line options used next to ``--job``, but also configuration settings -that are specified via an [EasyBuild configuration file](configuration.md#configuration-files) or through -[``$EASYBUILD_*`` environment variables](configuration.md#easybuild_-environment-variables). +that are specified via an [EasyBuild configuration file](1_07_configuration#configuration-files) or through +[``$EASYBUILD_*`` environment variables](1_07_configuration#easybuild_-environment-variables). This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables that are in place in the job environment are most likely *irrelevant*, since configuration settings @@ -330,4 +330,4 @@ $ module avail AUGUSTUS --- -*[[next: Module naming schemes]](3_04_module_naming_scheme.md)* +*[[next: Module naming schemes]](3_04_module_naming_scheme)* From 064afbee75500b1939a44c3460f066891e32cb20 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 7 Apr 2022 17:26:56 +0200 Subject: [PATCH 118/159] Restructuring for nicer navigation bar on the left. --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 12 - .../2022-CSC_and_LO/1_01_what_is_easybuild.md | 321 ----- docs/2022-CSC_and_LO/1_02_Lmod.md | 820 ----------- docs/2022-CSC_and_LO/1_03_CPE.md | 418 ------ .../1_04_LUMI_software_stack.md | 101 -- docs/2022-CSC_and_LO/1_05_terminology.md | 294 ---- docs/2022-CSC_and_LO/1_06_installation.md | 545 -------- docs/2022-CSC_and_LO/1_07_configuration.md | 610 --------- docs/2022-CSC_and_LO/1_08_basic_usage.md | 1216 ----------------- .../1_Intro/1_01_what_is_easybuild.md | 2 +- docs/2022-CSC_and_LO/2_00_part2_using.md | 8 - docs/2022-CSC_and_LO/2_01_troubleshooting.md | 545 -------- .../2_02_creating_easyconfig_files.md | 1027 -------------- docs/2022-CSC_and_LO/2_03_external_modules.md | 85 -- .../2_04_implementing_easyblocks.md | 754 ---------- docs/2022-CSC_and_LO/3_00_part3_advanced.md | 15 - .../2022-CSC_and_LO/3_01_easybuild_library.md | 301 ---- docs/2022-CSC_and_LO/3_02_hooks.md | 213 --- docs/2022-CSC_and_LO/3_03_slurm_jobs.md | 333 ----- .../3_04_module_naming_schemes.md | 569 -------- .../3_05_github_integration.md | 460 ------- docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md | 1 - 22 files changed, 1 insertion(+), 8649 deletions(-) delete mode 100644 docs/2022-CSC_and_LO/1_00_part1_intro.md delete mode 100644 docs/2022-CSC_and_LO/1_01_what_is_easybuild.md delete mode 100644 docs/2022-CSC_and_LO/1_02_Lmod.md delete mode 100644 docs/2022-CSC_and_LO/1_03_CPE.md delete mode 100644 docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md delete mode 100644 docs/2022-CSC_and_LO/1_05_terminology.md delete mode 100644 docs/2022-CSC_and_LO/1_06_installation.md delete mode 100644 docs/2022-CSC_and_LO/1_07_configuration.md delete mode 100644 docs/2022-CSC_and_LO/1_08_basic_usage.md delete mode 100644 docs/2022-CSC_and_LO/2_00_part2_using.md delete mode 100644 docs/2022-CSC_and_LO/2_01_troubleshooting.md delete mode 100644 docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md delete mode 100644 docs/2022-CSC_and_LO/2_03_external_modules.md delete mode 100644 docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md delete mode 100644 docs/2022-CSC_and_LO/3_00_part3_advanced.md delete mode 100644 docs/2022-CSC_and_LO/3_01_easybuild_library.md delete mode 100644 docs/2022-CSC_and_LO/3_02_hooks.md delete mode 100644 docs/2022-CSC_and_LO/3_03_slurm_jobs.md delete mode 100644 docs/2022-CSC_and_LO/3_04_module_naming_schemes.md delete mode 100644 docs/2022-CSC_and_LO/3_05_github_integration.md diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md deleted file mode 100644 index 0674be92..00000000 --- a/docs/2022-CSC_and_LO/1_00_part1_intro.md +++ /dev/null @@ -1,12 +0,0 @@ -# Part I: Introduction to EasyBuild on Cray systems - -*[[back to start page]](index.md)* - -- [What is EasyBuild?](1_01_what_is_easybuild.md) -- [The Lmod module system](1_02_Lmod.md) -- [The HPE Cray Programming Environment](1_03_CPE.md) -- [LUMI software stacks](1_04_LUMI_software_stack.md) -- [Terminology](1_05_terminology.md) -- [Installation](1_06_installation.md) -- [Configuration](1_07_configuration.md) -- [Basic usage](1_08_basic_usage.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md deleted file mode 100644 index 979c6e90..00000000 --- a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md +++ /dev/null @@ -1,321 +0,0 @@ -# What is EasyBuild? - -*[[back: Overview]](index.md)* - -

EasyBuild logo

- -**[EasyBuild](http://easybuild.io)** is a **software build and installation framework** -that manages (scientific) software in an efficient way, without compromising on performance. -Easybuild is especially tailored towards [**High Performance Computing (HPC)**](https://en.wikipedia.org/wiki/Supercomputer) systems, -and provides a **command-line interface** to facilitate the installation and management of a scientific software stack. - ---- - - - - - -## Elevator pitch - -EasyBuild intends to relieve HPC support teams from manually managing -software installations while at the same time **providing a consistent and well performing -scientific software stack** to end users of HPC systems. - -It serves as a **uniform interface for installing scientific software** -and saves valuable time (and frustration) by the automation of tedious, boring and repetitive tasks. - -In addition, EasyBuild can **empower scientific researchers to self-manage their software stack**, -and it can serve as a tool that can be leveraged for **building optimized container images**. - -The project has grown to become a **platform for collaboration** among HPC sites worldwide. - ---- - -On LUMI we pay a lot of attention to empowering users to create their custom software stack which -is best suited for their project without being confronted with hundreds of modules that are not -useful to them. It also enables them to easily customize the build instructions to create a -project-specific build or even to install software with custom patches to adapt to their needs. -For this we offer both [EasyBuild](https://easybuild.io/) and [Spack](https://spack.io/) -as tools but obviously this tutorial concentrates on EasyBuild alone. - -EasyBuild can also be used to [generate container recipes](https://docs.easybuild.io/en/latest/Containers.html) -that will then use EasyBuild when building -the container to install platform-optimised software in the container. However, as that functionality -is still under development on LUMI it is not yet covered in this LUMI-specific version of the tutorial. - - ---- - -## Key features - -EasyBuild is capable of **fully autonomously installing (scientific) software**, -including making sure that all necessary dependencies are installed, -and automatically generating environment module files. - -***No*** **admin privileges are required**: it is sufficient to have write permissions -to the preferred software installation prefix. - -EasyBuild is [**highly configurable**](https://docs.easybuild.io/en/latest/Configuration.html) via configuration files, -environment variables, and command line options. Various elements of the functionality can be -[**dynamically extended**](https://docs.easybuild.io/en/latest/Including_additional_Python_modules.html) via plugins, -and [**hooks**](https://docs.easybuild.io/en/latest/Hooks.html) are available for further site-specific customizations. - -The installation procedure executed by EasyBuild is thoroughly -[**logged**](https://docs.easybuild.io/en/latest/Logfiles.html), and is fully transparent via support for -[**dry runs**](https://docs.easybuild.io/en/latest/Extended_dry_run.html) and -[**tracing**](https://docs.easybuild.io/en/latest/Tracing_progress.html) the software installation procedure -as it is performed. - -EasyBuild supports using a **custom module naming scheme**, allows for -*hierarchical* module naming schemes, and **integrates with various other tools** ranging from resource managers ([Slurm](https://slurm.schedmd.com) and [GC3Pie](https://github.com/gc3pie/gc3pie)), -container tools ([Singularity](https://github.com/hpcng/singularity) and [Docker](https://www.docker.com)), -packaging tools ([FPM](https://fpm.readthedocs.io)), and so on. - ---- - -## Focus points - -EasyBuild was created specifically for **installing scientific software on HPC systems**, -which is reflected in some of the design choices that were made. - - -### Performance - -EasyBuild strongly prefers to **build software from source code**, whenever possible. - -This is important to ensure that the binaries that are installed can maximally exploit -the capabilities of the system architecture on which the software will be run. - -For that same reason, EasyBuild out-of-the-box **optimizes software for the processor architecture of the build host** -by default, via compiler options like ``-march=native`` (GCC), ``-xHost`` (Intel compilers), etc. -This behaviour [can be changed via the ``--optarch`` configuration setting](https://docs.easybuild.io/en/latest/Controlling_compiler_optimization_flags.html). -When using the HPE Cray Programming Environment, the target modules are used instead. -In principle it is possible to cross-compile with EasyBuild, but this comes with some pitfalls. -By default EasyBuild tries to run tests if the build process of a package provides that -functionality, or can run simple tests of its own, and of course these tests will fail if -the host CPU does not support the full instruction set of the target CPU. Secondly, more -and more software has broken build processes where developers thought that it was a good -idea to hard-code options such as ``-march=native`` or ``--xHost`` in a way that EasyBuild -cannot easily overwrite those. - - -### Reproducibility - -In addition to performance, **reproducibility of installations** is a core aspect of EasyBuild. - -TODO correct link in the next lines. - -Most software installations performed with EasyBuild use a **particular compiler toolchain**, -with which we aim to be in control over the build environment and avoid relying on tools and libraries -provided by the operating system. For similar reasons, we try to **provide all required dependencies through EasyBuild** as well, -with a few notable exceptions, like ``OpenSSL`` for security reasons, and Infiniband and GPU drivers which -are too closely intertwined with the operating system. - -For both toolchains and dependencies, **fixed software versions** are specified in the -easyconfig files. That way, easyconfig files can easily be shared with others: -if they worked for you it is very likely that they will work for others too, because the vast majority of the -software stack is controlled by EasyBuild. - -As we shall see when discussing the HPE Cray Programming Environment, the reproducibility is a bit more limited -in that environment due to the way the programming environment selects shared libraries at runtime. - - -### Community effort - -In a number of different ways, we try to encourage EasyBuild users to **collaborate** and help each other out. - -We actively recommend people to report problems and bugs, to submit ideas for additional features and improvements, -and to [**contribute back**](https://docs.easybuild.io/en/latest/Contributing.html) when possible, be it -by opening pull requests to the [GitHub repositories](https://github.com/easybuilders) or the [documentation](https://docs.easybuild.io). - -The EasyBuild community also provides packages for two standard toolchains, called the -[**common toolchains**](https://easybuild.readthedocs.io/en/latest/Common-toolchains.html). -The so-called ``foss`` is based on free and open software (GNU compilers, FlexiBlas with OpenBLAS backend, LaPACK, FFTW and Open MPI) -while the ``intel`` toolchain uses the full stack of Intel compilers, libraries and MPI. Currently these toolchains -cannot be supported on LUMI though as getting Open MPI to run on LUMI is challenging and as the Intel libraries and -some MPI versions also produce slow or incorrect results on AMD CPUs (and it is not clear if Intel MPI is compatible -with SlingShot 11). -Focusing the efforts on those two popular common toolchains makes it possible to support a lot of build recipes that are -collected in the [central repository](https://github.com/easybuilders/easybuild-easyconfigs). -They can still serve as a basis to develop build recipes for the HPE Cray Programming Environment. - -Last but not least, EasyBuild provides various [**GitHub integration features**](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) -that greatly facilitate the contribution process: opening, updating, and testing pull requests, -reviewing incoming contributions, and much more can all be done directly from the EasyBuild -command line. This not only saves time, effort, brain cycles, and mouse clicks for contributors, -but it also makes the review process for *maintainers* significantly easier. -All together this leads to improved stability and consistency. - ---- - -## What EasyBuild is ***not*** - -EasyBuild is ***not*** **YABT (Yet Another Build Tool)**: it does *not* replace established build -tools like CMake or ``make``, it wraps around them. -If the installation procedure of a software package involves running some unholy trinity of tools while whispering -the correct magic incantations, EasyBuild automates this process for you. - -It is ***not*** **a replacement for traditional Linux package managers** like ``yum``, ``dnf`` or ``apt``. -EasyBuild relies on certain tools and libraries provided by the operating system. This includes glibc, OpenSSL, -drivers for Infiniband and GPUs, and so on. It is required that these tools are installed and managed by other means. -This is typically done via the package management tool that comes with your Linux distribution. - -Finally, EasyBuild is ***not*** **a magic solution to all your (software installation) problems**. -You may, and probably still will occasionally, run into compiler errors unless somebody has already taken care of the problem for you. -This is even more likely to happen in the Cray environment as builds are not often tested with the Cray compilers -or even with the AMD compilers. - ---- - -## Requirements - -Since EasyBuild is primarily intended for installing (scientific) software on HPC systems, the main target is -systems running some flavor of **GNU/Linux**. EasyBuild also works on *macOS*, -but significant effort would be required to use it extensively there. - - - -EasyBuild is implemented in [**Python**](https://www.python.org), and is compatible with Python 2.7 and 3.5+. -Since [Python 2 is no longer maintained](https://blog.python.org/2020/04/python-2718-last-release-of-python-2.html), -it is recommended to use a recent version of Python 3 for running EasyBuild. - -An [**environment modules**](https://en.wikipedia.org/wiki/Environment_Modules_(software)) tool is required for running EasyBuild. -The modern Lua-based implementation **[Lmod](https://lmod.readthedocs.io)** is strongly recommended, -but other (Tcl-based) implementations are also supported. -Both Tcl and Lua are supported as syntax for environment module files. -On LUMI Lmod is used as the module tool. - ---- - -## Development - - -EasyBuild is **actively developed** by a worldwide community under the -[**GPLv2 open source license**](https://www.gnu.org/licenses/old-licenses/gpl-2.0.html) -via the [**``easybuilders``** organisation on GitHub](https://github.com/easybuilders), in the following repositories: - -- [easybuild-framework](https://github.com/easybuilders/easybuild-framework) -- [easybuild-easyblocks](https://github.com/easybuilders/easybuild-easyblocks) -- [easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs) -- [easybuild](https://github.com/easybuilders/easybuild) (website, documentation) -- [easybuild-tutorial](https://github.com/easybuilders/easybuild-tutorial) (the generic EasyBuild tutorials) - -On LUMI the generic EasyBuild installation is used but extended and adapted using LUMI-specifics -plugins and hooks maintained in the [LUMI software stack repository](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack). - -Stable versions of EasyBuild have been released roughly every 6-8 weeks since 2012, -and are published via the *Python Package Index (PyPI)*, -see [https://pypi.org/project/easybuild](https://pypi.org/project/easybuild/). - -**Comprehensive testing** practices are applied throughout the -development cycle, with extensive suites of unit and integration tests being run in a CI environment (GitHub Actions), -consistent testing of incoming contributions (pull requests), and thorough regression testing before every release. - ---- - -## A worldwide community - -EasyBuild has grown into a **community project** since it was first released publicy in 2012, -and is used and developed by dozens (hundreds?) of HPC centres and consortia worldwide. - -Since 2016 the EasyBuild community has gathered for a yearly [**EasyBuild User Meeting**](https://easybuild.io/eum), -which is an open and highly interactive event that provides a great opportunity to meet fellow EasyBuild users, -developers, and maintainers, and to discuss related topics, learn about new aspects of the tool, and how other HPC -sites leverage it, and integrate it with other tools like [Jenkins](https://www.youtube.com/watch?v=bzW45cfRUMU), -[Ansible](https://www.youtube.com/watch?v=Qr2udsAjkG0), [CernVM-FS](https://www.youtube.com/watch?v=_0j5Shuf2uE), -[ReFrame](https://www.youtube.com/watch?v=KN9QwQIrso4) etc. - -HPC sites, consortia, and companies that use EasyBuild include (but are not limited to): - -VSC logo -JSC logo -HPCNow! logo -Fred Hutch logo -Vienna BioCenter logo -CÉCI logo -Compute Canada logo -UiO logo -BEAR logo -SNIC logo -National University of Singapore logo -IT4Innovations logo -Texas A&M logo -TU Dresden logo -University of Melbourne logo -University of Luxembourg logo -EPCC logo -Cyprus Institute logo -University of Basel logo -Microsoft logo - ---- - -## Getting help - -### *Documentation* - -If you have any questions about EasyBuild, make sure to check the EasyBuild documentation first, -which is available at [**https://docs.easybuild.io**](https://docs.easybuild.io). - -It is not unlikely that an answer to your question is available there already. -The [search box](https://docs.easybuild.io/en/latest/search.html) on the top left may be helpful. - -### GitHub - - - -You can open an issue in one of the EasyBuild repositories in [the `easybuilders` GitHub organisation](https://github.com/easybuilders/) to -ask questions, report problems or share suggestions and ideas for additional -improvements. - -Please try to post the issue in the most suitable repository. If you are -not sure which repository is most suited just go ahead and open the issue anyway, -the EasyBuild maintainers can move your issue if deemed appropriate. - -Also here you can use the search box on the top left, to see if somebody else -has already asked a similar question or reported a similar problem. - -### Mailing list - -Feel free to ask any questions or provide feedback via the EasyBuild mailing list -`easybuild@lists.ugent.be`. - -Keep in mind that **you must be subscribed to the mailing list in order to -post messages there!** You can subscribe via [this link](https://lists.ugent.be/wws/subscribe/easybuild). - -In 2016-2017 over 1,000 messages were posted to the EasyBuild mailing list. -More recently this has decreased to about 700 messages per year, mostly due -to the EasyBuild Slack channel. - -### Slack - -Slack logo - -The [EasyBuild Slack](https://easybuild.slack.com/) is a quick and easy way -to get in touch with the EasyBuild community. -Over 400 people have joined it so far, and there is usually somebody awake in there -who may be able to help you out. - -**To join the EasyBuild Slack request an invitation via -[https://easybuild-slack.herokuapp.com](https://easybuild-slack.herokuapp.com/).** - -If you are not a big fan of Slack, you can also consider joining the `#easybuild` -IRC channel (available on the [Freenode IRC network](https://freenode.net/)). -A bot relays messages between this IRC channel and the `#general` channel in the EasyBuild Slack. - - -### Conference calls - -Every other week on Wednesday an EasyBuild conference call is set up to discuss recent -developments, provide an outlook to the next EasyBuild release, and -provide answers to questions that can be raised by the people attending the call. - -These conference calls are open to anyone interested in the EasyBuild project, -and have been organised since November 2013. - -More information, including the dates and times of the next calls, is available -[on the EasyBuild Wiki](https://github.com/easybuilders/easybuild/wiki/Conference-calls). - ---- - -*[[next: The Lmod module system]](1_02_Lmod.md)* diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md deleted file mode 100644 index d6d73a6c..00000000 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ /dev/null @@ -1,820 +0,0 @@ -# The Lmod module system - -*[[back: What is EasyBuild?]](1_01_what_is_easybuild.md)* - ---- - -## Modules - -*Module* is a massively overloaded term in (scientific) software and IT in general -(kernel modules, Python modules, and so on). -In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. - -[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept -on HPC systems: it is a way to specify changes that should be made to one or more -[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a -[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file -is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or -[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, -and specifies which environment variables should be updated, and how (append, -prepend, (re)define, undefine, etc.) upon loading the environment module. -Unloading the environment module will restore the shell environment to its previous state. - -Environment module files are processed via a **modules tool**, of which there -are several conceptually similar yet slightly different implementations. -The oldest module tool still in use today is Environment Modules 3.2, implemented in C and -supporting module files written in Tcl. After a gap in developement, Xavier Delaruelle of CEA -developed [Environment Modules 4 and 5](https://sourceforge.net/projects/modules/) which is -fully implemented on Tcl. An alternative module tool is [Lmod](https://lmod.readthedocs.io), -developed by Robert McLay at TACC and implemented in LUA. This tool supports natively LUA -module files but also offers a high degree of compatibility with Tcl-based module files -developed for Environment Modules fia a translation layer and some API translation. - -The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no -packages or official support for Environment Modules 4 or 5. At the user level, -Environment Modules 3.2 and Lmod have many commands in common, but with different options. -Lmod also has some powerful features that are lacking in Environment Modules 3.2. - -!!! Note "The Cray PE on LUMI" - On LUMI, Lmod was selected as the module tool. One area where there are significant - differences between Environment Modules 3.2 (and also the newer versions) and Lmod is - in the commands for discovering modules on the system. If you are not familiar with Lmod - and its commands for users, it is worthwile to read the - [LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). - Some of those commands are also discussed on this page. - ---- - -## Lmod hierarchy - -### User view - -Lmod supports a module hierarchy. In a hierarchy, there is a distinction between the *installed -modules* and the *available modules*. Available modules are those that can be loaded directly -without first loading any other module, while the installed modules is the complete set of -modules that one could load one way or another. A typical use case -is a hierarchy to deal with different compilers on a system and different MPI implementations. -After all, it is a common practice to only link libraries and application code compiled with the -same compiler to avoid compatibility problems between compilers (and to be able to use advanced -features such as link time optimization). This is even more important for MPI, as Open MPI and -MPCIH-derived MPI implementations have incompatible Application Binary Interfaces. This would lead -to a hierarchy with 3 levels: - -1. The ``Core`` level containing the modules for the compilers themselves, e.g., one or more versions - of the GNU compiler suite and one or more versions of LLVM-based compilers. - - Loading a compiler module would then make the next level available: - -2. The ``Compiler`` level, containing modules for libraries and packages that only rely on the compilers - but do not use MPI, as well as the MPI modules, e.g., a version of Open MPI and a version of MPICH. - - Loading one of the MPI modules would then make the next level available: - -3. The ``MPI`` level, containing libraries and applications that depend on the compiler used and the MPI - implementation. - -??? Example "A simple Lmod hierarchy with a single compiler" - - Here is a simple example of such a 3-level module hierarchy - (that almost could have been generated by EasyBuild): - -
- - In this example the ``Core`` level only includes a single module `GCC/9.3.0`, - while the ``Compiler`` level includes two modules: `OpenMPI/4.0.3` and `MPICH/3.3.2`. - In the ``MPI `` level, three modules are available: one for `FFTW`, one for `ScaLAPACK`, - and one for `HDF5`. - - Initially only the modules on the top level of a module hierarchy are available for loading. - If you run "`module avail`", the command that is used to view all modules that are available - for loading, with this example module hierarchy, you will only see the `GCC/9.3.0` module. - - Some modules in the top level of the hierarchy act as a "gateway" to modules in the - next level below. - To make additional modules available for loading one of these gateway modules has to be loaded. - In our example, loading the `GCC/9.3.0` module results in two additional modules coming into - view from the ``Compiler`` level, as indicated by the arrows: the modules for `OpenMPI` and `MPICH`. - These correspond to installations of `OpenMPI` - and `MPICH` that were built using `GCC/9.3.0`. - - Similarly, the `OpenMPI/4.0.3` module serves as a gateway to the three modules in the ``MPI`` level. - Only by loading the `OpenMPI` module will these additional three modules become - available for loading. They correspond to software installations built using the ``GCC/9.3.0`` - compiler with ``OpenMPI/4.0.3``. - -Now assume that we have two compilers in the hierarchy, Compiler_A and Compiler_B. Their modules would reside -at the ``Core`` level. Both compilers provide the same MPI implementation, MPI_C. So there would be two modules -for ``MPI_C`` in two different subdirectories at the ``Compiler`` level. And further assume that we have an -application, Appl_E, compiled with both Compiler_A and Compiler_B and using MPI_C. For that application there would -also be two module files at the ``MPI`` level, one in a subdirectory corresponding ao Compiler_A and MPI_C and one -in a subdirectory corresponding to Compiler_B and MPI_C. - -```mermaid -graph TD; -A[Compiler_A] --> AC[MPI_C]; -A --> AD[MPI_D] -B[Compiler_B] --> BC[MPI_C]; -AC --> ACE[Appl_E]; -AD --> ADE[Appl_E] -BC --> BCE[Appl_E]; -``` - - -To be able to load the module for Appl_E, a user should -first load Compiler_A, then load MPI_C and only then is it possible to load the module for Appl_E: - -```bash -module load Compiler_A MPI_C Appl_E -``` - -What is interesting is what happens if the user now loads Compiler_B: - -```bash -module load Compiler_B -``` - -In a properly designed and implemented hierarchy, Lmod will unload Compiler_A which will also trigger the unloading/deactivation -of MPI_C and Appl_E. It will then load the module for Compiler_B and proceed with looking if it can find another module for -MPI_C. That will then be loaded which now makes a different module for Appl_E available, which Lmod will proceed to load. If it -cannot find an exact match for the version, Lmod will even try to locate a different version. Hence the situation after loading -Compiler_B is that now modules are loaded for Compiler_B, MPI_C for Compiler_B and Appl_E for Compiler_A with MPI_C. -All this requires very little effort from the module file programmer and very little logic in the module files. E.g., rather -then implementing a single module file for Appl_E that would require logic to see which compiler and MPI implementation is loaded -and depending on those adapt the path to the binaries, several very simple modules need to be written with very little -logic, and one could add an Appl_E module for a different compiler or MPI implementation without touching any of the already -existing module files for that application. - -Similarly, if after - -```bash -module load Compiler_A MPI_C Appl_E -``` - -one does - -```bash -module load MPI_D -``` - -thent MPI_C gets unloaded, Lmod notices that it also has to unload/deactivate Appl_E, then will load MPI_D for Compiler_A and -finally will notice that there is an equivalent Appl_E module available again, and Lmod will load that one also. However, -now loading Compiler_B will cause a warning that MPI_D and Appl_E have been deactivated as there is no module name MPI_D in -any version for Compiler_B. - - -### Building blocks - -Some mechanisms in Lmod make implementing a hierarchy fairly easy (though there are a lot of hidden pitfalls) - -- The *MODULEPATH* environment variable determines which modules are available. MODULEPATH is different from any other - path-style variable in Lmod in that any change will immediately trigger a re-evaluation of which modules are available - and trigger deactivating modules that are no longer available when a directory is removed from the MODULEPATH or - looking for alternatives for deactivated modules when a directory is added to the MODULEPATH. - -- The *"one name rule"*: Lmod cannot have two modules loaded with the same name (but a different version). By default, when loading - a module with the name of an already loaded module, Lmod will automatically swap the old one with the new one, i.e., unload the - already loaded module and load the new one. - -- The *family* concept: It is possible to declare a module to be part of a family using a command in the module file. No two modules - of the same family can be loaded at the same time, and Lmod will again by default auto-swap the already loaded one with the one - being loaded. The procedure is different though as Lmod now first has to read the new module file to discover the family, and this - may lead to more side effects. But that discussion is outside the scope of this tutorial. - - The family concept was for a long time a unique feature of Lmod, but it has been added now also to Environment Modules version 5.1. - - -### Implementation details - -The above example could be implemented using 8 module files: One for each compiler, three for the MPI modules -(two for MPI_C and one for MPI_D) and -three for the application modules. - -``` -moduleroot -├── Core -│ ├── Compiler_A -│ │ └── version_A.lua -│ └── Compiler_B -│ └── version_B.lua -├── Compiler -│ ├── Compiler_A -│ │ └── version_A -│ │ ├── MPI_C -│ │ │ └── version_C.lua -│ │ └── MPI_D -│ │ └── version_D.lua -│ └── Compiler_B -│ └── version_B -│ └── MPI_C -│ └── version_C.lua -└── MPI - ├── Compiler_A - │ └── version_A - │ ├── MPI_C - │ │ └── version_C - │ │ └── Appl_E - │ │ └── version_E.lua - │ └── MPI_D - │ └── version_D - │ └── Appl_E - │ └── version_E.lua - └── Compiler_B - └── version_B - └── MPI_C - └── version_C - └── Appl_E - └── version_E.lua -``` - -Besides the module functions needed to create the environment needed to run the compiler, the module file for -Compiler_A would need only two lines to implement the hierarchy: - -```Lua -family('Compiler') -prepend_path('MODULEPATH', 'moduleroot/Compiler/Compiler_A/version_A') -``` - -There are now two different ``version_C.lua`` files. One contains the necessary calls to module functions to -initialise the environment to use the version compiled with Compiler_A/version_A while the other contains the -necessary functions to do that for Compiler_B/version_B. Again, two more lines are needed to implement the hierarchy. -E.g., for ``moduleroot/Compiler/Compiler_A/version_A/MPI_C/version_C.lua``: - -```Lua -family('MPI') -prepend_path('MODULEPATH', 'moduleroot/MPI/Compiler_A/version_A/MPI_C/version_C') -``` - -Finally two vesions of the ``version_E.lua`` file are needed, one to prepare the environment for using the -package with Compiler_A anmd MPI_C and one for using the package with Compiler_B and MPI_C. However, these -are just regular modules and no additions are needed to work for the hierarchy. - -Both EasyBuild and Spack support Lmod hierarchies and with these tools it is also fairly automatic to create -different versions of the module files for each compiler and MPI library used to build the application. When -hand-writing modules it may be more interesting to have a generic module which would work for all those cases -and that is also possible with Lmod. Lmod does have a range of *introspection functions* that a module can use -to figure out its name, version and place in the module tree. All that would be needed is that the various -instances of the module file are at the correct location in the module tree and link to the generic file which -can be outside the module tree. In fact, this feature is used on LUMI to implement the modules that load a -particular version of the hardware for a particular section of LUMI. - ---- - -## Finding modules - -In a hierarchical setup, not all modules are available at login. This implies that a user cannot use -``module avail`` to discover which software is available on the system. To this end Lmod has powerful -search commands. It is important to understand how these commands work to ensure that the proper information -is included in the module files to improve discoverability of software. - -!!! Note "Documentation in the LUMI documentation" - Extensive information on search commands with examples of how to use them on LUMI can be found - in the [LUMI documentation]([https://)](https://docs.lumi-supercomputer.eu/), in - [the computing section, "Module environment page", "Finding modules" section](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/#finding-modules). - - -### module spider command - -The available modules at any point in time are often only a subset of all installed modules on a -system. However, Lmod provides the ``module spider`` command to search for a module with a given name -among all installed modules and to tell you how this module can be loaded (i.e., which other modules -need to be loaded to make the module available). - -The ``module spider`` command has three levels, producing different outputs: - - 1. ``module spider`` without further arguments will produce a list of all - installed software and show some basic information about those packages. - Some packages may have an ``(E)`` behind their name and will appear in blue - (in the default colour scheme) which means that they are part of a different - package. These are called *extensions* of packages or modules. - This is explained a little further in this page. - - Note that ``module spider`` will also search in packages that are hidden from - being displayed. These packages can be loaded and used. However administrators - may have decided to hide them - either because they are not useful to regular users or because they think that - they will rarely or never be directly loaded by a user and want to avoid - overloading the module display. - - 2. ``module spider `` will search for the specific package. This - can be the name of a module, but it will also search some other information - that can be included in the modules. The search is also case-insensitive. - E.g., on LUMI - ```bash - module spider GNUplot - ``` - will show something along the lines of - ``` - ------------------------------------------------------------------ - gnuplot: - ------------------------------------------------------------------ - Description: - Gnuplot is a portable command-line driven graphing utility - - Versions: - gnuplot/5.4.2-cpeCray-21.08 - gnuplot/5.4.2-cpeGNU-21.08 - ``` - so even though the capitalisation of the name was wrong, it can tell us that - there are two versions of gnuplot. The ``cpeGNU-21.08`` and ``cpeCray-21.08`` - tell that the difference is the compiler that was used to install gnuplot, - being the GNU compiler (PrgEnv-gnu) and the Cray compiler (PrgEnv-cray) - respectively. - - In some cases, if there is no ambiguity, `module spider` will actually - already produce help about the package, which is the next level. - - 3. `module spider /` will show more help information - about the package, including information on which other modules need to be - loaded to be able to load the package. E.g., - ```bash - module spider git/2.35.1 - ``` - will return something along the lines of - ``` - ------------------------------------------------------------------- - git: git/2.35.1 - ------------------------------------------------------------------- - Description: - Git is a free and open source distributed version control - system - - You will need to load all module(s) on any one of the lines below - before the "git/2.35.1" module is available to load. - - CrayEnv - LUMI/21.12 partition/C - LUMI/21.12 partition/D - LUMI/21.12 partition/G - LUMI/21.12 partition/L - - Help: - ``` - (abbreviated output). Note that it also tells you which other modules need - to be loaded. You need to choose the line which is appropriate for you and - load all modules on that line, not the whole list of in this case 9 - modules. - -!!! failure "Known issue" - The Cray PE uses Lmod in an unconventional manner with the hierarchy not - build fully in the way Lmod expects. As a consequence Lmod is not always - able to generate the correct list of modules that need to be loaded to make - a package available, and the list of ways to make a module available may - also be incomplete. - - The problem is somewhat aggrevated on LUMI because the Cray PE hierarchy sits - next to the hierarchy of the software stack as the Cray PE is installed - separately and hence cannot be integrated in the way the Lmod developer had - in mind. - - -#### Module extensions - -Certain packages, e.g., Python, Perl or R, get a lot of their functionality through -other packages that are installed together with them and extend the functionity, -e.g., NumPy and SciPy for Python. Installing all those packages as separate modules -to make it easy to see if they are installed or not on a system would lead to an -overload of modules on the system. - -Similary, admins of a software stack may chose to bundle several libraries or tools -that are often used together in a single module (and single installation directory), -e.g., to reduce module clutter but also to reduce the length of the search paths for -binaries, libraries or manual pages to speed up loading of applications. - -Lmod offers a way to make those individual packages installed in a module discoverable -by declaring them as *extensions* of the module. The ``module spider`` command will -search for those too. - - 1. ``module spider`` without further arguments: The output may contain lines similar - to - ``` - ----------------------------------------------------------------------- - The following is a list of the modules and extensions currently available: - ----------------------------------------------------------------------- - Autoconf: Autoconf/2.71 (E) - - CMake: CMake/3.21.2 (E), CMake/3.22.2 (E) - ``` - which tells that ``Autoconf`` and ``CMake`` are not available as modules themselves - but as extensions of another module, and it also tells the versions that are available, - though that list may not be complete (and is not always complete for modules either - as it is limited to one line of output). - -2. ``module spider `` will search for extensions also. E.g., - ``` - module spider CMake - ``` - on LUMI will return something along the lines of - ``` - ----------------------------------------------------------------------- - CMake: - ----------------------------------------------------------------------- - Versions: - CMake/3.21.2 (E) - CMake/3.22.2 (E) - ``` - (output abbreviated). - This tells that there is no ``CMake`` module on the system but that two versions - of ``CMake`` are provided in another module. - - 3. `module spider /` will show more information on the - extension, including which module provides the extension and which other modules - have to be loaded to make that module available. E.g., on LUMI, - ``` - module spider CMake/3.22.2 - ``` - will output something along the lines of - ``` - ----------------------------------------------------------------------- - CMake: CMake/3.22.2 (E) - ----------------------------------------------------------------------- - This extension is provided by the following modules. To access the - extension you must load one of the following modules. Note that any - module names in parentheses show the module location in the software - hierarchy. - - buildtools/21.12 (LUMI/21.12 partition/L) - buildtools/21.12 (LUMI/21.12 partition/G) - buildtools/21.12 (LUMI/21.12 partition/D) - buildtools/21.12 (LUMI/21.12 partition/C) - buildtools/21.12 (CrayEnv) - ``` - (output abbreviated and slightly reformatted for readability). This tells that - ``CMake/3.22.2`` is provided by the ``bvuildtools/21.12`` module and that there - are 5 different ways to make that package available. - -??? Bug "Restrictions with older Lmod versions" - At the time of development of this tutorial, Cray is still using the pretty old - 8.3.1 version of Lmod. Even though extensions were supported since Lmod version 8.2.5, - Lmod 8.3.1 has several problems: - - - It is not possible to hide extensions in the output of ``module avail``, a feature - that only became available in version 8.5. This may be annoying to many users as - the extension list of packages such as Python, R and Perl can be very long (the - default EasyBuild installation of R contains on the order of 600 packages). - - For that reason on LUMI extensions are only used for some modules. - - - ``module avail`` also shows extensions for modules that are not available which - makes no sense. This bug was only corrected in Lmod 8.6.13 and 8.6.14. - - -### module keyword - -Another search command that is sometimes useful is `module keyword`. It really -just searches for the given word in the short descriptions that are included in -most module files and in the name of the module. The output is not always -complete since not all modules may have a complete enough short description. - -Consider we are looking for a library or package that supports MP3 audio -encoding. -```bash -module keyword mp3 -``` -will return something along the lines of -``` ----------------------------------------------------------------- - -The following modules match your search criteria: "mp3" ----------------------------------------------------------------- - - LAME: LAME/3.100-cpeCray-21.08, LAME/3.100-cpeGNU-21.08 - LAME is a high quality MPEG Audio Layer III (mp3) encoder -``` -though the output will depend on the version of Lmod. This may not be the most -useful example on a supercomputer, but the library is in fact needed to be able -to install some other packages even though the sound function is not immediately -useful. - -??? bug "Know issue: Irrelevant output" - At the moment of the development of this tutorial, this command actually - returns a lot more output, referring to completely irrelevant extensions. - This is a bug in the HPE-Cray-provided version of Lmod (8.3.1 at the time - of development of this tutorial) that was only solved in more recent versions. - - -### module avail - -The `module avail` command is used to show only available modules, i.e., modules -that can be loaded directly without first loading other modules. It can be used -in two ways: - - 1. Without a further argument it will show an often lengthy list of all - available modules. Some modules will be marked with `(D)` which means that - they are the default module that would be loaded should you load the module - using only its name. - - 2. With the name of a module (or a part of the name) it will show all modules - that match that (part of) a name. E.g., - ```bash - module avail gnuplot - ``` - will show something along the lines of - ``` - ------ EasyBuild managed software for software stack LUMI/21.08 on LUMI-L ------ - gnuplot/5.4.2-cpeCray-21.08 gnuplot/5.4.2-cpeGNU-21.08 (D) - - Where: - D: Default Module - (output abbreviated). - ``` - but - ```bash - module avail gnu - ``` - will show you an often lengthy list that contains all packages with gnu - (case insensitive) in their name or version. - - -### Getting help - -One way to get help on a particular module has already been discussed on this -page: ``module spider /`` will produce help about the package as -soon as it can unambiguously determine the package. It is the only command that -can produce help for all installed packages. The next two commands can only -produce help about available packages. - -A second command is ``module whatis`` with the name or name and version of a -module. It will show the brief description of the module that is included in -most modules on the system. If the full version of the module is not given, it -will display the information for the default version of that module. - -The third command is ``module help``. Without any further argument it will display -some brief help about the module command. However, when used as -``module help `` or ``module help /`` it will produce help for either the -default version of the package (if the version is not specified) or the -indicated version. - - -### Implementation details - -Lmod works by executing the module file. However, the actions of all Lmod-defined -functions will depend upon the mode in which Lmod is executing the module function, -and the module file can also detect in which mode it is executing. -Modes include "load", "unload"but also "spider". E.g., when te mode is "load", the -``setenv`` function will set an environment variable to the indicated value while in -"unload" mode that environment variable will be unset, and in "spider" mode the -environment variable is left untouched. The working of ``prepend_path``, a function -that modifies PATH-style variables, depends a bit on how Lmod is configured (as it is -possible to work with reference counts), but in its most basic mode, ``prepend_path`` -will add a given directory to a given PATH-style environment variable (or move it to -the front of the PATH-style variable if the directory is already in there), while in -"unload" mode that specific directory will be removed from the PATH (but no error will -be generated should the directory that is used as the argument not be part of the path -in that PATH-style variable). When the mode is "spider", the function has special behaviour -if it is used to change the ``MODULEPATH``. It will then note the change and add that -directory to the list of directories that has to be searched for module files. -This makes ``module spider`` a very expensive command as it may have to traverse a lot -of directories and has to execute all module files in there. Therefor Lmod will build -a so-called spider cache which can be pre-built in the system for certain directories -and otherwise will be build in the user's home directory (in the ``.lmod.d/.cache`` -subdirectory). Our experience is that this cache tends to be rather fragile, -in particular on Cray systems (and that has been confirmed in discussions with -people with access to some other Cray systems) so from time to time Lmod fails to -note changes to the modules, at least when using commands such as ``module spider``. -The actual loading and unloading of the module is not based on cached information. - -Lmod has several functions that can be used in module files to provide the information -that Lmod needs for the search-related and help commands. - -The ``help`` function defines the long help text used by ``module help`` and by -``module spider`` as soon as there is no ambiguity anymore about which module is -being searched for. - -The ``whatis`` function is used to provide short information about a module. That -information is then used by ``module whatis`` and ``module keyword`` , but also -for brief information shown by ``module spider`` when multiple modules or versions -of modules are found by the command. A module file can contain multiple ``whatis`` -commands and the Lmod manuel suggests to use those lines as a kind of database -record. See, e.g., -[the Lmod manual page with module file examples](https://lmod.readthedocs.io/en/latest/100_modulefile_examples.html?highlight=whatis). -One such example is -```Lua -whatis("Name: valgrind") -whatis("Version: 3.7.0") -whatis("Category: tools") -whatis("URL: http://www.valgrind.org") -whatis("Description: memory usage tester") -``` -It is not all that important to include all those lines in a module file, but some of -those lines get a special treatment from Lmod. The line starting with ``Description`` -is used by ``module spider`` to provide some brief information about the module if it -is not totally resolved. This comes with a limitation though: It is not show for each -version of the module, so ideally all "GROMACS" modules should contain the same -description line and use other lines to provide further information about what -distinguished a particular version. -Likewise the ``Category:`` line is used by the ``spider_decoration`` hook that can be -used to add decoration to the spider level 1 output. -All in all the ``whatis`` function if often overlooked in Lmod-based module functionx -but it is a very useful function to include in the proper way in module files. - -A third function that provides information to the search commands is ``extensions``. -It can be used to list up the extensions supported by the module. The argument list -may seem strange as it takes only a single argument, a string of comma-separated ``extension/version`` -elements, but that is because the number of arguments to a function is limited in -Lua and that limit can actually be met easily by modules for Python, Perl or R packages. - - ---- - -## Some warnings about writing modulefiles - -***This section is very technical and only useful if you want to manually implement -modules that depend on each other one way or another.*** - -Lmod cannot guarantee that the order of unloading the modules will be the inverse of -the order in which they were loaded. Moreover, unloading a module is not done by reverting -stored actions done when loading the module, but by executing the modulefile again -in a mode that reverts certain actions. This can lead to subtle problems when modulefiles -communicate with each other through environment variables or by detecting which other -modules are loaded. These problems are usually solved by using a proper hierarchy -and basing actions of modulefiles on their position in the hierarchy. - -One case where passing information between modules through environment variables will -go wrong is when that environment variable is subsequently used to compute a directory -name that should be added to a PATH-like variable. Assume we have two versions of -a ``MyPython`` module, e.g., ``MyPython/2.7.18`` and ``MyPython/3.6.10``. That module then -sets an environment variable ``PYTHON_API_VERSION`` to either ``2.7`` or ``3.6``. -Next we have a module ``MyPythonPackage`` that makes a number of Python packages available -for both Python modules. However, as some Python packages have to be installed separately -for each Python version, it does so by adding a directory to the environment variable -``PYTHON_PATH`` that contains the version which it gets by using the Lua function -``os.getenv`` to request the value of ``PYTHON_API_VERSION``. - -One problem becomes clear in the following scenario: -``` bash -module load MyPython/2.7.18 -module load MyPythonPackage/1.0 -module load MyPython/3.6.10 -``` -The ``module load MyPythonPackage`` will find the environment variable ``PYTHON_PACKAGE_API`` -with the value ``2.7`` as set by ``module load MyPython/2.7.18`` and hence add the directory -for the packages for version 2.7 to ``PYTHON_PATH``. The ``module load MyPython/3.6.10`` -command will trigger two operations because of the *"one name rule"*: First it will -automatically unload ``MyPython/2.7.18`` (which will unset ``PYTHON_API_VERSIUON``) and -next it will load ``MyPython/3.6.10`` which will set ``PYTHON_API_VERSION`` to ``3.6``. -However, ``MyPythonPackage`` is not reloaded so the ``PYTHON_PATH`` variable will now point -to the wrong directory. One would be tempted to think that the easy fix for the user would -be to reload ``MyPythonPackage/1.0``: -``` bash -module load MyPythonPackage/1.0 -``` -Because of the *"one name rule"* this will again trigger an unload followed by a load -of the module. The problem is in the unload. One would expect that first unloading -``MyPythonPackage`` would remove the 2.7 directory from the ``PYTHON_PATH`` but it -will not. Lmod does not remeber that last time it loaded ``MyPythonPackage`` it added -the 2.7 directory to ``PythonPath``. Instead it will execute the commands in the -modulefile and reverse certain commands. Since ``PYTHON_API_VERSION`` has now the value -``3.6``, it will try to remove the directory for version ``3.6`` which is not in the -``PYTHON_PATH``. The subsequent load will then add the 3.6 directory to ``PYTHON_PATH`` -so the environment variable now contains both directories. - -In this simple case, a ``module purge`` after the first two ``module load`` commands would -still work as Lmod is able to figure out the right order to unload modules, but in more -complicated examples this may also go wrong. However, a ``module purge`` command after -the load of ``MyPython/3.6.10`` would also fail to clean up the environment as it would -still fail to remove the 2.7 directory from ``PYTHONPATH``. - -??? Note "Running the example" - To test this example for yourself, create a directory and add that directory to - the ``MODULEPATH`` using ``module use``. In that directory, create the following - subdirectories and files: - 1. ``MyPython/2.7.18.lua`` with content: - ``` lua - LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) - setenv( 'PYTHON_API_VERSION', '2.7' ) - ``` - 2. ``MyPython/3.6.10.lua`` with content: - ``` lua - LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) - setenv( 'PYTHON_API_VERSION', '3.6' ) - ``` - 3. ``MyPythonPackage/1.0.lua`` with content: - ``` lua - LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) - LmodMessage( 'PYTHON_API_VERSION = ' .. ( os.getenv( 'PYTHON_API_VERSION' ) or '') ) - prepend_path( 'PYTHON_PATH', 'someroot/python' .. - ( os.getenv( 'PYTHON_API_VERSION' ) or 'TTT' ) .. '/packages' ) - LmodMessage( 'PYTHON_PATH = ' .. ( os.getenv( 'PYTHON_PATH' ) or '') ) - ``` -??? Note "Solution with a hierarchy" - The better way in Lmod to implement the above scenario would be in a module hierarchy. - - Just to show the power of Lmod introspection functions combined with a proper hierarchy - we present a solution using only one version of the code for ``MyPython`` and one version - of the code for ``MyPythonPackages``. - - It is best to start from a clean directory. In that directory, create: - - 1. The files ``level1/MyPython/2.7.18.lua`` and ``level1/MyPython/3.6.10.lua``, - both with the same contents: - ``` lua - LmodMessage( '\nIn ' .. myModuleFullName() .. ' in mode ' .. mode() ) - - local api_version = myModuleVersion():match( '(%d+%.%d+)%..*' ) - - -- Set the variable PYTHON_API_VERSION but not for internal use in the modules. - setenv( 'PYTHON_API_VERSION', api_version ) - - local module_root = myFileName():match( '(.*)/level1/' .. myModuleFullName() ) - prepend_path( 'MODULEPATH', pathJoin( module_root, 'level2/PythonAPI', api_version ) ) - LmodMessage( 'MODULEPATH is now\n ' .. - os.getenv( 'MODULEPATH' ):gsub( ':', '\n ' ) ) - ``` - - 2. The files ``level2/PythonAPI/2.7/MyPythonPackage/1.0.lua`` and - ``level2/PythonAPI/3.6/MyPythonPackage/1.0.lua``, both with the contents: - ``` lua - LmodMessage( '\nIn ' .. myFileName() .. ' in mode ' .. mode() ) - - local python_api_version = myFileName():match( '.*/level2/PythonAPI/([^/]+)/.*' ) - LmodMessage( 'Detected Python API version from hierarchy: ' .. python_api_version ) - LmodMessage( 'Detected Python API version from environment: ' .. - ( os.getenv( 'PYTHON_API_VERSION' ) or '' ) ) - - prepend_path( 'PYTHON_PATH', 'someroot/python' .. python_api_version .. '/packages' ) - - LmodMessage( 'PYTHON_PATH = ' .. (os.getenv( 'PYTHON_PATH' ) or '') ) - ``` - - Now add the ``level1`` subdirectory to ``MODULEPATH``, e.g., if you're in the directory - containing the ``level1`` and ``level2`` subdirectories: - ``` bash - module use $PWD/level1 - ``` - and then try the following commands: - ``` bash - module avail - module load MyPython/2.7.18 - module avail - module load MyPythonPackage/1.0 - module load MyPython/3.6.10 - ``` - and pay attention to the output. - - Initially ``module avail`` will show none of the ``MyPythonPackage`` modules. These are - installed modules but not available modules. ``module load MyPython/2.7.18`` will set the - environment variable ``PYTHON_API_VERSION`` to ``2.7`` and also add a directory to the front - of the ``MODULEPATH`` with the directory name ending on ``level2/PythonAPI/2.7``. Now - ``module avail`` will show the ``MyPythonPackage/1.0`` module. - - The ``MyPythonPackage`` shows two ways to get the version of the Python API to use for - determining the right directory to add to ``PYTHON_PATH``. The fragile way is to enquire - the value of the environment variable ``PYTHON_API_VERSION`` set by loading ``MyPython/2.7.18``. - The more robust way is to use the Lmod introspection function ``myFileName()`` which returns - the full path and file name of the module file that is executing, and extracting the version - from the path with a pattern matching function. In this particular situation both computed - values are the same so both would have worked to correctly add - ``somedir/python2.7/packages`` to the front of ``PYTHON_PATH``. - - The next command, ``module load MyPython/3.6.10`` triggers a chain of events. - - First, Lmod notices that there is already a module loaded with the same name, so it will - unload ``MyPython/2.7.18``. This will unset the environment variable ``PYTHON_API_VERSION`` - (the inverse operation of ``setenv``) and will remove the ``.../level2/PythonAPI/2.7`` - subdirectory from the ``MODULEPATH`` (the inverse action of ``prepend_path``). - - Now due to - the change of the ``MODULEPATH`` the ``MyPythonPackage/1.0`` module which was loaded from - ``.../level2/PythonAPI/2.7`` is no longer available so Lmod will continue with unloading - that module. The interesting bit now is that ``PYTHON_API_VERSION`` is unset. So had we - computed the name of the directory to add to ``PYTHON_PATH`` using the value of that - environment variable, the module would have failed to compute the correct directory name - to remove so ``prepend_path`` would have left the ``PYTHON_PATH`` environment variable - untouched. However, by computing that value from the directory of the modulefile, we get - the right value and can correctly remove ``somedir/python2.7/packages`` from ``PYTHON_PATH``. - Lmod will also remember that the module was only unloaded due to a change in the - ``MODULEPATH`` and not because a user explicitly unloaded the module. I.e., it considers - the module as deactivated but not as unloaded. - - Lmod proceeds with loading the ``MyPython/3.6.10`` module. This will now set - ``PTHON_API_VERSION`` to ``3.6`` and add a directory with name ending on - ``level2/PythonAPI/3.6`` to ``MODULEPATH``. - - Things are not done yet though. As the ``MODULEPATH`` has changed, Lmod looks at its list - of deactivated modules and notices that a different version of ``MyPythonPackage/1.0`` is - now available. Hence it will now automatically load that module from the - ``.../level2/PythonAPI/3.6`` subdirectory so that that module now correctly detects - that ``somedir/python3.6/package`` should be added to ``PYTHON_PATH``. - - Hence at the end of the cycle we have again a correctly configured environment with no - trace of the ``2.7`` version that was loaded initially and with no action required from - the user to ensure that ``MyPythonPackage`` is unloaded and reloaded to ensure the - correct configuration. - - This idea is used on LUMI to implement the various versions of the software stack with - for each software stack also optimised binaries for each of the node types. - - ---- - -## Further reading - -- [Lmod documentation](https://lmod.readthedocs.io/en/latest/index.html) -- [Lmod on LUMI in the LUMI documentation](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/) -- [Documentation of Environment Modules 5](https://modules.readthedocs.io/en/latest/), - an alternative to Lmod (though not currently supported by HPE Cray) - - ---- - -*[[next: The Cray Programming Environment]](1_03_CPE.md)* diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md deleted file mode 100644 index d3a2ff84..00000000 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ /dev/null @@ -1,418 +0,0 @@ -# The HPE Cray Programming Environment - -*[[back: The Lmod module system]](1_02_Lmod.md)* - ---- - -On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated -as Cray PE). The environment provides several tools, including compilers, communication libraries, -optmized math libraries and various other libraries, analyzers and debuggers. - -The Cray PE is made available through *environment modules* tha allow to select particular versions of -tools and to configure the environment in a flexible way. - ---- - -## Cray PE components - -### Cray compiler environments - -Cray PE supports multiple compilers, Cray and third party compilers as well: AOCC, Cray, Intel, GNU. - -Users can access the compilers loading a programming environment module (see below). - -Compilers in the Cray Programming Environment are almost always used through the Cray compiler wrappers. In fact, -until recently, some compilers could not be used without the wrappers as they failed to even detect their own -include files and some components. The behaviour of the wrapper will depend on the other modules that are -loaded in the environment. E.g., there is no separate wrapper for MPI as the MPI header files and libraries are -automatically activated through the regular wrappers as soon as a Cray MPI module is loaded. - -The commands to invoke compiler wrappers are `ftn` (Fortran), `cc` (C), `CC` (C++). They wrap automatically -to the right compilers based on the programming environment module that is loaded and the compiler module. - -The online help can be accessed with the `-help` option. E.g.: `cc -help`, `CC -help`. -One of the most frequently used options of the compiler wrappers is `-craype-verbose`: -``` - ftn -help | grep verbose -A 1 - -craype-verbose Print the command which is forwarded - to compiler invocation -``` -More information is available with the `info` or `man` commands. -E.g.: both `info cc` or `man cc` will display the man page of the C compiler wrapper. - -The compiler wrappers call the correct compiler in the currently loaded programming environment, -with appropriate options to build and link applications with relevant libraries, as required by the modules loaded. -Besides a number of generic options, they also will pass all other options to the underlying compile, -so you can still use the regular compiler options also. - -The compiler wrappers should replace direct calls to compiler drivers in Makefiles and build scripts -to ensure that the proper libraries are used. - -!!! Note - For system libraries, only dynamic linking is supported by compiler wrappers on the Cray EX system - ---- - -#### Cray Compiling Environment (CCE) - -The Cray Compiling Environment is set by the module `PrgEnv-cray`, which is loaded by default at login. -The compiler itself is contained in the `cce` module. - -CCE consists of Cray compilers performing code analysis during compilation to generate highly optimized code. -Supported languages include Fortran, C and C++, and UPC (Unified Parallel C). -The Cray C/C++ compiler is basically the Clang/LLVM compiler with a back-end configured by HPE Cray that also -adds some Cray-specific options while the Cray Fortran compiler uses an HPE-Cray specific front-end -with a back-end based on LLVM. The Fortran compiler supports most of Fortran 2018 and tends to be -considerably stricter than the GNU or Intel Fortran compilers. - -The CCE compilers will also support OpenMP offload to NVIDIA and AMD GPUs but that is still very immature and -very much work in progress at the time of the development of this tutorial. -The Fortran compiler (but not the C/C++ compiler) also supports -OpenACC for offloading to GPU and is being updated to the newest versions of this standard. - -The classic pre-LLVM Cray C/C++ compiler is not available on Cray EX systems. - -Compiler-specific manpages can be accessed on the system with `man crayftn`, `man craycc` or `man crayCC`. - -More details are given in the -[Cray Fortran Reference Manual](https://support.hpe.com/connect/s/search?language=en_US#q=Cray%20Fortran%20Reference%20manual&t=All&sort=relevancy&numberOfResults=25) - and the - [Cray Compiling Environment Release](https://support.hpe.com/connect/s/search?language=en_US#q=Cray%20Compiling%20Environment%20Release&t=All&sort=relevancy&numberOfResults=25) -that unfortunately are hidden deep in the new [support pages of HPE](https://support.hpe.com/). -The [Clang Compiler User’s Manual](https://clang.llvm.org/docs/UsersManual.html) -is another source of information for the Cray C and C++ Clang compilers. - -For more information about compiler pragmas and directives, see `man intro_directives` on the system. - ---- - -### Third-Party Compilers - -#### GNU - -The GNU C/C++ and Fortran compilers are probably the best supported third party -compiler in the Cray PE. - -Compiler-specific manpages can be accessed on the system with `man gfortran`, `man gcc` or `man g++`. - -More details are provided by the [GCC online documentation](https://gcc.gnu.org/onlinedocs). - - -#### AOCC - -There AOCC compilers, the AMD Optimizing C/C++ Compiler and matching fortran compilers, -AMD's compiler offering for CPU-only systems, have a matching programming environment module -and are a full citizen of the Cray PE. - -Compiler-specific documentation is available in the [AOCC User Guide](https://developer.amd.com/amd-aocc/#userguide). - -Cray provides a bundled package of support libraries to install into the PE environment to enable AOCC, -and Cray PE utilities such as debuggers and performance tools work with AOCC. - - -#### AMD ROCm compilers - -The AMD ROCm compilers (in the AMD world sometimes known as AOMP) are supported on systems -with AMD CPUs. However, at the time of writing of this tutorial they are not yet available in the -LUMI environment and the integration with the Cray environment still seems to be work in progress. - - -#### Intel (not on LUMI) - -The Cray PE also provides a programming environment module to enable -the Intel® oneAPI compiler and tools. - -The documentation is available in the [Intel® oneAPI Programming Guide](https://software.intel.com/content/www/us/en/develop/documentation/oneapi-programming-guide/top/oneapi-development-environment-setup.html) - -Cray provides a bundled package of support libraries to install into the Cray PE to enable the Intel compiler, allowing -utilities such as debuggers and performance tools to work with it. - - -#### NVIDIA HPC toolkit (not on LUMI) - -The NVIDIA HPC Toolkit compilers (formerly PGI) are supported on systems with NVIDIA GPUs. - - -### Cray Scientific and Math Library - -* Manpages: `intro_libsci`, `intro_fftw3` - -The Cray Scientific and Math Libraries (CSML, also known as LibSci) are a collection of numerical routines -optimized for best performance on Cray systems. - -These libraries satisfy dependencies for many commonly used applications on Cray systems for a wide variety of domains. - -When the module for a CSML package (such as `cray-libsci` or `cray-fftw`) is loaded, -all relevant headers and libraries for these packages are added to the compile -and link lines of the `cc`, `ftn`, and `CC` compiler wrappers, so linking with them is -completely transparant (to the extent that users wonder where the libraries are). - -The CSML collection contains the following Scientific Libraries: - -* BLAS (Basic Linear Algebra Subroutines) -* BLACS (Basic Linear Algebra Communication Subprograms) -* CBLAS (Collection of wrappers providing a C interface to the Fortran BLAS library) -* IRT (Iterative Refinement Toolkit) -* LAPACK (Linear Algebra Routines) -* LAPACKE (C interfaces to LAPACK Routines) -* ScaLAPACK (Scalable LAPACK) -* `libsci_acc` (library of Cray-optimized BLAS, LAPACK, and ScaLAPACK routines) -* HDF5 -* NetCDF (Network Common Data Format) -* FFTW3 (the Fastest Fourier Transforms in the West, release 3) - - -### Cray Message Passing Toolkit - -* Manpages: `intro_mpi` - -* Website: [http://www.mpi-forum.org](http://www.mpi-forum.org) - -MPI is a widely used parallel programming model that establishes a practical, portable, efficient, -and flexible standard for passing messages between ranks in parallel processes. - -Cray MPI is derived from Argonne National Laboratory MPICH and implements the MPI-3.1 standard -as documented by the MPI Forum in MPI: A Message Passing Interface Standard, Version 3.1. - -Support for MPI varies depending on system hardware. To see which functions and environment variables the -system supports, please have a look at the corresponding man pages with `man intro_mpi` on the system. - -Note that though on LUMI at the time of the tutorial both the OpenFabric Interface (OFI) based and UCX-based -versions of the library are supported, the finalised LUMI system will only support OFI. - - -### DSMML - -* Modules: `cray-dsmml` -* -* Manpages: `intro_dsmml` -* -* Website: [https://pe-cray.github.io/cray-dsmml](https://pe-cray.github.io/cray-dsmml) - -Distributed Symmetric Memory Management Library (DSMML) is a HPE Cray proprietary memory management library. - -DSMML is a standalone memory management library for maintaining distributed shared symmetric memory -heaps for top-level PGAS languages and libraries like Coarray Fortran, UPC, and OpenSHMEM. - -DSMML allows user libraries to create multiple symmetric heaps and share information with other libraries. - -Through DSMML, interoperability can be extracted between PGAS programming models. - -Further details are available in the man page on the system with `man intro_dsmml`. - - -### Other components - -- **Cray Performance and Analysis Tools**: - - Tools to analyze the performance and behavior of programs that are run on Cray systems, and a Performance API (PAPI). - -- **Cray Debugging Support Tools**: - - Debugging tools, including `gdb4hpc` and `Valgrind4hpc`. - - ---- - -## Configuring the Cray PE through modules - -Multiple releases of the HPE Cray PE can be installed simultaneously on the system and users can mix-and-match -components. However, it is also possible to load only components for a specific release of the PE. -Cray PE versions have version numbers of the form ``yy.mm``, e.g., ``22.02`` for the version released in -February of 2022. However, each of the components have their own version number and it is not easy to see -which version of a component came with which version(s) of the Cray PE. - -Below we only discuss those modules that are important when building software with EasyBuild. Debuggers, -profilers, etc., are not included in the list. - - -### The programming environment modules - -The usual way to initialise the Cray PE is by loading one of the ``PrgEnv-*`` modules. This module will then -load the appropriate compiler, compiler wrappers and other libraries. Some of the components that will be -loaded are configured through the ``/etc/cray-pe.d/cray-pe-configuration.sh`` file, so the list of modules -may depend on the actual system that you are using. - -The Cray PE supports the following ``PrgEnv-*`` modules. On LUMI, only the first three are currently available: - -| Module | Compiler module | What? | -|:---------------|-----------------|:--------------------------------------------| -| PrgEnv-cray | cce | The Cray Compiling Environment compilers | -| PrgEnv-gnu | gcc | The GNU compilers | -| PrgEnv-aocc | aocc | AMD compilers for CPU-only systems | -| PrgEnv-amd | rocm | AMD ROCm compilers for GPU systems | -| PrgEnv-intel | intel | The Intel compilers | -| PrgEnv-nvidia | nvidia | NVIDIA HPC toolkit compilers (formerly PGI) | - -All ``PrgEnv-*`` modules belong to the same module family. Hence Lmod will automatically unload any -already loaded ``PrgEnv-*`` module when you load a different one. - - -### Selecting the version through the cpe meta-module - -The Cray PE on the EX system provides the meta-module `cpe`: the purpose of the meta-module is -similar to the scope of the `cdt` and `cdt-cuda` meta-modules available on the XC systems. - -Loading one of the ``cpe/yy.mm`` modules (e.g., ``cpe/22.02``) has the following effects: - -- It sets the default versions of each of the Cray PE modules to the version that comes with that - particular release of the Cray PE. E.g., - - ``` bash - module load cpe/22.02 - module load cce - ``` - - would load that version of the ``cce`` compiler that comes with the 22.02 release of the Cray PE. - -- It will reload all already loaded Cray PE modules and switch them over to the version corresponding - to that particular release of the Cray PE. - -???+ Bug "Limitations and bugs" - Due to the way Lmod works and implementation bugs in the ``cpe`` modules, loading the ``cpe`` module - does not always have the desired effect. - - - The Cray PE sets the default version of each module by adding a file to the list of files in the - ``LMOD_MODULERCFILE`` environment variable. This is because Lmod does not re-evaluate the visibility - of modules and the internal list of default version immediately when the value of ``LMOD_MODULERCFILE`` - is changed but only the next time the ``module`` command is executed. Hence - ``` bash - module load cpe/22.02 ; module load cce - ``` - and - ``` - module load cpe/22.02 cce - ``` - do not have the same effect. In the first version, the version of ``cce`` loaded is the version that - corresponds to the 22.02 release of the Cray PE. In the second case however the default version of the - ``cce`` module is determined by whatever list of default modules was used when calling the ``module`` - command so may or may not be the one of the 22.02 release. - - - Loading the ``cpe`` module after loading the other Cray PE modules also does not always have the - desired effect in many versions of the Cray PE. This is because of a bug in the ``cpe `` module that - reloads the modules in the wrong order which may trigger the reload of a module with whatever version - was the default when the ``module`` command was called rather than the version the the ``cpe`` module - intends to (re-)load. - - -### The compiler wrapper module craype - -The ``craype`` module loads the compiler wrappers. There is only one set of compiler wrappers for all compilers. -Which compiler will be called, which libraries will be included, but also processor and GPU target options will -be used, is all determined by other modules. Hence it is in principle possible to use a single Makefile for -a project and still reconfigure the build by loading certain modules. - - -### Target modules - -The targets for CPU and GPU optimization, the network library for MPI bt also some other compiler options, -can be set through target modules: - -- ``craype-x86-*`` (and similar options can be expected on ARM-based systems) set the target for CPU - optimisations. For LUMI, the ``crape-x86-rome``, ``craype-x86-milan`` and ``craype-x86-trento`` modules - are relevant. - - This can also be used to cross-compile to a different target architecture unless the compiler target - gets overwritten by a compiler flag added to the command line through the Makefile or other means, - something that unfortunately happens more and more often in faulty software installation procedures. - -- ``creype-accel-*`` sets the target for OpenMP offload (and likely other technologies in the future). E.g., - loading ``craype-accel-amd-gfx90a`` tells the compilers to target AMD MI200 family GPUs, while loading - ``craype-accel-host`` tells the compiler to use the CPU instead (according to the documentation, the latter - is for ``PrgEnv-cray`` only). - -- ``craype-network-*`` selects the communication library to be used by Cray MPICH. On Slingshot 11 EX systems, - only ``craype-network-ofi`` is supported, but Slingshot 10 EX systems also offer support for UCX through the - ``craype-network-ucx`` module. - -- The ``craype-hugepages*`` modules enable Cray Huge Pages support. To fully enable this support they have to - be used at link-time and at run-time. At link time, support is compiled into the binary wile at run-time they - are used to set the actual size of the huge pages. - - The ``craype-hugepages*`` modules are not supported by all compilers. E.g., the AOCC compiler does not support - huge pages at the moment, and loading the module at link time will cause an error message from the linker. - - -### The compiler modules - -The compiler modules have already been discussed with the ``PrgEnv-*`` modules above. The different regular -compiler modules also all belong to the same family so no two different compilers can be loaded simultaneously -and Lmod will automatically unload the other compiler when a new one is loaded. - - -### The MPI modules - -To load the Cray MPI libraries, both one of the ``craype-network-*`` modules and a compiler module has to be -loaded as the MPI libraries are both network- and compiler specific. - -For some unknown reason, the MPI module for the libfabric (``craype-network-ofi``) transport is called ``cray-mpich`` -while the library for the UCX transport (``craype-network-ucx``) is called ``craype-network-ucx``. As a result, -the MPI module fails to reload automatically when switching between both transports, but it does reload automatically -when switching compilers. - -Loading an MPI module will also automatically configure the regular compiler wrappers to compile with support for -that MPI module. However, the libfabric and UCX versions of the MPI library have compatible interfaces, so it is -always possible to swap between those versions at runtime. - - -### The Cray Scientific libraries - -The Cray Scientific Libraries are loaded through the ``cray-libsci`` module (or ``cray-libsci_acc`` for the GPU -versions). Loading this module makes the BLAS, LAPACK, and ScaLAPACK libraries available, and also the Cray IRT -(Iterative Refinement Toolkit), a Cray-specific library. It will also configure the compiler wrappers to link -with these libraries, so no additional include or link options are needed. - - -### The ``fftw`` module - -As FFTW is a third party library, it gets its own module (``cray-fftw``) and is not included in the ``cray-libsci`` module. - -The ``cray-fftw`` module can only be loaded if one of the processor target modules (the ``craype-x86-*`` modules) -is loaded first. - - ---- - -## Some unexpected behaviour of the Cray PE - -On Cray EX systems, dynamic linking is the preferred way of linking applications. Whith that comes some -unexpected behaviour of the Cray modules. EasyBuild users expect that at run time the versions of the -libraries that are used, are the ones from the modules that are loaded. This is not always the case -for the runtime libraries of the Cray PE. By default the Cray PE will use a default set of libraries -that is determined by the system default version of the Cray PE (which is set by the sysadmins, not -determined by the ``cpe`` module). This can only be avoided by either using\ -rpath-linking (which is also special in the Cray PE as the wrappers activate rpath linking if the -environment variable ``CRAY_ADD_RPATH`` is defined and set to ``yes``) or by manually adjusting -the library path after loading the modules: -``` bash -export LD_LIBRARY_PATH=${CRAY_LD_LIBRARY_PATH}:$LD_LIBRARY_PATH -``` -The latter cannot be easily automated in modulefiles. Any technique that can be used (without actually -reworking the Cray PE modules) has nasty side effects in some scenarios. - -The net result of this feature is that some reproducibility of the results is lost. Programs will -react differently if the system default version of the Cray PE is changed as that will change the -set of default libraries that will be used at runtime unless rpath-linking is used or users -redfine ``LD_LIBRARY_PATH``. - - ---- - -## Further reading - -- [LUMI documentation: "Developing" section](https://docs.lumi-supercomputer.eu/development/) -- The Cray PE is mostly documented through man pages. There used to be some documentation on the - Cray web site also but the documentation system got reworked after the merger with HPE. - The documentation is now in the [HPE Support Centre](https://support.hpe.com/) where it - is very difficult to find the right version of the documents. -- The [PE-Cray](https://github.com/PE-Cray) GitHub project also provides some additional - documentation, including - - [some whitepapers](https://github.com/PE-Cray/whitepapers), - - [documentation for Cray OpenSMHEMX](https://github.com/PE-Cray/cray-openshmemx) and - - [information about Cray DSMML](https://github.com/PE-Cray/cray-dsmml) - though much of that information ia also not updated regularly anymore. - - ---- - -*[[next: LUMI software stacks]](1_04_LUMI_software_stack.md)* diff --git a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md deleted file mode 100644 index 3bfae70c..00000000 --- a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md +++ /dev/null @@ -1,101 +0,0 @@ -# LUMI software stacks (technical) - -*[[back: The Cray Programming Environment]](1_03_CPE.md)* - ---- - -The user-facing documentation on how to use the LUMI software stacks is -avialable in [the LUMI documentation](https://docs.lumi-supercomputer.eu/computing/softwarestacks/). -On this page we focus more on the technical implementation behind it. - ---- - -# An overview of LUMI - -LUMI has different node types providing compute resources: - -- LUMI has 16 login nodes, though many of those are reserved for special purposes and not - available to all users. TODO - - ---- - -## CrayEnv and LUMI modules - -On LUMI, two types of software stacks are currently offered: - - - ``CrayEnv`` (module name) offers the Cray PE and enables one to use - it completely in the way intended by HPE-Cray. The environment also offers a - limited selection of additional tools, often in updated versions compared to - what SUSE Linux, the basis of the Cray Linux environment, offers. Those tools - are installed and managed via EasyBuild. However, EasyBuild is not available - in that partition. - - It also rectifies a problem caused by the fact that there is only one - configuration file for the Cray PE on LUMI, so that starting a login shell - will not produce an optimal set of target modules for all node types. - The ``CrayEnv`` module recognizes on which node type it is running and - (re-)loading it will trigger a reload of the recommended set of target - modules for that node. - - - ``LUMI`` is an extensible software stack that is mostly managed through - [EasyBuild][easybuild]. Each version of the LUMI software stack is based on - the version of the Cray Programming Environment with the same version - number. - - A deliberate choice was made to only offer a limited number of software - packages in the globally installed stack as the setup of redundancy on LUMI - makes it difficult to update the stack in a way that is guaranteed to not - affect running jobs and as a large central stack is also hard to manage. - However, the EasyBuild setup is such that users can easily install - additional software in their home or project directory using EasyBuild build - recipes that we provide or they develop, and that software will fully - integrate in the central stack (even the corresponding modules will be made - available automatically). - - Each ``LUMI`` module will also automatically activate a set of application - modules tuned to the architecture on which the module load is executed. To - that purpose, the ``LUMI`` module will automatically load the ``partition`` - module that is the best fit for the node. After loading a version of the - ``LUMI`` module, users can always load a different version of the ``partition`` - module. - -Note that the ``partition`` modules are only used by the ``LUMI`` module. In the -``CrayEnv`` environment, users should overwrite the configuration by loading their -set of target modules after loading the ``CrayEnv`` module. - - ---- - -## The ``partition`` module - -The ``LUMI`` module currently supports four partition modules, but that number may -be reduced in the future: - -| Partition | CPU target | Accelerator | -|:----------------|-----------------------|:----------------------------| -| ``partition/L`` | ``craype-x86-rome`` | ``craype-accel-host`` | -| ``partition/C`` | ``craype-x86-milan`` | ``craype-accel-host`` | -| ``partition/G`` | ``craype-x86-trento`` | ``craype-accel-amd-gfx90a`` | -| ``partition/D`` | ``craype-x86-rome`` | ``craype-accel-nvidia80`` | - -All ``partition`` modules also load `craype-network-ofi``. - -``pattition/D`` may be dropped in the future as it seems we have no working CUDA setup -and can only use the GPU nodes in the LUMI-D partition for visualisation and not with CUDA. - -Furthermore if it would turn out that there is no advantage in optimizing for Milan -specifically, or that there are no problems at all in running Milan binaries on Rome -generation CPUs, ``partition/L`` and ``partition/C`` might also be united in a single -partition. - - - - - - - ---- - -*[[next: Terminology]](1_05_terminology.md)* - diff --git a/docs/2022-CSC_and_LO/1_05_terminology.md b/docs/2022-CSC_and_LO/1_05_terminology.md deleted file mode 100644 index 8e05012f..00000000 --- a/docs/2022-CSC_and_LO/1_05_terminology.md +++ /dev/null @@ -1,294 +0,0 @@ -# Terminology - -*[[back: LUMI software stacks]](1_04_LUMI_software_stack.md)* - ---- - -Over the years, we have come up with some terminology specific to EasyBuild -to refer to particular components, which we use alongside established terminology relevant to the context -of building and installing software. - -It is important to be familiar with these terms, so we'll briefly cover them one by one. - ---- - -## Toolchains - -A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, -which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. - -We refer to the different parts of a toolchain as **toolchain components**. - -The *compiler component* typically consists of [C](https://en.wikipedia.org/wiki/C_(programming_language)), -[C++](https://en.wikipedia.org/wiki/C%2B%2B), and [Fortran](https://en.wikipedia.org/wiki/Fortran) -compilers in the context of HPC, but additional compilers (for example, -a [CUDA](https://developer.nvidia.com/cuda-zone) compiler for -[GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units) software) -can also be included. - -Additional toolchain components are usually special-purpose libraries: - -- an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/) - or vendor-specific MPI implementations like Intel MPI in a toolchain based on the Intel compilers); - -- libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), - [LAPACK](http://performance.netlib.org/lapack/), [ScaLAPACK](http://www.netlib.org/scalapack/)); - - These libraries are included because they tend to provide a common API, so mixing them in a single - executable would cause name conflicts when linking. - -- a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); - -These libraries are included because they are some of the most popular libraries in scientific software, -but also because they provide an API that is available through different implementations. Mixing -implementations in a single build is a recipe for problems as the various implementations might -define the same symbols. - -A toolchain that includes all of these libraries is referred to as a **full toolchain**, while -a **subtoolchain** is a toolchain that is missing one or more of these libraries. -A **compiler-only toolchain** only consists of compilers (no additional libraries). - -The toolchain concept also maps nicely on the Programming Environment concept in the Cray -Programming Environment. Each ``PrgEnv-*`` module in fact provides a full toolchain in a -typical Cray PE installation, except for the FFTW library. Note that systems could configure -the contents of a ``PrgEnv-*`` module differently as the MPI library and scientific library are -optional. In the Cray PE, the MPI library is provided by the ``cray-mpic`` module, -the BLAS, LAPACK and ScaLAPACK libraries by the ``cray-libsci`` module (for CPU-only nodes) -and the FFTW library through the ``cray-fftw`` module. - -Many EasBuild toolchains are organised in a hierarchy. Each toolchain can use components compiled -with itself or with one of the subtoolchains. The Cray toolchains however are not currently -organised in a hierarchy. - - -### System toolchain - -The **`system` toolchain** is a special case which corresponds to using the compilers and libraries -*provided by the operating system*, rather than using toolchain components that were installed using EasyBuild. - -It used sparingly, mostly to install software where no actual compilation is done or -to build a set of toolchain compilers and its dependencies, since the versions of the system compilers -and libraries are beyond the control of EasyBuild, which could affect the reproducibility of the installation. - -On LUMI however it is used a bit more and it takes some of the functions of the ``GCCcore`` toolchain in -other EasyBuild toolchain hierarchies. - - -### Common toolchains - -The `foss` and `intel` toolchains are also known as the *common toolchains*, -because they are widely adopted by the EasyBuild community. - -The `foss` toolchain consists of all open source components (hence the name: -"FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), -[ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). -In recent versions (since 2021a), [FlexiBLAS](https://www.mpi-magdeburg.mpg.de/projects/flexiblas) -is used as the BLAS library, with OpenBLAS and LAPACK as the backend. - -The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version -controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. - -Roughly every 6 months, a new version of these common toolchains is agreed upon -in the EasyBuild community, after extensive testing. - -The common toolchains are also organised in a toolchain hierarchy: - -```mermaid -graph TD; -A[SYSTEM] --> B[GCCcore]; -B --> C[GCC]; -B --> D[iccifort]; -C --> E[gompi: Adds OpenMPI]; -D --> F[iimpi: Adds Intel MPI]; -D --> G[imkl: Adds Intel MKL]; -E --> H[foss: Adds OpenBLAS, LAPACK, ScaLAPACK, FFTW]; -F --> I[intel]; -G --> I; -``` - -More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). - - ---- - -## EasyBuild framework - -EasyBuild is written in Python and organised in three layers, represented by the three blocks in the logo: -the EasyBuild framework, easyblocks and Easyconfig files. - -The EasyBuild *framework* consists of a set of Python modules organised in packages (``easybuild.framework``, -``easybuild.toolchains``, ``easybuild.tools``, etc.) that collectively form **the core of EasyBuild**, -and is developed in the [``easybuild-framework`` repository on GitHub](https://github.com/easybuilders/easybuild-framework). - -It implements the **common functionality that you need when building software from source**, -providing functions for unpacking source files, applying patch files, collecting the output produced -by shell commands that are being run and checking their exit code, generating environment module files, etc. - -The EasyBuild framework does *not* implement any specific installation procedure, it only provides -the necessary functionality to facilitate this. - - ---- - -## Easyblocks - -An *easyblock* is **a Python module that implements a specific software installation procedure**, -and can be viewed as a plugin to the EasyBuild framework. Easyblocks can be either *generic* or *software-specific*. - -A **generic easyblock** implements an installation procedure that can be used for -multiple different software packages. Commonly used examples include the ``ConfigureMake`` easyblock -which implements the ubiquitous ``configure``-``make``-``make install`` procedure, and the -``PythonPackage`` easyblock that can be used to install a Python package. - -*In Spack this functionality is provided in the core of the package.* - -A **software-specific easyblock** implements an installation procedure that is specific to a particular software packages. -Infamous examples include the easyblocks we have for ``GCC``, ``OpenFOAM``, ``TensorFlow``, ``WRF``, ... - -The installation procedure performed by an easyblock can be controlled by defining -**easyconfig parameters** in an easyconfig file. - -*In Spack, this functionality is provided by the ``package.py`` files for the various packages. However, EasyBuild -does not need a software-specific easyblock for every package as for many packages the combination of a -generic easyblock with the much simpler description in the easyconfig file is sufficient. -The software-specific easyblocks can be a bit troublesome on Cray PE systems. Some easyblocks contain code that -is specific for certain compilers or toolchains and do not support all compilers or all toolchains.* - -A collection of (generic and software-specific) easyblocks is developed by the EasyBuild community -in the [``easybuild-easyblocks`` repository on GitHub](https://github.com/easybuilders/easybuild-easyblocks). - ---- - -## Easyconfig parameters - -An **easyconfig parameter** specifies a particular aspect of a software installation that should be performed by -EasyBuild. - -Some easyconfig parameters are **mandatory**. The following parameters *must* be defined in *every* easyconfig file: - -* ``name`` and ``version``, which specify the name and version of the software to install; -* ``homepage`` and ``description``, which provide key metadata for the software; -* ``toolchain``, which specifies the compiler toolchain to use to install the software (see - ``toolchains`` tab); - -Other easyconfig parameters are **optional**: they can be used to provide required information, -or to control specific aspects of the installation procedure performed by the easyblock. - -Some commonly used optional easyconfig parameters include: - -* ``easyblock``, which specifies which (generic) easyblock should be used for the installation; -* ``sources`` and ``source_urls``, which specify the list of source files and where to download them; -* ``dependencies`` and ``builddependencies``, which specify the list of (build) dependencies; -* ``configopts``, ``buildopts``, and ``installopts``, which specify options for the configuration/build/install commands, respectively; - -If no value is specified for an optional easyconfig parameter, the corresponding default value will be used. - -There are two groups of easyconfig parameters. *General* easyconfig parameters can be defined for any software -package, and (usually) control a specific aspect of the installation. *Custom* easyconfig parameters are -only supported by certain easyblocks, and only make sense for particular (types of) software. - ---- - -## Easyconfig files - -*Easyconfig files* (or *easyconfigs* for short), are **simple text files written in Python syntax -that specify what EasyBuild should install**. -Each easyconfig file defines the set of **easyconfig parameters** that collectively form a complete specification -for a particular software installation. - -The **filename** of an easyconfig file usually ends with the ``.eb`` extension. -In some contexts the filename is expected to be determined by the value of a handful of key -easyconfig parameters: ``name``, ``version``, ``toolchain`` and ``versionsuffix``. The general format for -the filename of an easyconfig file is: ``--.eb``, -where the toolchain part is omitted when a ``system`` toolchain is used, and the `` can be empty. - -The filename of easyconfig files is particularly relevant when EasyBuild is searching for easyconfig files to resolve dependencies, -since it does this purely based on filenames: interpreting the contents of every (potential) easyconfig file -it encounters would be too expensive. - -In the [``easybuild-easyconfigs`` repository on GitHub](https://github.com/easybuilders/easybuild-easyconfigs), -the EasyBuild community maintains a large (and growing) collection of easyconfig files, for a wide range of -(scientific) software. - -*EasyBuild does not have an equivalent of the concretiser of Spack. EasyBuild always starts from a complete, -precise configuration of a package to install, determined by the easyconfig parameters and possibly a -custom easyblock. Both approaches have their pros and cons. The concretiser in Spack combined with the -configuration of a software package via command line parameters makes it easy to quickly generate a -number of different configurations of a package to test performance or to check which combinations work -well. But one also has to be careful to not cause an explosion of modules and software installations, and -take care of using the appropriate features to ensure that one can generate the same configurations again. -The precise configurations of EayBuild can sometimes slow down things as they may need a lot of file editing, -but when it comes to building a reproducible software stack and sharing precise building instructions, this -approach may be preferred.* - ---- - -## Easystack files - -[**Easystack files**](https://docs.easybuild.io/en/latest/Easystack-files.html) -are a new concept in EasyBuild, providing a way to define a *software stack* -that should be installed by EasyBuild. - -They are written in [YAML syntax](https://yaml.org/), and include a list of *software specifications* -which correspond to a list of easyconfig files, with support for providing specific EasyBuild -configuration options for particular software packages, and including or excluding specific software -packages based on labels. - -The support for using easystack files is currently marked as *experimental*, -which means it is subject to change in future EasyBuild releases, and may be prone to errors. - ---- - -## Extensions - -*Extensions* is the collective term we use for **additional software packages that can be installed -on top of another software package**. Common examples are *Python packages*, *R libraries*, and *Perl modules*. - -As you can tell the common terminology here is a bit messy, so we came up with a unifying term... - -Extensions can be installed in different ways: - -* *stand-alone*, as a separate installation on top of one or more other installations; -* as a part of a *bundle* of extensions that collectively form a separate installation; -* or as an actual *extension* to a specific installation to yield a "batteries included" - type of installation (for examples by adding a bunch of Python packages from PyPI into - a Python installation); - ---- - -## Dependencies - -A *dependency* is a common term in the context of software. It refers to **a software -package that is either strictly required by other software, or that can be leveraged to -enhance other software** (for example to support specific features). - -There are three main types of dependencies for computer software: - -* a **build dependency** is only required when building/installing a software package; - once the software package is installed, it is no longer needed to *use* that software - (examples: `CMake`, `pkg-config`); -* a **link-time dependency** is somewhere in between a build and runtime dependency: - it is only needed when *linking* a software package; it can become either a build or runtime - dependency, depending on exactly how the software is installed (example: `OpenBLAS`); -* a **run-time dependency** (often referred to simply as *dependency*) is a software package that is - required to *use* (or *run*) another software package (example: `Python`); - -EasyBuild currently doesn't treat link-time dependencies differently from run-time dependencies -and hence has no specific easyconfig parameter for them. - ---- - -## Bringing it all together - -The EasyBuild **framework** leverages **easyblocks** to automatically build and install -(scientific) software, potentially including additional **extensions**, using a particular compiler **toolchain**, -as specified in **easyconfig files** which each define a set of **easyconfig parameters**. - -EasyBuild ensures that the specified **(build) dependencies** are in place, -and automatically generates a set of **(environment) modules** that facilitate access to the installed software. - -An **easystack file** can be used to specify a collection of software to install with EasyBuild. - ---- - -*[[next: Installation]](1_06_installation.md)* diff --git a/docs/2022-CSC_and_LO/1_06_installation.md b/docs/2022-CSC_and_LO/1_06_installation.md deleted file mode 100644 index 24c71d5e..00000000 --- a/docs/2022-CSC_and_LO/1_06_installation.md +++ /dev/null @@ -1,545 +0,0 @@ -# Installing EasyBuild - -*[[back: Terminology]](1_05_terminology.md)* - ---- - -Before deep diving into the wonderful world of EasyBuild and getting your hands dirty with -the hands on exercises coming up in this tutorial, you will need to install EasyBuild. - -In this section we outline a couple of different ways of doing this, and also the things you should pay attention to. -By the end, you will have a functional EasyBuild installation that you can use -for the remainder of this tutorial. - -!!! Remark "EasyBuild on LUMI" - - EasyBuild is already installed for each of the LUMI software stacks. The LUMI software - stacks are based on the Cray PE (using the Cray-specific toolchains). - Moreover, in case you want to install a full shadow stack rather than build on top - of one of the LUMI software stacks, scripts are provided to initialise a new version - of the LUMI software stack based on a specific version of the Cray PE, and those - scripts also take care of the installation of EasyBuild. - - Hence this section of the tutorial is only useful for people who want more background - knowledge or who want to experiment with other EasyBuild toolchains. - - -!!! summary - - * *Requirements*: Linux, Python 2.7 or 3.5+, environment modules tool (Lmod is recommended). - * *Installation methods*: - * ``pip install easybuild`` - * installing EasyBuild with EasyBuild - * development setup - * Verify EasyBuild installation using `eb --version` and `eb --help`. - - ---- - -## Requirements - -### Linux - -The main target platform for EasyBuild is *Linux*, since it is heavily focused on installing software -on HPC system where Linux is dominant operating system (to the point where 100% of the current Top500 list of supercomputers are running Linux). - -EasyBuild is also compatible with *macOS*, but the included easyconfig files are heavily focused -on Linux so most software installations supported by EasyBuild won't work out-of-the-box on macOS. -You can still use the EasyBuild command line interface on macOS for other tasks though, -like development or contributing back to the project. - -### Python - -EasyBuild is implemented in Python, and is **compatible with both Python 2.7 and Python 3.5+** -(that is, Python 3.5 or a newer version of Python 3). - -To check which Python version you have, use: - -```shell -python -V -``` - -*No additional Python packages are required by EasyBuild*, the ones that come with the standard -Python distribution are sufficient. Some additional Python packages *can* be leveraged for specific features. -More on that later. - -### Environment modules tool - -**An *environment modules tool* is required for using EasyBuild.** - -We strongly recommend using [Lmod](https://lmod.readthedocs.io), a Lua-based modern environment -modules implementation and the most commonly used modules tool in the EasyBuild community. -Other implementations, like the original Tcl-based one, are also supported. - -To check if you have a modules tool installed, use: - -``` -module --version -``` - -If this produces output that starts with something like "`Modules based on Lua: Version 8.3.1`" you have Lmod installed, -which is the default modules tool used by EasyBuild, and you are all set for installing and using EasyBuild. -Any sufficiently recent Lmod version (8.x or even 7.x) should be fine. - -If you see output that starts with a line like "`VERSION=3.2.10`" or "`Modules Release 4.5.0`", -you have the original Tcl-based environment modules tool installed, either in its original -C-based imnplementation (``VERSION=3.2.10``) or the new Tcl-based implementation developed at CEA -(``Modules Release 4.5.0``) -and *EasyBuild will need to be configured to use it after installation*. - -If the `module` function is not defined either you do not have a modules tool installed -or your environment is not properly set up to use it. In this case, -please refer to the EasyBuild documentation here for more information. - -HPE Cray supports both the old-style Environments Modules (based on version 3.2.10) and Lmod. On LUMI, -we use Lmod as the default module system for users, and EasyBuild is configured to use that one. - ---- - -## EasyBuild as a Python package - -EasyBuild consists of a number of interdependent Python packages, -and is available via both GitHub at https://github.com/easybuilders, -as well as via the standard Python Package Index (PyPI) at -https://pypi.org/project/easybuild. - -As you may be aware the Python packaging ecosystem is bit convoluted, -which is reflected in the many different ways in which you can install a Python package. -In addition, EasyBuild is packaged in 3 components (framework, easyblocks, easyconfigs) -which slightly complicates the installation. - -
- -Nevertheless, you don't need to be a rocket scientist to install EasyBuild (and even if you are, -that's OK too), so don't worry. - -You can install EasyBuild just like you can install any other Python software that is released -via the standard *Python Package Index* (PyPI), through one of the standard Python installation tools -(like `pip`, `virtualenv`, `pipenv`, ...). -And since EasyBuild is a software installation tool in its own right, we actually have a couple -of additional tricks up our sleeve! - -### Python 2 or Python 3? - -For EasyBuild it does not matter much whether you install it on top of Python 2 or Python 3. -Since version 4.5, Python 3 does offer some optional features (requiring additional packages) -that are not available with Python 2. -Since Python 2 is end-of-life, -we strongly recommend using Python 3 if you have the choice. - -By default EasyBuild will use the `python` command to run, -but you can control this if needed via ``$EB_PYTHON``. - - -## Installing EasyBuild - -We present three methods for installing EasyBuild. -It is up to you which one you prefer, both result a fully functional EasyBuild installation. - -Time to get your hands dirty! - -* Method 1: Using ``pip`` -* Method 2: installing EasyBuild with EasyBuild -* Method 3: Development setup - ---- - -### Method 1: Using `pip` - -Since EasyBuild is released as a [Python package on PyPI](https://pypi.org/project/easybuild) -you can install it using `pip`, the most commonly used tool for installing Python packages. - -You may need to take additional steps after the installation, depending on the exact installation command. - -!!! note - There are various other ways of installing Python packages, which we won't cover here. - If you are familiar with other tools like `virtualenv` or `pipenv`, feel free to use those - instead to install EasyBuild. - -#### Running `pip install` - -Installing EasyBuild with `pip` is as simple as running the following command: - -```shell -pip install easybuild -``` - -However, you may need to slightly change this command depending on the context and your personal preferences: - -* To install EasyBuild *system-wide*, you can use `sudo` (if you have admin privileges): - ```shell - sudo pip install easybuild - ``` - -* To install EasyBuild *in your personal home directory*, you can use the `--user` option: - ```shell - pip install --user easybuild - ``` - This will result in an EasyBuild installation in `$HOME/.local/`. - -* To install EasyBuild in a *specific directory* you can use the `--prefix` option: - ```shell - pip install --prefix _PREFIX_ easybuild - ``` - In this command, you should replace '`_PREFIX_`' with the location where you want to have EasyBuild installed - (for example, `$HOME/tools` or `/tmp/$USER`). - -#### `pip` vs `pip3` - -On systems where both Python 2 and Python 3 are installed you may also have different `pip` commands -available. Or maybe `pip` is not available at all, and only "versioned" `pip` commands like `pip3` are -available. - -If you (only) have `pip3` available, you can replace `pip` with `pip3` in any of the `pip install` commands -above. - -If you want to ensure that you are using the ``pip`` installation that corresponds to the Python 3 installation -that you intend to use, you can use ``python3 -m pip`` rather than ``pip3``. - -#### Updating your environment - -If you used the `--user` or `--prefix` option in the `pip install` command, -or if you installed EasyBuild with a `pip` version that does not correspond -to your default Python installation, -you will need to update your environment to make EasyBuild ready for use. -This is not required if you did a system-wide installation in a standard location with the default Python version. - -!!! note - Keep in mind that you will have to make these environment changes again if you start a new shell session. - To avoid this, you can update one of the shell startup scripts in your home directory (`.bashrc` for example). - -#### Updating ``$PATH`` - -Update the `$PATH` environment variable to make sure the `eb` command is available: -```shell - -export PATH=_PREFIX_/bin:$PATH -``` -**Replace '`_PREFIX_`' in this command** with the directory path where EasyBuild was installed into -(use `$HOME/.local` if you used `pip install --user`). - -This is not required if you installing EasyBuild in a standard system location. - -You can check with the `which eb` command to determine whether or not you need to update the ``$PATH`` environment variable. - -#### Updating ``$PYTHONPATH`` - -If you installed EasyBuild to a non-standard location using `pip install --prefix`, -you also need to update the Python search path environment variable -[`$PYTHONPATH`](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH) to instruct Python where -it can find the EasyBuild Python packages. - -This is not required if you used the `--user` option, since Python will automatically consider -`$HOME/.local` when searching for installed Python packages, or if you installed EasyBuild in a standard -system-wide location. - -Update `$PYTHONPATH` by running a command like: - -```shell - -export PYTHONPATH=_PREFIX_/lib/pythonX.Y/site-packages:$PYTHONPATH -``` - -Here, you need to replace the `X` and `Y` with the major and minor version of your Python installation, -which you can determine by running `python -V`. -For example, if you are using Python 3.6, make sure you are using `/python3.6/` in the command to update `$PYTHONPATH`. - -And of course, you again need to **replace '`_PREFIX_`'** with the installation prefix where EasyBuild was installed -into. - -For example: - -```shell -# update $PYTHONPATH if EasyBuild was installed in $HOME/tools with Python 3.6 -export PYTHONPATH=$HOME/tools/lib/python3.6/site-packages:$PYTHONPATH -``` - -#### Setting ``$EB_PYTHON`` - -If you want to control which Python version is used to run EasyBuild, -you can specify the name or the full path to the `python` command that should be used by the `eb` command -via the `$EB_PYTHON` environment variable. - -This may be required when you installing EasyBuild with a version of `pip` that does not correspond -with the default Python version. - -For example, to ensure that `eb` uses `python3.6`: - -```shell -export EB_PYTHON=python3.6 -``` - -#### Setting ``$EB_VERBOSE`` - -To determine which `python` commands are being considered by the `eb` command, -you can define the `$EB_VERBOSE` environment variable. For example: - -```shell -$ EB_VERBOSE=1 eb --version ->> Considering 'python3.6'... ->> 'python3' version: 3.6.8, which matches Python 3 version requirement (>= 3.5) ->> Selected Python command: python3 (/usr/bin/python3.6) ->> python3.6 -m easybuild.main --version -This is EasyBuild 4.3.3 (framework: 4.3.3, easyblocks: 4.3.3) on host example -``` - -!!! Tip "Debugging startup problems" - - Using `ÈB_VERBOSE` is useful if EasyBuild fails to start up and complains it - cannot find a suitable Python executable. Rather often the error message is - caused by a failure elsewhere in EasyBuild. - - ---- - -### Method 2: Installing EasyBuild with EasyBuild - -!!! note - This section covers an alternative method for installing EasyBuild. - - If you already have EasyBuild installed through ``pip``, - you can skip ahead to the next section. - -If you prefer having EasyBuild available through an environment module file, -you can consider installing EasyBuild with EasyBuild. This can be done in 3 steps: - -* Step 1: Installing EasyBuild with ``pip`` into a temporary location (only needed if EasyBuild is not installed yet) -* Step 2: Using EasyBuild to install EasyBuild as a module -* Step 3: Loading the EasyBuild module - -!!! note - A [bootstrap script](https://docs.easybuild.io/en/latest/Installation.html#bootstrapping-easybuild) - is available that automates this procedure, - but is known to be problematic in some contexts, and is not being actively - maintained anymore. - - As a result, we do not recommend using the bootstrap script anymore. - - -#### Step 1: Installing EasyBuild into a temporary location - -If you don't have EasyBuild installed yet, you need to install it in a temporary location first. -The recommended way of doing this is using ``pip``. - -For example, to install EasyBuild into a subdirectory `/tmp/$USER` using the default Python 3 version: - -```shell -# pick installation prefix, and install EasyBuild into it -export EB_TMPDIR=/tmp/$USER/eb_tmp -python3 -m pip install --ignore-installed --prefix $EB_TMPDIR easybuild -``` - -```shell -# update environment to use this temporary EasyBuild installation -export PATH=$EB_TMPDIR/bin:$PATH -export PYTHONPATH=$(/bin/ls -rtd -1 $EB_TMPDIR/lib*/python*/site-packages | tail -1):$PYTHONPATH -export EB_PYTHON=python3 -``` - -#### Step 2: Using EasyBuild to install EasyBuild - -Once you have a working (recent) temporary EasyBuild installation, you can use it to -install EasyBuild as a module. Usually this is done in the location where you would -like to install other software too. - -You can use the ``eb --install-latest-eb-release`` command for this, -combined with the ``--prefix`` option to control which directories are used by EasyBuild for the installation. - -For example, to install the latest version of EasyBuild as a module into ``$HOME/easybuild``: - -```shell -eb --install-latest-eb-release --prefix $HOME/easybuild -``` - -!!! note - You may see a harmless deprecation warning popping up when performing this installation, just ignore it. - -#### Step 3: Loading the EasyBuild module - -Once step 2 is completed, you should be able to load the module that was generated alongside -the EasyBuild installation. You will need to do this every time you start a new shell session. - -First, make the module available by running the following command (which will update the module search path -environment variable `$MODULEPATH`): - -```shell - -module use _PREFIX_/modules/all -``` - -**Replace '`_PREFIX_`'** with the path to the directory that you used when running step 2 -(for example, ``$HOME/easybuild``). - -Then, load the `EasyBuild` module to update your environment and make EasyBuild available for use: - -```shell - -module load EasyBuild -``` - -Note that in this case, we don't need to make any changes to our environment for EasyBuild to work correctly. -The environment module file that was generated by EasyBuild specifies all changes that need to be made. - -### Method 3: Development setup - -If you are planning to make changes to EasyBuild, or if you prefer using the latest *bleeding edge* -version of EasyBuild that is being developed, you can consider *cloning* the 3 main EasyBuild repositories -from GitHub, and updating your environment to run EasyBuild from there. - -This can be done as follows (into ``$HOME/easybuild``): - -```shell -mkdir -p $HOME/easybuild -cd $HOME/easybuild -``` - -```shell -# clone EasyBuild repositories from GitHub -git clone https://github.com/easybuilders/easybuild-framework.git -git clone https://github.com/easybuilders/easybuild-easyblocks.git -git clone https://github.com/easybuilders/easybuild-easyconfigs.git -``` -```shell -# update environment for running EasyBuild from there -export PATH=$HOME/easybuild/easybuild-framework:$PATH -export PYTHONPATH=$HOME/easybuild/easybuild-framework:$HOME/easybuild/easybuild-easyblocks:$HOME/easybuild/easybuild-easyconfigs:$PYTHONPATH - -# control which Python command is used to run EasyBuild -export EB_PYTHON=python3 -``` - - -### Approach on LUMI - -*Documentation on the inner workings of the LUMI software stack can be found in the -[LUMI-SoftwareStack GitHub, docs subdirectory](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/docs).* - -To keep the different versions of the LUMI software stack as independent from one another as possible, -EasyBuild is bootstrapped for each software stack by the ``prepare_LUMI_stack.sh`` script. We use the -system Python 3 for running EasyBuild. That Python version currently doesn't have ``pip`` installed. -EasyBuild is first installed in a temporary work directory from files downloaded from PyPi. These -are each untarred and then installed into their location by running -``python3 setup.py install --prefix=...`` in their unpacked directory. -We do so only for the framework and easyblocks files as the easconfig files are not used to -install EasyBuild. Instead we create our own EasyConfig file for EasyBuild which contains -some additional packages that enable extra features in EasyBuild and also provide more information -to Lmod. Next the configuration module for EasyBuild (see the next section, -["Configuring EasyBuild"](1_07_configuration), for more information) and use the temporary -installation of EasyBuild with our own EasyConfig file to do a proper installation of EasyBuild -with module in the final location. - - -TODO: follow development setup but with downloading rather than cloning, then use that one with the -EasyBuild configuration module to install EasyBuild with itself. - - - -## Verifying the installation - -Regardless of how EasyBuild was installed, you can now run a couple of basic commands to verify the installation: - -#### Checking the version - -To check which EasyBuild version you have installed, run: - -```shell - -eb --version -``` - -The output should match with the latest EasyBuild version. - - -#### Consulting the help output - -You can consult the help output of the `eb` command, which produces a long list of available options -along with a short informative message. - -```shell -eb --help -``` - -#### Showing the default EasyBuild configuration - -To inspect the current EasyBuild configuration, you can use this command: - -```shell -eb --show-config -``` - -This should tell you that EasyBuild (ab)uses `$HOME/.local/easybuild` as a default location. -More on configuring EasyBuild in the [next part of the tutorial](configuration). - -#### System information - -You ask EasyBuild to collect and print some information about the -system you are using it on (OS, CPU, Python, etc.) using this command: - -```shell -eb --show-system-info -``` - -## Updating EasyBuild - -Before we wrap up here, a brief word about updating EasyBuild. - -Once you have EasyBuild installed, the easiest way to update to a newer version is by instructing EasyBuild -to install the latest available version as a module: - -``` - -eb --install-latest-eb-release -``` - -This will result in a *new* EasyBuild installation, which is entirely separate from the EasyBuild installation -you are currently using (so it is *not* an in-place update). -The location where this new EasyBuild version will be installed is determined by the active -EasyBuild configuration. - -If you have installed EasyBuild through ``pip``, and you prefer updating that installation, -you can use ``pip install --upgrade easybuild`` (perhaps with additional options like ``--user`` or ``--prefix``). - -!!! Note "EasyBuild upgrade policy on LUMI" - - Even minor version or patch level updates of EasyBuild may break things (and this has happned - in the past on other systems managed previously by LUMI User Support Team members). Hence a very - conservative upgrade policy is used on LUMI. - - In general we fix the version of EasyBuild within a particular version of the LUMI software stack - and only transition to the latest version when starting a new software stack. The reason is that we - want to be able to rebuild an existing software stack with as little unexpected problems as possible. - A move to a newer version of EasyBuild is acceptable should we run into major problems with an existing - version that cannot be solved in an easyconfig file or LUMI-specific custom easyblock, but in those - cases we will first rebuild the whole software stack on a test setup to ensure that no problems are - introduced by upgrading EasyBuild. - - ---- - -## Exercise (optional) - -*This exercise is not needed to proceed the course when working on LUMI as we will -use the installation that is already present.* - -Install EasyBuild in your home directory. - -Make sure that the EasyBuild installation uses the `python3` command to run, -rather than the standard `python` command. - -Choose your own adventure, or try all these installation methods! - -* install EasyBuild with `pip` (or another very similar command...) using either the `--user` or `--prefix` option; -* perform a (manual) "bootstrap" installation into `$HOME/easybuild`, as outlined in installation method 2; -* set up a development installation; - -Check that the installation works by running the verification commands outlined -[above](#verifying-the-installation). - ---- - -**You'll need a working custom EasyBuild installation for some of the -optional generic EasyBuild exercises, but in general you can use the -installation provided by LUMI.** - - -*[[next: Configuring EasyBuild]](1_07_configuration.md)* diff --git a/docs/2022-CSC_and_LO/1_07_configuration.md b/docs/2022-CSC_and_LO/1_07_configuration.md deleted file mode 100644 index a1c2f3de..00000000 --- a/docs/2022-CSC_and_LO/1_07_configuration.md +++ /dev/null @@ -1,610 +0,0 @@ -# Configuring EasyBuild - -*[[back: Installation]](1_06_installation.md)* - ---- - -After installing EasyBuild, you should **configure** it. - -EasyBuild should work fine out-of-the-box if you use Lmod as your modules tool. -If you are not using Lmod, please see here for more information. - -Nevertheless, we strongly recommend you to inspect the default configuration, -and to configure EasyBuild according to your preferences and the system on which you will use it. - -!!! Note "EasyBuild configuration on LUMI" - - On LUMI serveral configurations of EasyBuild are already available. - - The ``EasyBuild-user`` module is the most important one. It will configure EasyBuild - to install software in either a default location in the user's home directory - (``$HOME/EasyBuild``) or the location pointed to by the environment variable - ``EBU_USER_PREFIX``. The latter can be used to install software in the project directory - so that it is available to all users of the project. It enables the creation of a custom - software environment build on top of the LUMI software stack. Software in this environment is - build in exactly the same way as it would if it were installed in the central repository, but - one can now also easily create custom configurations without confusing other projects with - different or non-standard configurations of software. - - There are two more admin-only configuration modules for EasyBuild. The main one is - ``EasyBuild-production`` which is used for software installations in the central software stack. - The other one is ``EasyBuild-infrastructure`` which is used to install infrastructure modules - that need to be installed in a pure Lmod hierarchy, e.g., the modules that load the toolchains. - - ---- - -## Available configuration settings - -One of the central policies in the EasyBuild project is to avoid hardcoded settings in the codebase. -While this significantly increases the ability to configure EasyBuild to your liking, it also results -in a large amount of available configuration settings. - -The full list of configuration settings can be consulted via `eb --help`, -which shows the corresponding command line option accompanied by a short description. -At the time of writing, *over 240 different configuration settings* are supported by EasyBuild. - -For the sake of this tutorial we will focus on a specific subset of configuration settings, -and cover only the most prominent and important ones. - -We will refer to EasyBuild configuration settings using the names as they appears in the output of `eb --help`, -and omit the leading dashes (`--`) for the sake of clarity. - -Keep in mind that *every* configuration setting can be defined in 3 different ways, -see below for more details. - ---- - -### Overall prefix - -*(default: `HOME/.local/easybuild`)* - -The `prefix` configuration setting specifies the overall prefix that EasyBuild should use, -which **determines the default value for various other configuration settings**: - -* `installpath`: `` -* `buildpath`: `/build` -* `sourcepath`: `/sources` -* `repositorypath` (easyconfigs archive): `/ebfiles_repo` -* `containerpath`: `/containers` - -Here, `` represents the value of the `prefix` configuration setting. - -If one of the configuration settings affected by `prefix` is defined specifically, -the `prefix` value becomes irrelevant for that specific configuration setting. - ---- - -### Install path - -*(default: ``)* - -The **location for both the software installation directories and generated module files** -can be controlled via the `installpath` configuration setting. -Software installation directories will be placed in `/software`, while -`/modules/all` will be used for generated module files. - -The `installpath` location is usually set to a directory on a *shared filesystem* when installing -software for an HPC cluster. Of course, software can also be installed on a local filesystem, -which is particularly useful to test and evaluate software installations. - -Separate configuration settings are available for both software and modules locations, -as well as for controlling the name of the `software` and `modules/all` subdirectories. - -*The EasyBuild community recommends to only change the `installpath` configuration setting to control the location -of software installations and accompanying module files, -such that the software and modules directories are located in the same parent directory, -and the default `software` and `modules/all` names for the subdirectories are used.* - -!!! Note "Path for software and for modules on LUMI" - - On LUMI we do not follow that recommendation. There is a three for the software installations - themselves with subdirectories based on the version of the software stack and LUMI hardware - partition, and a separate tree for the modules organised in a similar way. - - This makes it slightly easier to organise the module tree with user-friendly labeling, but above - all also makes the synchronisation process of the 4 instances of the software directory more robust - as it is now easy to synchonise all modules in the last step, which is a much quicker process than - syncrhonising the software installations. - - We also use short paths for software installations (to avoid overrunning the maximum length of a - shebang line in scripts) while we use longer, more descriptive names for subdirectories in the - module tree. - - ---- - -### Build path - -*(default: `/build`)* - -For each installation it performs, EasyBuild creates a **separate build directory** where software will be compiled -before installing it. This directory is cleaned up automatically when the installation is successfully completed. -To control the location where these build directories are created, you can use the `buildpath` configuration setting. - -Keep in mind that build directories may grow out to several GBs in size during an installation, -and that the commands that run in there can be fairly I/O-intensive since they may involve -manipulating lots of small files. In addition, a build directory that corresponds to a failing installation -is *not* cleaned up automatically, but it will be cleaned up and recycled when the same installation is re-attempted. -Running out of disk space in the location where build directories are created will result in failing -installations. - -It is strongly recommend to use the path to a directory on a *local filesystem* for the value of the -`buildpath` configuration setting, since using a shared filesystem like Lustre or GPFS is known to cause -problems when building certain software packages. Using an in-memory location (like `/dev/shm/$USER`) can -significantly speed up the build process, but may also lead to problems (due to space limitations, -or specific mount options like `noexec`). - -!!! Note "buildpath on LUMI" - - The configuration modules on LUMI will use a RAM disk for the build path. On the login nodes, - ``$XDG_RUNTIME_DIR`` is used as that space is automatically cleared when the last session of a user - ends. However, on the compute nodes a job- or user-specific subdirectory of ``/dev/shm`` is currently used - as ``$XDG_RUNTIME_DIR`` does not exist. - - ---- - -### Source path - -*(default: `/sources`)* - -For most supported software, EasyBuild can **automatically download the source files** required for the installation. -Before trying to download a source file, EasyBuild will first check if it is already present in the source path. - -The locations considered by EasyBuild when checking for available source files, as well as the location to -store downloaded source files, can be controlled via the ``sourcepath`` configuration setting. - -The `sourcepath` value is a colon (`:`) separated list of directory paths. -Each of these paths will be considered in turn when checking for available source files, -until one of them provides the desired source file. Searching for source files is done -based on filename, and a couple of subdirectories are considered. -For example, for a software package named '`Example'`, EasyBuild will consider locations -like `/e/Example/`, `/Example/`, and so on. - -The first path listed in `sourcepath` is the location where EasyBuild will store downloaded source files, -organised by software name through subdirectories, so EasyBuild expects to have write permissions to this path. -For the other paths listed in `sourcepath` only read permissions are required. - -*Make sure you have write permissions to the first path listed in `sourcepath`, so EasyBuild is able -to store downloaded files there. Feel free to list additional paths if you already have a cache of downloaded -files available somewhere.* - -*Storing the downloaded files not only reduces the amount of downloads while developing new easyconfig files -but also greatly helps when recompiling a software stack, as it is not that uncommon that download sites change -of files become unavailable.* - - ---- - -### Easyconfigs archive - -*(default: `/ebfiles_repo`)* - -EasyBuild keeps track of the easyconfig files that were used for installations in the easyconfigs -archive, the location of which is specified by the `repositorypath` configuration setting. - -By default the specified path is assumed to be a regular directory, but using a Git repository -as easyconfigs archive is also supported (for more details, see -[the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Configuration.html#easyconfigs-repository-repository-repositorypath)). - - ---- - -### Modules tool & module syntax - -*(default: `Lmod` as modules tool, and `Lua` as module syntax)* - -By default, EasyBuild assumes you are using **[Lmod](https://lmod.readthedocs.io) as modules tool**. -In addition, it will generate module files in [Lua](https://www.lua.org/) syntax, as supported by Lmod -(next to `Tcl` syntax). - -To diverge from this, you can define the `modules-tool` configuration setting to indicate you -are using a different modules tool; see the output of `eb --avail-modules-tools` for a list of supported -modules tools. -Note that for anything other than Lmod, you *must* make -sure that the actual modules tool binary command is available through `$PATH` (more information -on this [in the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Configuration.html#modules-tool-modules-tool)). - -If you prefer that EasyBuild generates module files in [Tcl](https://tcl.tk/) syntax, you can instruct -it to do so via the `module-syntax` configuration setting. Note that this is *required* if you are using -a modules tool other than Lmod, since only Lmod supports module files in `Lua` syntax. - -*We recommend using EasyBuild with the defaults settings: Lmod as modules tool, and Lua as module syntax.* - ---- - -### Robot search path - -*(default: `robot-paths` specifies the location to the easyconfig files included with EasyBuild installation; -dependency resolution is* ***not*** *enabled)* - -When EasyBuild needs to locate one or more easyconfig files, it will do so via the **robot search path**. -This applies to both easyconfig files that were specified using only their filename as an argument to the -`eb` command, as well as to easyconfigs required to resolve dependencies (more on that later). - -To control the robot search path, two configuration settings are available: `robot` and `robot-paths`. -Both accept a colon-separated list of locations to consider when looking for easyconfig files, -with `robot` having a higher priority than `robot-paths` (i.e., the directories specified with `robot` -appear before those specified by `robot-paths` in the robot search path). - -The key difference between these two configuration settings is that defining `robot` also enables dependency resolution, -and hence makes EasyBuild install missing dependencies, alongside -specifying a list of paths to consider when searching for easyconfig files. -On the other hand, defining `robot-paths` does not have the side effect of enabling dependency resolution. - -In addition, you can use the `--robot` command line option without specifying any paths to it, -to only enable dependency resolution. ***This option is needed on LUMI if you want automatic -dependency resolution as it is disabled on LUMI to prevent accidental mistakes when installing -software.*** - -!!! note "Easyconfig files included with EasyBuild and the robot search path" - Keep in mind that when either of the `robot` or `robot-paths` configuration settings are - defined, the default value corresponding to the location of easyconfigs included with the - EasyBuild is *no longer considered*. - - There are ways around this however, which are outside of the scope of this tutorial. - - For more information, - [see the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Using_the_EasyBuild_command_line.html#controlling-robot-search-path). - - *On LUMI the included easyconfig files cannot be used without changes as they are for the - common toolchains which are currently not supported in the LUMI software stacks.* - - -!!! Note "Easyconfigs archive on LUMI and the robot search path" - - On LUMI we also include the easyconfigs archive at the front of the search path for easyconfig - files. This has the advantage that EasyBuild when checking dependencies will always find the - configuration that is actually installed on the system, even if the easyconfig file in one of - the regular repositories has changed. It also has the disadvantage that one may accidentally - be re-installing with the already installed configuration while one really wants to re-install - the module in a different configuration. - - - ---- - -### Module naming scheme - -*(default: `EasyBuildMNS`)* - -EasyBuild will use a specific naming scheme for the module files it generates for each of the software installations. -This ensures consistency, regardless of who uses EasyBuild to perform the installation. - -Different types of module naming schemes are supported (flat, hierarchical, ...) and you can provide an implementation of -your own custom module naming scheme if desired. A number of different naming schemes are included -with EasyBuild, which you can consult via `eb --avail-module-naming-schemes`. - -The default `EasyBuildMNS` module naming scheme roughly corresponds to the filename of easyconfig files, -and consists of the software name followed by a combination of the software version, toolchain and -an optional label (which corresponds to the value of the `versionsuffix` easyconfig parameter): -`/<-toolchain>`. Just like with [names of easyconfig -files](1_05_terminology#easyconfig-files), the `<-toolchain>` part is omitted when the `system` toolchain is used, -and the `` value is empty by default. - -!!! Note "Module naming scheme on LUMI" - - On LUMI we use a custom variant of the standard EasyBuild flat naming scheme. The EasyBuild default - naming scheme creates an ``all`` subdirectory in the module directory containing all modules, but also - a directory per category, with the categories determined from the ``moduleclass`` easyconfig parameter. - However, that choice is often rather arbitrary as modules can belong to different categories depending - on who you ask. E.g., GROMACS is used in physics, chemistry and life sciences while EasyBuild can only - assign a single category. - - Hence we do not generate the categories and drop the ``all`` level in the module subdirectory. - - ---- - -## Configuration levels - -Configuring EasyBuild can be done in different ways: - -* through one or more configuration files; -* via `$EASYBUILD_*` environment variables; -* using `eb` command line options; - -Each of the methods corresponds to a *configuration level*. - -*Every* configuration setting can be defined via one of these -mechanisms, without exception! - ---- - -### Configuration level hierarchy - -There is a strict **hierarchy between the different configuration levels** supported by EasyBuild. - -Settings defined via a *configuration file* only override default values. - -A configuration setting that is defined via the corresponding *`$EASYBUILD_*` environment variable* -takes precedence over the value specified in a configuration file (if any). - -Finally, values specified through `eb` command line options **always** win, -regardless of whether the corresponding -configuration setting was already defined some other way, be it via a configuration file or -an environment variable. - -For example, let us consider a fictional configuration setting named `magic`: - -* If a value for `magic` is specified in an EasyBuild configuration file, - then this value will only be used if the corresponding environment variable - (`$EASYBUILD_MAGIC`) is *not* defined **and** if the `--magic` command line option is *not* used; -* If the `$EASYBUILD_MAGIC` environment *is* defined however, then its value - will be used for the `this-is-magic` configuration setting; -* *Unless* the `--magic` command line option is used, - since in that case the value provided as an argument there will be used instead. - ---- - -### Configuration files - -*Configuration files* are the most basic way of configuring EasyBuild. -Two types of are supported by EasyBuild: *user-level* and *system-level* -configuration files. The output of `eb --show-default-configfiles` tells you which locations -are considered, and whether any configuration files were found. - -EasyBuild configuration files are written in the standard [INI format](https://en.wikipedia.org/wiki/INI_file), -and the configuration settings are grouped into different *sections*. - -To create an EasyBuild configuration file, the output of `eb --confighelp` is very useful: -it produces the list of all supported configuration settings which are readily grouped in sections -and with every entry commented out, along with accompanying descriptive comments mentioning the default values, -and in the correct syntax. - -*Configuration files are the recommended mechanism to define configuration settings -that should always be in place, regardless of the software you are installing.* - -#### EasyBuild configuration files vs easyconfig files - -EasyBuild configuration files are sometimes confused with easyconfig files, -due to the similar name. However, these are two entirely different concepts! - -EasyBuild configuration files (usually `*.cfg`) are a way of *configuring the general behaviour of EasyBuild* -across different software installations. They define *configuration settings*, -such as the location where software should be installed, or the syntax that should -be used when generating module files. - -An [easyconfig file](1_05_terminology#easyconfig-files) (`*.eb`) on the other hand *specifies the details for one particular software installation*. -It does this by defining a set of *easyconfig parameters*, which tell EasyBuild the name and version -of the software to install, which [toolchain](1_05_terminology#toolchains) and [easyblock](1_05_terminology#easyblocks) to use, etc. - -For each software installation performed by EasyBuild, there is a corresponding easyconfig file. -There typically are only a handful of configuration files used however, for example a system-level -configuration file, perhaps combined with a user-level one. Or there may be no configuration files involved -at all, since EasyBuild can also be configured through other mechanisms: environment variables and command line -options. - ---- - -### `$EASYBUILD_*` environment variables - -A particularly easy way to configure EasyBuild is through *environment variables*. - -At startup, EasyBuild will pick up any environment variable of which the name starts with '`EASYBUILD_`'. -For each of these, it will determine the corresponding configuration setting (or exit with an error if -none was found). - -Mapping the name of a configuration setting to the name of the corresponding environment variable is straightforward: -use capital letters, replace dashes (`-`) with underscores (`_`), and prefix with `EASYBUILD_`. - -For example: the `module-syntax` configuration setting can be specified by defining -the `$EASYBUILD_MODULE_SYNTAX` environment variable: - -```shell -export EASYBUILD_MODULE_SYNTAX=Tcl -``` - -*Configuring via environment variables is especially practical for controlling the EasyBuild configuration -in a more dynamic way. For example, you can implement a simple shell script that defines `$EASYBUILD_*` -environment variables based on the current context (user, hostname, other environment variables), and -configure EasyBuild through [sourcing](https://bash.cyberciti.biz/guide/Source_command) it.* - -!!! note - Keep in mind that environment variables are only defined for the shell session you are currently working in. - If you want to configure EasyBuild through environment variables in a more persistent way, - you can leverage one of the [shell startup scripts](https://bash.cyberciti.biz/guide/Startup_scripts) (for example `$HOME/.bash_profile` or `$HOME/.bashrc`). - ---- - -### `eb` command line options - -Finally, you can also configure EasyBuild by specifying one or options to the `eb` command. - -As mentioned earlier, the values for configuration settings defined this way override the value that -is specified through any other means. So if you want to be sure that a particular configuration setting -is defined the way you want it to be, you can use the corresponding command line option. - -There are various configuration settings for which it only makes sense to use the command line option. -An example of this is letting the `eb` command print the EasyBuild version (via `eb --version`). -Although you could configure EasyBuild to always print its version and then exit whenever the `eb` command is -run, that would not be very useful... - -*Command line options are typically used to define configuration settings that are only relevant to -that particular EasyBuild session. One example is doing a test installation into a temporary directory:* - -```shell -eb --installpath /tmp/$USER example.eb -``` - -## Inspecting the current configuration (`--show-config`) - -Given the large amount of available configuration settings in EasyBuild and the different configuration levels, -you can easily lose track of exactly how EasyBuild is configured. - -Through the `--show-config` command line option you can -easily inspect the currently active EasyBuild configuration. - -The output of `--show-config` includes a sorted list of all configuration settings that are defined to a -*non-default* value, -along with a couple of important ones that are always shown (like `buildpath`, `installpath`, `sourcepath`, and so on). -In addition, it also indicates at which configuration level each setting was defined, -so you can trace down *where* it was defined if needed. - -This is the output produced by `eb --show-config` for the default EasyBuild configuration, -where EasyBuild was installed via `pip install --user` (which results in the value shown for the -`robot-paths` configuration setting): - -```shell -# -# Current EasyBuild configuration -# (C: command line argument, D: default value, E: environment variable, F: configuration file) -# -buildpath (D) = /home/example/.local/easybuild/build -containerpath (D) = /home/example/.local/easybuild/containers -installpath (D) = /home/example/.local/easybuild -repositorypath (D) = /home/example/.local/easybuild/ebfiles_repo -robot-paths (D) = /home/example/.local/easybuild/easyconfigs -sourcepath (D) = /home/example/.local/easybuild/sources -``` - -As shown here, all configuration settings shown follow the default `prefix` value (`$HOME/.local/easybuild`), -and none of the values diverge from the default value, since all entries are marked with `(D)` for "default value"). - -**Example** - -Now let us do some basic configuring and inspect the resulting output of `--show-config`. - -First, create a user-level EasyBuild configuration file to define the `prefix` configuration setting: - -```shell -mkdir -p $HOME/.config/easybuild -echo '[config]' > $HOME/.config/easybuild/config.cfg -echo 'prefix=/apps' >> $HOME/.config/easybuild/config.cfg -``` - -In addition, define the `buildpath` configuration setting using the corresponding -environment variable: - -```shell -export EASYBUILD_BUILDPATH=/tmp/$USER -``` - -Then run `--show-config` while you specify that the `installpath` configuration -setting should be defined as `/tmp/$USER`: - -```shell -$ eb --installpath=/tmp/$USER --show-config -# -# Current EasyBuild configuration -# (C: command line argument, D: default value, E: environment variable, F: configuration file) -# -buildpath (E) = /tmp/easybuild -containerpath (F) = /apps/containers -installpath (C) = /tmp/easybuild -packagepath (F) = /apps/packages -prefix (F) = /apps -repositorypath (F) = /apps/ebfiles_repo -robot-paths (D) = /home/example/.local/easybuild/easyconfigs -sourcepath (F) = /apps/sources -``` - -The output indicates that the `installpath` setting was specified through a command line option (indicated -with `(C)`), that the `buildpath` setting was defined via an environment -variable (indicated with `(E)`), that the `robot-paths` setting still has the default value (indicated with `(D)`), and that all other configuration -settings were specified via a configuration file, some of which indirectly through the `prefix` value (indicated with -`(F)`). - ---- - -## Exercises (optional) - -*These exercises are not very relevant for LUMI as LUMI already offers a complete configuration -also for user installations of software. However, if you are a very advanced user, you may still -want to make changes to that configuration, and all three options (configuration files, -environment variables and command line parameters) are available to users in the LUMI setup.* - -***Exercise C.1* - Configure EasyBuild** - -Configure EasyBuild to use the `easybuild` subdirectory in your home directory for everything, except for: - -* The location of the build directories: use `/tmp/$USER` for this; -* The locations that should be considered when searching for source files: - include both `$HOME/easybuild/sources` and `/easybuild/sources`, but make - sure that source files that are downloaded by EasyBuild are stored in - `$HOME/easybuild/sources`. - -Leave other configuration settings set to their default value. - -??? success "(click to show solution)" - - This is pretty straightforward. - - Here we just define the corresponding environment variables: - - ```shell - export EASYBUILD_PREFIX=$HOME/easybuild - export EASYBUILD_BUILDPATH=/tmp/$USER - export EASYBUILD_SOURCEPATH=$HOME/easybuild/sources:/easybuild/sources - ``` - - The location where EasyBuild should download source files to - must be listed first in the `sourcepath` configuration setting. - - The output of `--show-config` should look like this: - - ```shell - buildpath (E) = /tmp/example - containerpath (E) = /home/example/easybuild/containers - installpath (E) = /home/example/easybuild - packagepath (E) = /home/example/easybuild/packages - prefix (E) = /home/example/easybuild - repositorypath (E) = /home/example/easybuild/ebfiles_repo - robot-paths (D) = /home/example/easybuild/easyconfigs - sourcepath (E) = /home/example/easybuild/sources:/easybuild/sources - ``` - ---- - -***Exercise C.2* - Install a trivial software package with EasyBuild** - -Try running the following command: - -```shell -eb bzip2-1.0.6.eb -``` - -Where do you expect to find the installation? - -??? success "(click to show solution)" - - The software was installed in `$HOME/easybuild`, - since that's how we configured EasyBuild in *Exercise 3.1*: - - ```shell - $ ls $HOME/easybuild - ebfiles_repo modules software sources - ``` - - The actual installation is in `$HOME/easybuild/software`, - while the module file was generated in `$HOME/easybuild/modules/all`: - - ```shell - $ ls $HOME/easybuild/software - bzip2 - $ ls $HOME/easybuild/software/bzip2 - 1.0.6 - $ ls $HOME/easybuild/software/bzip2/1.0.6 - bin easybuild include lib man - ``` - - ```shell - $ ls $HOME/easybuild/modules/all - bzip2 - $ ls $HOME/easybuild/modules/all/bzip2 - 1.0.6.lua - ``` - - The source file for bzip2 1.0.6 was downloaded to `$HOME/easybuild/sources`: - - ```shell - $ ls $HOME/easybuild/sources/b/bzip2 - bzip2-1.0.6.tar.gz - ``` - - We will discuss this in more detail in the next part of the tutorial. - ---- - -*[[next: Basic usage]](1_08_basic_usage.md)* diff --git a/docs/2022-CSC_and_LO/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_08_basic_usage.md deleted file mode 100644 index f5031e48..00000000 --- a/docs/2022-CSC_and_LO/1_08_basic_usage.md +++ /dev/null @@ -1,1216 +0,0 @@ -# Basic usage - -*[[back: Configuring EasyBuild]](1_07_configuration.md)* - ---- - -Now that we have installed and configured EasyBuild, we can start using it for what it is intended for: -getting scientific software installed without breaking a sweat, or having to resist the urge to -shout out four-letter words. - -We will look at the high-level workflow first, and then cover each aspect in more detail. - -A couple of exercises are available at the end to help to make you more familiar with the -EasyBuild command line interface, so pay attention! - ---- - -## Overall workflow - -Installing software with EasyBuild is as easy as specifying to the **`eb` command** what we -want to install, and then sitting back to enjoy a coffee or tea (or whatever beverage you prefer). - -This is typically done by **specifying the name of one or more easyconfig files**, often combined -with the `--robot` (or `-r`) option to let EasyBuild also install missing dependencies. - -It is recommended to first assess the current situation before letting EasyBuild install the software, -and to check which **dependencies** are already installed and which are still missing. In addition, -you may want to inspect the specifics of the **installation procedure** that will be performed by EasyBuild, -and ensure that the active EasyBuild configuration is what it should be. - ---- - -## Specifying easyconfigs - -Letting EasyBuild know what should be installed can be done by specifying one or more easyconfig files, -which is also the most common way. Alternative methods like using the `--software-name` option won't be -covered in this tutorial, since they are not commonly used. - -Arguments passed to the `eb` command, being anything that is *not* an option (which starts with `-` or `--`) or -is a value for a preceding configuration option, are assumed to refer to easyconfig files (with some exceptions). -These could be: - -* the *(absolute or relative) path* to an easyconfig file; -* the *name* of an easyconfig file; -* the path to a *directory* containing easyconfig files; - -Specified paths to files must of course point to existing files; if not, EasyBuild will print an appropriate error message: - -```shell -$ eb /tmp/does_not_exist.eb -ERROR: Can't find path /tmp/does_not_exist.eb -``` - -When only the *name* of an easyconfig file is specified, EasyBuild will automatically try and locate it. -First, it will consider the *current directory*. If no file with the specified name is found there, -EasyBuild will search for the easyconfig file in the [robot search path](../1_07_configuration/#robot-search-path). - -If the path to an existing *directory* is provided, EasyBuild will walk through the entire directory -(including all subdirectories), retain all files of which the name ends with `.eb`, and (try to) use these -as easyconfig files. - - -#### Example command - -Suppose we have the current situation in our home directory: - -* two (easyconfig) files named `example1.eb` and `example2.eb`; -* a subdirectory named `some_deps`, which has two easyconfig files `dep1.eb` and `dep2.eb` - alongside a text file named `list.txt`; -* a subdirectory named `more_deps` located *in* the `some_deps` subdirectory, - which contains another easyconfig file `dep3.eb`; - -Or, visually represented: - -```shell -example1.eb -example2.eb -some_deps -├── deb2.eb -├── dep1.eb -├── list.txt -└── more_deps - └── dep3.eb -``` - -In this context, we run the following EasyBuild command from our home directory: - -```shell -eb example1.eb bzip2-1.0.6.eb $HOME/example2.eb some_deps -``` - -EasyBuild will interpret each of these arguments as follows: - -* `example1.eb` is the name of a file in the current directory, so it can be used directly; -* `bzip2-1.0.6.eb` is the name of an easyconfig file to locate via the robot search path - (since it does not exist in the current directory); -* `$HOME/example2.eb` specifies the full path to an existing file, which can be used directly; -* `some_deps` is the relative path to an existing directory, so EasyBuild will scan it and find three - easyconfig files: `some_deps/dep1.eb`, `some_deps/dep2.eb` and `some_deps/more_deps/dep3.eb`, - The `list.txt` file will be ignored since its name does not end with `.eb`. - ---- - -### Easyconfig filenames - -Note that the `eb` command does not care how easyconfig files are named, at least to some extent: the `.eb` -file extension *does* matter when easyconfig files are being picked up in subdirectories. - -File names for easyconfigs being mostly irrelevant is only correct with respect to the arguments passed to -the `eb` command however. As we will learn soon, the name of easyconfig files *does* matter (a lot) -when EasyBuild needs to locate easyconfigs that can be used to resolve a specified dependency -(see [here](#enabling-dependency-resolution)). - -This explains why easyconfig files usually adher to a very specific naming scheme, -corresponding to `--.eb`, where: - -* `` represents the software name; -* `` represents the software version; -* `` represents the toolchain used in the easyconfig file, which consists of the toolchain name - and version separated with a dash (`-`), and which is omitted (including the preceding `-`) when the - [`system` toolchain](../1_05_terminology#system-toolchain) is used; -* `` represents the value of the `versionsuffix` easyconfig parameter, - which is sometimes used to distinguish multiple variants of particular software installations - (and is empty by default); - ---- - -### Searching for easyconfigs - -You will frequently need to determine the exact name of an easyconfig file you want to install, -or just check which easyconfigs are available for a given software package. -This can be done by searching for easyconfigs using **`eb --search`** or **`eb -S`**. - -By default all directories listed in the [robot search path](../1_07_configuration#robot-search-path) will be -searched. If you want to search in additional directories without changing the robot search path, -you can use the `search-paths` configuration setting, or you can change the robot search path via either -the `robot` or `robot-paths` configuration options. - -Both the `--search` and `-S` options trigger the same search operation, but yield different output: -`eb --search` will print the full path to each easyconfig file that matches the specified search pattern, -while `eb -S` produces a more concise output. - -For example, let's check which easyconfig files are available for OpenFOAM 8 with a `foss` toolchain: - -```shell -$ eb --search openfoam-8-foss - * /home/example/.local/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-8-foss-2020a.eb - * /home/example/.local/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-8-foss-2020b.eb -``` - -The output is a bit more condensed when using `eb -S`: - -```shell -$ eb -S openfoam-8-foss -CFGS1=/home/example/.local/easybuild/easyconfigs/o/OpenFOAM - * $CFGS1/OpenFOAM-8-foss-2020a.eb - * $CFGS1/OpenFOAM-8-foss-2020b.eb -``` - -Note that the search is performed *case-insensitive*. - -The search pattern can include wildcards like `.*` and/or character groups like `[0-9]`, -or other special characters like `^` or `$` to mark the start/end of the filename, -but you need to be careful that `bash` does not expand these before the `eb` command is started, -so it is recommended to wrap the search pattern in single quotes (`'...'`) when using wildcards. - -For example, to check which easyconfigs are available to install TensorFlow 2.4.1 with the `2020b` version of a toolchain: - -```shell -$ eb -S '^tensorflow-2.4.1.*2020b' -CFGS1=/home/example/.local/easybuild/easyconfigs/t/TensorFlow - * $CFGS1/TensorFlow-2.4.1-foss-2020b.eb - * $CFGS1/TensorFlow-2.4.1-fosscuda-2020b.eb -``` - ---- - -#### Search index - -When searching for easyconfig files, you may see a message like this pop up: - -``` -== found valid index for , so using it... -``` - -This indicates that a **search index** was used for this particular directory which significantly speeds -up the search procedure, especially when the easyconfig files are located on a shared filesystem where metadata -operations involving lots of (small) files can be slow. - -For the easyconfig files included with an EasyBuild release, a search index is readily provided. -For other directories, you can create a search index using `eb --create-index `. -[See the EasyBuild documentation for more information](https://easybuild.readthedocs.io/en/latest/Easyconfigs_index.html). - ---- - -## Inspecting easyconfigs - -Once you have determined the name of the easyconfig file that corresponds to the software you want to install, -you may want to take a closer look at its contents before employing it. - -Since easyconfig files are simple text files (in Python syntax), you could use the ubiquitous `cat` command -or your favorite text editor (`vim`, what else). To avoid that you need to locate the easyconfig file first -and copy-paste the full path to it, you can use **`eb --show-ec`**. - -!!! Hint - To follow the examples below on LUMI, load ``LUMI/21.12`` and ``EasyBuild-user`` - (though results may differ or the examples not work anymore as the software installation - on LUMI evolves). - -For example, let's inspect the contents of the `bzip2-1.0.6.eb` easyconfig file: - -```shell -$ eb --show-ec bzip2-1.0.8-cpeCray-21.12.eb -== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-53o823qb/easybuild-xn6nmt61.log -== Contents of /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeCray-21.12.eb: -# Contributed by Kurt Lust, LUMI project & UAntwerpen - -local_bzip2_version = '1.0.8' # http://www.bzip.org/downloads.html - -name = 'bzip2' -version = local_bzip2_version - -homepage = 'https://www.sourceware.org/bzip2/' - -whatis = [ - 'Description: bzip2 is a freely available, patent free, high-quality data compressor.', - 'The module contains both executables and libraries.' - 'Keywords: BZ2', -] - -description = """ -bzip2 is a freely available, patent free, high-quality data compressor. It -typically compresses files to within 10% to 15% of the best available techniques -(the PPM family of statistical compressors), whilst being around twice as fast -at compression and six times faster at decompression. It is based on the -Burrows-Wheeler block-sorting text compression algorithm and Huffman coding. -""" - -usage = """ -Check the man pages for the available commands or the web-based documentation for the -library functions. -""" - -docurls = [ - 'Web-based documentation: http://www.bzip.org/docs.html', - 'Man pages available for bzcmp, bzdiff, bzegrep, bzfgrep, bzgrep, bzip2, bunzip2, bzless and bzmore', -] - -toolchain = {'name': 'cpeCray', 'version': '21.12'} -toolchainopts = {'pic': True} - -source_urls = ['https://sourceware.org/pub/%(name)s/'] -sources = [SOURCE_TAR_GZ] -patches = ['bzip2-%(version)s-pkgconfig-manpath.patch'] -checksums = [ - 'ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269', # bzip2-1.0.8.tar.gz - 'de11269dc6e4917023af4cee9ff83b204953ad9cde561dbc9d6fc70d9b9578e3', # bzip2-1.0.8-pkgconfig-manpath.patch -] - -builddependencies = [ # Create a reproducible build environment. - ('buildtools', '%(toolchain_version)s', '', True), -] - -local_bzip2_major_minor = '.'.join(local_bzip2_version.split('.')[:2]) - -sanity_check_paths = { - 'files': [ 'lib/pkgconfig/bzip2.pc', 'lib/libbz2.a', 'lib/libbz2.%s' % SHLIB_EXT, - 'lib/libbz2.%s.%s' % (SHLIB_EXT, local_bzip2_major_minor), - 'lib/libbz2.%s.%s' % (SHLIB_EXT, local_bzip2_version), - 'include/bzlib.h' ] + - [ 'bin/b%s' % x for x in ['unzip2', 'zcat', 'zdiff', 'zgrep', 'zip2', 'zip2recover', 'zmore'] ] + - [ 'share/man/man1/bz%s.1' % x for x in ['cmp', 'diff', 'egrep', 'fgrep', 'grep', 'ip2', 'less', 'more'] ], - 'dirs': [] -} - -sanity_check_commands = [ - 'bzip2 --help', - 'pkg-config --libs bzip2', -] - -moduleclass = 'tools' - -== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-53o823qb/easybuild-xn6nmt61.log* have been removed. -== Temporary directory /run/user/10012026/easybuild/tmp/eb-53o823qb has been removed. -``` -The output may actually be longer for an easyconfig file that is already installed on the system -as a new easyconfig file is generated in the repository with some information about the installation -added to it and as on LUMI these are at the front of the robot search path to ensure that the system -finds the right easyconfig file matching with a module on the system. - -We'll get back to what all of this means later... - ---- - -## Checking dependencies - -!!! note - In some of the examples below, we assume that some software is already installed with EasyBuild. - - If you are following hands-on in a prepared environment, make sure these installations are visible - in the output of "`module avail`". - - -Before kicking off an installation, it is good practice to check which of the required dependencies -are already installed, and which ones are still missing. - -This can be helpful to ensure that your EasyBuild configuration is set up correctly, -and to prevent from accidentally installing an entirely new software stack from scratch -in an unintended location (like `$HOME/.local/easybuild`, the -[default installation path](../configuration/#install-path)). - ---- - -### Dry run - -To get a complete overview of all required dependencies, and see which ones are already installed -and which ones aren't yet, you can use **`eb --dry-run`**. - -Since `--dry-run` produces rather verbose output including the full path to each easyconfig file, -there is a more concise equivalent available as well: `eb --dry-run-short`, which is equivalent with **`eb -D`**. - -For example, to check which of the required dependencies for `SAMtools-1.11-GCC-10.2.0.eb` are already installed: - -```shell -$ eb SAMtools-1.14-cpeGNU-21.12.eb -D -== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-oo0lj9lq/easybuild-2cyomy8v.log -Dry run: printing build status of easyconfigs and dependencies -CFGS=/appl/lumi - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeGNU/cpeGNU-21.12.eb (module: cpeGNU/21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ncurses/ncurses-6.2-cpeGNU-21.12.eb (module: ncurses/6.2-cpeGNU-21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/zlib/zlib-1.2.11-cpeGNU-21.12.eb (module: zlib/1.2.11-cpeGNU-21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeGNU-21.12.eb (module: bzip2/1.0.8-cpeGNU-21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12-minimal.eb (module: gettext/0.21-cpeGNU-21.12-minimal) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/XZ/XZ-5.2.5-cpeGNU-21.12.eb (module: XZ/5.2.5-cpeGNU-21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/Brotli/Brotli-1.0.9-cpeGNU-21.12.eb (module: Brotli/1.0.9-cpeGNU-21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cURL/cURL-7.78.0-cpeGNU-21.12.eb (module: cURL/7.78.0-cpeGNU-21.12) - * [ ] $CFGS/LUMI-EasyBuild-contrib/easybuild/easyconfigs/h/HTSlib/HTSlib-1.14-cpeGNU-21.12.eb (module: HTSlib/1.14-cpeGNU-21.12) - * [ ] $CFGS/LUMI-EasyBuild-contrib/easybuild/easyconfigs/s/SAMtools/SAMtools-1.14-cpeGNU-21.12.eb (module: SAMtools/1.14-cpeGNU-21.12) -== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-oo0lj9lq/easybuild-2cyomy8v.log* have been removed. -== Temporary directory /run/user/10012026/easybuild/tmp/eb-oo0lj9lq has been removed. -``` - -This output tells you that most of the dependencies required by ``SAMtools-1.14-cpeGNU-21.12.eb`` are -already installed, since they are marked with ``[x]``. However, the easyconfig files for -``HTSLib-1.14-cpeGNU-21.12.eb`` and SAMtools itself are not installed yet, denoted by the -lack of an ``x`` in ``[ ]``. - - ---- - -### Missing dependencies - -If you are only interested in which dependencies are still *missing*, -you can consult the output of **`eb --missing`**, or the equivalent **`eb -M`**. - -For example, for the SAMtools easyconfig file used in the previous example we get (with -some lines removed from the output): - -```shell -$ eb SAMtools-1.14-cpeGNU-21.12.eb -M -2 out of 11 required modules missing: - -* HTSlib/1.14-cpeGNU-21.12 (HTSlib-1.14-cpeGNU-21.12.eb) -* SAMtools/1.14-cpeGNU-21.12 (SAMtools-1.14-cpeGNU-21.12.eb) -``` - -That should be pretty self-explanatory: out of the 113 required dependencies (which includes the `cpeGNU` toolchain -and everything needed to install it), only 2 dependencies (including SAMtools itself) are missing. Great! - ---- - -## Inspecting install procedures - -Finally, before actually installing something you may want to assess *how* exactly EasyBuild is going -to install the software. - -Perhaps you don't trust EasyBuild yet (you will eventually though, hopefully), -or maybe you just want to double check that you have made the right choice before going through with -the actual installation. - -Using **`eb --extended-dry-run`**, or just **`eb -x`** for short, -you can get a **detailed overview of the installation procedure that would be performed by EasyBuild**, -**in a matter of seconds**. - -By means of example, let's inspect some parts of the installation procedure for ``HTSLib-1.14-cpeGNU-21.12.eb``: - -```shell -$ eb HTSlib-1.14-cpeGNU-21.12.eb -x -... - -*** DRY RUN using 'ConfigureMake' easyblock (easybuild.easyblocks.generic.configuremake @ /appl/lumi/SW/LUMI-21.12/common/EB/EasyBuild/4.5.3/lib/python3.6/site-packages/easybuild/easyblocks/generic/configuremake.py) *** - -== building and installing HTSlib/1.14-cpeGNU-21.12... -fetching files... [DRY RUN] - -... - -[prepare_step method] -Defining build environment, based on toolchain (options) and specified dependencies... - -Loading toolchain module... - -module load cpeGNU/21.12 - -Loading modules for dependencies... - -module load buildtools/21.12 -module load zlib/1.2.11-cpeGNU-21.12 -module load bzip2/1.0.8-cpeGNU-21.12 -module load XZ/5.2.5-cpeGNU-21.12 -module load cURL/7.78.0-cpeGNU-21.12 - -... - -Defining build environment... - -... - - export CC='cc' - export CFLAGS='-O2 -ftree-vectorize -fno-math-errno' - -... - -configuring... [DRY RUN] - -[configure_step method] - running command "./configure --prefix=/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/HTSlib/1.14-cpeGNU-21.12" - (in /run/user/10012026/easybuild/build/HTSlib/1.14/cpeGNU-21.12/HTSlib-1.14) - -building... [DRY RUN] - -[build_step method] - running command "make -j 256" - (in /run/user/10012026/easybuild/build/HTSlib/1.14/cpeGNU-21.12/HTSlib-1.14) - -testing... [DRY RUN] - -[test_step method] - -installing... [DRY RUN] - -... - -sanity checking... [DRY RUN] - -[sanity_check_step method] -Sanity check paths - file ['files'] - * bin/bgzip - * bin/tabix - * lib/libhts.so -Sanity check paths - (non-empty) directory ['dirs'] - * include -Sanity check commands - * bgzip --version - * htsfile --version - * tabix --version - -... -``` - -We've obviously trimmed the generated output a bit, but it should be sufficient. - -An overview of the installation procedure is shown, following the installation steps as they would be -performed by EasyBuild. The output above shows: - -* how the build environment will be set up during the `prepare` step, by loading the module for both the - toolchains and the dependencies, and defining a set of environment variables like `$CC`, `$CFLAGS`, etc. -* which command will be executed during the configuration step, and in which directory; -* the list of files and directories that will be checked during the sanity check step; - -If you were concerned about EasyBuild being too much of a black box, that is hopefully resolved now. - -!!! note - It is important to highlight here that the reported installation procedure *may* not be 100% correct, - since the [easyblock](1_05_terminology.md#easyblocks) can change its mind based on the output of shell commands - that were executed, or based on the contents of a file that was generated during the installation. - Since all "actions" that would be performed during the installation are actually skipped when using `eb -x`, - the reported installation procedure could be partially incorrect. - - In addition, the easyblock may trip over the fact that the installation procedure is not actually being - executed, which sometimes leads to an unexpected error. These situations are handled gracefully by `eb -x` - though, which will report any errors that occurred and then continue anyway with inspecting the remainder of the - installation procedure. Although this obviously limits the value of the generated output, - it doesn't make it completely useless. - ---- - -## Installing software - -You should now be able to make an informed decision on which easyconfig file you should use to -make EasyBuild install the software you require. - -As mentioned before, installing an easyconfig is as simple as passing it to the `eb` command. - -So, let's try to install libdap version 3.20.9 with the cpeGNU/21.12 toolchain. - -Let's first check if it has any dependencies that still need to be installed: - -```shell -$ eb libdap-3.20.9-cpeGNU-21.12.eb -D -== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-wm_bk3j6/easybuild-puyu_559.log -Dry run: printing build status of easyconfigs and dependencies - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeGNU/cpeGNU-21.12.eb (module: cpeGNU/21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/zlib/zlib-1.2.11-cpeGNU-21.12.eb (module: zlib/1.2.11-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libtirpc/libtirpc-1.3.2-cpeGNU-21.12.eb (module: libtirpc/1.3.2-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/Brotli/Brotli-1.0.9-cpeGNU-21.12.eb (module: Brotli/1.0.9-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ICU/ICU-69.1-cpeGNU-21.12.eb (module: ICU/69.1-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cURL/cURL-7.78.0-cpeGNU-21.12.eb (module: cURL/7.78.0-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeGNU-21.12.eb (module: bzip2/1.0.8-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ncurses/ncurses-6.2-cpeGNU-21.12.eb (module: ncurses/6.2-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libreadline/libreadline-8.1-cpeGNU-21.12.eb (module: libreadline/8.1-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12-minimal.eb (module: gettext/0.21-cpeGNU-21.12-minimal) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/PCRE2/PCRE2-10.37-cpeGNU-21.12.eb (module: PCRE2/10.37-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/XZ/XZ-5.2.5-cpeGNU-21.12.eb (module: XZ/5.2.5-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libxml2/libxml2-2.9.12-cpeGNU-21.12.eb (module: libxml2/2.9.12-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libxslt/libxslt-1.1.34-cpeGNU-21.12.eb (module: libxslt/1.1.34-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12.eb (module: gettext/0.21-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/file/file-5.41-cpeGNU-21.12.eb (module: file/5.41-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/util-linux/util-linux-2.37.1-cpeGNU-21.12.eb (module: util-linux/2.37.1-cpeGNU-21.12) - * [ ] /pfs/lustrep3/users/kurtlust/LUMI/LUMI-EasyBuild-contrib/easybuild/easyconfigs/l/libdap/libdap-3.20.9-cpeGNU-21.12.eb (module: libdap/3.20.9-cpeGNU-21.12) -== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-wm_bk3j6/easybuild-puyu_559.log* have been removed. -== Temporary directory /run/user/10012026/easybuild/tmp/eb-wm_bk3j6 has been removed. -``` - -and now instal the library: - -```shell -$ eb libdap-3.20.9-cpeGNU-21.12.eb -== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-kfphjoi8/easybuild-kcs00ai5.log -== processing EasyBuild easyconfig -/pfs/lustrep3/users/kurtlust/LUMI/LUMI-EasyBuild-contrib/easybuild/easyconfigs/l/libdap/libdap-3.20.9-cpeGNU-21.12.eb -== building and installing libdap/3.20.9-cpeGNU-21.12... -== fetching files... -== ... (took 2 secs) -== creating build dir, resetting environment... -== unpacking... -== patching... -== preparing... -== ... (took 6 secs) -== configuring... -== ... (took 1 min 6 secs) -== building... -== ... (took 53 secs) -== testing... -== installing... -== ... (took 5 secs) -== taking care of extensions... -== restore after iterating... -== postprocessing... -== sanity checking... -== ... (took 3 secs) -== cleaning up... -== creating module... -== ... (took 2 secs) -== permissions... -== packaging... -== COMPLETED: Installation ended successfully (took 2 mins 20 secs) -== Results of the build can be found in the log file(s) -/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/libdap/3.20.9-cpeGNU-21.12/easybuild/easybuild-libdap-3.20.9-20220329.154535.log -== Build succeeded for 1 out of 1 -== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-kfphjoi8/easybuild-kcs00ai5.log* have been removed. -== Temporary directory /run/user/10012026/easybuild/tmp/eb-kfphjoi8 has been removed. -``` - -That was... easy. Is that really all there is to it? Well, almost... - -### Enabling dependency resolution - -The libdap installation worked like a charm, but remember that all required dependencies were already -available (see [above](#dry-run)). - -If we try this with the `SAMtools-1.14-cpeGNU-21.12.eb`, for which the required `HTSlib` dependencies is not available yet, it's less successful: - -```shell -$ eb SAMtools-1.14-cpeGNU-21.12.eb -M - -2 out of 11 required modules missing: - -* HTSlib/1.14-cpeGNU-21.12 (HTSlib-1.14-cpeGNU-21.12.eb) -* SAMtools/1.14-cpeGNU-21.12 (SAMtools-1.14-cpeGNU-21.12.eb) -``` - -```shell -$ eb SAMtools-1.14-cpeGNU-21.12.eb -... -== preparing... -== FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12): build failed -(first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12 (took 3 secs) -== Results of the build can be found in the log file(s) -/run/user/10012026/easybuild/tmp/eb-rgj1v43y/easybuild-SAMtools-1.14-20220329.155911.ZtDcX.log - -ERROR: Build of /appl/lumi/LUMI-EasyBuild-contrib/easybuild/easyconfigs/s/SAMtools/SAMtools-1.14-cpeGNU-21.12.eb failed (err: 'build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12') -``` - -Oh my, what's this all about? - -If we filter the output a bit and focus on the actual error, the problem is clear: - -``` -Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12') -``` - -The required dependency `HTSlib/1.14-cpeGNU-21.12` is not installed yet, -and EasyBuild does not automatically install missing dependencies unless it is told to do so -(which we didn't do in the configuration for LUMI). - -It helpfully suggests to use the `--robot` (or '-r') command line option, so let's try that: - -```shell -$ eb SAMtools-1.14-cpeGNU-21.12.eb --robot -... -== resolving dependencies ... -... -== building and installing HTSlib/1.14-cpeGNU-21.12... -... -== COMPLETED: Installation ended successfully (took 13 sec) -... -== building and installing SAMtools/1.14-cpeGNU-21.12... -... -== COMPLETED: Installation ended successfully (took 8 sec) -... -== Build succeeded for 2 out of 2 -``` - -With dependency resolution enabled the `HTSlib` module gets installed first, -before EasyBuild proceeds with installing `SAMtools`. Great! - ---- - -### Trace output - -As you may have noticed if you tried the previous example hands-on, -some installations take a while. -An installation can be spending quite a bit of time during the build step, but what is actually going on there? - -To provide some more feedback as the installation progresses, you can enable the "`trace`" configuration setting. -Let's do this by defining the `$EASYBUILD_TRACE` environment variable, just to avoid having to type `--trace` -over and over again. - -We will redo the installation of `SAMtools-1.14-cpeGNU-21.12.eb` by passing the `--rebuild` -option to the `eb` command (try yourself what happens if you don't use the `--rebuild` option!): - -```shell -$ export EASYBUILD_TRACE=1 -$ eb SAMtools-1.14-cpeGNU-21.12.eb --rebuild -... -== configuring... - >> running command: - [started at: 2022-03-29 18:46:31] - [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] - [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-g7vd83qv.log] - /users/kurtlust/LUMI-user-appl/sources/generic/eb_v4.5.3/ConfigureMake/config.guess - >> command completed: exit 0, ran in < 1s - >> running command: - [started at: 2022-03-29 18:46:31] - [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] - [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-k0etfv8i.log] - ./configure --prefix=/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/SAMtools/1.14-cpeGNU-21.12 --build=x86_64-pc-linux-gnu ---host=x86_64-pc-linux-gnu --with-htslib=$EBROOTHTSLIB - >> command completed: exit 0, ran in 00h00m03s -== ... (took 3 secs) -== building... - >> running command: - [started at: 2022-03-29 18:46:34] - [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] - [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-svcps0yj.log] - make -j 256 CC="cc" CXX="CC" CFLAGS="-O2 -ftree-vectorize -fno-math-errno -fPIC" CXXFLAGS="-O2 -ftree-vectorize -fno-math-errno --fPIC" - >> command completed: exit 0, ran in 00h00m06s -== ... (took 6 secs) -``` - -That's a bit more comforting to stare at... - -SAMtools uses a custom easyblock that is derived from the generic `ConfigureMake` easyblock. -During the *configure* step, the `./configure` command is run with `--build` and `--host` -options added by the generic ConfigureMake easyblock (and the other command, `config.guess` plays -a role in determining the value of those flags). The `--with-htslib=$EBROOTHTSLIB` flag is added -via our easyconfig file to tell SAMtools to use an already available version of HTSlib rather -than the built-in one. - -During the *build* step, the software is actually being compiled -by running the `make` command. EasyBuild automatically uses the available cores on the system (in this case 256). - -We even get a pointer to a log file that contains the output of the command being run, -so we can use `tail -f` to see in detail how it progresses. - -Once the `make` command completes, we get a message that the command completed with a exit code 0 -(implying success), and that it took 3 seconds to run. That's good to know. - -Later during the installation, we now also see this output during the sanity check step: - -``` -== sanity checking... - >> file 'bin/blast2sam.pl' found: OK - >> file 'bin/bowtie2sam.pl' found: OK - >> file 'bin/export2sam.pl' found: OK - >> file 'bin/interpolate_sam.pl' found: OK -... - >> loading modules: SAMtools/1.14-cpeGNU-21.12... - >> running command 'samtools version' ... - >> result for command 'samtools version': OK -``` - -Thanks to enabling trace mode, EasyBuild tells us which files (& directories, but there are non in this case) -it is checking for -in the installation, and which command it is trying to run before declaring it a success. Nice! - -The extra output you get when trace mode is enabled is concise and hence not overwhelming, -while it gives a better insight into what is going on during the installation. -It may also help to spot unexpected actions being taken during the installation early on, -so you can interrupt the installation before it completes, if deemed necessary. - ---- - -## Using installed software - -So far, we have already installed 4 different software packages (SAMtools, HTSlib, libdap); -we even installed SAMtools twice! - -A lot was going on underneath the covers: locating and unpacking -the source tarballs, setting up the build environment, configuring the build, compiling, -creating and populating the installation directory, performing a quick sanity check on the installation, -cleaning things up, and finally generating the environment module file corresponding to the installation. - -That's great, but how do we now actually *use* these installations? - -This is where the generated module files come into play: they form the access portal to the software -installations, and we'll use the ubiquitous `module` command to digest them. - -First, we need to make sure that the modules tool is aware of where the module files for -our installations are located. On LUMI, when using the EasyBuild-user module to configure EasyBuild, -everything is taken care of for you and the LUMI modules will also automatically add the -suitable module directories for user-installed software to the search path for modules. -By default, EasyBuild-config will install in `$HOME/EasyBuild`, but it is possible to -build the installation in a different directory by pointing to it with the environment -variable `EBU_USER_PREFIX`. Of course this variable needs to be set before loading the `LUMI` -module. (Note that one reason why we don't load a software stack by default is that in -the current setup of LUMI this module would be loaded before the user gets the chance to set -that environment variable in `.bash_profile` or `.bashrc`.) - -When loading the `EasyBuild-user` module, the module command will show you were EasyBuild -will install the software and put the modules, and also put its repository of -processed easyconfig file. - -```shell -ml EasyBuild-user - -EasyBuild configured to install software from the LUMI/21.12 software stack for the LUMI/L -partition in the user tree at /users/kurtlust/LUMI-user-appl. - * Software installation directory: /users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L - * Modules installation directory: /users/kurtlust/LUMI-user-appl/modules/LUMI/21.12/partition/L - * Repository: /users/kurtlust/LUMI-user-appl/ebrepo_files/LUMI-21.12/LUMI-L - * Work directory for builds and logs: /run/user/10012026/easybuild - Clear work directory with clear-eb -``` - -EasyBuild will copy each easyconfig file it installs to -the repository and add some lines to it with information about the installation. It -also has some options that may edit the source easyconfig, e.g., when asking EasyBuild -to try to build with another toolchain. - -You can always check where EasyBuild is installing stuff by checking the output of -`eb --show-config`. - -If you're unsure where EasyBuild is installing stuff at this point, -check the output of `eb --show-config`; the value of the `installpath` configuration setting is what we are interested in now: - -```shell -$ eb --show-config -... -buildpath (E) = /run/user/XXXXXXXX/easybuild/build -... -installpath-modules (E) = /users/XXXXXXXX/LUMI-user/modules/LUMI/21.12/partition/L -installpath-software (E) = /users/XXXXXXXX/LUMI-user/SW/LUMI-21.12/L... -... -repositorypath (E) = /users/XXXXXXXX/LUMI-user/ebrepo_files/LUMI-21.12/LUMI-L -... -sourcepath (E) = /users/XXXXXXXX/LUMI-user/sources:/appl/lumi/sources/easybuild -... -``` - -This is sligthly different from the default EasyBuild setup, where the modules, software, -repository and sources would be installed in respectively the subdirectories `modules`, -`software`, `ebfiles_repo` and `sources` of the directory pointed to by the `installpath` -line. - -The modules directory is also a simplified one from the standard EasyBuild one as that also -provides a module categorisation besides a directory containing all modules. As this categorisation -is largely arbitrary and hard to use in the module system, we decided simply not to use it in -our installation and use a custom naming scheme. - -However, if you would be using EasyBuild on another system with its default configuration, the -above setup would be used. For more information, we refer to the generic EasyBuild tutorials on the -[EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). - -Now the modules tool should be aware of our brand new installations: - -```shell -$ module avail -... --- EasyBuild managed user software for software stack LUMI/21.12 on LUMI-L --- - HTSlib/1.14-cpeGNU-21.12 libdap/3.20.9-cpeGNU-21.12 - SAMtools/1.14-cpeGNU-21.12 - ------ EasyBuild managed software for software stack LUMI/21.12 on LUMI-L ----- -... -``` - -This output shows both the modules for our own installations as well as the "central" installations -(which we omitted for brevity). - -Now we can load these modules and start using these software installations. - -Let's test this for SAMtools. In our current environment, the `samtools` command is not available yet: - -```shell -$ module list - -Currently Loaded Modules: - 1) perftools-base/21.12.0 - 2) cce/13.0.0 - 3) craype/2.7.13 - 4) cray-dsmml/0.2.2 - 5) cray-mpich/8.1.12 - 6) cray-libsci/21.08.1.2 - 7) PrgEnv-cray/8.2.0 - 8) ModuleLabel/label (S) - 9) init-lumi/0.1 (S) - 10) craype-x86-rome - 11) craype-accel-host - 12) libfabric/1.11.0.4.106 - 13) craype-network-ofi - 14) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta - 15) partition/L (S) - 16) LUMI/21.12 (S) - - Where: - S: Module is Sticky, requires --force to unload or purge -$ samtools --bash: samtools: command not found -``` - -Loading the module for SAMtools changes that: - -```shell -$ module load SAMtools/1.14-cpeGNU-21.12 - -Lmod is automatically replacing "cce/13.0.0" with "gcc/11.2.0". -Lmod is automatically replacing "PrgEnv-cray/8.2.0" with "cpeGNU/21.12". - -Due to MODULEPATH changes, the following have been reloaded: - 1) cray-mpich/8.1.12 - -$ module list - -Currently Loaded Modules: - 1) perftools-base/21.12.0 - 2) ModuleLabel/label (S) - 3) init-lumi/0.1 (S) - 4) craype-x86-rome - 5) craype-accel-host - 6) libfabric/1.11.0.4.106 - 7) craype-network-ofi - 8) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta - 9) partition/L (S) - 10) LUMI/21.12 (S) - 11) gcc/11.2.0 - 12) craype/2.7.13 - 13) cray-mpich/8.1.12 - 14) cray-libsci/21.08.1.2 - 15) cray-dsmml/0.2.2 - 16) cpeGNU/21.12 - 17) ncurses/6.2-cpeGNU-21.12 - 18) zlib/1.2.11-cpeGNU-21.12 - 19) bzip2/1.0.8-cpeGNU-21.12 - 20) gettext/0.21-cpeGNU-21.12-minimal - 21) XZ/5.2.5-cpeGNU-21.12 - 22) Brotli/1.0.9-cpeGNU-21.12 - 23) cURL/7.78.0-cpeGNU-21.12 - 24) HTSlib/1.14-cpeGNU-21.12 - 25) SAMtools/1.14-cpeGNU-21.12 - - Where: - S: Module is Sticky, requires --force to unload or purge - -$ samtools --version -samtools 1.14 -Using htslib 1.14 -Copyright (C) 2021 Genome Research Ltd.... -... -``` - -Note that the modules for the required dependencies, including the compiler toolchain (which provides runtime libraries -like `libstdc++.so`), are loaded automatically. The "`module load`" command changes the active environment, -by updating environment variables like `$PATH` for example, to make the software available for use. - -##### Resetting your environment - -To restore your environment to a pristine state in which no modules are loaded, you can either -unload the loaded modules one by one using "`module unload`", or you can unload all of them at once using -"`module purge`". On LUMI, `module purge` will unload all application modules but will not undo the selection -of the software stack. It will reset the software stack though to use those modules that fit best -with the hardware of the current node (i.e., you may find a different `partition` module). - -```shell -$ module purge -The following modules were not unloaded: - (Use "module --force purge" to unload all): - - 1) LUMI/21.12 6) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta - 2) craype-x86-rome 7) partition/L - 3) craype-accel-host 8) init-lumi/0.1 - 4) libfabric/1.11.0.4.106 9) ModuleLabel/label - 5) craype-network-ofi -$ module list - -Currently Loaded Modules: - 1) LUMI/21.12 (S) - 2) craype-x86-rome - 3) craype-accel-host - 4) libfabric/1.11.0.4.106 - 5) craype-network-ofi - 6) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta - 7) partition/L (S) - 8) init-lumi/0.1 (S) - 9) ModuleLabel/label (S) - - Where: - S: Module is Sticky, requires --force to unload or purge -``` - -Runnin `module --force purge` instead will remove all modules, including the `init-lumi` -module which does part of the initialisation. You will not be able to use the software -stacks completely as before without first loading `init-lumi` in its most recent (or default) -version again! - -```shell -$ module --force purge -$ module list -No modules loaded -``` - ---- - -## Stacking software - -Maybe you have overlooked how the software we are playing around with was not only installed across multiple -different installation directories per software, we are also "stacking" our own installations (in `$HOME/EasyBuild` -or `$EBU_USER_PREFIX`) on top of installations that are provided in a totally different location (`/appl/lumi`). - -**EasyBuild doesn't care *where* software is installed: as long as the module file that provides access to it -is available, it is happy to pick it up and use it when required.** - -This implies that end users of LUMI can easily install their -own small software stack on top of what is provided centrally by the LUMI User Support, -for example. They can even -"replace" a central software installation for their purposes if they need to, since the modules tool will -load the first module file that matches the request being made (there are some caveats with this, but we -won't go into those here). - ---- - -## Exercises - -TODO: Rework for LUMI and provide the necessary EasyConfig files. - -***Guidelines*** - -Do yourself a favor: don't peek at the solution until you have made an attempt to solve the exercise yourself! - -Please do not spoil solutions to others before they have been discussed by the tutorial organisers. - -The exercises are based on the easyconfig files included with EasyBuild 4.3.3. - ---- - -***Exercise U.0**** - Making installed software available* - -Before working on the exercises for this part of the tutorial, -make sure that the software that is already installed in the prepared environment is available. - -We will assume that you have a small software stack installed using the `2020b` version of the `foss` toolchain. - -**Tip:** execute a "`module use`" command, and verify with "`module avail`" that a bunch of software modules -are available for loading. - -??? success "(click to show solution)" - - Use the following command to make the modules for the software available that is pre-installed - in the prepared environment: - ```shell - module use /easybuild/modules/all - ``` - - If software is installed in a different location than `/easybuild/` in your environment, - you should adjust the command accordingly. - - ---- - -***Exercise U.1**** - Searching easyconfigs* - -See if EasyBuild provides any easyconfig files for installing GROMACS version 2020/5. - -??? success "(click to show solution)" - To check for available easyconfig files, we can use `eb --search` or `eb -S`: - ```shell - $ eb -S gromacs-2020.5 - == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... - CFGS1=/home/example/.local/easybuild/easyconfigs/g/GROMACS - * $CFGS1/GROMACS-2020.5-fosscuda-2020a-Python-3.8.2.eb - * $CFGS1/GROMACS-2020.5_fix_threads_gpu_Gmxapitests.patch - ``` - This actually shows one easyconfig file but also a patch file. We can also search specifically - for GROMACS 2020.5 in the `foss` and `fosscuda` toolchains using - ```shell - $ eb -S gromacs-2020.5-foss - == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... - CFGS1=/home/example/.local/easybuild/easyconfigs/g/GROMACS - * $CFGS1/GROMACS-2020.5-fosscuda-2020a-Python-3.8.2.eb - ``` - and now we find a single easyconfig file. - ---- - -***Exercise U.2**** - Checking dependencies* - -Check which dependencies are missing to install QuantumESPRESSO version 6.6 with the `2020b` version of the `foss` toolchain. - -??? success "(click to show solution)" - First, we need to determine the name of the easyconfig file for QuantumESPRESSO version 6.6: - ```shell - $ eb -S 'QuantumESPRESSO-6.6.*foss-2020b' - == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... - CFGS1=/home/example/.local/easybuild/easyconfigs/q/QuantumESPRESSO - * $CFGS1/QuantumESPRESSO-6.6-foss-2020b.eb - ``` - To determine which dependencies are missing to install this QuantumESPRESSO easyconfig file, we can use `--missing`: - ```shell - $ eb QuantumESPRESSO-6.6-foss-2020b.eb --missing - - 3 out of 58 required modules missing: - - * libxc/4.3.4-GCC-10.2.0 (libxc-4.3.4-GCC-10.2.0.eb) - * ELPA/2020.11.001-foss-2020b (ELPA-2020.11.001-foss-2020b.eb) - * QuantumESPRESSO/6.6-foss-2020b (QuantumESPRESSO-6.6-foss-2020b.eb) - ``` - (some nonessential output removed). - ---- - -***Exercise U.3**** - Performing a dry run* - -Figure out which command EasyBuild would use to compile -the software provided by the `Bowtie2-2.4.2-GCC-9.3.0.eb` easyconfig file, -without actually installing `Bowtie2`. - -Also, which binaries will EasyBuild check for to sanity check the installation? - -??? success "(click to show solution)" - To inspect the installation procedure, we can use `eb -x Bowtie2-2.4.2-GCC-9.3.0.eb`. - - The output for the build step shows the actual compilation command that would be performed (`make ...`): - - ```shell - [build_step method] - >> running command: - [started at: 2021-03-08 20:15:08] - [working dir: /local_scratch/hkenneth/eb-1wodfohg/__ROOT__/local_scratch/hkenneth/Bowtie2/2.4.2/GCC-9.3.0/Bowtie2-2.4.2] - [output logged in /local_scratch/hkenneth/eb-1wodfohg/easybuild-run_cmd-haojzisn.log] - make -j 48 CC="gcc" CPP="g++" CXX="g++" RELEASE_FLAGS="-O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -std=gnu++98" - (in /local_scratch/hkenneth/Bowtie2/2.4.2/GCC-9.3.0/Bowtie2-2.4.2) - ``` - - If the output you get is less detailed, you may not have set `export EASYBUILD_TRACE=1`. - - The output for the sanity check step shows which binaries are expected to be installed: - ``` - [sanity_check_step method] - Sanity check paths - file ['files'] - * bin/bowtie2 - * bin/bowtie2-align-l - * bin/bowtie2-align-s - * bin/bowtie2-build - * bin/bowtie2-build-l - * bin/bowtie2-build-s - * bin/bowtie2-inspect - * bin/bowtie2-inspect-l - * bin/bowtie2-inspect-s - ``` - ---- - -***Exercise U.4**** - Installing software* - -Install version 3.1.0 of the `h5py` Python package and all missing dependencies, -using the `foss/2020b` toolchain, into `/tmp/$USER/easybuild`, -while leveraging the already installed software available from `/easybuild`. - -Enable trace output so you can see which parts of the installation take a while. - -??? success "(click to show solution)" - First, determine the easyconfig file we can use for this: - ```shell - $ eb -S 'h5py-3.1.0.*foss-2020b' - CFGS1=/home/example/.local/easybuild/easyconfigs/h/h5py - * $CFGS1/h5py-3.1.0-foss-2020b.eb - ``` - - Make sure the pre-install software in `/easybuild/` is available: - ```shell - module use /easybuild/modules/all - ``` - - Check which dependencies are missing to install this `h5py` easyconfig: - ```shell - $ eb h5py-3.1.0-foss-2020b.eb --missing - - 2 out of 63 required modules missing: - - * pkgconfig/1.5.1-GCCcore-10.2.0-python (pkgconfig-1.5.1-GCCcore-10.2.0-python.eb) - * h5py/3.1.0-foss-2020b (h5py-3.1.0-foss-2020b.eb) - ``` - - Install `h5py` by specifying the easyconfig file and enabling dependency resolution via `--robot`, - while indicating that we want to install the software into `/tmp/$USER/easybuild` using the `--installpath` - option. Also make sure that trace mode is enabled by defining the `$EASYBUILD_TRACE` environment variable. - ```shell - $ export EASYBUILD_TRACE=1 - $ eb h5py-3.1.0-foss-2020b.eb --robot --installpath /tmp/$USER/easybuild - ... - == building and installing pkgconfig/1.5.1-GCCcore-10.2.0-python... - ... - == building and installing h5py/3.1.0-foss-2020b... - ... - == building... - >> running command: - [started at: 2020-06-10 21:47:32] - [working dir: /tmp/example/h5py/3.1.0/foss-2020b/h5py-3.1.0] - [output logged in /tmp/eb-rjjkbqe1/easybuild-run_cmd-d_dkc4iz.log] - python setup.py configure --mpi --hdf5=$EBROOTHDF5 && /easybuild/software/Python/3.8.6-GCCcore-10.2.0/bin/python setup.py build - >> command completed: exit 0, ran in 00h01m27s - ... - == COMPLETED: Installation ended successfully (took 2 min 46 sec) - ... - == Build succeeded for 2 out of 2 - ``` - - The trace output shows that most time is spent in the build command, - which runs both `python setup.py configure` and `python setup.py build`. - ---- - -***Exercise U.5**** - Using installed software* - -Using the `h5py` installation from the previous exercise to create an empty HDF5 file, -using the following Python statements: - -```python -import h5py -f = h5py.File("empty.hdf5", "w") -f.close() -``` - -Check the resulting file using the `h5stat` command. - -??? success "(click to show solution)" - First, we need to make the modules tool aware of the module files that were installed into `/tmp/$USER/easybuild`: - ```shell - module use /tmp/$USER/easybuild/modules/all - ``` - - Then we can check the `h5py` module is available, and load it: - ```shell - $ module avail h5py - ------------ /tmp/example/easybuild/modules/all ------------ - h5py/3.1.0-foss-2020b - ``` - - ```shell - module load h5py/3.1.0-foss-2020b - ``` - - The Python code snippet can be run directly on the command line using "`python -c '...'`", since it's tiny: - ```shell - python -c 'import h5py; f = h5py.File("empty.hdf5", "w"); f.close()' - ``` - Of course you can also copy the Python code snippet in a file named `test_h5py.py`, - and then run it with `python test_h5py.py`. - - Checking with the `h5stat` command shows that the resulting `empty.hdf5` is indeed a valid HDF5 file: - ```shell - $ ls -l empty.hdf5 - -rw-rw-r-- 1 example example 800 Jun 10 21:54 empty.hdf5 - - $ h5stat empty.hdf5 - Filename: empty.hdf5 - File information - # of unique groups: 1 - # of unique datasets: 0 - ... - ``` - ---- - -If you've made it through the hands-on exercises, congratulations! - -If not, don't worry too much about it. We covered a lot of ground here, -and it's a lot to take in at once, take your time... - ---- - -*[[next: Part 2: Using EasyBuild]](2_00_part2_using.md)* diff --git a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md index fa5075f5..d6ed3f61 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md @@ -225,7 +225,7 @@ sites leverage it, and integrate it with other tools like [Jenkins](https://www. HPC sites, consortia, and companies that use EasyBuild include (but are not limited to): -VSC logo +a href="https://www.vscentrum.be">VSC logo JSC logo HPCNow! logo Fred Hutch logo diff --git a/docs/2022-CSC_and_LO/2_00_part2_using.md b/docs/2022-CSC_and_LO/2_00_part2_using.md deleted file mode 100644 index c50036be..00000000 --- a/docs/2022-CSC_and_LO/2_00_part2_using.md +++ /dev/null @@ -1,8 +0,0 @@ -# Part II: Using EasyBuild - -*[[back to start page]](index.md)* - -* [Troubleshooting](2_01_troubleshooting.md) *(hands-on)* -* [Creating easyconfig files](2_02_creating_easyconfig_files.md) *(hands-on)* -* [Using external modules from the Cray PE](2_03_external_modules.md) -* [Implementing easyblocks](2_04_implementing_easyblocks.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/2_01_troubleshooting.md b/docs/2022-CSC_and_LO/2_01_troubleshooting.md deleted file mode 100644 index 30f2f5cc..00000000 --- a/docs/2022-CSC_and_LO/2_01_troubleshooting.md +++ /dev/null @@ -1,545 +0,0 @@ -# Troubleshooting - -*[[back: Overview]](index.md)* - ---- - -Whatever tool you use, when installing scientific software you'll -be running into problems rather sooner than later. - -In this part we take a look at how you can **troubleshoot a failing installation**, -and focus on a couple of EasyBuild aspects that can be helpful in that context, -like the **error messages produced by EasyBuild**, the detailed **log file** that is -available for the installation and the **build directory** where most software is -being compiled before it actually gets installed. - -At the end of this part an exercise is available in which you get -to fix a failing installation, so pay attention! - -## EasyBuild error messages - -When EasyBuild detects that something went wrong, it will usually produce a -short (and hopefully helpful) error message. - -Things that could go wrong during an installation include: - -* missing source or patch files; -* a checksum error on a downloaded source or patch file; -* required dependencies that are not specified in the easyconfig file; -* failing shell commands; -* running out of available memory or disk space; -* a segmentation fault caused by a flipped bit triggered by a cosmic ray ([really, it happens!](https://blogs.oracle.com/linux/attack-of-the-cosmic-rays-v2)); - -Unfortunately this is not an exhaustive list, there are plenty of other -potential problems that could result in a failing installation... - -For each of the shell commands that EasyBuild executes during an -installation, it will check the exit status. -If the exit status is zero, EasyBuild will usually assume that the shell command -ran correctly, and it will continue with the rest of the installation procedure. -If the exit status is anything but zero, EasyBuild will assume that a problem -has occurred, and the installation will be interrupted. - -### Example - -Here is an example of an EasyBuild error message (slightly reformatted for clarity): - -``` -$ eb example.eb -... -== building... -== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): -build failed (first 300 chars): cmd "make" exited with exit code 2 and output: -/usr/bin/g++ -O2 -ftree-vectorize -march=znver2 -fno-math-errno -c -o core.o core.cpp -cc1plus: error: bad value (‘znver2’) for ‘-march=’ switch (took 1 sec) -== Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log -ERROR: Build of /home/easybuild/subread.eb failed (err: ...) -``` - -Let's break this down a bit: during the `build` step of the installation -procedure EasyBuild was running `make` as a shell command, which -failed (exit code 2, so not zero). -The `make` command tripped over the compilation of `core.cpp` that failed because -`-march=znver2` is not a known option to the `g++` command. - -OK fine, but now what? Can you spot something suspicious here? -Wait a minute... Why is `make` using `/usr/bin/g++` for the compilation?! -That's not where our toolchain compiler is installed, -that's somewhere under `/opt/cray/pe/gcc`. - -Let's see what `/usr/bin/g++` is: - -``` -$ /usr/bin/g++ --version -g++ (SUSE Linux) 7.5.0 -``` - -Oh my that's an ancient compiler (7.5 was released on November 14, 2019, -a few months after the release of the Zen2 architecture, but -the base version, 7.1, is really from May 2, 2017, long before the Zen2 architecture was around) -That could definitely explain why it doesn't know about the Zen2 architecture yet... - -Your next step in this case should probably be figuring -out why `/usr/bin/g++` is being used rather than just `g++`, which would -result in using the right compiler version because EasyBuild sets up the build -environment carefully. - -This is a fictitious example of course, but hopefully it gives you a feeling -of how errors that occur during the installation are handled. - -## EasyBuild log files - -Finding the cause of a problem that made the installation fail is, unfortunately, not always that straightforward... - -EasyBuild includes the first 300 characters of the output produced by a failing -shell command in the error message, which is a simple way to try include -actionable information in the error message while not flooding your terminal -with the full output of the command that failed. - -In some cases there won't be any useful information in there however, -since the actual error message(s) could only appear way later, perhaps even after -the command was already running for several minutes. - -In that case, you will have to dive into the log file that is created by EasyBuild for -every installation, which is located in the unique temporary directory for the EasyBuild session. - -See for example this output line from our earlier example error message: - -``` -== Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log -``` - -You can open this file with your favorite text editor or a tool like `less` -to take a look at the information collected in the log file, which includes -things like: - -* informative messages produced by both the EasyBuild framework and the easyblock - describing how the installation is progressing; -* how the build environment was set up: which modules were loaded, which environment variables were set; -* the exact shell commands that were executed, and in which directory they were run; -* the full output produced by these commands, and their exit code; - -Note that the installation log is also copied into each software installation -directory for successful installation, into the `easybuild` subdirectory. -For example: - -``` -/appl/lumi/SW/LUMI-21.12/L/EB/ncurses/6.2-cpeGNU-21.12/easybuild/easybuild-ncurses-6.2-20220302.110244.log -``` - -### Last log - -The `eb` command supports a handy little option that prints the location -to the most recently updated build log. You can leverage this to quickly -open the build log of the last ***failed*** EasyBuild session in an editor: - -``` -vim $(eb --last-log) -``` - -### Navigating log files - -Usually you want to go to the end of the log file and then work your way up, -either by scrolling or by searching for specific patterns. Here are a couple -of suggestions of patterns you can use to locate errors: - -* `ERROR` -* `Error 1` -* `error: ` -* `failure` -* `not found` -* `No such file or directory` -* `bazel` -* `Segmentation fault` - -Using "`error`" as a search pattern is not very useful: you will hit a lot of log lines -that are not actually errors at all (like the compilation of an `error.c` file), -and you'll miss others that do include errors but mention `ERROR` or `Error` -rather than `error`. - -When using `less` to view a log file, you can navigate it by: - -* hitting '`$`' followed by '`G`' to go to the end of the log file; -* using your arrow keys to scroll up/down; -* typing '`?`' followed by some text and Enter to search backwards for a particular -pattern ('`/`' to search forwards, '`n`' for next match); -* hitting '`q`' to exit; - -It can also be helpful to zoom in on a specific step of the installation procedure, -which you can do by looking for step markers like these: - -``` -== 2020-06-13 01:34:48,816 example INFO configuring... -== 2020-06-13 01:34:48,817 example INFO Starting configure step -... -== 2020-06-13 01:34:48,823 main.EB_HPL INFO Running method configure_step part of step configure -``` - -If you want to look at the start of the output produced by a particular command, -you can look for the log message that looks like this (this is from the installation -log for `HDF5`): - -``` -== 2020-06-09 13:11:19,968 run.py:222 INFO running cmd: make install -== 2020-06-09 13:11:25,186 run.py:538 INFO cmd " make install " exited with exit code 0 and output: -Making install in src -make[1]: Entering directory `/tmp/example/HDF5/1.10.7/gompi-2020b/hdf5-1.10.7/src' -... -``` - -It can be useful to look for the *first* error that occurred in the output of a command, since subsequent errors are -often fallout from earlier errors. You can do this by first navigating -to the start of the output for a command using "`INFO running cmd`" as a search pattern, and then looking for patterns -like "`error:`" from there. - -## Inspecting the build directory - -When an installation fails the corresponding build directory is *not* cleaned up -automatically, that is only done for successful installations. -This allows you to dive in and check for clues in the files that are stored there. - -The location of the build directory is mentioned in the EasyBuild error message: - -``` -== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): ... -``` - -For software using a classic `configure` script, you may have to locate -and inspect the `config.log` file in the build directory to determine the underlying cause of an error. For software using CMake as a configuration tool you often have to check in -`CMakeOutput.log` or `CMakeError.log` for clues, which are sneakily hidden by CMake in a `CMakeFiles` subdirectory of the build directory. - -As a side note here: as EasyBuild does not clean out old and failed builds you will need to eventually manually remove these build directories from the `buildpath` directory. On the login nodes of LUMI this is currently not much of an issue as `$XDG_RUNTIME_DIR` is (ab)used for build and -temporary files and that directory is cleaned automatically. However, when building on compute nodes, where `$XDG_RUNTIME_DIR` is -not available, it is essential to manually clean that directory as the space is not automatically cleaned when your session -ends. The `EasyBuild-user` module does define the bash function `clear-eb` that can be used to clear that space. - -## Exercise - -Let's work our way through a less smooth software installation, -using the easyconfig file that is provided below. - -Don't worry if most of this is still unclear to you, we'll get -to writing easyconfig files from scratch [later in this tutorial](2_02_creating_easyconfig_files). - -```python -easyblock = 'MakeCp' - -name = 'Subread' -version = '2.0.1' - -homepage = 'http://subread.sourceforge.net' -description = "High performance read alignment, quantification and mutation discovery" - -toolchain = {'name': 'PrgEnv-gnu', 'version': '21.10'} - -# download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz -sources = ['subread-%(version)s-source.tar.gz'] -checksums = ['d808eb5b1823c572cb45a97c95a3c5acb3d8e29aa47ec74e3ca1eb345787c17b'] - -start_dir = 'src' - -# -fcommon is required to compile Subread 2.0.1 with GCC 10, -# which uses -fno-common by default (see https://www.gnu.org/software/gcc/gcc-10/porting_to.html) -buildopts = '-f Makefile.Linux CFLAGS="-fast -fcommon"' - -files_to_copy = ['bin'] - -sanity_check_paths = { - 'files': ['bin/featureCounts', 'bin/subread-align'], - 'dirs': ['bin/utilities'], -} - -sanity_check_commands = ["featureCounts --version"] - -moduleclass = 'bio' -``` - -Do you spot any potential problems yet with this easyconfig file? - -***Preparation*** - -Start by copying the text above in a file named `subread.eb` -(which does not follow the EasyBuild conventions but that is not a problem for this exercise), -so you can gradually fix the problem you'll encounter. - -Also make sure that the pre-installed software stack is available, -and that the EasyBuild-user module is loaded (unless you installed EasyBuild -yourself): - -``` -module load LUMI/21.12 -module load EasyBuild-user -``` - -This will configure EasyBuild correctly for this exercise, though if you already have -an existing EasyBuild user installation you may want to work in a different one -by pointing `$EBU_USER_PREFIX` to the desired work directory before loading -`LUMI/21.12`. - -Check your configuration via `eb --show-config`. - -Strictly speaking the configuration doesn't matter much for the sake of this -exercise, but it may help with the step-wise approach we'll take and -grasping the solutions. - -Remember though: *no peeking* before you tried to solve each step yourself! - ---- - -***Exercise T.1**** - Sources* - -Try to install the `subread.eb` easyconfig file, see what happens. - -Can you fix the problem you run into, perhaps without even changing -the easyconfig file? - -??? success "(click to show solution)" - - The installation fails because the source file `subread-2.0.1-source.tar.gz` - is not found: - ``` - $ eb subread.eb - ... - == FAILED: Installation ended unsuccessfully (build directory: /run/user/XXXXXXXX/easybuild/build/Subread/2.0.1/cpeGNU-21.12): build failed (first 300 chars): - Couldn't find file subread-2.0.1-source.tar.gz anywhere, and downloading it didn't work either... - Paths attempted (in order): ... - ``` - - In this case, the problem is that the easyconfig file does not specify - where the sources can be downloaded from. Not automatically at least, - but there is a helpful comment included: - ```python - # download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz - sources = ['subread-%(version)s-source.tar.gz'] - ``` - - We can download the source tarball ourselves, - and move it to the location where EasyBuild expects to find it - (in the `sourcepath` directory): - ``` - curl -OL https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz - mkdir -p $EBU_USER_PREFIX/sources/s/Subread - mv subread-2.0.1-source.tar.gz $EBU_USER_PREFIX/sources/s/Subread/ - ``` - - Or, we can change the easyconfig file to specify the location where - the easyconfig file can be downloaded from: - ```python - source_urls = ['https://download.sourceforge.net/subread/'] - sources = ['subread-%(version)s-source.tar.gz'] - ``` - Note that the `source_urls` value is a *list* of candidate download URLs, - *without* the filename of the source file itself. - - This way, EasyBuild will download the source file when running `eb subread.eb`. - - The source tarball is fairly large (23MB), so don't be alarmed if the download takes a little while. - - ```shell - $ ls -lh $EBU_USER_PREFIX/sources/s/Subread - total 23M - -rw-rw-r-- 1 XXXXXXXX XXXXXXXX 23M Mar 30 16:08 subread-2.0.1-source.tar.gz - ``` - ---- - -***Exercise T.2**** - Toolchain* - -After fixing the problem with missing source file, try the installation again. - -What's wrong now? How can you fix it quickly? - -Take into account that we just want to get this software package installed, -we don't care too much about details like the version of the dependencies or -the toolchain here... - - -??? success "(click to show solution)" - - The installation fails because the easyconfig specifies that GCC 8.5.0 - should be used as toolchain: - - ``` - $ eb subread.eb - ... - ERROR: Failed to process easyconfig /pfs/lustrep3/users/kurtlust/easybuild-tutorial/Troubleshooting/subread.eb: Toolchain PrgEnv-gnu not found, - available toolchains: ... - ... - ``` - - `PrgEnv-gnu` is an HPE Cray PE module that may look like a toolchain - it certainly has - the same function: provide compiler, MPI and basic math libraries - but it is not - recognised as a toolchain by EasyBuild. EasyBuild prefers to manage its own modules so that it knows - well what is in it which is not the case with the `PrgEnv-*` modules from the Cray PE - as the content may differe between systems and as the versions of the compilers etc. that - are loaded differ on other modules that are loaded. Hence we created Cray-specific toolchains. - You'll actually find two series of Cray toolchains in the list of available toolchains. The - `CrayGNU`, `CrayIntel`, `CrayPGI` and `CrayCCE` are included with the EasyBuild distribution - and where developed by CSCS for their systems using Environment Modules. These were not compatible - with the initial releases of the Cray PE with Lmod modules so new ones were developed on which we - also built for the LUMI toolchains. Those are called `cpeCray`, `cpeGNU`, `cpeAOCC` and `cpeAMD` - and are maintained by LUST and available via the LUMI repositories. - - Changing the toolchain name to `cpeGNU` is not enough to solve all problems though: - - ``` - $ eb subread.eb - ... - ERROR: Build of /pfs/lustrep3/users/kurtlust/easybuild-tutorial/Troubleshooting/subread.eb failed (err: 'build failed (first 300 chars): - No module found for toolchain: cpeGNU/21.10') - ... - ``` - - We don't have this `cpeGNU` version installed, but we do have GCC 21.12: - - ```shell - $ module avail cpeGNU/ - ----- Infrastructure modules for the software stack LUMI/21.12 on LUMI-L ----- - cpeGNU/21.12 - ... - ``` - - So let's try using that instead. - - Edit the easyconfig file so it contains this: - - ```python - toolchain = {'name': 'cpeGNU', 'version': '21.12'} - ``` ---- - -***Exercise T.3**** - Build step* - -With the first two problems fixed, now we can actually try to build the software. - -Can you fix the next problem you run into? - -??? success "(click to show solution)" - - The compilation fails, but the error message we see is incomplete due to - EasyBuild truncating the command output (only the 300 first characters of the output are shown): - ``` - == FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12): build failed - (first 300 chars): cmd " make -j 256 -f Makefile.Linux CFLAGS="-fast -fcommon"" exited with exit code 2 and output: - gcc -mtune=core2 -O3 -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\""2.0.1"\" -D_FILE_OFFSET_BITS=64 -fmessage-length=0 - -ggdb -fast -fcommon -I/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86 (took 4 secs) - ``` - - If you open the log file (e.g., with `view $(eb --last-log)`) and scroll to the end, - the error is pretty obvious: - ``` - gcc: error: unrecognized command line option -fast; did you mean -Ofast? - make: *** [core.o] Error 1 - ``` - - The easyconfig file hard specifies the `-fast` compiler flag via the `CFLAGS` argument to the build command: - ```python - # -fcommon is required to compile Subread 2.0.1 with GCC 10, - # which uses -fno-common by default (see https://www.gnu.org/software/gcc/gcc-10/porting_to.html) - buildopts = '-f Makefile.Linux CFLAGS="-fast -fcommon"' - ``` - - EasyBuild sets up the build environment, so there should be no need - to hard specify compiler flags (certainly not incorrect ones). - The comment above the `buildopts` definition makes it clear that the `-fcommon` - flag *is* required though, because GCC 10 became a bit stricter by - using `-fno-common` by default (and we're using GCC 11 in `cpeGNU/21.12`). - Note that we are using `-fcommon` - as an escape mechanism here: it would be better to fix the source code - and create a patch file instead. - - An easy way to fix this problem is to replace the `-fast` with `-Ofast`, - as the compiler error suggests. - - In this case it is advised to change the `CFLAGS` argument that is added - to be build command to replace the `-fast` with `$CFLAGS`, - which is defined in the build environment by EasyBuild. - - ```python - buildopts = '-f Makefile.Linux CFLAGS="$CFLAGS -fcommon"' - ``` - - Note that we need to be careful with quotes here: we use inner double quotes - to ensure that `$CFLAGS` will be expanded to its value when the build command is run. - As you can see after the change by doing a dry-run: - - ``` - $ eb subread.eb -x - ... - - Defining build environment... - - export BLAS_INC_DIR='/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/include' - ... - export CC='cc' - export CFLAGS='-O2 -ftree-vectorize -fno-math-errno' - ... - [build_step method] - running command "make -j 256 -f Makefile.Linux CFLAGS="$CFLAGS -fcommon"" - (in /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12/Subread-2.0.1/src) - ... - ``` - - EasyBuild will launch the command - ``` - make -j 256 -f Makefile.Linux CFLAGS="$CFLAGS -fcommon" - ``` - in a shell where `CFLAGS` is defined and set to an appropriate value (determined by - defaults in EasyBuild, settings in the EasyBuild configuration and settings in the - easyconfig file that we shall discuss later). - ---- - -***Exercise T.4**** - Sanity check* - -After fixing the compilation issue, you're really close to getting the installation working, we promise! - -Don't give up now, try one last time and fix the last problem that occurs... - -??? success "(click to show solution)" - - Now the installation itself works but the sanity check fails, - and hence the module file does not get generated: - ``` - $ eb subread.eb - ... - == FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12): - build failed (first 300 chars): Sanity check failed: sanity check command featureCounts --version exited with code 255 - (output: featureCounts: unrecognized option '--version' - ... - ... - ``` - - If you look at the full output in the log file you can see - that the correct option to check the version of the `featureCounts` command - is "`-v`" rather than "`--version`", so we need to fix this in the easyconfig file. - - Make the following change in the easyconfig file: - ```python - sanity_check_commands = ["featureCounts -v"] - ``` - - After doing so, **you don't have to redo the installation - from scratch**, you can use the `--module-only` option to only run the - sanity check and generate the module file again: - ``` - eb subread.eb --module-only - ``` - ---- - -In the end, you should be able to install Subread 2.0.1 with the cpeGNU 21.12 toolchain by fixing the problems with the `subread.eb` easyconfig file. - -Check your work by manually loading the module and checking the version -via the `featureCounts` command, which should look like this: - -```shell -$ module load Subread/2.0.1-cpeGNU-21.12 -... -$ featureCounts -v -featureCounts v2.0.1 -``` - ---- - -*[[next: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md deleted file mode 100644 index f9bec25a..00000000 --- a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md +++ /dev/null @@ -1,1027 +0,0 @@ -# Creating easyconfig files - -*[[back: Troubleshooting]](2_02_troubleshooting)* - ---- - -We already know how to install easyconfig files that are provided by LUMI, -but what about installing software for which no easyconfig is available yet, -or for which we need to adapt an existing easyconfig? - -To do this we will need to **create additional easyconfig files**, -since every software installation performed by EasyBuild is done based on an easyconfig file. - -In this part of the tutorial we will look at the guts of easyconfig files and even create some ourselves! - -## Easyconfigs vs easyblocks - -Before we dive into writing [easyconfig files](../1_05_terminology/#easyconfig-files), -let us take a brief look at how they relate to [easyblocks](../1_05_terminology/#easyblocks). - -As we discussed [earlier](../1_05_terminology/#terminology), an easyconfig file (`*.eb`) is required -for each installation -performed by EasyBuild which specifies the details of the installation (which software -version, toolchain, etc.), while the installation procedure is implemented -in an easyblock (a Python module). - -When can we leverage a *generic easyblock*, perhaps via a "fat" easyconfig file that includes -a lot of carefully defined easyconfig parameters, and when should we use a minimal easyconfig file -together with a custom *software-specific* easyblock? - -This is not an easy question to answer in a general sense, since it depends on several factors: -the complexity of the software you want to get installed, how much flexibility you want, -how "intelligent" the installation procedure should be with respect to the compiler toolchain and dependencies -that are used for the installation, whether you want an easyconfig that is more user-focused (with easy-to-understand -parameters) or one that is more focused towards people who understand installation commands, etc. - -In a nutshell, custom software-specific easyblocks are "do once and forget": they are central solution to peculiarities in the installation procedure of a particular software package. However, they also hide a lot from direct view, so if things go -wrong, it is often harder to debug the exact problem. And since a single easyblock has to cover -multiple software versions, multiple toolchains and multiple possible configurations of the package, -they are often harder to develop and certainly harder to test. As a result many of the easyblocks -included with EasyBuild work poorly on HPE Cray systems, e.g., because they want to add compiler flags -specific for a certain compiler and don't recognise the Cray compilers. - -Reasons to consider implementing a software-specific easyblock rather than using a generic easyblock include: - -- 'critical' values for easyconfig parameters required to make installation succeed; -- toolchain-specific aspects of the build and installation procedure (e.g., configure options); -- interactive commands that need to be run; - *For example: The easyblock to [install maple](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/m/maple.py) - responds to a number of interactive questions.* -- custom (configure) options for dependencies; - *For example, the [easyblock for VMD](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/v/vmd.py) - will add configure options depending on the dependency list of the package.* -- having to create or adjust specific (configuration) files; -- 'hackish' usage of a generic easyblock; -- complex or very non-standard installation procedure; - *For example, the [easyblock to install the gcc compilers from source](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/g/gcc.py), - bootstrapping with the system compiler and then re-installing with itself.* - -For implementing easyblocks we refer to the -["Implementing easyblocks" section of this tutorial](2_03_implementing_easyblocks) -and the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). - -## Writing easyconfig files - -Writing an easyconfig file boils down to defining a set of easyconfig parameters in a text file, -which we give a '`.eb`' extension by convention. The name of an easyconfig file doesn't matter -when using it directly to install software, but it does matter when EasyBuild needs to find it -to resolve a dependency for example (as we [discussed earlier](../1_08_basic_usage/#easyconfig-filenames)). - -The syntax for easyconfig files is *Python syntax*: you are basically defining a bunch of Python variables -that correspond to easyconfig parameters. - -The order in which the easyconfig parameters are defined doesn't matter, but we generally try to stick to a particular -order which roughly corresponds to the order in which the easyconfig parameters are used during the installation. -That is mostly helpful for humans staring at easyconfig files or comparing them. - -### Mandatory parameters - -A limited number of easyconfig parameters are *mandatory*, they must be defined in every easyconfig file: - -- `name`: the name of the software to install; -- `version`: the version of the software to install; -- `homepage`: a URL to the website of the software; -- `description`: a description of the software; -- `toolchain`: the compiler toolchain to use for the installation; - -#### `name`, `version` - -It should be no surprise that specifying the name and version of the software you want to install is mandatory. -This information may influence the value of several other easyconfig parameters (like the name of the source file), and is also used to the determine the name of the module file to install. - -```python -name = 'example' -version = '1.0' -``` - -#### `homepage`, `description` - -The homepage and description are included in the generated module file for the installation. -That way the "`module show`" command provides some useful high-level information about the installation. - -```python -homepage = 'https://example.org' -description = "This is just an example." -``` - -Usually it does not matter whether you use single or double quotes to specify string values. -There are cases where it is important to use -the right type of quotes however, we will get back to that later (keep it in mind for the exercises!). - -For multi-line descriptions, you will need to use "triple quoting" (which is standard Python syntax): - -```python -description = """This is an example - of a multi-line description. - It is spread across multiple lines.""" -``` - -If no homepage is known for the package, the convention in the EasyBuild community is to use -`(none)` as the value. - -The `description` field is used in two different places in the LUA module file generated by -EasyBuild: -- In the help block, for the first section of the help information -- If there is no `whatis` parameter in the easyconfig file, it is also used to generate - a whatis-command with the description in the module file. - -*Note:* We ask people who contribute easyconfigs to LUMI to instead use a decent description -that is useful in the help information, and to specify a short description via the `whatis` -parameter: - -```python -whatis = ['Description: Blosc is an extremely fast, multi-threaded, meta-compressor library'] -``` - -Note that in this case the word "Description:" has to be explicitly specified. - - -#### `toolchain` - -EasyBuild also requires that the [compiler toolchain](../1_05_terminology/#toolchains) is specified, via the `toolchain` -easyconfig parameter. - -This can either be the [`system` toolchain](../1_05_terminology/#system-toolchain), for which a constant named `SYSTEM` is available: - -```python -toolchain = SYSTEM -``` - -Usually we specify a 'proper' toolchain like the `cpeGNU/21.12` toolchain we have used bvefore. -The name and version of the toolchain can be specified using a small Python dictionary, -for example: - -```python -toolchain = {'name': 'cpeGNU', 'version': '21.12'} -``` - -### Commonly used parameters - -You will often need to specify additional easyconfig parameters to get something useful done. -We will cover the most commonly used ones here, but keep in mind that these are *not* mandatory. - -A full overview of all known easyconfig parameters can be obtained via "`eb --avail-easyconfig-params`" -or just "`eb -a`" for short, or can be consulted in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_parameters.html). - -#### Sources, patches, and checksums - -In most easyconfig files you will see that a list of source files is specified via the `sources` -easyconfig parameter, usually combined -with one or more URLs where these sources can be downloaded specified via `source_urls`. -There also may be patch files listed (specified via `patches`), -and checksums for both the source files and patches (specified via `checksums`). - -The `sources` easyconfig parameter is commonly defined but it is *not* mandatory, -because some easyconfig files only specify bundles of software packages and hence only -serve to generate a module file. - -Here is an example of how these easyconfig parameters can be specified: - -```python -source_urls = [ - 'https://example.org/download/', - 'https://example.org/download/archive/', -] -sources = ['example-1.0-src.tar.gz'] -patches = ['example-fix.patch'] -checksums = [ - '9febae18533d035ac688d977cb2ca050e6ca8379311d7a14490ad1ef948d45fa', - '864395d648ad9a5b75d1a745c8ef82b78421d571584037560a22a581ed7a261c', -] -``` - -Each of these require a *list* of values, so even if there is only a single source file or download URL -you must use square brackets as shown in the example. The default value for each of these is an empty list (`[]`). - -Some things worth pointing out here: - -* The download URLs specified via `source_urls` do *not* include the name of the file, that is added - automatically by EasyBuild when it tries to download the file (only if it's not available already.) -* If multiple download URLs are specified, they are each tried once in order until the download of the source file was - successful. This can be useful to include backup locations where source files can be downloaded from. -* Names of source files and patches should not include hardcoded software versions, they usually use a - template value like `%(version)s` instead: - ```python - sources = ['example-%(version)s-src.tar.gz'] - ``` - EasyBuild will use the value of the `version` easyconfig parameter to determine the actual name of the source - file. This way the software version is only specified in one place and the easyconfig file is easier to - update to other software versions. A list of template values can be consulted via the EasyBuild command - line via the `--avail-easyconfig-templates` option, or in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_templates.html). -* Source files can also be specified in ways other than just using a filename, see the - [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-param-sources-alt) for more information. - It is also possible to download a given commit from a GitHub repository. -* Specified checksums are usually SHA256 checksum values, but - [other types are also supported](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html?highlight=checksums#checksums). - - -#### Easyblock - -The easyblock that should be used for the installation can be specified via the `easyblock` easyconfig parameter. - -This is not mandatory however, because by default EasyBuild will determine the easyblock to use based on the -name of the software. If '`example`' is specified as software name, EasyBuild will try to locate a -software-specific easyblock named `EB_example` (in a Python module named `example.py`). Software-specific -easyblocks follow the convention that the class name starts with `'EB_`', followed by the software name -(where some characters are replaced, like '`-`' with '`_minus_`'). It is possible to use different -naming conventions for software-specific easyblocks, but then EasyBuild will not automatically detect -that there is one for the package and it will also need to be specified via the `easyblock` parameter. - -**Generic easyblocks** - -Usually the `easyblock` value is the name of a *generic* easyblock, if it is specified. The name of -a generic easyblock does *not* start with '`EB_`', so you can easily distinguish it from a software-specific -easyblock. - -Here are a couple of commonly used generic easyblocks: - -* `ConfigureMake`: implements the standard `./configure`, `make`, `make install` installation procedure; -* `CMakeMake`: same as `ConfigureMake`, but with `./configure` replaced with `cmake` for the configuration step; -* `PythonPackage`: implements the installation procedure for a single Python package, by default using - "`python setup.py install`" but other methods like using "`pip install`" are also supported; -* `Bundle`: a simple generic easyblock to bundle a set of software packages together in a single installation directory; -* `PythonBundle`: a customized version of the `Bundle` generic easyblock to install a bundle of Python packages - in a single installation directory; - -A full overview of the available generic easyblocks is available in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/generic_easyblocks.html). You can also consult the output of -`eb --list-easyblocks`, which gives an overview of *all* known easyblocks, and how they relate to each other. - -**Custom easyconfig parameters** - -Most generic easyblocks provide additional easyconfig parameters to steer their behaviour. -You can consult these via "`eb -a --easyblock`" or just "`eb -a -e`", which results in an -additional "`EASYBLOCK-SPECIFIC`" section to be added. See the (partial) output of this command for example: - -```shell -$ eb -a -e ConfigureMake -Available easyconfig parameters (* indicates specific to the ConfigureMake easyblock): -... -EASYBLOCK-SPECIFIC ------------------- -build_cmd* Build command to use [default: "make"] -build_type* Value to provide to --build option of configure script, e.g., x86_64-pc-linux-gnu (determined by config.guess shipped with EasyBuild if None, False implies to leave it up to the configure script) [default: None] -configure_cmd* Configure command to use [default: "./configure"] -configure_cmd_prefix* Prefix to be glued before ./configure [default: ""] -host_type* Value to provide to --host option of configure script, e.g., x86_64-pc-linux-gnu (determined by config.guess shipped with EasyBuild if None, False implies to leave it up to the configure script) [default: None] -install_cmd* Build command to use [default: "make install"] -prefix_opt* Prefix command line option for configure script ('--prefix=' if None) [default: None] -tar_config_opts* Override tar settings as determined by configure. [default: False] -``` - -#### Dependencies - -You will often need to list one or more [dependencies](../1_05_terminology/#dependencies) that are required -to install or run the software. -We distinguish between two main different types of dependencies: runtime dependencies and build dependencies. - -*Runtime dependencies* are required for using the installed software, and may also have to be available -during the installation. These dependencies can be specified via the `dependencies` easyconfig parameter. -EasyBuild will load the modules for these dependencies when setting up the build environment, -and will include load statements for them in the generated module file. - -*Build dependencies* are only required during the installation of the software, not for using the -software once it is installed. The modules for these dependencies will be loaded in the build environment -set up by EasyBuild during the installation, but they will *not* be loaded by the generated module file. -You can specify build dependencies via the `builddependencies` easyconfig parameter. -One typical example of a build dependency is `CMake`, which is only needed for configuring -the build. On LUMI we define a `buildtools` module for each version of the `LUMI` software stack -which contains most popular build tools. This ensures that we use a consistent set of build tools -and that they do not change over the life time of a stack, also in case of OS upgrades, to -increase the reproducibility of the build process - -Here is a simple example of specifying dependencies: - -```python -builddependencies = [ - ('buildtools', '%(toolchain_version)s', '', True) -] - -dependencies = [ - ('Python', '3.8.2'), - ('HDF5', '1.10.6'), - ('SciPy-bundle', '2020.03', '-Python-%(pyver)s'), -] -``` - -Both `builddependencies` and `dependencies` require a list of tuples, -each of which specifying one dependency. -The name and version of a dependency is specified with a 2-tuple (a tuple with two string values). - -In some cases additional information may have to be provided, as is shown in the example above for the `SciPy-bundle` -dependency where a 3rd value is specified corresponding to the `versionsuffix` value of this dependency. -If this is not specified, it is assumed to be the empty string (`''`). - -Note how we use the '`%(pyver)s'` template value in the `SciPy-bundle` dependency -specification, to avoid hardcoding the Python version in different places. (Though this -specific parameter is less useful on LUMI as we currently try to build on top of `cray-python`.) - -The `buildtools` build dependency shows that there is a fourth parameter specifying the toolchain -used for that dependency and is needed if that toolchain is different from the one used in the example. -As it is not possible to load several Cray toolchains together (they are not in a hierachical relation) -the only useful value on LUMI is `True` which tells that `buildtools` is build with the `SYSTEM` -toolchain. Here also we use a template, `%(toolchain_version)s` which - as its name suggests - expands -to the version of the toolchain, as we version our `buildtools` modules after the version of the Cray -toolchains for which they are intended. - -See also the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#dependencies) -for additional options on specifying dependencies. That page specifies two more dependency types: - -* `hiddendependencies` are currently not used on LUMI, and if we would use them in the future, it will likely - be through a way that does not require this parameter. -* `osdependencies` can be used to let EasyBuild check if certain needed OS packages are installed. - See, e.g., the [easyconfigs for the `buildtools` package](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/easybuild/easyconfigs/b/buildtools) - on LUMI. - - -#### Version suffix - -In some cases you may want to build a particular software package in different configurations, -or include a label in the module name to highlight a particular aspect -of the installation. - -The `versionsuffix` easyconfig parameter can be used for this purpose. -The name of this parameter implies that this label will be added after the -software version (and toolchain label) in the standard module naming scheme. - -If you are configuring the software to build with a particular non-default value, -you can indicate this as follows for example: - -```python -versionsuffix = '-example-label' -``` - -This mechanism is used frequently to indicate that a software installation depends -on a particular version of Python, where the `%(pyver)s` template comes in useful again: - -```python -versionsuffix = '-Python-%(pyver)s' -... -dependencies = [('Python', '2.7.18')] -``` - -Even though Python 2 is officially dead and -buried some scientific software still requires it, and mixing modules where -some use Python 2 and other use Python 3 doesn't work well. -The `versionsuffix` label is helpful to inform the user that a particular Python version is -required by the installation. - -#### Customizing configure, build, test, and install commands - -When using a generic easyblock like `ConfigureMake` or `CMakeMake`, you will often -find yourself having to specify options to the configure, build, test, or install commands, or to inject additional commands right before them. - -For this the following standard easyconfig parameters are available: - -* `preconfigopts`: string value that is glued *before* the configure command; -* `configopts`: string value that is added *after* the configure command, which can be used to specify configuration options; - -Equivalent easyconfig parameters are available for the `build`, `test` and `install` steps: `prebuildopts`, `buildopts`, `pretestopts`, `testopts`, `preinstallopts`, and `installopts`. - -Here is a fictitious example of how they can be used: - -```python -easyblock = 'ConfigureMake' -... -dependencies = [('HDF5', '1.10.6')] -... -configopts = '--enable-hdf5-support' - -prebuildopts = 'export HDF5_PREFIX="$EBROOTHDF5" && ' - -installopts = "PREFIX='%(installdir)s'" -``` -(again an imaginary example as on LUMI we advise to use the Cray-provided HDF5 modules.) - -Here we are: - -* Adding the `--enable-hdf5-support` configure option, to convince the `ConfigureMake` easyblock to run the following - command during the configure step: - ```shell - ./configure --prefix ... --enable-hdf5-support - ``` - (where the '`...`' represents the path to installation directory where the software should be installed). - -* Specifying that an additional command has to be run before running `make` in the build step. We use '`&&`' to glue the - command to the `make` command, so `make` will only be run if the command we specified ran correctly. So, the build step will run something like: - ```shell - export HDF5_PREFIX="$EBROOTHDF5" && make -j 4 - ``` - The '`4`' value passed to the `-j` option shown here, which specifies how many commands `make` can run in parallel, is automatically determined by EasyBuild based on the number of available cores (taking into account `ulimit` settings, and cpuset and cgroup restrictions). - -* Passing the location where the software should be installed via the `PREFIX` argument to the `make install` command during the installation step. This results in the - following command being run: - ```shell - make install PREFIX=... - ``` - (where the '`...`' again represents the path to installation directory). - Even though the - installation directory is already specified in the configure command, it is - apparently blatantly ignored by the software we are installing here, and we are expected to specify it - this way instead. How rude! - -The `$EBROOTHDF5` environment variable that we are using in `prebuildopts` corresponds to the path of -the installation directory of the HDF5 dependency. EasyBuild includes a statement -to define an `$EBROOT*` environment variable -like this in every environment module file it generates (see the output of "`module show HDF5`"). - -#### Sanity check - -One seemingly trivial yet important aspect of the installation procedure that EasyBuild performs -is the sanity check step. - -By default EasyBuild does a simple sanity check that verifies whether there is a non-empty `bin` subdirectory -in the installation, next to a non-empty `lib` or `lib64` directory (either is sufficient). - -It is recommended to customize the sanity check and check for something more specific, like a particular -binary or directory, or making sure that a trivial command (like `example -V` or `example --help`) -runs correctly. - -To specify a custom set of files and/or directories to check, -you can use the `sanity_check_paths` easyconfig parameter. The expected value is Python dictionary -with two keys: `files` and `dirs`. For example: - -```python -sanity_check_paths = { - 'files': ['bin/example'], - 'dirs': ['examples/one', 'examples/two'], -} -``` - -In addition, you can specify one or more commands that should be working without a problem (that is, have a zero exit status) via the `sanity_check_commands` easyconfig parameter. -These commands will be run just like a user would: after loading the module that was generated for this installation. -Here is an example: - -```python -sanity_check_commands = [ - "example --version", - "example --help", -] -``` - - -#### Module class - -Finally, you will usually see the `moduleclass` easyconfig parameter to be defined as well, for example: - -```python -moduleclass = 'lib' -``` - -This is done to categorize software, and it is used to group the generated module files into smaller sets -([remember what we saw when installing software earlier](../1_08_basic_usage/#using-installed-software)). - -This is currently not used on LUMI since we feel that (a) it is not easy to explain to users how they can -then only make certain classes that are useful to them visible and more importantly (b) since it is not -always intuitive to decide which moduleclass should be used for a package, or from a user 's perspective, -in which category to look for a package. - -## Generating tweaked easyconfigs - -Sometimes you may want to install software that differs only slightly from an -existing easyconfig file, like a newer software version or using a different -compiler toolchain. Do we need to create an easyconfig file for this too? - -We do, but EasyBuild does provide some help so you don't need to *manually* -create the easyconfig file. You can use one of the `--try-*` options provided -by the `eb` command to make EasyBuild *generate* a new easyconfig file based on -an existing one. - -For example, to try installing a different software version you can use the `--try-software-version` option: - -```shell -eb example-1.2.3.eb --try-software-version 1.2.4 -``` - -Or, to try using a different compiler toolchain you can use `--try-toolchain`: - -```shell -eb example-1.2.3-foss-2020b.eb --try-toolchain cpeCray,21.12 -``` - -It is important to keep in mind the *"try"* aspect here: while easyconfigs that -are generated by EasyBuild via a `--try-*` option often do work fine, there is -no strong guarantee they will. Newer software versions may come with changes to -the installation procedure, additional dependencies that are required, etc. -Using a different compiler toolchain may be as simple as just switching one for -another, but it may require additional changes to be made to configure options, for example. - -## Copying easyconfigs - -One additional handy command line option we want to highlight is `--copy-ec`, which can be used to -copy easyconfig files to a specific location. That may sound trivial, but -keep in mind that you can specify easyconfigs to the `eb` command using only -the filename, and letting the robot search mechanism locate them. - -So to copy an easyconfig file, we would have to use `eb --search` first to -get the full location to it, copy-paste that, and then use the `cp` command. - -It is a lot easier with `--copy-ec`: - -```shell -$ eb --copy-ec SAMtools-1.11-GCC-10.2.0.eb SAMtools.eb -... -SAMtools-1.10-GCC-10.2.0.eb copied to SAMtools.eb -``` - -If you omit the target location, the easyconfig file will simply be copied -to the current working directory, retaining the original filename. - -You can copy multiple easyconfig files at once, as long as the target location -is an existing directory. - -## Example - -By means of example, we are going to puzzle together an easyconfig file to install the -example software package `eb-tutorial`. - -The sources for `eb-tutorial` version 1.0.1 are available at: - -``` -https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.0.1.tar.gz -``` - -You can consult the unpacked sources at [https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1). - -### Preparation - -Make sure that a proper version of the `LUMI` software stack is loaded -(we recommend `LUMI/21.12` for the session for which this tutorial was designed) -and also `EasyBuild-user` is loaded to configure EasyBuild to install in -`$HOME/EasyBuild` or `$EBU_USER_PREFIX`. - -``` shell -module load LUMI/21.12 -module load EasyBuild-user` -``` - - -### Easyblock - -Let's start by getting the mandatory easyconfig parameters defined in the easyconfig file: - -```python -name = 'eb-tutorial' -version = '1.0.1' - -homepage = 'https://easybuilders.github.io/easybuild-tutorial' - -whatis = [ 'Description: EasyBuild tutorial example'] - -description = """ -This is a short C++ example program that can be buid using CMake. -""" -``` - -Let's see what EasyBuild does with this very minimal easyconfig file: - -```shell -$ eb eb-tutorial.eb -== Temporary log file in case of crash /run/user/XXXXXXXXX/easybuild/tmp/eb-k_82wotb/easybuild-wg0k_reb.log -ERROR: Failed to process easyconfig /pfs/lustrep4/users/XXXXXXXX/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v1.eb: -No software-specific easyblock 'EB_eb_minus_tutorial' found for eb-tutorial -``` - -It is not mandatory to specify an easyblock in the easyconfig. However, in the absense of that -specification, EasyBuild goes looking for an application-specific easyblock with the standard name, -in this case `EB_eb_minus_tutorial`, which it does not have. Does that mean we have to implement an easyblock? - -In this simple case it doesn't, since we can leverage one of the available *generic easyblocks*. -But, which one? - -Build instructions are usually included in a `README` file, or in the documentation. -In this case, there's indeed a minimal [`README` -file](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1/README) available, -which tells us that we should use the `cmake` command to configure the installation, followed by `make` and `make install`. - -[We briefly discussed](#easyblock) a generic easyblock that does exactly this: `CMakeMake`. - -```python -easyblock = 'CMakeMake' -``` - -The "`easyblock =`" line is usually at the top of the easyconfig file, but strictly speaking -the order of the parameter definitions doesn't matter (unless one is defined in terms of another one). - - -### Mandatory parameters - -When trying this improved easyconfig file, EasyBuild will inform us that we failed to specify -one of the mandatory easyconfig parameters: `toolchain`: - -``` -$ eb example.eb -== Temporary log file in case of crash /run/user/XXXXXXXXX/easybuild/tmp/eb-22om7hut/easybuild-ipozjipf.log -ERROR: Failed to process easyconfig /pfs/lustrep4/users/XXXXXXXX/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v2.eb: -mandatory parameters not provided in pyheader: toolchain -``` - -We will use `cpeCray/21.12` as toolchain, so we also define the `toolchain` easyconfig parameter: - -```python -toolchain = {'name': 'cpeCray', 'version': '21.12'} -``` - -In addition, we'll also specify the `moduleclass`. -This is not required, but it is usually set to a sensible value: - -```python -moduleclass = 'tools' -``` - -The default value is '`base`', at least '`tools`' has *some* meaning. - -### CMake build dependency - -The `CMakeMake` easyblock needs the `cmake` command. On LUMI we are lucky as -`cmake` is already installed in the OS. It may be a somewhat older version, but for this -program is is probably enough. Yet in general it is better to use sufficiently recent -build tools, and `cmake` is one of those tools that is typically entered as a build -dependency. After all, the less you rely on the OS, the more likely it becomes that -your easyconfig is useful for other sites also. - -In the [section on Lmod](../1_02_Lmod#module-extensions) we've already seen that on LUMI -the `cmake` command is available through the `buildtools` modules, and as discussed in -other examples on this page, LUMI has one for every `LUMI` software stack with its version -number the same as the stack and corresponding toolchains. It is a good practice to -add this module as a build dependency: - -```python -builddependencies = [ - ('buildtools', '%(toolchain_version)s', '', True) -] -``` - -In a more traditional EasyBuild setup with the common toolchains, there is usually no need to specify -the toolchain for (build) dependencies. EasyBuild will automatically consider -[subtoolchains](../1_05_terminology/#toolchains) compatible with the specified toolchain to locate -modules for the dependencies. However, the Cray PE toolchains on LUMI are currently not part of -such a hierarchy and the `SYSTEM` toolchain we used for `buildtools` is not automatically considered -which is why we need the 4-element version of the dependency specification. - -You can verify that EasyBuild now locates the dependency via `eb -D` (equivalent with `eb --dry-run`): - -``` -$ eb eb-tutorial.eb -D - ... - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeCray/cpeCray-21.12.eb (module: cpeCray/21.12) - * [ ] /pfs/lustrep4/users/kulust/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v4.eb (module: eb-tutorial/1.0.-cpeCray-21.12) -``` - -### Sources - -If you try again after adding `buildtools` as a build dependency, you will see the installation fail again in the -configuration step. Inspecting the log file reveals this: - -``` -CMake Error: The source directory "/run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12" does not appear to contain CMakeLists.txt. -``` - -Wait, but there *is* a `CMakeLists.txt`, we can see it in the -[unpacked sources](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1)! - -Let's inspect the build directory: - -``` -$ ls /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12 -easybuild_obj -$ ls /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12/easybuild_obj -$ -``` - -There's nothing there at all! And that's not strange because we didn't actually -specify any sources in our easyconfig file... - -The `sources` easyconfig parameter is commonly defined but it is *not* mandatory, -because some easyconfig files only specify bundles of software packages and hence only -serve to generate a module file. - -So we need to specify one or more source files that should be used, -via the `sources` easyconfig parameter which specifies a *list* of -names of source files: - -```python -sources = ['eb-tutorial-1.0.1.tar.gz'] -``` - -We can avoid hardcoding the version number here by using a *template value*: - -```python -sources = ['eb-tutorial-%(version)s.tar.gz'] -``` - -And since this is a standard way of naming software files, there's -even a constant available that we can use: - -```python -sources = [SOURCE_TAR_GZ] -``` - -That way, we only have the software version specified *once* in the easyconfig file, -via the `version` easyconfig parameter. That will come in useful later (see [Exercise 7.2](#exercises))... - -If now we try installing the easyconfig file again, EasyBuild complains -that it can't find the specified source file anywhere: - -``` -Couldn't find file eb-tutorial-1.0.1.tar.gz anywhere, and downloading it didn't work either... -Paths attempted (in order):... -``` - -To let EasyBuild automatically download the source file if it is not available yet, -we have to specify *where* it can be downloaded. This is done via `source_urls`: - -```python -source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] -sources = [SOURCE_TAR_GZ] -``` - -### Required configure option - -With `sources` and `source_urls` defined, we can try again. Yet again we see the configure step fail. -Is this a ruse to make you hate CMake with a passion? Maybe... - -Here's what we find in the log file: - -``` -CMake Error at CMakeLists.txt:7 (message): - EBTUTORIAL_MSG is not set! -``` - -Apparently the `eb-tutorial` software has a required configure option. It's almost as if that -was done on purpose, how silly! - -Options to the configure command can be specified by the `configopts` easyconfig parameter. -To define the value of a CMake option, we need to use `-DNAME_OF_OPTION`, so: - -```python -configopts = "-DEBTUTORIAL_MSG='Hello from the EasyBuild tutorial!' " -``` - -We need to be a little bit careful with quotes here. If we use outer double quotes, -we have to use single quotes to specify the actual value for the `EBTUTORIAL_MSG` configure option. -That works fine here, but that's not always the case! -In some cases we will have to use inner doubles quotes, for example to get environment variables -expanded when the configure command is run (see [Exercise 7.1](#exercises)). - -### Sanity check - -Hopefully that brings us closer to getting the installation to work... - -``` -$ eb eb-tutorial.eb -.... -== sanity checking... -== ... (took 2 secs) -== FAILED: Installation ended unsuccessfully (build directory: /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12): -build failed (first 300 chars): Sanity check failed: no (non-empty) directory found at 'lib' or 'lib64' in -/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 (took 7 secs) -``` - -It got all the way to the sanity check step, that's great! - -The sanity check failed because no '`lib`' or `'lib64'` directory was found. -Indeed: - -``` -$ ls /users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 -bin -$ ls /users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 -eb-tutorial -``` - -There is only a binary named `eb-tutorial` in the `bin` subdirectory. -So we need to customize the standard sanity check: - -```python -sanity_check_paths = { - 'files': ['bin/eb-tutorial'], - 'dirs': [], -} -``` - -Since we want to obtain a *working* installation, we might as well try to run this `eb-tutorial` command as well: - -```python -sanity_check_commands = ['eb-tutorial'] -``` - -Let us now retry, but use `--module-only` rather than redoing the whole installation. -`--module-only` still sanity checks the installation, so if it creates -a module, we know it will work as expected. -By enabling trace mode via `--trace` we can get some more information too: - -```shell -$ eb eb-tutorial.eb --module-only --trace -... -== sanity checking... - >> file 'bin/eb-tutorial' found: OK - >> loading modules: eb-tutorial/1.0.1-cpeCray-21.12... - >> running command 'eb-tutorial' ... - >> result for command 'eb-tutorial': OK -... -== COMPLETED: Installation ended successfully (took 4 sec) -``` - -Yes, great success! - -To convince yourself that the installation works as intended, try to load the `eb-tutorial` module and -run the `eb-tutorial` command yourself: - -``` -$ module load eb-tutorial -$ eb-tutorial -Hello from the EasyBuild tutorial! -``` - -### Complete easyconfig - -Here is the complete easyconfig we puzzled together for this example: - -```python -easyblock = 'CMakeMake' - -name = 'eb-tutorial' -version = '1.0.1' - -homepage = 'https://easybuilders.github.io/easybuild-tutorial' -description = "EasyBuild tutorial example" - -toolchain = {'name': 'GCC', 'version': '10.2.0'} - -source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] -sources = [SOURCE_TAR_GZ] -checksums = ['d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9'] - -builddependencies = [('CMake', '3.18.4')] - -configopts = "-DEBTUTORIAL_MSG='Hello from the EasyBuild tutorial!' " - -sanity_check_paths = { - 'files': ['bin/eb-tutorial'], - 'dirs': [], -} - -sanity_check_commands = ['eb-tutorial'] - -moduleclass = 'tools' -``` - -One additional easyconfig parameter we defined here is `checksums`, -which specifies SHA256 checksums for source (and patch) files, -so EasyBuild can verify them before performing an installation. - -You can let EasyBuild determine *and* inject these SHA256 checksums -automatically via `eb --inject-checksums`: - -``` -$ eb eb-tutorial.eb --inject-checksums -... -== injecting sha256 checksums for sources & patches in example.eb... -== * eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 -``` - -Finally, we should consider changing the name of the easyconfig that we just developed -to align with the EasyBuild conventions as otherwise it would not be found when used as -a dependency of another package. In this case, the name should be -`eb-tutorial-1.0.0-cpeCray-21.12.eb`. In fact, EasyBuild stored a processed version -of our easyconfig with that name in the repository: - -``` -$ ls $EASYBUILD_REPOSITORYPATH -eb-tutorial -$ ls $EASYBUILD_REPOSITORYPATH/eb-tutorial -eb-tutorial-1.0.1-cpeCray-21.12.eb -$ cat $EASYBUILD_REPOSITORYPATH/eb-tutorial/eb-tutorial-1.0.1-cpeCray-21.12.eb -... -``` - -Note that EasyBuild has added an additional parameter to the easyconfig file, `buildstats`, -with a lot of information about how easybuild was called and properties of the node on which -it was run. This file is still a valid easyconfig file though from which we can build the -program again. - -Let's also inspect the installation directory when the whole build process has finished successfully. -This is very easy after loading the module, as EasyBuild-generated modules define a number of environment -variables for each module: - -``` -$ module load eb-tutorial -$ env | grep TUTORIAL -EBVERSIONEBMINTUTORIAL=1.0.1 -EBDEVELEBMINTUTORIAL=/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12/easybuild/eb-tutorial-1.0.1-cpeCray-21.12-easybuild-devel -EBROOTEBMINTUTORIAL=/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 -``` -The most interesting one of those variables is the `EBROOT` variable which points to the installation directory. -As variable names cannot contain minus signs, the minus in the module name is replaced with `MIN` in the name -of the variable (which is not the most consistent thing however as in the name of an easyblock it is replaced -with `minus`). - -Let's have a look in that directory: - -``` -$ ls $EBROOTEBMINTUTORIAL -bin easybuild -$ ls EBROOTEBMINTUTORIAL/easybuild -easybuild-eb-tutorial-1.0.1-20220401.184518.log -easybuild-eb-tutorial-1.0.1-20220401.184518_test_report.md -eb-tutorial-1.0.1-cpeCray-21.12-easybuild-devel -eb-tutorial-1.0.1-cpeCray-21.12.eb -reprod -$ ls EBROOTEBMINTUTORIAL/easybuild/reprod -easyblocks -eb-tutorial-1.0.1-cpeCray-21.12.eb -eb-tutorial-1.0.1-cpeCray-21.12.env -hooks -ls $EBROOTEBMINTUTORIAL/easybuild/reprod/easyblocks -cmakemake.py configuremake.py -``` - -As you can see, EasyBuild has also created the `easybuild` subdirectory (and it actually told us about that -at the end of the installation) which contains a lot of information about the build, also to make it easier -to reproduce a build process afterwards. - - ---- - -## Exercises - ---- - -***Exercise E.1**** - Making `eb-tutorial` a bit more personal* - -Change the easyconfig file for `eb-tutorial` to make the message printed by the `eb-tutorial` command -a bit more personal: include the username of the account that was used to install the software in it -(using the `$USER` environment variable). - -??? success "(click to show solution)" - For this we need to change the value that is passed to the `EBTUTORIAL_MSG` configure option: - ```python - configopts = '-DEBTUTORIAL_MSG="Hello from the EasyBuild tutorial! I was installed by $USER." ' - ``` - Here we have to use inner double quotes, to ensure that the `$USER` environment variable is expanded - by the shell when running the `cmake` configure command. - - When you run the `eb-tutorial` command yourself, you should get output like this (not a message that - includes a literal '`$USER`' string): - - ```shell - Hello from the EasyBuild tutorial! I was installed by example. - ``` - - To re-install the `example.eb` easyconfig, you will need to use `eb --rebuild` or `eb --force`. - EasyBuild will also complain about modules that are loaded already if you tested the program above. - A good way to get rid of all those modules on LUMI is to simply use `module purge`. You don't need - to reload the software stack, but you will need to load `EasyBuild-user` again. - ---- - -***Exercise E.2**** - Installing eb-tutorial version 1.1.0* - -Install version 1.1.0 of the `eb-tutorial` example software, -which is a trivial version bump compared to version 1.0.1. - -The sources are available via: - -``` -https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.1.0.tar.gz -``` - -You can leverage the `eb-tutorial` easyconfig file we have composed in the example above, -**but you should not make any manual changes to it!** - -??? success "(click to show solution)" - You can use the `--try-software-version` option for this: - ```shell - $ eb eb-tutorial.eb --try-software-version 1.1.0 - ... - == building and installing eb-tutorial/1.1.0-cpeCray-21.12... - ... - == COMPLETED: Installation ended successfully (took 4 sec) - ``` - To test: - ``` - $ module load eb-tutorial/1.1.0-cpeCray-21.12 - $ eb-tutorial - I have a message for you: - Hello from the EasyBuild tutorial! I was installed by XXXXXXXX. - ``` - (`eb-tutorial` version 1.0.1 doesn't print "`I have a message for you:`") - - EasyBuild has also created a new easyconfig for this configuration and stored - in the repository and the `easybuild` subdirectory from the installation - directory. As on LUMI the repository is in the search path we can actually copy - the file back to the current directory: - ``` - eb --copy-ec eb-tutorial-1.1.0-cpeCray-21.12. ``` - Some of the formatting is lost though and the checksum is still missing, so you may want - to do some cleaning up. - ``` - eb eb-tutorial-1.1.0-cpeCray-21.12.eb --inject-checksum - ``` - - ---- - -*[[next: Using external modules from the Cray PE]](2_03_external_modules)* diff --git a/docs/2022-CSC_and_LO/2_03_external_modules.md b/docs/2022-CSC_and_LO/2_03_external_modules.md deleted file mode 100644 index 80a29712..00000000 --- a/docs/2022-CSC_and_LO/2_03_external_modules.md +++ /dev/null @@ -1,85 +0,0 @@ -# Using external modules from the Cray PE - -*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files)* - ---- - -## What are external modules? - -EasyBuild supports the use of modules that were not installed via EasyBuild. -We refer to such modules as [external modules](https://docs.easybuild.io/en/latest/Using_external_modules.html). -External modules do not define the `EBROOT*` and `EBVERSION*` environment variables that EasyBuild would define -and uses internally in several easyblocks and some easyconfig files. - -External modules are used extensively on Cray systems to interface with the Cray PE (which comes with its own -modules and cannot be installed via EasyBuild): -[external modules can be used as dependencies](https://docs.easybuild.io/en/latest/Using_external_modules.html#using-external-modules-as-dependencies), -by including the module name in the dependencies list, -along with the `EXTERNAL_MODULE` constant marker. - -For example, to specify the module `cray-fftw` as a dependency, you should write the following in your easyconfig file: -``` python -dependencies = [('cray-fftw', EXTERNAL_MODULE)] -``` - -For such dependencies, EasyBuild will: - -* load the module before initiating the software build and install procedure - -* include a `module load` statement in the generated module file (for runtime dependencies) - -!!! Note - The default version of the external module will be loaded unless a specific version is given as dependency, - and here that version needs to be given as part of the name of the module and not as the second element in the - tuple. - - ```python - dependencies = [('cray-fftw/3.3.8.12', EXTERNAL_MODULE)] - ``` - -If the specified module is not available, EasyBuild will exit with an error message stating that the dependency -can not be resolved because the module could not be found, without searching for a matching easyconfig file -from which it could generate the module. - - ---- - -## EasyBuild Metadata for external modules - -[Metadata](https://docs.easybuild.io/en/latest/Using_external_modules.html#metadata-for-external-modules) -can be supplied to EasyBuild for external modules: using the `--external-modules-metadata` -configuration option, the location of one or more metadata files can be specified. - -The files are expected to be in INI format, with a section per module name -and key-value assignments specific to that module. - -The external modules metadata file can be also defined with the corresponding environment variable: -``` -echo $EASYBUILD_EXTERNAL_MODULES_METADATA -/apps/common/UES/jenkins/production/easybuild/cpe_external_modules_metadata-21.04.cfg -``` - -The following keys are -[supported by EasyBuild](https://docs.easybuild.io/en/latest/Using_external_modules.html#supported-metadata-values): - -* name: software name(s) provided by the module -* version: software version(s) provided by the module -* prefix: installation prefix of the software provided by the module - -For instance, the external module version loaded by the dependency `cray-fftw` can be specified as follows: -```ini -[cray-fftw] -name = FFTW -prefix = FFTW_DIR/.. -version = 3.3.8.10 -``` - -The environment variable `$EBROOTFFTW` will also be defined according to the `prefix` specified in the metadata file. - -On LUMI, users in generally don't need to be too concerned about the metadata file as the EasyBuild-user (and other hidden -EasyBuild configuration modules) take care of pointing to the right metadata file, which is specific for each version of the -Cray PE and hence each version of the LUMI software stack. - ---- - -*[[next: Implementing easyblocks]](2_04_implementing_easyblocks.md)* diff --git a/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md deleted file mode 100644 index a91b0dc8..00000000 --- a/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md +++ /dev/null @@ -1,754 +0,0 @@ -# Implementing easyblocks - -*[[back: Using external modules from the Cray PE]](2_03_external_modules.md)* - ---- - -## The basics - -An *easyblock* is a Python module that implements a software build and installation procedure. - -This concept is essentially implemented as a Python script that plugs into the EasyBuild framework. - -EasyBuild will leverage easyblocks as needed, depending on which software packages it needs to install. Which easyblock is required is determined by the ``easyblock`` easyconfig parameter, if it is present, or by the software name. - - -## Generic vs software-specific easyblocks - -Easyblocks can either be *generic* or *software-specific*. - -*Generic* easyblocks implement a "standard" software build and installation procedure that is used by multiple different -software packages. -A commonly used example is the -[``ConfigureMake``](https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/generic/configuremake.py) -generic easyblock, which implements the standard ``configure`` - ``make`` - ``make install`` installation procedure used -by most GNU software packages. - -*Software-specific* easyblocks implement the build and installation procedure for a particular software package. -Typically this involves highly customised steps, for example specifying dedicated configuration options, creating -or adjusting specific files, executing non-standard shell commands, etc. Usually a custom implementation of the -sanity check is also included. Much of the work done in software-specific easyblocks can often also be done -in generic easyblocks using parameters such as ``confdigopts`` etc., but a software-specific easyblock can -hide some of that complexity from the user. Other software-specific easyblocks implement very specific -installation procedures that do not fit in one of the generic ones. - -Using a generic easyblock requires specifying the ``easyblock`` parameter in the easyconfig file. -If it is not specified, EasyBuild will try and find the software-specific easyblock derived from the software name. - -The distinction between generic and software-specific easyblocks can be made based on the naming scheme that is used -for an easyblock (see below). - - -## Naming - -Easyblocks need to follow a strict naming scheme, to ensure that EasyBuild can pick them up automatically as needed. -This involves two aspects: - -* the name of the Python class; -* the name and location of the Python module file. - -### Python class name - -The name of the Python class is determined by the *software name* for software-specific easyblocks. -It consists of a prefix '``EB_``', followed by the (encoded) software name. - -Because of limitations in Python on characters allowed in names of Python classes, -only alphanumeric characters and underscores (``_``) are allowed. Any other characters are replaced following an encoding scheme: - -* spaces are replaced by underscores (``_``); -* dashes ``-`` are replaced by ``_minus_`` (note the inconsistency with the naming of ``EBROOT`` and ``EBVERSION`` variables); -* underscores are replaced by ``_underscore_``; - -The ``encode_class_name`` function provided in ``easybuild.tools.filetools`` returns the expected class name -for a given software name; for example: - -```shell -$ python3 -c "from easybuild.tools.filetools import encode_class_name; print(encode_class_name('netCDF-Fortran'))" -EB_netCDF_minus_Fortran -``` - -**Python class name for *generic* easyblocks** - -For *generic* easyblocks, the class name does *not* include an ``EB_`` prefix (since there is no need for an escaping -mechanism) and hence the name is fully free to choose, taking into account the restriction to alphanumeric characters -and underscores. - -For code style reasons, the class name should start with a capital letter and use CamelCasing. - -Examples include ``Bundle``, ``ConfigureMake``, ``CMakePythonPackage``. - -### Python module name and location - -The *filename* of the Python module is directly related to the name of Python class it provides. - -It should: - -* *not* include the ``EB_`` prefix of the class name for software-specific easyblocks; -* consists only of lower-case alphanumeric characters (``[a-z0-9]``) and underscores (``_``); - * dashes (``-``) are replaced by underscores (``_``); - * any other non-alphanumeric characters (incl. spaces) are simply dropped; - -Examples include: - -* ``gcc.py`` (for *GCC*) -* ``netcdf_fortran.py`` (for *netCDF-Fortran*) -* ``gamess_us.py`` (for *GAMESS (US)*) - -The ``get_module_path`` function provided by the EasyBuild framework in the -``easybuild.framework.easyconfig.easyconfig`` module returns the (full) -module location for a particular software name or easyblock class name. For example: - -```python ->>> from easybuild.framework.easyconfig.easyconfig import get_module_path ->>> get_module_path('netCDF-Fortran') -'easybuild.easyblocks.netcdf_fortran' ->>> get_module_path('EB_netCDF_minus_Fortran') -'easybuild.easyblocks.netcdf_fortran' -``` - -The location of the Python module is determined by whether the easyblock is generic or software-specific. -Generic easyblocks are located in the ``easybuid.easyblocks.generic`` namespace, while software-specific easyblocks -live in the ``easybuild.easyblocks`` namespace directly. - -To keep things organised, the actual Python module files -for software-specific easyblocks are kept in 'letter' subdirectories, -rather than in one large '``easyblocks``' directory -(see -[https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/](https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/)), -but this namespace is collapsed transparently by EasyBuild (you don't need to import from letter subpackages). - -To let EasyBuild pick up one or more new or customized easyblocks, you can use the [``--include-easyblocks``](https://docs.easybuild.io/en/latest/Including_additional_Python_modules.html#including-additional-easyblocks-include-easyblocks) -configuration option. As long as both the filename of the Python module and the name of the Python class -are correct, EasyBuild will use these easyblocks when needed. - -On LUMI, the EasyBuild configuration modules take care of setting this parameter (using the corresponding environment -variable), pointing to custom easyblocks in the LUMI software stack itself and a repo (with a fixed name) that users -can create themselves. At this moment it does not yet include possible other easyblock repositories in other repositories. - - -## Structure of an easyblock - -The example below shows the overal structure of an easyblock: - -```python -from easybuild.framework.easyblock import EasyBlock -from easybuild.tools.run import run_cmd - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def configure_step(self): - """Custom implementation of configure step for Example""" - - # run configure.sh to configure the build - run_cmd("./configure.sh --install-prefix=%s" % self.installdir) -``` - - -Each easyblock includes an implementation of a ``class`` that (directly or indirectly) derives from the abstract -``EasyBlock`` class. - -Typically some useful functions provided by the EasyBuild framework are imported at the top of the Python module. - -In the class definition, one or more '``*_step``' methods (and perhaps a couple of others) are redefined, -to implement the corresponding step in the build and installation procedure. - -Each easyblock *must* implement the ``configure``, ``build`` and ``install`` steps, since these are not implemented -in the abstract ``EasyBlock`` class. This could be done explicitly by redefining the corresponding ``*_step`` methods, -or implicitly by deriving from existing (generic) easyblocks. - -The full list of methods that can be redefined in an easyblock can be consulted in -the [API documentation](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyblock.html). - -## Deriving from existing easyblocks - -When implementing an easyblock, it is common to derive from an existing (usually generic) easyblock, -and to leverage the functionality provided by it. This approach is typically used when only a specific part -of the build and installation procedure needs to be customised. - -In the (fictitious) example below, we derive from the generic ``ConfigureMake`` easyblock to redefine the ``configure`` -step. In this case, we are *extending* the ``configure`` step as implemented by ``ConfigureMake`` rather than -redefining it entirely, since we call out to the original ``configure_step`` method at the end. - -```python -from easybuild.easyblocks.generic.configuremake import ConfigureMake -from easybuild.tools.filetools import copy_file - - -class EB_Example(ConfigureMake): - """Custom easyblock for Example""" - - def configure_step(self): - """Custom implementation of configure step for Example""" - - # use example make.cfg for x86-64 - copy_file('make.cfg.x86', 'make.cfg') - - # call out to original configure_step implementation of ConfigureMake easyblock - super(EB_Example, self).configure_step() -``` - -## Easyconfig parameters - -All of the easyconfig parameters that are defined in an easyconfig file -are available via the ``EasyConfig`` instance that can be accessed through ``self.cfg`` in an easyblock. -For instance, if the easyconfig file specifies - -```python -name = 'example' -version = '2.5.3' -versionsuffix = '-Python-3.7.4' -``` - -then these three parameters are accessible within an easyblock via ``self.cfg['name']``, ``self.cfg['version']`` -and ``self.cfg['versionsuffix']``. - -A few of the most commonly used parameters can be referenced directly: - -* **``self.name``** is equivalent with ``self.cfg['name']``; -* **``self.version``** is equivalent with ``self.cfg['version']``; -* **``self.toolchain``** is equivalent with ``self.cfg['toolchain']``; - - -### Updating parameters - -You will often find that you need to *update* some easyconfig parameters in an easyblock, -for example ``configopts`` which specifies options for the configure command. - -Because of implementation details (related to -how template values like ``%(version)s`` are handled), you need to be a bit careful here... - -To completely redefine the value of an easyconfig parameter, you can use simple assignment. For example: - -```python -self.cfg['example'] = "A new value for the example easyconfig parameter." -``` - -If want to *add* to the existing value however, you *must* use the ``self.cfg.update`` method. For example: - -```python -self.cfg.update('some_list', 'example') -``` - -This will ***not*** work (because ``self.cfg['some_list']`` does not return a reference to the original value, -but to a *temporary copy* thereof): - -```python -# anti-pattern, this does NOT work as expected! -self.cfg['some_list'].append('example') -``` - -### Custom parameters - -Additional custom easyconfig parameters can be defined in an easyblock to steer its behaviour. -This is done via the ``extra_options`` *static* method. Custom parameters can be specified to be mandatory. - -The example below shows how this can be implemented: - -```python -from easybuild.easyblocks.generic.configuremake import ConfigureMake -from easybuild.framework.easyconfig import CUSTOM, MANDATORY - - -class EB_Example(ConfigureMake): - """Custom easyblock for Example""" - - @staticmethod - def extra_options(): - """Custom easyconfig parameters for Example""" - extra_vars = { - 'required_example_param': [None, "Example required custom parameter", MANDATORY], - 'optional_example_param': [None, "Example optional custom parameter", CUSTOM], - } - return ConfigureMake.extra_options(extra_vars) -``` - -The first element in the list of a defined custom parameter corresponds to the default value for that parameter -(both ``None`` in the example above). The second element provides some informative help text -(which can then be displayed with ``eb -a -e ``, eg, ``eb -a -e EB_GCC``), -and the last element -indicates whether the parameter is mandatory (``MANDATORY``) or just an optional custom parameter (``CUSTOM``). - -## Easyblock constructor - -In the ``class`` constructor of the easyblock, i.e. the ``__init__`` method, one or more class variables -can be initialised. These can be used for sharing information between different ``*_step`` methods in the easyblock. - -For example: - -```python -from easybuild.framework.easyblock import EasyBlock - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def __init__(self, *args, **kwargs): - """Constructor for Example easyblock, initialises class variables.""" - - # call out to original constructor first, so 'self' (i.e. the class instance) is initialised - super(EB_Example, self).__init__(*args, **kwargs) - - # initialise class variables - self.example_value = None - self.example_list = [] -``` - -## File operations - -File operations is a common use case for implementing easyblocks, hence the EasyBuild framework provides a -number of useful functions related to this, including: - -* ``read_file()``: read file at a specified location and returns its contents; - -* ``write_file(, )`` at a specified location with provided contents; - to append to an existing file, use ``append=True`` as an extra argument; - -* ``copy_file(, )`` to copy an existing file; - -* ``apply_regex_substitutions(, )`` to patch an existing file; - -All of these functions are provided by the [``easybuild.tools.filetools``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html) module. - -## Executing shell commands - -For executing shell commands two functions are provided by the -[``easybuild.tools.run``](https://docs.easybuild.io/en/latest/api/easybuild.tools.run.html) module: - -* ``run_cmd()`` to run a non-interactive shell command; - -* ``run_cmd_qa(, )`` to run an interactive shell command; - -Both of these accept a number of optional arguments: - -* ``simple=True`` to just return ``True`` or ``False`` to indicate a successful execution, - rather than the default return value, i.e., a tuple that provides the command output and the exit code (in that - order); - -* ``path=`` to run the command in a specific subdirectory; - -The ``run_cmd_qa`` function takes two additional specific arguments: - -* ``no_qa=`` to specify a list of patterns to recognize non-questions; - -* ``std_qa=`` to specify regular expression patterns for common questions, and the matching answer; - -## Manipulating environment variables - -To (re)define environment variables, the ``setvar`` function provided by the -[``easybuild.tools.environment``](https://docs.easybuild.io/en/latest/api/easybuild.tools.environment.html) -module should be used. - -This makes sure that the changes being made to the specified environment variable are kept track of, -and that they are handled correctly under ``--extended-dry-run``. - -## Logging and errors - -It is good practice to include meaningful log messages in the ``*_step`` methods being customised in the easyblock, -to enrich the EasyBuild log with useful information for later debugging or diagnostics. - -For logging, the provided ``self.log`` logger class should be used. -You can use the ``self.log.info`` method to log an informative message. -Similar methods are available for logging debug messages (``self.log.debug``), which are -only emitted when ``eb`` is run with debugging mode enabled (``--debug`` or ``-d``), -and for logging warning messages (``self.log.warning``). - -If something goes wrong, you can raise an ``EasyBuildError`` instance to report the error. - -For example: - -```python -from easybuild.framework.easyblock import EasyBlock -from easybuild.tools.build_log import EasyBuildError -from easybuild.tools.run import run_cmd - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def configure_step(self): - """Custom implementation of configure step for Example""" - - cmd = "./configure --prefix %s" % self.installdir) - out, ec = run_cmd(cmd) - - success = 'SUCCESS' - if success in out: - self.log.info("Configuration command '%s' completed with success." % cmd) - else: - raise EasyBuildError("Pattern '%s' was not found in output of '%s'." % (success, cmd)) -``` - -## Custom sanity check - -For software-specific easyblocks, a custom sanity check is usually included to verify that the installation was -successful or not. - -This is done by redefining the ``sanity_check_step`` method in the easyblock. For example: - -```python -from easybuild.framework.easyblock import EasyBlock - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def sanity_check_step(self): - """Custom sanity check for Example.""" - - custom_paths = { - 'files': ['bin/example'], - 'dirs': ['lib/examples/'], - } - custom_commands = ['example --version'] - - # call out to parent to do the actual sanity checking, pass through custom paths and commands - super(EB_Example, self).sanity_check_step(custom_paths=custom_paths, custom_commands=custom_commands) -``` - -You can both specify file paths and subdirectories to check for, which are specified relative to the installation directory, -as well as simple commands that should execute successfully after completing the installation and loading the generated module file. - -It is up to you how extensive you make the sanity check, but it is recommended to make the check as complete -as possible to catch any potential build or installation problems that may occur, while ensuring that it can -run relatively quickly (in seconds, or at most a couple of minutes). - -## Version-specific parts - -In some cases version-specific actions or checks need to be included in an easyblock. -For this, it is recommended to use ``LooseVersion`` rather than directly comparing version numbers using string values. - -For example: - -```python -from distutils.version import LooseVersion -from easybuild.framework.easyblock import EasyBlock - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def sanity_check_step(self): - """Custom sanity check for Example.""" - - custom_paths = { - 'files': [], - 'dirs': [], - } - - # in older versions, the binary used to be named 'EXAMPLE' rather than 'example' - if LooseVersion(self.version) < LooseVersion('1.0'): - custom_paths['files'].append('bin/EXAMPLE') - else: - custom_paths['files'].append('bin/example') - - super(EB_Example, self).sanity_check_step(custom_paths=custom_paths) -``` - -## Compatibility with ``--extended-dry-run`` and ``--module-only`` - -Some special care must be taken to ensure that an easyblock is fully compatible with ``--extended-dry-run`` / ``-x`` -(see [Inspecting install procedures](../basic_usage/#inspecting-install-procedures)) and ``--module-only``. - -For compatibility with ``--extended-dry-run``, you need to take into account that specified operations -like manipulating files or running shell commands will not actually be executed. You can check -whether an easyblock is being run in dry run mode via ``self.dry_run``. - -For example: - -```python -from easybuild.framework.easyblock import EasyBlock -from easybuild.tools.build_log import EasyBuildError -from easybuild.tools.run import run_cmd - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def configure_step(self): - """Custom implementation of configure step for Example""" - - cmd = "./configure --prefix %s" % self.installdir) - out, ec = run_cmd(cmd) - - success = 'SUCCESS' - if success in out: - self.log.info("Configuration command '%s' completed with success." % cmd) - - # take into account that in dry run mode we won't get any output at all - elif self.dry_run: - self.log.info("Ignoring missing '%s' pattern since we're running in dry run mode." % success) - - else: - raise EasyBuildError("Pattern '%s' was not found in output of '%s'." % (success, cmd)) -``` - -For ``--module-only``, you should make sure that both the ``make_module_step``, including the ``make_module_*`` submethods, -and the ``sanity_check_step`` methods do not make any assumptions about the presence of certain environment variables, or that class variables have been defined already. - -This is required because under ``--module-only`` the large majority of the ``*_step`` functions are -simply *skipped* entirely. So, if the ``configure_step`` method is responsible for defining class variables that are -picked up in ``sanity_check_step``, the latter may run into unexpected initial values like ``None``. -A possible workaround is to define a separate custom method to define the class variables, and to call out to this -method from ``configure_step`` and ``sanity_check_step`` (for the latter, conditionally, i.e., only if the class -variables still have the initial values). - -For example: - -```python -from easybuild.framework.easyblock import EasyBlock - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def __init__(self, *args, **kwargs): - """Easyblock constructor.""" - super(EB_Example, self).__init__(*args, **kwargs) - - self.command = None - - def set_command(self): - """Initialize 'command' class variable.""" - # $CC environment variable set by 'prepare' step determines exact command - self.command = self.name + '-' + os.getenv('CC') - - def configure_step(self): - """Custom configure step for Example.""" - - self.set_command() - self.cfg.update('configopts', "COMMAND=%s" % self.command) - - super(EB_Example, self).configure_step() - - def sanity_check_step(self): - """Custom implementation of configure step for Example""" - - if self.command is None: - self.set_command() - - super(EB_Example, self).sanity_check_step(custom_commands=[self.command]) -``` - - -## Easyblocks in the Cray ecosystem - -The generic easyblocks are usually rather independent of compilers etc. and tend to work well with all toolchains. -However, software-specific easyblocks may contain code that is specific for certain toolchains and are often only -tested with the common toolchains (foss and intel and their subtoolchains). Many of those easyblocks will fail -on Cray systems (or any system that uses other toolchains) as they don't recognise the compiler and rather than -implementing some generic behaviour that may or may not work, produce an error message instead that the compiler -toolchain is not supported. - -Several packages on LUMI therefore use generic easyblocks rather than the software-specific easyblocks that may -exist for those applications. Adapting those software-specific easyblocks for LUMI poses an interesting maintenance -problem. Either one could decide to not contribute back to the community, but this implies then that all modifications -made to the corresponding easyblocks in the EasyBuild distribution should be monitored and implemented in the custom -easyblocks for Cray also. On the other hand, contributing back to the community also poses two problems. First it -would also require to implement the Cray toolchains as used on LUMI in the core of EasyBuild (which already contains -a different set of toolchains targeted more at how the Cray PE works with the regular environment modules), and that -only makes sense if these toolchains are first extended to not only cover the programming environments supported on -LUMI but also the Intel and NVIDIA programming environments. Second, the EasyBuild community has no easy way of testing -any modification made to such an easyblock on a Cray PE system. Hence every update made in the community may break -the Cray PE support again. - - -## Exercise - -### Exercise I.1 - -Try implementing a new custom easyblock for ``eb-tutorial``, which derives directly -from the base ``EasyBlock`` class. - -Your easyblock should: - -* define a custom mandatory easyconfig parameter named ``message``; -* run `cmake` to configure the installation, which includes at least: - * specifying the correct installation prefix (using the `-DCMAKE_INSTALL_PREFIX=...` option); - * passing down the value of ``message`` easyconfig parameter via `-DEBTUTORIAL_MSG=...` -* run `make` to build `eb-tutorial`; -* run `make install` to install the generated binary; -* perform a custom sanity check to ensure the installation is correct; -* pick up on commonly used easyconfig parameters like `configopts` and `preinstallopts` where appropriate; - -??? success "(click to show solution)" - - Here's a complete custom easyblock for ``eb-tutorial`` that derives from the base ``EasyBlock`` class, - which should be included in a file named ``eb_tutorial.py``. - - We need to implement the ``configure_step``, ``build_step``, and ``install_step`` methods in - order to have a fully functional easyblock. - - The configure, build, and install steps take into account the corresponding easyconfig - parameters that allow customizing these commands from an easyconfig file. - - ```python - from easybuild.framework.easyblock import EasyBlock - from easybuild.framework.easyconfig import MANDATORY - from easybuild.tools.run import run_cmd - - - class EB_eb_minus_tutorial(EasyBlock): - """Custom easyblock for eb-tutorial.""" - - @staticmethod - def extra_options(): - extra = EasyBlock.extra_options() - extra.update({ - 'message': [None, "Message that eb-tutorial command should print", MANDATORY], - }) - return extra - - def configure_step(self): - """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" - - cmd = ' '.join([ - self.cfg['preconfigopts'], - 'cmake', - '-DCMAKE_INSTALL_PREFIX=\'%s\'' % self.installdir, - '-DEBTUTORIAL_MSG="%s"' % self.cfg['message'], - self.cfg['configopts'], - ]) - run_cmd(cmd) - - def build_step(self): - """Build step for eb-tutorial""" - - cmd = ' '.join([ - self.cfg['prebuildopts'], - 'make', - self.cfg['buildopts'], - ]) - run_cmd(cmd) - - def install_step(self): - """Install step for eb-tutorial""" - - cmd = ' '.join([ - self.cfg['preinstallopts'], - 'make install', - self.cfg['installopts'], - ]) - run_cmd(cmd) - - def sanity_check_step(self): - custom_paths = { - 'files': ['bin/eb-tutorial'], - 'dirs': [], - } - custom_commands = ['eb-tutorial'] - return super(EB_eb_minus_tutorial, self).sanity_check_step(custom_paths=custom_paths, - custom_commands=custom_commands) - ``` - - We also need to adapt our easyconfig file for ``eb-tutorial``: - - - The ``easyblock`` line is no longer needed as we will rely on the automatic selection of the - software-specific easyblock. - - We don't need to define the message through ``configopts`` but via the easyblock-specific - configuration parameter ``message``. In fact, we were so careful when implementing the ``configure_step`` - that even variable expansion will still work so we can still include ``$USER`` in the message. - - The sanity check is also no longer needed as it is done by the software-specific easyblock. - - So the easyconfig file simplifies to: - - ```python - name = 'eb-tutorial' - version = "1.1.0" - - homepage = 'https://easybuilders.github.io/easybuild-tutorial' - - whatis = [ 'Description: EasyBuild tutorial example'] - - description = """ - This is a short C++ example program that can be buid using CMake. - """ - - toolchain = {'name': 'cpeCray', 'version': '21.12'} - - builddependencies = [ - ('buildtools', '%(toolchain_version)s', '', True) - ] - - source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] - sources = [SOURCE_TAR_GZ] - checksums = ['def18b69b11a3ec34ef2a81752603b2118cf1a57e350aee41de9ea13c2e6a7ef'] - - message = 'Hello from the EasyBuild tutorial! I was installed by $USER.' - - moduleclass = 'tools' - - ``` - - Running this example on LUMI is a little tricky as using ``--include-easyblocks`` to point EasyBuild to - our new easyblock interfers with settings already made by the EasyBuild configuration modules (``EasyBuild-user``) - and causes error messages about the toolchains. So either the easyblock needs to be copied to the user location - that can be found by looking at the output of ``eb --show-config`` or we simply need to extend the list of - easyblocks that EasyBuild searches with the easyblocks in the current directory: - - ``` shell - EASYBUILD_INCLUDE_EASYBLOCKS="$EASYBUILD_INCLUDE_EASYBLOCKS,./*.py" - ``` - - - - -### Exercise I.2 - -Try implementing another new custom easyblock for ``eb-tutorial``, -which derives from the generic ``CMakeMake`` easyblock. - -Your easyblock should only: - -* define a custom mandatory easyconfig parameter named ``message``; -* pass down the value of ``message`` easyconfig parameter via `-DEBTUTORIAL_MSG=...` -* perform a custom sanity check to ensure the installation is correct; - -??? success "(click to show solution)" - - When deriving from the ``CMakeMake`` generic easyblock, there is a lot less to worry about. - - We only need to customize the ``configure_step`` method to ensure that the ``-DEBTUTORIAL_MSG`` configuration - option is specified; the ``CMakeMake`` easyblock already takes care of specifying the location of - the installation directory (and a bunch of other configuration options, like compiler commands and flags, etc.). - - Implementing the ``build_step`` and ``install_step`` methods is no longer needed, - the standard procedure that is run by the ``CMakeMake`` generic easyblock is fine, - and even goes beyond what we did in the previous exercise (like building in parallel with ``make -j``). - - ```python - from easybuild.easyblocks.generic.cmakemake import CMakeMake - from easybuild.framework.easyconfig import MANDATORY - from easybuild.tools.run import run_cmd - - class EB_eb_minus_tutorial(CMakeMake): - """Custom easyblock for eb-tutorial.""" - - @staticmethod - def extra_options(): - extra = CMakeMake.extra_options() - extra.update({ - 'message': [None, "Message that eb-tutorial command should print", MANDATORY], - }) - return extra - - def configure_step(self): - """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" - self.cfg.update('configopts', '-DEBTUTORIAL_MSG="%s"'% self.cfg['message']) - - super(EB_eb_minus_tutorial, self).configure_step() - - def sanity_check_step(self): - custom_paths = { - 'files': ['bin/eb-tutorial'], - 'dirs': [], - } - custom_commands = ['eb-tutorial'] - return super(EB_eb_minus_tutorial, self).sanity_check_step(custom_paths=custom_paths, - custom_commands=custom_commands) - ``` - - This is a much simpler easyblock as we already use all the logic that has been written for us to build - with CMake. - -*[[next: Part 3: Overview]](3_00_part3_advanced.md)* diff --git a/docs/2022-CSC_and_LO/3_00_part3_advanced.md b/docs/2022-CSC_and_LO/3_00_part3_advanced.md deleted file mode 100644 index 28a24990..00000000 --- a/docs/2022-CSC_and_LO/3_00_part3_advanced.md +++ /dev/null @@ -1,15 +0,0 @@ -# Part III: Advanced topics - -*[[back to start page]](index.md)* - -In this section we mostly cover "good to know that they exist" features as they are not used -on LUMI, or not really accessible to regular user installations that are performed with the -LUMI ``EasyBuild-user`` module. Hooks are used on LUMI, but it is not really advised to -overwrite the centrally defined hooks with a local file. And the whole structure of the -EasyBuild integration is also set up to make use of the GitHub integration in the future. - -* [Using EasyBuild as a library](3_01_easybuild_library.md) *(hands-on)* -* [Using hooks to customise EasyBuild](3_02_hooks.md) *(hands-on)* -* [Submitting installations as Slurm jobs](3_03_slurm_jobs.md) *(hands-on)* -* [Module naming schemes (incl. hierarchical)](3_04_module_naming_schemes.md) *(hands-on)* -* [GitHub integration to facilitate contributing to EasyBuild](3_05_github_integration.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/3_01_easybuild_library.md b/docs/2022-CSC_and_LO/3_01_easybuild_library.md deleted file mode 100644 index f0c255eb..00000000 --- a/docs/2022-CSC_and_LO/3_01_easybuild_library.md +++ /dev/null @@ -1,301 +0,0 @@ -# Using EasyBuild as a library - -*[[back: Configuring EasyBuild]](1_07_configuration.md)* - ---- - -You can use EasyBuild as a Python library and implement a script to automate a particular task. - -All the functionality provided by the EasyBuild framework can be leveraged, -so you only have to glue things together and add the missing pieces. - - -## Setting up the EasyBuild configuration - -Before you call any of the functions provided by the EasyBuild framework, -you should initialize EasyBuild by **setting up the configuration**. -Some of the EasyBuild framework functions assume that EasyBuild has been initialized properly, -and calling them without doing so will result in errors. - -To set up the EasyBuild configuration, you should call the ``set_up_configuration`` function -that is provided by the ``easybuild.tools.options`` module. - -This function takes a couple of optional arguments that are useful in the context of -a Python script that uses EasyBuild as a library: - -* Via ``args`` you can provide a list of command line arguments that should be passed to the option parser. - By default, the standard command line arguments are picked up via ``sys.args``, which may not be what you want. -* By passing ``silent=True`` you can specify that the ``set_up_configuration`` function should not print - anything when it is called. By default, it will print the location to the temporary EasyBuild log file. - -### Cleaning up the temporary directory - -When EasyBuild is configured, a temporary directory specific to that EasyBuild session will be created automatically. -You should clean up that directory, especially if you will be creating temporary files, or if the script will be run -often. - -Note that cleaning up the temporary directory implies removes the temporary log files, -so you probably only want to do this if no errors occurred. - -### Minimal example script - -Here is a minimal (and pretty useless) example Python script, which sets up the EasyBuild configuration -and cleans up the temporary directory: - -```python -#!/usr/bin/env python3 -from easybuild.tools.filetools import remove_dir -from easybuild.tools.options import set_up_configuration - -opts, _ = set_up_configuration(args=[], silent=True) - -remove_dir(opts.tmpdir) -``` - - -## Example use cases - -Once the EasyBuild configuration has been set up, the functions provided by the EasyBuild framework -can be called from a Python script (or directly from the Python interpreter). - -A full overview of all functions is available via the [EasyBuild API documentation](https://docs.easybuild.io/en/latest/api/easybuild.html). - -We highlight a couple commonly used functions in the sections below. - -### File operations - -The [``easybuild.tools.filetools``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html) -module provides a bunch of functions for performing file operations. - -That includes straightforward things like reading, writing, and copying files -(see [``read_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.read_file), [``write_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.write_file), [``copy_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.copy_file)), but also more specific functionality like applying a patch file ([``apply_patch``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.apply_patch)), -computing different types of checksums for a file ([``compute_checksum``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.compute_checksum)), and downloading files -([``download_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.download_file)). - -Here is a trivial example where we download a file from a specified URL to a temporary directory, -compute the SHA256 checksum, and then copy the downloaded file to the current working directory: - -```python -#!/usr/bin/env python3 -import os -import sys -import tempfile - -from easybuild.tools.filetools import download_file, compute_checksum, copy_file, remove_dir -from easybuild.tools.options import set_up_configuration - -url = sys.argv[1] - -opts, _ = set_up_configuration() - -fn = os.path.basename(url) -tmp_path = os.path.join(opts.tmpdir, fn) -download_file(fn, url, tmp_path) - -sha256 = compute_checksum(tmp_path, checksum_type='sha256') -print("SHA256 checksum for %s: %s" % (fn, sha256)) - -target_dir = os.getcwd() -copy_file(tmp_path, target_dir) -print("%s copied to %s" % (fn, target_dir)) - -# cleanup temp dir -remove_dir(opts.tmpdir) -``` - -Example output: -```shell -$ export TEST_URL='https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.0.1.tar.gz' -$ python3 download_and_copy.py $TEST_URL -== Temporary log file in case of crash /tmp/eb-or_xhaw8/easybuild-ewooa00c.log -SHA256 checksum for eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 -eb-tutorial-1.0.1.tar.gz copied to /home/example -``` -We are deliberately *not* specifying the `args` optional argument to the `set_up_configuration` call here, -so we can pass EasyBuild configuration options as command line arguments to this script. -Of course, only those configuration options that are taken into account by the EasyBuild -framework functions being called have any impact. For example: - -```shell -# wait for max. 1h for the download to complete -python3 download_and_copy.py $TEST_URL --download-timeout 3600 -``` - -### Running shell commands - -To run shell commands, we can leverage the ``run_cmd`` functions from the ``easybuild.tools.run`` module. - -Here is a simple example for running the '`make`' command via a Python script: - -```python -#!/usr/bin/env python3 -import sys - -from easybuild.tools.filetools import remove_dir, which -from easybuild.tools.run import run_cmd -from easybuild.tools.options import set_up_configuration - -opts, _ = set_up_configuration(args=[]) - -cmd = 'make' -cmd_path = which(cmd) -if cmd_path: - print(">>> '%s' command found at %s" % (cmd, cmd_path)) -else: - sys.stderr.write("ERROR: '%s' command not found!\n" % cmd) - sys.exit(1) - -cmd = ' '.join(["make"] + sys.argv[1:]) -out, ec = run_cmd(cmd) - -print("\n>>> Output of '%s' (exit code %s):\n\n%s" % (cmd, ec, out)) - -remove_dir(opts.tmpdir) -``` - -Note that in this case it is important to use `args=[]` when calling `set_up_configuration`, -to avoid that the additional script arguments which are options for the '`make`' command -are picked up by the EasyBuild option parser. - -Example usage: - -```shell -$ python3 run_make.py --version -== Temporary log file in case of crash /tmp/eb-zfvbceg6/easybuild-7tynetaj.log - ->> Output of 'make --version' (exit code 0): - -GNU Make 3.81 -``` - -### Interacting with the modules tool - -You can interact with the environment modules tool in a Python script using the -[``easybuild.tools.modules``](https://docs.easybuild.io/en/latest/api/easybuild.tools.modules.html) -Python module that is a part of the EasyBuild framework. -The modules tool that will be used is determined by the active EasyBuild configuration. - -Here is an example script that checks which modules are available and currently loaded, -loads the default module file for ``bzip2``, and inspects the resulting changes to the environment. - -```python -#!/usr/bin/env python3 -import os -import sys - -from easybuild.tools.filetools import remove_dir -from easybuild.tools.modules import get_software_root_env_var_name, modules_tool -from easybuild.tools.options import set_up_configuration - -opts, _ = set_up_configuration() - -# obtain ModulesTool instance for preferred modules tool (determined by active EasyBuild configuration) -mod_tool = modules_tool() -print("Active modules tool: %s version %s" % (mod_tool.NAME, mod_tool.version)) - -avail_modules = mod_tool.available() -print("Found %d available modules in total" % len(avail_modules)) - -avail_eb_modules = mod_tool.available('EasyBuild') -print("Found %d available modules for EasyBuild: %s" % (len(avail_eb_modules), ', '.join(avail_eb_modules))) - -loaded_modules = mod_tool.loaded_modules() -print("%d modules are currently loaded: %s" % (len(loaded_modules), ', '.join(loaded_modules))) - -# load default module for bzip2, check changes to environment -name = 'bzip2' -env_var_name = get_software_root_env_var_name(name) -if any(m.startswith(name + '/') for m in avail_modules): - - print("Current $%s value: %s" % (env_var_name, os.getenv(env_var_name, '(no set)'))) - print("Loading (default) '%s' module..." % name) - - mod_tool.load([name]) - print("Loaded modules: %s" % ', '.join(mod_tool.loaded_modules())) - - # inspect $_LMFILES_ environment variable to determine path to loaded bzip2 module file - for mod_file_path in os.getenv('_LMFILES_').split(':'): - if name in mod_file_path: - print("Path to loaded %s module: %s" % (name, mod_file_path)) - break - - # $EBROOTBZIP2 should be set now (if the bzip2 module was installed with EasyBuild) - print("Current $%s value: %s" % (env_var_name, os.getenv(env_var_name, '(no set)'))) -else: - sys.stderr.write("No modules available for %s\n" % name) - sys.exit(1) - -remove_dir(opts.tmpdir) -``` - -### Parsing easyconfig files - -Here is another small example Python script, which uses the EasyBuild framework functionality -to locate and parse an easyconfig file, and inspect the value of specific easyconfig parameters. - -We define a small helper function named ``parse_easyconfig``, because the EasyBuild framework API -is a bit awkward to use for this simple use case. - -```python -#!/usr/bin/env python3 -import sys - -from easybuild.framework.easyconfig.tools import det_easyconfig_paths, parse_easyconfigs -from easybuild.tools.options import set_up_configuration - - -def parse_easyconfig(ec_fn): - """ - Helper function: find and parse easyconfig with specified filename, - and return parsed easyconfig file (an EasyConfig instance). - """ - # determine path to easyconfig file - ec_path = det_easyconfig_paths([ec_fn])[0] - - # parse easyconfig file; - # the 'parse_easyconfigs' function expects a list of tuples, - # where the second item indicates whether or not the easyconfig file was - # automatically generated or not - ec_dicts, _ = parse_easyconfigs([(ec_path, False)]) - - # only retain first parsed easyconfig, ignore any others (which are unlikely - # anyway) - return ec_path, ec_dicts[0]['ec'] - - -# check whether required arguments are provided -if len(sys.argv) < 3: - sys.stderr.write("ERROR: Usage: %s ") - sys.exit(1) - -ec_fn = sys.argv[1] -keys = sys.argv[2:] - -set_up_configuration(args=[], silent=True) - -ec_path, ec = parse_easyconfig(ec_fn) - -print("Inspecting %s ..." % ec_path) -for key in keys: - print("%s: %s" % (key, ec[key])) -``` - -Example usage (ensure that ``EasyBuild-user`` is loaded as EasyBuild should be -configured properly!): - -``` -$ ./inspect_easyconfig.py GMP-6.2.1-cpeCray-21.12.eb name version sources sanity_check_paths -Inspecting /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/GMP/GMP-6.2.1-cpeCray-21.12.eb ... -name: GMP -version: 6.2.1 -sources: ['gmp-6.2.1.tar.bz2'] -sanity_check_paths: {'files': ['include/gmp.h', 'include/gmpxx.h', 'lib/libgmp.a', 'lib/libgmp.la', 'lib/libgmp.so', 'lib/libgmpxx.a', 'lib/libgmpxx.la', 'lib/libgmpxx.so', 'lib/pkgconfig/gmp.pc', 'lib/pkgconfig/gmpxx.pc'], 'dirs': ['share/info']} -``` - -This feature can be used, e.g., to extract information from easyconfig files to generate documentation -for the library of easyconfig files. - - ---- - -*[[next: Using hooks to customise EasyBuild]](3_02_hooks.md)* diff --git a/docs/2022-CSC_and_LO/3_02_hooks.md b/docs/2022-CSC_and_LO/3_02_hooks.md deleted file mode 100644 index f91f7d7c..00000000 --- a/docs/2022-CSC_and_LO/3_02_hooks.md +++ /dev/null @@ -1,213 +0,0 @@ -# Using hooks to customise EasyBuild - -*[[back: Using EasyBuild as a library]](3_01_easybuild_library.md)* - ---- - -Sometimes you may have the need to customise the behaviour of EasyBuild, -while you want to avoid copying easyconfig files in order to make small changes -(like adding an additional configuration option), having to maintain a set -of modified easyblocks, or using a patched version of the EasyBuild framework. - -EasyBuild provides support for implementing so-called *hooks*, -which are small functions that are triggered automatically at specific times. - -Through these hook functions you can take additional actions, or even tweak -internal data structures, to change the software installation procedure that EasyBuild -would normally execute. - -This is a very powerful feature, but it is important to aware of some details in order -to make good use of it. - -Support for hooks is also -[covered in detail in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Hooks.html). - -## Why hooks? - -Hooks can be implemented for various reasons. Some examples used by some sites in the EasyBuild -community: - -- Enforce some site policies on easyconfig files. A practical example for LUMI could be imposing - the use of the `whatis` parameter and ensuring that it contains a `Description` line to have a - more consistent format for module files. -- Ensuring that some information is always in the module file. E.g., on LUMI the `site_contacts` - parameter is added automatically when installing in the central stack if it is not already in - the easyconfig file to refer to the LUST support pages. -- Modify the behaviour of standard easyconfig files that come with EasyBuild to adapt them to the system - while allowing users to simply use the standard easyconfig files. E.g., we could consider this option - to offer the common EasyBuild FOSS toolchain and its subtoolchains to users on LUMI-C, while using hooks to - adapt, e.g., the easyconfig files for Open MPI to work on LUMI. - -Some site have really large hook files to implement policies and modify standard EasyBuild build recipes, e.g., -- [The hooks file from Jülich Supercomputing Centre](https://github.com/easybuilders/JSC/blob/2022/Custom_Hooks/eb_hooks.py) -- ComputeCanada [cc_hooks_common.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_common.py) - and [cc_hooks_gento.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_gentoo.py) -- [The hooks file from the EESSI software stack](https://github.com/EESSI/software-layer/blob/main/eb_hooks.py). - [EESSI](https://www.eessi-hpc.org/) is an effort to build a software stack distributed via - [CernVM-FS](https://cernvm.cern.ch/fs/) using EasyBuild to build all software. - -## Implementing and using hooks - -Using hooks is done by implementing functions with specific names in a Python module, -and configuring EasyBuild to use them by specifying the path to the file that contains them -via the ``hooks`` configuration option. - -For example, if the file named ``my_eb_hooks.py`` in your home directory contains the implementation -of your hooks, you can configure EasyBuild to use them by defining the ``$EASYBUILD_HOOKS`` environment -variable: - -```shell -export EASYBUILD_HOOKS=$HOME/my_eb_hooks.py -``` - -## Available hooks - -EasyBuild supports various hooks, the full list can be consulted via ``eb --avail-hooks``. - -There are 3 different types of hooks: - -- start/end hooks -- parse hook -- pre/post-step hooks - -These are discussed in more details in the sections below. - -### Start/end hooks - -The first type of hooks are the ``start`` and ``end`` hooks, which are respectively triggered -at the very start of the EasyBuild session (right after setting up the EasyBuild configuration), -and at the very end of the session (right before stopping the logger and doing final cleanup). - -These hooks are only called *once* for every EasyBuild session, no matter how many installations -are performed during that session. - -The ``end`` hook is *not* triggered if an error occurred during one of the installations, -only on successful completion of the EasyBuild session. - -These hooks can be implementing by defining a function named ``start_hook`` and ``end_hook`` -in the Python module file that is provided via the ``hooks`` configuration option. -No arguments are provided when calling these hooks. - -### Parse hook - -The ``parse`` hook is triggered right after an easyconfig file is being parsed, -*before* EasyBuild sets up its internal data structures based on the parsed easyconfig. - -If you want to dynamically change one or more easyconfig parameters without changing the corresponding -easyconfig files, using this hook may be appropriate. - -Note that parsing an easyconfig file can happen for a variety of reasons, -not only when the easyconfig file will actually be installed. EasyBuild will also -parse easyconfig files to check whether they resolve required dependencies, -to check whether the corresponding module file is already installed, etc. - -This hook can be implemented via a function named ``parse_hook``, and exactly one -argument is provided when it is called: the [``EasyConfig``](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyconfig.easyconfig.html#easybuild.framework.easyconfig.easyconfig.EasyConfig) -instance that represents the parsed easyconfig file. - -### Pre/post-step hooks - -The third type of hooks are *pre/post-step* hooks, which are triggered right before or -after a particular installation step is executed. - -EasyBuild performs each software installation by stepping through over a dozen different methods, -and for each of these steps there a pre- and post-hook is triggered, which results in over 30 -additional hooks. - -To use any of these hooks, you need to implement a function that follow a strict naming scheme: -``__hook``. For example, the hook that is triggered right before the ``configure`` -step is run is a function named ``pre_configure_hook``. - -Every time these hooks are called, a single argument is provided: an [``EasyBlock``](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyblock.html#easybuild.framework.easyblock.EasyBlock) -instance that represents the easyblock that is being used to perform the installation. -The parsed easyconfig file can be accessed via the ``cfg`` class variable of the ``EasyBlock`` instance. - -These hooks are useful for influencing the installation procedure at a particular stage. - -## Caveats - -There are a couple of important caveats to take into account when implementing hooks. - -### Breaking EasyBuild with hooks - -Since hooks allow you to inject custom code into EasyBuild at runtime, -it is also easy to break EasyBuild by using hooks... - -Make sure to carefully test your hook implementations, and constrain the actions -you take a much as possible, for example by adding conditions to control for which -software names you will actually modify the installation procedure, etc. - -Any errors that are triggered or raised while a hook function is running -will interrupt the EasyBuild session. - -So don't forget: with great power comes great responsibility! - -### Template values - -Depending on the type of hook, you may observe "raw" values of easyconfig parameters where -template values have not been resolved yet, or values in which template values have been resolved already. - -In the ``parse`` hook, you will always see unresolved template values. - -In the pre/post-step hooks you will see resolved template values, -unless you explicitly disable templating. - -To obtain easyconfig parameter values with unresolved template values in step hooks, -you can use the ``disable_templating`` [context manager](https://docs.python.org/3/reference/compound_stmts.html#with). -For example: - -```python -from easybuild.framework.easyconfig.easyconfig import disable_templating -from easybuild.tools.build_log import print_warning - -def pre_source_hook(eb): - """Print warning when software version was found in 'raw' name of source file.""" - with disable_templating(eb.cfg): - for src in eb.cfg['sources']: - if eb.version in src: - msg = "Software version '%s' found in name of source file (%s), " % (eb.version, src) - msg += "please use %(version)s template value instead!" - print_warning(msg) -``` - - -### Manipulating easyconfig parameters - -If you want update a particular easyconfig parameter without overwriting the existing value, -a bit of care has to be taken: you should use the ``update`` method of the ``EasyConfig`` instance -for this, unless you disable template resolution. This is particularly important when -updating easyconfig parameters that have *mutable* value (like a ``list`` or ``dict``). - -Here's a correct example of a pre-install hook: - -```python -def pre_install_hook(eb): - if eb.name == 'pigz': - # always copy the README directory too when installing pigz - eb.cfg.update('files_to_copy', 'README') -``` - -This seemingly equivalent implementation will ***not*** work (the value of the `files_to_copy` -easyconfig parameter will *not* be updated): - -```python -def pre_install_hook(eb): - if eb.name == 'pigz': - # incorrect way of adding 'README' to 'files_to_copy' (DON'T USE THIS!) - eb.cfg['files_to_copy'].append('README') -``` - -To use this coding style successfully, you have to disable the templating mechanism -when updating the easyconfig parameter: - -```python -def pre_install_hook(eb): - if eb.name == 'pigz': - # this works, but it is better to use the 'update' method instead... - with disable_templating(eb.cfg): - eb.cfg['files_to_copy'].append('README') -``` - ---- - -*[[next: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* diff --git a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md deleted file mode 100644 index 283a53a0..00000000 --- a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md +++ /dev/null @@ -1,333 +0,0 @@ -# Submitting installations as Slurm jobs - -*[[back: Using hooks to customise EasyBuild]](3_02_hooks.md)* - ---- - -EasyBuild can submit jobs to different backends including Slurm to install software, -to *distribute* the often time-consuming installation of a set of software applications and -the dependencies they require to a cluster. Each individual package is installed in a separate -job and job dependencies are used to manage the dependencies between package so that no build -is started before the dependencies are in place. - -This is done via the ``--job`` command line option. - -It is important to be aware of some details before you start using this, which we'll cover here. - -!!! Warning "This section is not supported on LUMI, use at your own risk" - - EasyBuild on LUMI is currently not fully configured to support job submission via Slurm. Several - changes would be needed to the configuration of EasyBuild, including the location of the - temporary files and build directory. Those have to be made by hand. - - Due to the setup of the central software stack, this feature is currently useless to install - the central stack. For user installations, there are also limitations as the enviornment - on the compute nodes is different from the login nodes so, e.g., different locations for - temporary files are being used. These would only be refreshed if the EasyBuild configuration - modules are reloaded on the compute nodes which cannot be done currently in the way Slurm - job submission is set up in EasyBuild. - - Use material in this section with care; it has not been completely tested. - - -## Configuration - -The EasyBuild configuration that is active at the time that ``eb --job`` is used -will be *passed down* into the submitted job automatically, via command line options to the ``eb`` -command that is run in the job script. - -This includes not only command line options used next to ``--job``, but also configuration settings -that are specified via an [EasyBuild configuration file](1_07_configuration#configuration-files) or through -[``$EASYBUILD_*`` environment variables](1_07_configuration#easybuild_-environment-variables). - -This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables -that are in place in the job environment are most likely *irrelevant*, since configuration settings -they specify they will most likely be overruled by the corresponding command line options. -It does also imply however that the EasyBuild configuration that is in place when ``eb --job`` is used -does also work on the compute nodes to which the job is submitted. - - -## Using ``eb --job`` - -### Job backend - -The default job backend in EasyBuild v4.x is [``GC3Pie``](https://gc3pie.readthedocs.io). -To let EasyBuild submit jobs to Slurm instead, you should set the ``job-backend`` configuration setting -to ``Slurm``, for example by setting the corresponding environment variable: - -```shell -export EASYBUILD_JOB_BACKEND='Slurm' -``` - -On LUMI this is taken care of in the EasyBuild configuration modules such as ``EasyBuild-user``. - - -### Job resources - -To submit an installation as a job, simply use ``eb --job``: - -```shell -eb example.eb --job -``` - -By default, EasyBuild will submit single-core jobs requesting for 24 hours of walltime. -You can tweak the requested resources via the ``job-cores`` and ``job-max-walltime`` configuration options. -For example: - -```shell -# submit job to install example, using 5 cores and 2 hours of max. walltime -eb example.eb --job --job-cores 5 --job-max-walltime 2 -``` - -Note that not all ``job-*`` configuration settings apply to all job backends, -see the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Submitting_jobs.html) for more details. - -### Controlling Slurm submission options - -When using Slurm as a job backend, EasyBuild will automatically generate job scripts which -use the ``eb`` command to perform a single installation. These scripts will be submitted -using the ``sbatch`` command. - -EasyBuild currently doesn't provide away to customize the Slurm submission options, -for example to submit to a particular partition, or to use a particular account, -build you can set the corresponding ``$SBATCH_*`` environment variables prior to running ``eb --job``. - -For example, to specify a particular account that should be used for the jobs submitted by EasyBuild -(equivalent with using the ``-A`` or ``--account`` command line option for ``sbatch``): - -```shell -export SBATCH_ACCOUNT='project_XXXXXXXXX' -``` - -Or to submit to a particular Slurm partition (equivalent with the ``-p`` or ``--partition`` option for ``sbatch``): - -```shell -export SBATCH_PARTITION='small' -``` - -For more information about supported ``$SBATCH_*`` environment variables, -see the [Slurm documentation](https://slurm.schedmd.com/sbatch.html#lbAJ). - -## Combining ``--job`` and ``--robot`` - -If one or more dependencies are still missing for the software you want to install, -you can combine ``--job`` and ``--robot`` to get EasyBuild to submit a *separate* job -for each of the installations. These jobs will *not* ``--robot``, they will each only -perform a single installation. - -Dependencies between jobs will be "registered" at submission time, so Slurm will put jobs -on hold until the jobs that install the required (build) dependencies have completed successfully, -and cancel jobs if the job to install a dependency failed for some reason. - -## Attention points - -There are a couple of important things to keep an eye on when submitting installations as jobs... - -### Differences on cluster workernodes - -Sometimes the resources available on the login nodes and cluster workernodes are slightly different, -and you may need to take this into account in your EasyBuild configuration. - -For example, plenty of disk space may be available in the `/tmp` temporary filesystem on a login node, -while the workernodes require you to use a different location for temporary files and directories. -As a result, you may need to slightly change your EasyBuild configuration when submitting installations -as jobs, to avoid that they fail almost instantly due to a lack of disk space. - -Keep in mind that the active EasyBuild configuration is passed down into the submitted jobs, -so any configuration that is present on the workernodes may not have any effect. - -For example, on LUMI it is possible to use ``$XDG_RUNTIME_DIR`` on the login nodes which has -the advantage that any leftovers of failed builds will be cleaned up when the user ends their last -login session on that node, but it is not possible to do so on the compute nodes. - -```shell -# EasByuild is configured to use /tmp/$USER on the login node -uan01 $ eb --show-config | grep buildpath -buildpath (E) = /run/user/XXXXXXXX/easybuild/build - -# use /dev/shm/$USER for build directories when submitting installations as jobs -login01 $ eb --job --buildpath /dev/shm/$USER/easybuild example.eb --robot -``` - - -### Temporary log files and build directories - -The problems for the temporary log files are twofold. First, they may end up in a place -that is not available on the compute nodes. E.g., for the same reasons as for the build -path, the LUMI EasyBuild configuration will place the temporary files in a subdirectory of -``$XDG_RUNTIME_DIR`` on the loginnodes but a subdirectory of ``/dev/shm/$USER`` on the -compute nodes. The second problem however is that if an installation fails, those log files are -not even accessible anymore which may leave you wondering about the actual cause of the failing -installation... - -To remedy this, there are a couple of EasyBuild configuration options you can use: - -* You can use ``--tmp-logdir`` to specify a different location where EasyBuild should store temporary log files, - for example: - ```shell - $ eb --job example.eb --tmp-logdir $HOME/eb_tmplogs - ``` - This will move at least the log file to a suitable place. - -* If you prefer having the entire log file stored in the Slurm job output files, - you can use ``--logtostdout`` when submitting the jobs. This will result in extensive logging - to your terminal window when submitting the jobs, but it will also make EasyBuild - log to ``stdout`` when the installation is running in the job, and hence the log messages will be - captured in the job output files. - -The build directory of course also suffers from the problem of being no longer accessible if the -installation fails, but there it is not so easy to find a solution. Building on a shared file system -is not only much slower, but in particular on parallel file systems like GPFS/SpectrumScale, Lustre -or BeeGFS buiding sometimes fails in strange ways. One thing you can consider if you cannot do the -build on a login node (e.g., because the code is not suitable for cross-compiling or the configure -system does tests that would fail on the login node), is to rety the installation in an -interactive job, so you can inspect the build directory after the installation fails. - -### Lock files - -EasyBuild creates [locks](https://docs.easybuild.io/en/latest/Locks.html) -to prevent that the same installation is started multiple times on -different system to the same installation directory. - -If an installation fails or gets interrupted, EasyBuild cleans up those locks automatically. - -However, if a Slurm job that is using EasyBuild to install software gets cancelled (because it -ran out of walltime, tried to consume too much memory, through an ``scancel`` command, etc.), -EasyBuild will not get the chance to clean up the lock file. - -If this occurs you will need to either clean up the lock file (which is located in the `software/.locks` -subdirectory of ``installpath``) manually, or re-submit the job with ``eb --job --ignore-locks``. - -## Example - -As an example, we will let EasyBuild submit jobs to install ``AUGUSTUS`` with the ``foss/2020b`` toolchain. - -!!! Warning "This example does not work on LUMI" - - Note that this is an example using the FOSS common toolchain. For this reason it does not work on - LUMI. - -### Configuration - -Before using ``--job``, let's make sure that EasyBuild is properly configured: - -```shell -# Load the EasyBuild-user module (central installations will not work at all -# using job submission) -module load LUMI/21.12 -module load partition/C -module load EasyBuild-user - -# use ramdisk for build directories -export EASYBUILD_BUILDPATH=/dev/shm/$USER/build -export EASYBUILD_TMPDIR=/dev/shm/$USER/tmp - -# use Slurm as job backend -export EASYBUILD_JOB_BACKEND=Slurm -``` - - -We will also need to inform Slurm that jobs should be submitted into a particular account, and -in a particular partition: - -```shell -export SBATCH_ACCOUNT=project_XXXXXXXXX -export SBATCH_PARTITION='small' -``` - -This will be picked up by the ``sbatch`` commands that EasyBuild will run to submit the software installation jobs. - - -### Submitting jobs to install AUGUSTUS - -Now we can let EasyBuild submit jobs for AUGUSTUS. - -Let's first check what is still missing: - -```shell -$ eb AUGUSTUS-3.4.0-foss-2020b.eb --missing -... -11 out of 61 required modules missing: - -* HTSlib/1.11-GCC-10.2.0 (HTSlib-1.11-GCC-10.2.0.eb) -* lpsolve/5.5.2.11-GCC-10.2.0 (lpsolve-5.5.2.11-GCC-10.2.0.eb) -* Boost/1.74.0-GCC-10.2.0 (Boost-1.74.0-GCC-10.2.0.eb) -* GSL/2.6-GCC-10.2.0 (GSL-2.6-GCC-10.2.0.eb) -* SAMtools/1.11-GCC-10.2.0 (SAMtools-1.11-GCC-10.2.0.eb) -* BCFtools/1.11-GCC-10.2.0 (BCFtools-1.11-GCC-10.2.0.eb) -* METIS/5.1.0-GCCcore-10.2.0 (METIS-5.1.0-GCCcore-10.2.0.eb) -* BamTools/2.5.1-GCC-10.2.0 (BamTools-2.5.1-GCC-10.2.0.eb) -* MPFR/4.1.0-GCCcore-10.2.0 (MPFR-4.1.0-GCCcore-10.2.0.eb) -* SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0.eb) -* AUGUSTUS/3.4.0-foss-2020b (AUGUSTUS-3.4.0-foss-2020b.eb) -``` - -Several dependencies are not installed yet, so we will need to use ``--robot`` to ensure that -EasyBuild also submits jobs to install these first. - -To speed up the installations a bit, we will request 8 cores for each submitted job (via ``--job-cores``). -That should be sufficient to let each installation finish in (well) under 1 hour, -so we only request 1 hour of walltime per job (via ``--job-max-walltime``). - -In order to have some meaningful job output files, we also enable trace mode (via ``--trace``). - -``` -$ eb AUGUSTUS-3.4.0-foss-2020b.eb --job --job-cores 8 --job-max-walltime 1 --robot --trace -... -== resolving dependencies ... -... -== List of submitted jobs (11): Boost-1.74.0-GCC-10.2.0 (Boost/1.74.0-GCC-10.2.0): 1000011; GSL-2.6-GCC-10.2.0 (GSL/2.6-GCC-10.2.0): 1000004; SAMtools-1.11-GCC-10.2.0 (SAMtools/1.11-GCC-10.2.0): 1000005; HTSlib-1.11-GCC-10.2.0 (HTSlib/1.11-GCC-10.2.0): 1000006; BCFtools-1.11-GCC-10.2.0 (BCFtools/1.11-GCC-10.2.0): 1000001; lpsolve-5.5.2.11-GCC-10.2.0 (lpsolve/5.5.2.11-GCC-10.2.0): 1000007; BamTools-2.5.1-GCC-10.2.0 (BamTools/2.5.1-GCC-10.2.0): 1000008; METIS-5.1.0-GCCcore-10.2.0 (METIS/5.1.0-GCCcore-10.2.0): 1000009; MPFR-4.1.0-GCCcore-10.2.0 (MPFR/4.1.0-GCCcore-10.2.0): 1000010; SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0): 1000002; AUGUSTUS-3.4.0-foss-2020b (AUGUSTUS/3.4.0-foss-2020b): 1000003 -== Submitted parallel build jobs, exiting now -``` - -### Inspecting the submitted jobs - -Once EasyBuild has submitted the jobs, we can inspect them via Slurm's ``squeue`` command: - -``` -$ squeue -u $USER -la - JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON) -1000001 small BCFtools user123 PENDING 0:00 2:00:00 1 (Dependency) -1000002 small SuiteSpa user123 PENDING 0:00 2:00:00 1 (Dependency) -1000003 small AUGUSTUS user123 PENDING 0:00 2:00:00 1 (Dependency) -1000004 small GSL-2.6- user123 RUNNING 0:21 2:00:00 1 node003 -1000005 small SAMtools user123 RUNNING 0:21 2:00:00 1 node007 -1000006 small HTSlib-1 user123 RUNNING 0:21 2:00:00 1 node007 -1000007 small lpsolve- user123 RUNNING 0:21 2:00:00 1 node011 -1000008 small BamTools user123 RUNNING 0:21 2:00:00 1 node011 -1000009 small METIS-5. user123 RUNNING 0:21 2:00:00 1 node013 -1000010 small MPFR-4.1 user123 RUNNING 0:21 2:00:00 1 node029 -1000011 small Boost-1. user123 RUNNING 0:24 2:00:00 1 node029 -``` - -Note that 3 jobs can not be started yet, because those installations require on one or more -missing dependencies. As soon as the jobs for those dependencies (successfully) complete, -these jobs will be able to start. - -### Final result - -After about 20 minutes, AUGUSTUS and all missing dependencies should be installed: - -``` -$ ls -lrt $HOME/EasyBuild/modules/.../*.lua | tail -11 --rw-rw----. 1 example example 1634 Mar 29 10:13 /users/example/easybuild/modules/all/HTSlib/1.11-GCC-10.2.0.lua --rw-rw----. 1 example example 1792 Mar 29 10:13 /users/example/easybuild/modules/all/SAMtools/1.11-GCC-10.2.0.lua --rw-rw----. 1 example example 1147 Mar 29 10:13 /users/example/easybuild/modules/all/BamTools/2.5.1-GCC-10.2.0.lua --rw-rw----. 1 example example 957 Mar 29 10:13 /users/example/easybuild/modules/all/lpsolve/5.5.2.11-GCC-10.2.0.lua --rw-rw----. 1 example example 1549 Mar 29 10:13 /users/example/easybuild/modules/all/METIS/5.1.0-GCCcore-10.2.0.lua --rw-rw----. 1 example example 1525 Mar 29 10:14 /users/example/easybuild/modules/all/GSL/2.6-GCC-10.2.0.lua --rw-rw----. 1 example example 1221 Mar 29 10:15 /users/example/easybuild/modules/all/MPFR/4.1.0-GCCcore-10.2.0.lua --rw-rw----. 1 example example 1678 Mar 29 10:15 /users/example/easybuild/modules/all/BCFtools/1.11-GCC-10.2.0.lua --rw-rw----. 1 example example 1292 Mar 29 10:21 /users/example/easybuild/modules/all/Boost/1.74.0-GCC-10.2.0.lua --rw-rw----. 1 example example 1365 Mar 29 10:28 /users/example/easybuild/modules/all/SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0.lua --rw-rw----. 1 example example 2233 Mar 29 10:30 /users/example/easybuild/modules/all/AUGUSTUS/3.4.0-foss-2020b.lua - -$ module avail AUGUSTUS - --- EasyBuild managed user software for software stack ... -- - AUGUSTUS/3.4.0-foss-2020b -``` - ---- - -*[[next: Module naming schemes]](3_04_module_naming_scheme)* diff --git a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md deleted file mode 100644 index d0b4ebe8..00000000 --- a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md +++ /dev/null @@ -1,569 +0,0 @@ -# Module naming schemes - -*[[back: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* - ---- - -Up until now we have used the naming scheme in use on LUMI which is for many practical purposes -is indistinguishable from the default EasyBuild naming scheme (`EasyBuildMNS`). -It produces module files with names that closely resemble to the names of the -corresponding easyconfig files. -For example, when installing `zlib-1.2.11-cpeGNU-21.12.eb` the generated module was named -`zlib/1.2.11-cpeGNU-21.12`. - -EasyBuild supports several different module naming schemes: - -```shell -$ eb --avail-module-naming-schemes -List of supported module naming schemes: - EasyBuildMNS - LUMI_FlatMNS - MigrateFromEBToHMNS - HierarchicalMNS - CategorizedHMNS - CategorizedModuleNamingScheme -``` - -In this part of the tutorial we will take a closer look at `HierarchicalMNS`, -which is the standard **hierarchical** module naming scheme included with EasyBuild. - -We will also take a quick look at implementing our own custom module naming scheme. - -## Flat vs hierarchical - -!!! Note - - This text is very much based on the generic EasyBuild tutorials as we cannot demonstrate - all aspects on LUMI as it is configured today. - - On LUMI, the complete module scheme is partly hierarchical, but EasyBuild uses a flat naming - scheme. The two levels in the hierarchy that are present to deal with different versions of - the software stack and the various types of compute nodes, are not managed by EasyBuild. - -The default module naming scheme `EasyBuildMNS` -and the custom `LUMI_FlatMNS` naming scheme are both examples of regular *"flat"* -module naming schemes, which are characterized by: - -- all module files are directly available for loading; -- each module name uniquely identifies a particular installation; - -In contrast, a *hierarchical* module naming scheme -consists of a *hierarchy* of module files. -A fairly typical 3-level schme (``Core``, ``Compiler`` and ``MPI``) has been -discussed in the [section on Lmod](../1_02_Lmod#lmod-hierarchy). -This typical Lmod hierarcny would map very well on the EasyBuild common toolchains. - -In fact, for the example - -
- -software at the ``Core`` level would be installed with the ``GCCcore`` and ``SYSTEM`` -toolchains. Software at the ``Compiler`` level would be installed with the ``GCC`` -toolchain, while software at the ``MPI`` level after loading the ``OpenMPI`` module -would be installed using the regular ``foss`` toolchain or the ``gompi`` toolchain -(see the diagram in the -["Common toolchains" section of the page on terminology](../1_05_terminology/#common-toolchains)). - -On LUMI, where software is installed through the Cray Programming Environment with no real choice of -MPI implementation, a two-level arrangement would still make a lot of sense, with at the ``Core`` level -all software compiled with the SYSTEM toolchain while there could be a ``PrgEnv`` level for software -compiled with a particula programming enviornment aka cepGNU/cpeCray/cpeAOCC toolchain. Such a scheme -is used on the Cray systems at CSCS. - -To recap, the characteristics of a module hierarchy are: - -* not all module files are directly available for loading; -* some modules serve as a gateway to more modules; -* to access some software installations you will first need to load one or more gateway modules in order - to use them; - -You can probably think of other ways to organize module files in a hierarchical module tree, but here -we will stick to the standard core / compiler / MPI hierarchy. - -### Pros & cons - -So why go through all this trouble of organizing modules hierarchically? - -There are a couple of advantages to this approach: - -* shorter module names (or at least for the version part of the name); -* less overwhelming list of available modules; -* only compatible modules can be loaded together; - -However, the are some minor disadvantages too: - -* not all existing modules are directly visible, so users have to learn how to find - modules using ``module spider`` etc.; -* gateway modules may have little meaning to end users; - -#### Length of module names - -When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For a `HDF5` installation -with the EasyBuild common toolchains for example, -one might have `HDF5/1.10.6-gompi-2020a` as module name. The `-gompi-2020a` part of the name refers to the toolchain that was -used for this installation, but it may be confusing to some people (what kind of Pokémon is a "gompi"?!). - -In the example module hierarchy shown above, the module for `HDF5` could simply be named `HDF5/1.10.6` which is basically the bare -essentials: software name and version. That's way better, nice and clean! - -#### Amount of available modules - -The output of "`module avail`" can be quite overwhelming if lots of module files -are installed and a flat module naming scheme is used, since *all* modules are -*always* available. -EasyBuild makes it very easy to install lots of software, -so the number of installed modules can easily grow into the hundreds or even thousands... - -This often explosive growth of modules is less of an issue when using a hierarchical module naming scheme, since -initially only a small set of modules are available, and relatively limited -groups of additional modules become available as gateway modules are loaded. - -#### Loading compatible modules - -Since all modules are available at once when using a flat module naming scheme, you can easily load -modules together that are not compatible with each other. - -Imagine loading two modules that were built with a different compiler toolchain (different compiler, -different MPI library). That's likely to end in tears, unless you have the necessary technical expertise -to fully understand what is going on, *and* you are being very careful... - -In a module hierarchy this can be prevented, since modules for software that were installed with a -different compiler and/or a different MPI library are located in a different part of the module hierarchy, -and so these modules can not be loaded together easily. - - -#### Visibility of existing modules - -One downside of a module hierarchy is that not all existing modules are directly available for loading -or are even *visible* to the user, since the output of "`module avail`" only shows a subset of all modules. - -As we have discussed in [the Lmod section](../LMOD). -[Lmod](https://lmod.readthedocs.io) provides a powerful solution to search for modules through the -``module spider`` and to some extent also the ``module keyword`` commands. -So as long as the end users are -aware of this additional command, it should not be difficult to discover which software installations exist -and how they can be accessed. The "`module spider`" command will inform the user which of the gateway modules -need to be loaded in order to load a specific module file. - -#### Semantics of gateway modules - -An additional potential problem of a module hierarchy is that the semantics of the gateway modules may not be clear -to end users. They may wonder why they need to pick a specific compiler and MPI library, or which of the -available options is the best one. Maybe they are not even aware what exactly a "compiler" is, or how it is -relevant to the software they need in their bioinformatics pipeline... - -This can be partially resolved by loading a default compiler and MPI module so a particular set of modules -is available right after login, which could be the ones used in the most recent toolchain, or the -recommended versions. More experienced users could then leverage the "`module spider`" command to navigate -the module hierarchy. - -On LUMI this would mean loading a default software stack, but due to the the default modules are currently -loaded on LUMI this was not possible to accomplish without losing other functionality of the module tree. - -## Using a custom module naming scheme - -Next to the module naming schemes that are included with EasyBuild, -you can also define your own module naming scheme (MNS), and configure EasyBuild to use it -(which is eactly what has been done on LUMI to remove a feature of the default ``EasyBuildMNS`` scheme -that we do not use). - -### Implementation - -To use a custom module naming scheme, you need to implement a Python module file, -where you define a Python class that derives from the general [``ModuleNamingScheme``](https://docs.easybuild.io/en/latest/api/easybuild.tools.module_naming_scheme.mns.html#easybuild.tools.module_naming_scheme.mns.ModuleNamingScheme) class. - -For a flat module naming scheme, it is sufficient to implement the ``det_full_module_name`` method, -which should return a string value (the full module name). -You may also need to customize the ``is_short_modname_for`` method, which verifies whether -a given (short) module name is for the software with a particular given name (or not). - -The argument provided to ``det_full_module_name`` can be a small Python dictionary which only specifies -the value of a handful of easyconfig parameters (`name`, `version`, `toolchain`, and `versionsuffix`), -or an ``EasyConfig`` instance which represents a parsed easyconfig file and contains values for *all* -known easyconfig parameters. - -For simple module naming schemes, just have ``name``, ``version``, ``toolchain``, and ``versionsuffix`` -available is sufficient. If it is not, you can list which additional easyconfig parameters are required for -your module naming scheme via a class constant ``REQUIRED_KEYS``. - -A fairly straightforward example of a *flat* module name scheme is the default EasyBuild module naming scheme -[``EasyBuildMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/easybuild_mns.py). - -For a *hierarchical* module naming scheme, various additional methods have to be implemented. - -Some of these, like ``det_module_subdir`` and ``det_short_module_name``, determine which part of -the (full) module name determines the location of the module in the module hierarchy, and which part -is the user-facing (or "short") module name. Others, like ``det_modpath_extensions``, -determine which modules are *gateway* modules that open up an additional level of the hierarchy -(by *extending* the ``$MODULEPATH``). - -A typical example of a hierarchical module naming scheme is [``HierarchicalMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/hierarchical_mns.py) that comes with EasyBuild (which is quite complex because it includes quite a bit of special handling for particular compiler -toolchains), which implements the traditional ``Core``-``Compiler``-``MPI`` module hierarchy we discussed -above. - -### Configuring EasyBuild - -To let EasyBuild use a custom module naming scheme, you need to: - -* specify the path to the Python module file implementing it via the ``include-module-naming-schemes`` - EasyBuild configuration option; - -* indicate that you also want to *use* this custom module naming scheme via the ``module-naming-scheme`` - EasyBuild configuration option. - -For example: - -```shell -export EASYBUILD_INCLUDE_MODULE_NAMING_SCHEMES=$HOME/easybuild/example_mns.py -export EASYBUILD_MODULE_NAMING_SCHEME=ExampleMNS -``` - -#### Example custom module naming scheme - -Here is an example of a custom module naming scheme, where: - -* the ``versionsuffix`` goes directly after the ``version``; -* all dashes are replaced by underscores; -* all module names are lowercase; - -Note that we also need to customise the ``is_short_modname_for`` method, -to make sure it returns ``True`` when the EasyBuild framework checks whether -``scipy_bundle/2020.11_foss_2020b`` is a module name for ``SciPy-bundle``. - -```python -import os - -from easybuild.tools.module_naming_scheme.mns import ModuleNamingScheme - - -class ExampleMNS(ModuleNamingScheme): - - REQUIRED_KEYS = ['name', 'version', 'versionsuffix', 'toolchain'] - - def det_full_module_name(self, ec): - """ - Determine full module name: - - all lowercase - - replace all dashes with underscores - """ - parts = [ec['version']] - - # versionsuffix directly after version (but only if it's not empty) - if ec['versionsuffix']: - parts.append(ec['versionsuffix']) - - # only add toolchain name/version for non-system toolchain - tc = ec['toolchain'] - if tc['name'].lower() != 'system': - parts.extend([tc['name'], tc['version']]) - - modname = ec['name'] + '/' + '_'.join(parts) - - modname = modname.replace('-', '_').replace('__', '_') - - return modname.lower() - - def is_short_modname_for(self, short_modname, name): - """Determine whether short module name is a module for the software with specified name.""" - return short_modname.startswith(name.lower().replace('-', '_') + '/') -``` - -We can see what the module names with this module naming scheme would like like via ``eb -D``. -E.g., for the common toolchains (the example will not work on LUMI as the default easyconfig -files are not in the robot- and the search path): - -``` -$ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D - ... - * [ ] $CFGS/g/GCC/GCC-10.2.0.eb (module: gcc/10.2.0) - ... - * [ ] $CFGS/p/Python/Python-2.7.18-GCCcore-10.2.0.eb (module: python/2.7.18_gcccore_10.2.0) - ... - * [ ] $CFGS/o/OpenMPI/OpenMPI-4.0.5-GCC-10.2.0.eb (module: openmpi/4.0.5_gcc_10.2.0) - ... - * [ ] $CFGS/s/SciPy-bundle/SciPy-bundle-2020.03-foss-2020a-Python-2.7.18.eb (module: scipy_bundle/2020.03_python_2.7.18_foss_2020a) -``` - -## Example module hierarchy: HDF5 - -!!! Warning "Example not suitable for LUMI" - **This exercise is meant for a system where the common toolchains can be used and requires an - indpendent EasyBuild installation in your personal file space**, - because EasyBuild will try to copy the installation log file to each installation directory. - -Now that we know more about hierarchical module naming schemes, -let us see how EasyBuild can help us with generating a hierarchical module tree. - -In this example we will use EasyBuild to generate modules organised in a hierarchy -for some of the software that is already installed in the prepared environment. - -The good news is that the existing installations can be reused. There is absolutely no need -to reinstall the software, we are just creating a different "view" on these software installations. - -### Preparing the environment - -Before running EasyBuild to generate a hierarchical module tree, we have to be a bit careful -with preparing our environment. - -**We must absolutely avoid mixing modules from a flat and hierarchical module naming scheme!** - -Some module files will have the same name in both module trees (like `GCC/10.2.0` for example), -but their contents will be different. -Mixing modules from a flat and hierarchical module tree *will* trigger problems... - -So we have to make sure that the module files we already have in `/easybuild` are *not* visible. -The easiest way to do this is to unload all modules (using "`module purge`") -and resetting the module search path to be empty, which we can do with "`module unuse $MODULEPATH`". - -```shell -module purge -module unuse $MODULEPATH -``` - -In this part of the tutorial, we are assuming you are *not* using an EasyBuild installation provided through -a module. We have just made all modules unavailable, so we would have to first -install EasyBuild again in our hierarchical module tree before we can continue. - -**We strongly recommend using an EasyBuild installation that was [installed via "`pip install`" -or "`pip3 install`"](../1_06_installation#method-1-using-pip) in this part of the tutorial.** - -An easy way to do this is in the prepared environment is to run: - -```shell -pip3 install --user easybuild -export PATH=$HOME/.local/bin:$PATH -export EB_PYTHON=python3 -``` - -### Configuring EasyBuild - -First of all, we need to make sure that EasyBuild is properly configured. -We can do this by defining this set of environment variables: - -```shell -export EASYBUILD_PREFIX=$HOME/easybuild -export EASYBUILD_BUILDPATH=/tmp/$USER -export EASYBUILD_INSTALLPATH_SOFTWARE=/easybuild/software -export EASYBUILD_MODULE_NAMING_SCHEME=HierarchicalMNS -export EASYBUILD_INSTALLPATH_MODULES=$HOME/hmns/modules -``` - -To make sure we didn't make any silly mistakes, we double check using `eb --show-config`: - - -```shell -$ eb --show-config -# -# Current EasyBuild configuration -# (C: command line argument, D: default value, E: environment variable, F: configuration file) -# -buildpath (E) = /tmp/example -containerpath (E) = /home/example/easybuild/containers -installpath (E) = /home/example/easybuild -installpath-modules (E) = /home/example/hmns/modules -installpath-software (E) = /easybuild/software -module-naming-scheme (E) = HierarchicalMNS -packagepath (E) = /home/example/easybuild/packages -prefix (E) = /home/example/easybuild -repositorypath (E) = /home/example/easybuild/ebfiles_repo -robot-paths (D) = /home/example/.local/easybuild/easyconfigs -sourcepath (E) = /home/example/easybuild/sources -``` - -There are a couple of things worth pointing out here: - -* We have defined the `module-naming-scheme` configuration setting to `HierarchicalMNS`, - which makes EasyBuild use the included standard hierarchical module naming scheme (the classic - core / compiler / MPI one we discussed above). -* We have specified *different* locations for the software (via `installpath-software`) - and the module files (via `installpath-modules`). This is important because we want to - reuse the software that is already installed in `/easybuild/software` while we want to - generate an entirely new module tree for it (in `$HOME/hmns/modules`). - -The other configuration settings are the same as before, and mostly irrelevant for this example. - -### Generating modules for HDF5 - -Let us now generate a hierarchical module tree for `HDF5` and all of its dependencies, -including the toolchain. That sounds complicated, and it sort of is since there are -a lot of details you have to get right for the module hierarchy to works as intended, -but EasyBuild can do all the hard work for us. - -The steps we will have to go through are: - -* Tell EasyBuild we want to "install" the `HDF5-1.10.7-gompi-2020b.eb` easyconfig file; -* Enable dependency resolution via `--robot`; -* Assuming the software would have been installed already with the default naming scheme - in a different module directory, instruct EasyBuild to only generate the module files, - not to install the software (since it is - there already in `/easybuild/software`), via the `--module-only` option. - -These steps translate to this single `eb` command: - -``` -$ eb HDF5-1.10.7-gompi-2020b.eb --robot --module-only -... -== building and installing MPI/GCC/10.2.0/OpenMPI/4.0.5/HDF5/1.10.7... -... -== sanity checking... -== cleaning up [skipped] -== creating module... -... -== COMPLETED: Installation ended successfully (took 9 sec) -... -== Build succeeded for 41 out of 41 -``` - -This should take a couple of minutes in total, for generating 41 modules. -Remember that this also includes generating module files for the toolchain and -all of its components. - -In addition, there is a bit more going on one that just generating module files, -since the sanity check step is still being run for each of the installations -when using `--module-only` to ensure the installation is actually functional. -After all, there is no point in generating a module for an obviously broken -installation... - -### Loading the HDF5 module - -After generating the hierarchical module tree for HDF5, how do we access the HDF5 installation through it? - -Here's what the module tree looks like on disk: - -``` -$ ls $HOME/hmns/modules/all -Compiler Core MPI -``` - -Those are basically the 3 levels in the module hierarchy we showed in our example earlier. - -The starting point is the top level of the module hierarchy named `Core`: - -``` -module use $HOME/hmns/modules/all/Core -``` - -Let us see what that gives us in terms of available modules: - -``` -$ module avail - ---------------------- /home/easybuild/hmns/modules/all/Core --------------------- - Bison/3.5.3 GCCcore/10.2.0 flex/2.6.4 help2man/1.47.4 - Bison/3.7.1 (D) M4/1.4.18 gettext/0.21 ncurses/6.2 - GCC/10.2.0 binutils/2.35 gompi/2020b zlib/1.2.11 -``` - - -Nice and short module names, but only a limited set of them. - -We know a module file exists for `HDF5`, but we can't see it yet (and hence -we can't load it either). - -``` -$ module avail HDF5 -No module(s) or extension(s) found! -Use "module spider" to find all possible modules and extensions. -``` - -Let us see if `module spider` is of any help, as "`module avail`" so kindly suggests: - -``` -$ module spider HDF5 -... - - You will need to load all module(s) on any one of the lines below - before the "HDF5/1.10.7" module is available to load. - - GCC/10.2.0 OpenMPI/4.0.5 -``` - -This tells us we need to load two gateway modules before we can load the module -for HDF5. - -Let us start with loading the `GCC` compiler module: - -``` -module load GCC/10.2.0 -``` - -And then check again which modules are available: - -``` -$ module avail - --------------- /home/easybuild/hmns/modules/all/Compiler/GCC/10.2.0 -------------- - OpenMPI/4.0.5 - ------------- /home/easybuild/hmns/modules/all/Compiler/GCCcore/10.2.0 ------------ - Autoconf/2.69 UCX/1.9.0 libpciaccess/0.16 - ... - Szip/2.1.1 libfabric/1.11.0 zlib/1.2.11 (L,D) - Autoconf/2.69 XZ/5.2.5 libtool/2.4.6 - ---------------------- /home/easybuild/hmns/modules/all/Core --------------------- - Bison/3.5.3 GCCcore/10.2.0 (L) flex/2.6.4 help2man/1.47.4 - Bison/3.7.1 (D) M4/1.4.18 gettext/0.21 ncurses/6.2 - GCC/10.2.0 (L) binutils/2.35 gompi/2020b zlib/1.2.11 -``` - -Good news, we now have additional modules available! - -The compiler level of our hierarchy actually consists of two directories here: `Compiler/GCCcore/10.2.0` -and `Compiler/GCC/10.2.0`. The modules in the `GCCcore` directory are ones we can use in other compiler -toolchains that use GCC 10.2.0 as a base compiler (the details of that are out of scope here). - -The module we are interested in is `OpenMPI/4.0.5`, which is another gateway module. - -Remember that the "`module spider`" output told us that there does indeed exist a module for `HDF5`, but that -we need to load *both* the `GCC/10.2.0` and `OpenMPI/4.0.5` modules first. - -So, let us do exactly that (remember that `GCC/10.2.0` is already loaded): - -``` -module load OpenMPI/4.0.5 -``` - -If you now check the output of "`module avail`" again, you should see the `HDF5/1.10.7` module: - -``` -$ module avail - --------- /home/easybuild/hmns/modules/all/MPI/GCC/10.2.0/OpenMPI/4.0.5 ------- - HDF5/1.10.7 - ------------- /home/easybuild/hmns/modules/all/Compiler/GCC/10.2.0 ------------ - OpenMPI/4.0.5 (L) - -... -``` - -To use HDF5, we need to load this `HDF5/1.10.7` module. We can verify that the installation works -using one of the commands provided by HDF5, `h5dump` for example: - -``` -module load HDF5/1.10.7 -``` - -``` -$ h5dump --version -h5dump: Version 1.10.7 -``` - -If you now check which modules are loaded via "`module list`", you will notice that all module names -and nice and short now, which is one of the advantages of using a hierarchical module tree: - -```shell -$ module list - -Currently Loaded Modules: - 1) GCCcore/10.2.0 5) numactl/2.0.13 9) hwloc/2.2.0 13) PMIx/3.1.5 - 2) zlib/1.2.11 6) XZ/5.2.5 10) libevent/2.1.12 14) OpenMPI/4.0.5 - 3) binutils/2.35 7) libxml2/2.9.10 11) UCX/1.9.0 15) Szip/2.1.1 - 4) GCC/10.2.0 8) libpciaccess/0.16 12) libfabric/1.11.0 16) HDF5/1.10.7 -``` - - ---- - -*[[next: GitHub integration]](3_05_github_integration.md)* diff --git a/docs/2022-CSC_and_LO/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_05_github_integration.md deleted file mode 100644 index 6e19cb9a..00000000 --- a/docs/2022-CSC_and_LO/3_05_github_integration.md +++ /dev/null @@ -1,460 +0,0 @@ -# GitHub integration to facilitate contributing to EasyBuild - -*[[back: Module naming schemes]](3_04_module_naming_schemes.md)* - ---- - -!!! Warning "Preliminary not for LUMI and many other Cray systems" - - Not everything in this section is as useful for EasyBuild installations that build - on the Cray Programming Environment. E.g., the default easyconfig repository does - not contain easyconfigs for the Cray PE. - - Moreover, due to the restricted operating system on the compute nodes of big - Cray systems such as the EX series (COS instead of the full SUSE linux), - the GitHub integration does not work on the compute nodes of a typical setup as - one of the required Python packages fails to work as it needs the DBus daemon. - - Since GitHub integration is not yet really used on LUMI, this section is not fully - tested but left in the tutorial for completeness. - -To contribute changes to the EasyBuild code (framework or easyblocks) or easyconfigs, -you will need to be a bit familiar with Git and GitHub. Or maybe not? - -## Manual contribution procedure - -0) Create and setup a [GitHub account](https://github.com/join) (and register your SSH public key); - -1) Clone and fork the appropriate GitHub repository, for example when contributing an easyconfig file: - -```shell -git clone git@github.com:easybuilders/easybuild-easyconfigs.git -cd easybuild-easyconfigs -git remote add my_fork git@github.com:your_github_account/easybuild-easyconfigs.git -``` - -**You should change '`your_github_account`' in the last line to your own GitHub user name!** - -2) Create and check out a new branch, starting from the (up-to-date) ``develop`` branch: - -``` -git checkout develop -git pull origin develop -git checkout -b example -``` - -3) Stage the changes you want to contribute, after you make sure that your easyconfig file has the -[correct filename](../basic_usage/#easyconfig-filenames), and that it's located in the appropriate directory. - -```shell -mkdir -p easybuild/easyconfigs/e/example/ -mv example.eb easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb -git add easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb -``` - -4) Commit those changes with a sensible commit message: - -```shell -git commit -m "This is just an example" -``` - -5) Push your branch to your fork of the repository on GitHub: - -```shell -git push my_fork example -``` - -6) Open the pull request through the GitHub web interface, making sure that: - -* the target branch is correct (should be `develop`); -* an appropriate title is used; -* a short description of the changes is provided; -* the changes are indeed the ones you want to propose; -* clicking the (correct) green button; - -
Pfft
- -That didn't exactly motivate you to contribute, did it... - -## Github integration features - -Over the years we noticed that some people were keen on contributing to EasyBuild, -but they were not very familiar with Git or GitHub. That meant they had to overcome a -relatively steep learning curve before they could contribute... - -
Gandalf vs Git
- -In addition, the contribution workflow can be a bit daunting and time consuming, -even if you're already familiar with the procedure. You will have dozens of -branches flying around in no time, and if you get stuck in a weird corner -with `git` you may quickly end up demotivated. - -This is frustrating not only for the people who wanted to contribute but -also for the EasyBuild maintainers, and it doesn't agree with the philosophy of -a project that aims to *automate* tedious software installation procedures. - -At the end of 2015 efforts were made to tackle this issue by implementing -GitHub integration features in EasyBuild, which automate the contribution -workflow by running `git` commands and interacting with the [GitHub API](https://developer.github.com/v3/). - -We will briefly go over some of these features here, but they are also covered in detail [in the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html). - -### Requirements & configuration - -First of all, the GitHub integration features impose a couple of additional [requirements](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) -and configuration. - - -**Additional dependencies** - -Both the `GitPython` and `keyring` Python packages as well as the `keyrings.cryptfile` add-on package must be installed. - -!!! Note - You may experiences problems installing the ``cryptography`` Python packages, - which is a dependency of keyring. The underlying cause is that you need to have - the [``Rust``](https://www.rust-lang.org/) compiler installed to install the latest version - of ``cryptography`` (see [here](https://github.com/pyca/cryptography/issues/5771)). - - You can work around this issue using: - - ```shell - pip3 install --user 'cryptography<3.4' - ``` - -**SSH public key in GitHub account** - -You need to have a GitHub account that has your SSH public key registered in it -(via [https://github.com/settings/keys](https://github.com/settings/keys)). - -If you need to generate an SSH key pair, you can run the following command: - -```shell -ssh-keygen -t rsa -b 4096 -``` - -You can copy the SSH public key from the output of this command: - -```shell -cat .ssh/id_rsa.pub -``` - - -**Forked repository in GitHub** - -In addition, you must have *forked* the EasyBuild repository you want to contribute to -(for example [https://github.com/easybuilders/easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs)). - -**EasyBuild configuration, incl. GitHub token** - -You also have to configure EasyBuild a bit more, so it knows about your -GitHub user name *and* has a GitHub token available in order to perform actions -in GitHub with your credentials. - -To do this, you should define the `github-user` configuration option and -run the "`eb --install-github-token`" command: - -```shell -# replace 'ebtutorial' with your own GitHub username! -$ export EASYBUILD_GITHUB_USER=ebtutorial -$ eb --install-github-token -``` - -To create a GitHub token: - -* Visit [https://github.com/settings/tokens](https://github.com/settings/tokens). -* Click *"Personal access tokens"*. -* Click followed by *"Generate new token"*. -* Give the token a name (for example *"Token for EasyBuild"*). -* Select both the '`repo`' and '`gist`' scopes. -* Click the green *"Generate token"* button. -* Copy the generated token. -* Paste the token when asked by `--install-github-token` (and hit *Enter*). -* Enter a password to encrypt your GitHub token. - -The output should look something like this: - -```shell -$ eb --install-github-token -== temporary log file in case of crash /tmp/eb-9z0bdve9/easybuild-hfpti62w.log -Token: -Validating token... -Token seems to be valid, installing it. -Please set a password for your new keyring: -Please confirm the password: -Token 'fed..987' installed! -``` - - -**Checking status of GitHub integration** - -You can check the status of the GitHub integration using "`eb --check-github`": - -```shell -$ eb --check-github -== temporary log file in case of crash /tmp/eb-4ckdlyfy/easybuild-gp69ev2w.log - -Checking status of GitHub integration... - -Making sure we're online...OK - -* GitHub user...ebtutorial => OK -Please enter password for encrypted keyring: -* GitHub token...fed..987 (len: 40) => OK (validated) -* git command...OK ("git version 1.8.3.1; ") -* GitPython module...OK (GitPython version 3.1.3) -* push access to ebtutorial/easybuild-easyconfigs repo @ GitHub...OK -* creating gists...OK -* location to Git working dirs... not found (suboptimal) - -All checks PASSed! - -Status of GitHub integration: -* --from-pr: OK -* --new-pr: OK -* --review-pr: OK -* --update-pr: OK -* --upload-test-report: OK -``` - -If you see '`OK`' for each of the status checks, you're all set -to try out the GitHub integration features! - -!!! Note - If your SSH private key is protected with a password, you may need - to enter your password a couple of times when running "`eb --check-github`". - - You can avoid this by [using an SSH agent](https://help.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent). - -!!! Note - If you see the ``push access`` check fail with ``Failed to fetch branch 'main'``, - you will need to rename the ``master`` branch in your fork of the ``easybuild-easyconfigs`` - repository from ``master`` to ``main`` (this is required since EasyBuild v4.3.3). - - This can be done via the *pencil* icon at - [https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches](https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches) - (replace ``YOUR_GITHUB_ACCOUNT`` with the name of your GitHub account in this URL). - -### Creating pull requests - -The GitHub integration in EasyBuild allows you to **create pull requests -using the `eb` command**, without even leaving your shell environment. -How cool is that‽ - -To create a pull request to the `easybuild-easyconfigs` repository, -you can either do it in a single go by -running "`eb --new-pr`" and passing it one or more easyconfig files to add -into the pull request. - -The more detailed option is to first create a branch in your repository fork -in GitHub via "`eb --new-branch-github`" and then later open the pull request -via "`eb --new-pr-from-branch`". This method can be useful when preparing multiple -interdependent pull requests, or to check whether your changes pass the unit tests -(which are run automatically in the GitHub Actions CI environment for -all branches pushed to your fork). - -The `--new-pr` option can also be used to open pull requests to the easyblocks -and framework repositories, and it will even automatically determine the target -repository based on the contents of the files you provide. Whoa! - -You can control the target repository for your pull request using -`--pr-target-account` (default is `easybuilders`) and `--pr-target-repo`. - -If you want you can carefully double check your work before actually -opening the pull request by doing a dry run via "`eb --dry-run --new-pr`" -or "`eb -D --new-pr`". - -Finally, you can use "`eb --preview-pr`" to see how the easyconfig files -for which you plan to create a pull request differ from existing easyconfig -files. - -### Updating pull requests - -To update an existing pull request with additional changes -you can use "`eb --update-pr`" and pass the pull request ID, -alongside the paths to the updated files. - -If you have only created a branch (for example via `eb --new-branch-github`) -you can update it via `--update-branch-github` in the same way, -passing the branch name instead of a pull request ID. - -### Using a pull request - -Next to creating and updating branches and pull requests -you can also *use* easyconfig files and easyblocks from a pull request, -regardless of its status (open, merged, or closed). This is particularly -useful when testing contributions, or to install software for which -support is not yet included in the latest EasyBuild release. - -Using the `--from-pr` option you can install easyconfig files from the -pull request with specified ID. By default all easyconfig files that are -touched by the pull request will be installed, but you can specify -particular ones to use as well. It is generally advised to also use the -`--robot` option to ensure that the easyconfig files are installed in the -correct order with respect to dependencies. - -Similarly, using a new or updated easyblock from a pull request is as simple -as using the `--include-easyblocks-from-pr` option. And of course you can -combine it with `--from-pr`! - -Via `--upload-test-report` you can let EasyBuild submit a comment into the -easyconfig pull request to show that the installation worked on your system. This is -useful for others to know, in particular EasyBuild maintainers, since the comment -will include information about your system (OS, processor, etc.) and your EasyBuild configuration. - -## Demo - -!!! Warning "Not currently suited for LUMI" - - This is text from a previous (non-Cray) version of the tutorial. The demo does require - a number of files not installed on LUMI after going through this tutorial, - and someone who can master the repository used to clean up again. - However, having a look at the output (which was generated in the spring of 2021 on - the CSC system puhti) still offers some information, so the demo was left in this - version of the tutorial. - -That is a lot to digest, so let us make this a bit more concrete with an example: -we will open a pull request for the [`eb-tutorial` example software](../adding_support_software/#example) to *a fork* of the [`easybuild-easyconfigs` repository](https://github.com/easybuilders/easybuild-easyconfigs) using the `eb` command, -and submit a test report in it. - -!!! Note - Make sure that you have correctly configured the GitHub integration, - [see above](#requirements-configuration). - -### Creating pull request - -We first configure EasyBuild to target the `ebtutorial` GitHub account rather -than the default `easybuilders` GitHub organisation, -by defining the `pr-target-account` configuration setting: - -```shell -export EASYBUILD_PR_TARGET_ACCOUNT=ebtutorial -``` - -In the output of "`eb --show-config`" you should see a line like this: - -``` -pr-target-account (E) = ebtutorial -``` - -We only do this to avoid that lots of pull requests for the `eb-tutorial` -example software are opened in the [central easyconfigs repository](https://github.com/easybuilders/easybuild-easyconfigs). - -Opening a pull request is as simple as running "`eb --new-pr`" and passing -the easyconfig file: - -```shell -$ eb --new-pr example.eb -== temporary log file in case of crash /tmp/eb-ggr6scbq/easybuild-hnk271xj.log -== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... -== fetching branch 'develop' from https://github.com/ebtutorial/easybuild-easyconfigs.git... -== copying files to /tmp/eb-ggr6scbq/git-working-dirxwk1fzaw/easybuild-easyconfigs... -== pushing branch '20200622095415_new_pr_eb-tutorial100' to remote 'github_ebtutorial_qgtfU' (git@github.com:ebtutorial/easybuild-easyconfigs.git) -Enter passphrase for key '/home/example/.ssh/id_rsa': -Please enter password for encrypted keyring: - -Opening pull request -* target: ebtutorial/easybuild-easyconfigs:develop -* from: ebtutorial/easybuild-easyconfigs:20200622095415_new_pr_eb-tutorial100 -* title: "{tools}[GCC/10.2.0] eb-tutorial v1.0.1" -* labels: new -* description: -""" -(created using `eb --new-pr`) - -""" -* overview of changes: - easybuild/easyconfigs/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb | 26 ++++++++++++++++++++++++++ - 1 file changed, 26 insertions(+) - -Opened pull request: https://github.com/ebtutorial/easybuild-easyconfigs/pull/ -== Temporary log file(s) /tmp/eb-ggr6scbq/easybuild-hnk271xj.log* have been removed. -== Temporary directory /tmp/eb-ggr6scbq has been removed. -``` - -Take a moment to grasp what we did here: we ran **a single `eb` command** which -took care of the **[whole contribution procedure](#contribution-procedure)** for us, including: - -* Cloning the `easybuilders/easybuild-easyconfigs` repository and checking out the `develop` branch (in a temporary - directory); -* Picking a sensible name for a branch and creating it; -* Adding the `eb-tutorial` easyconfig file to the branch, in the correct location - (`easybuild/easyconfigs/e/eb-tutorial/`) and with the correct filename (`eb-tutorial-1.0.1-GCC-10.2.0.eb`); -* Pushing the branch to our fork (`example/easybuild-easyconfigs`); -* Actually opening the pull request, using an informative title. - -That is so... easy! - -This feature not only *significantly* lowers the bar for contributing, -it also saves quite a bit of time since you don't need to double check -various details (like targeting the `develop` branch) or spend time on -coming up with a nice looking title or funny branch name (although you -still can if you really want to). - -There are a couple of nice side effects too, like not having any local branches -to tidy up on once the pull request gets merged (since `--new-pr` created the -branch only in a temporary directory). - -If many contributions are made via `--new-pr` it also simplifies the task -of EasyBuild maintainers, since pull requests opened this way have a particular -structure to them and thus are easier to digest because they look familiar. - -### Uploading test report - -After opening the pull request, we should also upload a test report to show that the installation is working. -This is just as easy as creating the pull request. - -First make sure that the pre-installed software in the prepared environment -is available, since the required dependencies for `eb-tutorial` are already -installed there: - -```shell -module use /easybuild/modules/all -``` - -You can verify which dependencies are still missing using `--from-pr` combined with `--missing`: - -```shell -# change '1' to the ID of your own pull request (see output of --new-pr) -$ eb --from-pr 1 --missing -== temporary log file in case of crash /tmp/eb-ioi9ywm1/easybuild-e3v0xa1b.log -Please enter password for encrypted keyring: -== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... - -1 out of 20 required modules missing: - -* eb-tutorial/1.0.1-GCC-10.2.0 (eb-tutorial-1.0.1-GCC-10.2.0.eb) -``` - -Uploading a test report boils down to combining `--from-pr` with `--upload-test-report`: - -```shell -# change '1' to the ID of your own pull request (see output of --new-pr) -$ eb --rebuild --from-pr 1 --upload-test-report -Please enter password for encrypted keyring: -... -== processing EasyBuild easyconfig /tmp/eb-bnb1pv3n/files_pr65/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb -== building and installing eb-tutorial/1.0.1-GCC-10.2.0... -... -== COMPLETED: Installation ended successfully (took 2 sec) -... -Adding comment to easybuild-easyconfigs issue #65: 'Test report by @ebtutorial -**SUCCESS** -Build succeeded for 1 out of 1 (1 easyconfigs in this PR) -example - Linux centos linux 7.8.2003, x86_64, Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz (haswell), Python 3.6.8 -See https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 for a full test report.' -== Test report uploaded to https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 and mentioned in a comment in easyconfigs PR#1 -``` - -Note that we may need to use `--rebuild` here since `eb-tutorial` may already be installed. - -This results in a comment being added to the pull request: - -
Test report comment
- -The gist linked from this comment provides more detailed information: - -
Test report gist
- ---- - -*[[next: Additional reading]](4_00_additional_reading.md)* diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md b/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md index 74f3f82a..1e1de7c7 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md @@ -40,7 +40,6 @@ community: adapt, e.g., the easyconfig files for Open MPI to work on LUMI. Some site have really large hook files to implement policies and modify standard EasyBuild build recipes, e.g., - - [The hooks file from Jülich Supercomputing Centre](https://github.com/easybuilders/JSC/blob/2022/Custom_Hooks/eb_hooks.py) - ComputeCanada [cc_hooks_common.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_common.py) and [cc_hooks_gento.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_gentoo.py) From 71458709b1e7593e12cebc1f0e88671157beef78 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 7 Apr 2022 21:14:52 +0200 Subject: [PATCH 119/159] Correction of links. --- mkdocs.yml | 41 ++++++++++++++++++++--------------------- 1 file changed, 20 insertions(+), 21 deletions(-) diff --git a/mkdocs.yml b/mkdocs.yml index 8ce643c9..c7c97216 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -121,31 +121,30 @@ nav: - "CSC (spring '22)": - (overview): 2022-CSC_and_LO/index.md - Introduction to EasyBuild: - - (overview): 2022-CSC_and_LO/1_00_part1_intro.md - - What is EasyBuild?: 2022-CSC_and_LO/1_01_what_is_easybuild.md - - The Lmod module system: 2022-CSC_and_LO/1_02_Lmod.md - - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_03_CPE.md - - LUMI software stacks: 2022-CSC_and_LO/1_04_LUMI_software_stack.md - - Terminology: 2022-CSC_and_LO/1_05_terminology.md - - Installation: 2022-CSC_and_LO/1_06_installation.md - - Configuration: 2022-CSC_and_LO/1_07_configuration.md - - Basic usage: 2022-CSC_and_LO/1_08_basic_usage.md + - (overview): 2022-CSC_and_LO/1_Intro/index.md + - What is EasyBuild?: 2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md + - The Lmod module system: 2022-CSC_and_LO/1_Intro/1_02_Lmod.md + - The HPE Cray Programming Environment: 2022-CSC_and_LO/1_Intro/1_03_CPE.md + - LUMI software stacks: 2022-CSC_and_LO/1_Intro/1_04_LUMI_software_stack.md + - Terminology: 2022-CSC_and_LO/1_Intro/1_05_terminology.md + - Installation: 2022-CSC_and_LO/1_Intro/1_06_installation.md + - Configuration: 2022-CSC_and_LO/1_Intro/1_07_configuration.md + - Basic usage: 2022-CSC_and_LO/1_Intro/1_08_basic_usage.md - Using EasyBuild: - - (overview): 2022-CSC_and_LO/2_00_part2_using.md - - Troubleshooting: 2022-CSC_and_LO/2_01_troubleshooting.md - - Creating easyconfig files: 2022-CSC_and_LO/2_02_creating_easyconfig_files.md - - Using external modules from the Cray PE: 2022-CSC_and_LO/2_03_external_modules.md - - Implementing easyblocks: 2022-CSC_and_LO/2_04_implementing_easyblocks.md + - (overview): 2022-CSC_and_LO/2_Using/index.md + - Troubleshooting: 2022-CSC_and_LO/2_Using/2_01_troubleshooting.md + - Creating easyconfig files: 2022-CSC_and_LO/2_Using/2_02_creating_easyconfig_files.md + - Using external modules from the Cray PE: 2022-CSC_and_LO/2_Using/2_03_external_modules.md + - Implementing easyblocks: 2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md - Advanced topics: - - (overview): 2022-CSC_and_LO/3_00_part3_advanced.md - - Using EasyBuild as a library: 2022-CSC_and_LO/3_01_easybuild_library.md - - Using hooks to customise EasyBuild: 2022-CSC_and_LO/3_02_hooks.md - - Submitting installations as Slurm jobs: 2022-CSC_and_LO/3_03_slurm_jobs.md - - Module naming schemes (incl. hierarchical): 2022-CSC_and_LO/3_04_module_naming_schemes.md - - GitHub integration to facilitate contributing to EasyBuild: 2022-CSC_and_LO/3_05_github_integration.md + - (overview): 2022-CSC_and_LO/3_Advanced/index.md + - Using EasyBuild as a library: 2022-CSC_and_LO/3_Advanced/3_01_easybuild_library.md + - Using hooks to customise EasyBuild: 2022-CSC_and_LO/3_Advanced/3_02_hooks.md + - Submitting installations as Slurm jobs: 2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md + - Module naming schemes (incl. hierarchical): 2022-CSC_and_LO/3_Advanced/3_04_module_naming_schemes.md + - GitHub integration to facilitate contributing to EasyBuild: 2022-CSC_and_LO/3_Advanced/3_05_github_integration.md - Additional reading: 2022-CSC_and_LO/4_00_additional_reading.md - plugins: # show revision date at bottom of each page - git-revision-date-localized From 4a7d65d1c018a44ce3b1d9b5f73d59df13f8e433 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 10 May 2022 17:50:47 +0200 Subject: [PATCH 120/159] Multiple minor corrections. --- docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md b/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md index 1e1de7c7..74f3f82a 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md @@ -40,6 +40,7 @@ community: adapt, e.g., the easyconfig files for Open MPI to work on LUMI. Some site have really large hook files to implement policies and modify standard EasyBuild build recipes, e.g., + - [The hooks file from Jülich Supercomputing Centre](https://github.com/easybuilders/JSC/blob/2022/Custom_Hooks/eb_hooks.py) - ComputeCanada [cc_hooks_common.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_common.py) and [cc_hooks_gento.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_gentoo.py) From 6b0ebef152c2b1765fcecebd724ac6ddef51b1d8 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 7 Mar 2022 17:29:52 +0100 Subject: [PATCH 121/159] Initial commit, already contains the structure to work on the first tutorial. --- README.md | 2 +- docs/2021-LUST/index.md | 5 +++++ docs/2021-lust/index.md | 22 +--------------------- 3 files changed, 7 insertions(+), 22 deletions(-) create mode 100644 docs/2021-LUST/index.md diff --git a/README.md b/README.md index e84c0d54..3d6d3571 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -

+å

Welcome to the repository that hosts the sources of the official **[EasyBuild](https://easybuild.io) tutorial**, see https://easybuilders.github.io/easybuild-tutorial. diff --git a/docs/2021-LUST/index.md b/docs/2021-LUST/index.md new file mode 100644 index 00000000..c2adbc65 --- /dev/null +++ b/docs/2021-LUST/index.md @@ -0,0 +1,5 @@ +# EasyBuild tutorial for LUST + +Overview page of the introductory tutorial on [EasyBuild](https://easybuild.io) for the *[LUMI](https://www.lumi-supercomputer.eu) User Support team (LUST)*. + +The tutorial is available on [the EasyBuilders tutoral web site](https://easybuilders.github.io/easybuild-tutorial/2021-lust/). \ No newline at end of file diff --git a/docs/2021-lust/index.md b/docs/2021-lust/index.md index 53f3b9ac..c2adbc65 100644 --- a/docs/2021-lust/index.md +++ b/docs/2021-lust/index.md @@ -2,24 +2,4 @@ Overview page of the introductory tutorial on [EasyBuild](https://easybuild.io) for the *[LUMI](https://www.lumi-supercomputer.eu) User Support team (LUST)*. -- [Part I: **Introduction to EasyBuild**](part1_intro.md) *(Tue March 9th 2021, 9am-12 CET)* - * [What is EasyBuild?](what_is_easybuild.md) - * [Terminology](terminology.md) - * [Installation](installation.md) *(hands-on)* - * [Configuration](configuration.md) *(hands-on)* - * [Basic usage](basic_usage.md) *(hands-on)* -- [Part II: **Using EasyBuild**](part2_using.md) *(Tue March 23rd 2021, 9am-12 CET)* - * [Troubleshooting](troubleshooting.md) *(hands-on)* - * [Creating easyconfig files](creating_easyconfig_files.md) *(hands-on)* - * [Implementing easyblocks](implementing_easyblocks.md) *(hands-on)* -- [Part III: **Advanced topics**](part3_advanced.md) *(Tue March 30th 2021, 9am-12 CEST)* - * [Using EasyBuild as a library](easybuild_library.md) *(hands-on)* - * [Using hooks to customise EasyBuild](hooks.md) *(hands-on)* - * [Submitting installations as Slurm jobs](slurm_jobs.md) *(hands-on)* - * [Module naming schemes (incl. hierarchical)](module_naming_schemes.md) *(hands-on)* - * [GitHub integration to facilitate contributing to EasyBuild](github_integration.md) *(hands-on)* -- [Part IV: **EasyBuild on Cray systems**](part4_cray.md) *(Friday June 18th 2021, 09-12 CEST)* - * [Introduction to Cray Programming Environment](cray/introduction.md) *(hands-on)* - * [Cray External Modules](cray/external_modules.md) *(hands-on)* - * [Cray Custom Toolchains](cray/custom_toolchains.md) *(hands-on)* - * [EasyBuild at CSCS](cray/easybuild_at_cscs.md) *(hands-on)* +The tutorial is available on [the EasyBuilders tutoral web site](https://easybuilders.github.io/easybuild-tutorial/2021-lust/). \ No newline at end of file From 044517585e73b33c40465fcfb4427b588a458a1c Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 7 Mar 2022 18:30:05 +0100 Subject: [PATCH 122/159] Updated the "What is EasyBuild" page of the tutorial for LUMI." --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 12 + .../2022-CSC_and_LO/1_01_what_is_easybuild.md | 321 ++++++++++++++++++ 2 files changed, 333 insertions(+) create mode 100644 docs/2022-CSC_and_LO/1_00_part1_intro.md create mode 100644 docs/2022-CSC_and_LO/1_01_what_is_easybuild.md diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md new file mode 100644 index 00000000..0bfd0ad5 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -0,0 +1,12 @@ +# Part I: Introduction to EasyBuild on Cray systems + +*[[back to start page]](index.md)* + +*(Tue March 4th 2021, 9am-12 CET)* + +- [What is EasyBuild?](1_01_what_is_easybuild.md) + +- [Terminology](terminology.md) +- [Installation](installation.md) *(hands-on)* +- [Configuration](configuration.md) *(hands-on)* +- [Basic usage](basic_usage.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md new file mode 100644 index 00000000..678c8c4a --- /dev/null +++ b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md @@ -0,0 +1,321 @@ +# What is EasyBuild? + +*[[back: Overview]](index.md)* + +

EasyBuild logo

+ +**[EasyBuild](http://easybuild.io)** is a **software build and installation framework** +that manages (scientific) software in an efficient way, without compromising on performance. +Easybuild is especially tailored towards [**High Performance Computing (HPC)**](https://en.wikipedia.org/wiki/Supercomputer) systems, +and provides a **command-line interface** to facilitate the installation and management of a scientific software stack. + +--- + + + + + +### *Elevator pitch* + +EasyBuild intends to relieve HPC support teams from manually managing +software installations while at the same time **providing a consistent and well performing +scientific software stack** to end users of HPC systems. + +It serves as a **uniform interface for installing scientific software** +and saves valuable time (and frustration) by the automation of tedious, boring and repetitive tasks. + +In addition, EasyBuild can **empower scientific researchers to self-manage their software stack**, +and it can serve as a tool that can be leveraged for **building optimized container images**. + +The project has grown to become a **platform for collaboration** among HPC sites worldwide. + +--- + +On LUMI we pay a lot of attention to empowering users to create their custom software stack which +is best suited for their project without being confronted with hundreds of modules that are not +useful to them. It also enables them to easily customize the build instructions to create a +project-specific build or even to install software with custom patches to adapt to their needs. +For this we offer both [EasyBuild](https://easybuild.io/) and [Spack](https://spack.io/) +as tools but obviously this tutorial concentrates on EasyBuild alone. + +EasyBuild can also be used to [generate container recipes](https://docs.easybuild.io/en/latest/Containers.html) +that will then use EasyBuild when building +the container to install platform-optimised software in the container. However, as that functionality +is still under development on LUMI it is not yet covered in this LUMI-specific version of the tutorial. + + +--- + +### *Key features* + +EasyBuild is capable of **fully autonomously installing (scientific) software**, +including making sure that all necessary dependencies are installed, +and automatically generating environment module files. + +***No*** **admin privileges are required**: it is sufficient to have write permissions +to the preferred software installation prefix. + +EasyBuild is [**highly configurable**](https://docs.easybuild.io/en/latest/Configuration.html) via configuration files, +environment variables, and command line options. Various elements of the functionality can be +[**dynamically extended**](https://docs.easybuild.io/en/latest/Including_additional_Python_modules.html) via plugins, +and [**hooks**](https://docs.easybuild.io/en/latest/Hooks.html) are available for further site-specific customizations. + +The installation procedure executed by EasyBuild is thoroughly +[**logged**](https://docs.easybuild.io/en/latest/Logfiles.html), and is fully transparent via support for +[**dry runs**](https://docs.easybuild.io/en/latest/Extended_dry_run.html) and +[**tracing**](https://docs.easybuild.io/en/latest/Tracing_progress.html) the software installation procedure +as it is performed. + +EasyBuild supports using a **custom module naming scheme**, allows for +*hierarchical* module naming schemes, and **integrates with various other tools** ranging from resource managers ([Slurm](https://slurm.schedmd.com) and [GC3Pie](https://github.com/gc3pie/gc3pie)), +container tools ([Singularity](https://github.com/hpcng/singularity) and [Docker](https://www.docker.com)), +packaging tools ([FPM](https://fpm.readthedocs.io)), and so on. + +--- + +### *Focus points* + +EasyBuild was created specifically for **installing scientific software on HPC systems**, +which is reflected in some of the design choices that were made. + + +#### Performance + +EasyBuild strongly prefers to **build software from source code**, whenever possible. + +This is important to ensure that the binaries that are installed can maximally exploit +the capabilities of the system architecture on which the software will be run. + +For that same reason, EasyBuild out-of-the-box **optimizes software for the processor architecture of the build host** +by default, via compiler options like ``-march=native`` (GCC), ``-xHost`` (Intel compilers), etc. +This behaviour [can be changed via the ``--optarch`` configuration setting](https://docs.easybuild.io/en/latest/Controlling_compiler_optimization_flags.html). +When using the HPE Cray Programming Environment, the target modules are used instead. +In principle it is possible to cross-compile with EasyBuild, but this comes with some pitfalls. +By default EasyBuild tries to run tests if the build process of a package provides that +functionality, or can run simple tests of its own, and of course these tests will fail if +the host CPU does not support the full instruction set of the target CPU. Secondly, more +and more software has broken build processes where developers thought that it was a good +idea to hard-code options such as ``-march=native`` or ``--xHost`` in a way that EasyBuild +cannot easily overwrite those. + + +#### Reproducibility + +In addition to performance, **reproducibility of installations** is a core aspect of EasyBuild. + +TODO correct link in the next lines. + +Most software installations performed with EasyBuild use a **particular compiler toolchain**, +with which we aim to be in control over the build environment and avoid relying on tools and libraries +provided by the operating system. For similar reasons, we try to **provide all required dependencies through EasyBuild** as well, +with a few notable exceptions, like ``OpenSSL`` for security reasons, and Infiniband and GPU drivers which +are too closely intertwined with the operating system. + +For both toolchains and dependencies, **fixed software versions** are specified in the +easyconfig files. That way, easyconfig files can easily be shared with others: +if they worked for you it is very likely that they will work for others too, because the vast majority of the +software stack is controlled by EasyBuild. + +As we shall see when discussing the HPE Cray Programming Environment, the reproducibility is a bit more limited +in that environment due to the way the programming environment selects shared libraries at runtime. + + +#### Community effort + +In a number of different ways, we try to encourage EasyBuild users to **collaborate** and help each other out. + +We actively recommend people to report problems and bugs, to submit ideas for additional features and improvements, +and to [**contribute back**](https://docs.easybuild.io/en/latest/Contributing.html) when possible, be it +by opening pull requests to the [GitHub repositories](https://github.com/easybuilders) or the [documentation](https://docs.easybuild.io). + +The EasyBuild community also provides packages for two standard toolchains, called the +[**common toolchains**](https://easybuild.readthedocs.io/en/latest/Common-toolchains.html). +The so-called ``foss`` is based on free and open software (GNU compilers, FlexiBlas with OpenBLAS backend, LaPACK, FFTW and Open MPI) +while the ``intel`` toolchain uses the full stack of Intel compilers, libraries and MPI. Currently these toolchains +cannot be supported on LUMI though as getting Open MPI to run on LUMI is challenging and as the Intel libraries and +some MPI versions also produce slow or incorrect results on AMD CPUs (and it is not clear if Intel MPI is compatible +with SlingShot 11). +Focusing the efforts on those two popular common toolchains makes it possible to support a lot of build recipes that are +collected in the [central repository](https://github.com/easybuilders/easybuild-easyconfigs). +They can still serve as a basis to develop build recipes for the HPE Cray Programming Environment. + +Last but not least, EasyBuild provides various [**GitHub integration features**](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) +that greatly facilitate the contribution process: opening, updating, and testing pull requests, +reviewing incoming contributions, and much more can all be done directly from the EasyBuild +command line. This not only saves time, effort, brain cycles, and mouse clicks for contributors, +but it also makes the review process for *maintainers* significantly easier. +All together this leads to improved stability and consistency. + +--- + +### *What EasyBuild is* ***not*** + +EasyBuild is ***not*** **YABT (Yet Another Build Tool)**: it does *not* replace established build +tools like CMake or ``make``, it wraps around them. +If the installation procedure of a software package involves running some unholy trinity of tools while whispering +the correct magic incantations, EasyBuild automates this process for you. + +It is ***not*** **a replacement for traditional Linux package managers** like ``yum``, ``dnf`` or ``apt``. +EasyBuild relies on certain tools and libraries provided by the operating system. This includes glibc, OpenSSL, +drivers for Infiniband and GPUs, and so on. It is required that these tools are installed and managed by other means. +This is typically done via the package management tool that comes with your Linux distribution. + +Finally, EasyBuild is ***not*** **a magic solution to all your (software installation) problems**. +You may, and probably still will occasionally, run into compiler errors unless somebody has already taken care of the problem for you. +This is even more likely to happen in the Cray environment as builds are not often tested with the Cray compilers +or even with the AMD compilers. + +--- + +### *Requirements* + +Since EasyBuild is primarily intended for installing (scientific) software on HPC systems, the main target is +systems running some flavor of **GNU/Linux**. EasyBuild also works on *macOS*, +but significant effort would be required to use it extensively there. + + + +EasyBuild is implemented in [**Python**](https://www.python.org), and is compatible with Python 2.7 and 3.5+. +Since [Python 2 is no longer maintained](https://blog.python.org/2020/04/python-2718-last-release-of-python-2.html), +it is recommended to use a recent version of Python 3 for running EasyBuild. + +An [**environment modules**](https://en.wikipedia.org/wiki/Environment_Modules_(software)) tool is required for running EasyBuild. +The modern Lua-based implementation **[Lmod](https://lmod.readthedocs.io)** is strongly recommended, +but other (Tcl-based) implementations are also supported. +Both Tcl and Lua are supported as syntax for environment module files. +On LUMI Lmod is used as the module tool. + +--- + +### *Development* + + +EasyBuild is **actively developed** by a worldwide community under the +[**GPLv2 open source license**](https://www.gnu.org/licenses/old-licenses/gpl-2.0.html) +via the [**``easybuilders``** organisation on GitHub](https://github.com/easybuilders), in the following repositories: + +- [easybuild-framework](https://github.com/easybuilders/easybuild-framework) +- [easybuild-easyblocks](https://github.com/easybuilders/easybuild-easyblocks) +- [easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs) +- [easybuild](https://github.com/easybuilders/easybuild) (website, documentation) +- [easybuild-tutorial](https://github.com/easybuilders/easybuild-tutorial) (the generic EasyBuild tutorials) + +On LUMI the generic EasyBuild installation is used but extended and adapted using LUMI-specifics +plugins and hooks maintained in the [LUMI software stack repository](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack). + +Stable versions of EasyBuild have been released roughly every 6-8 weeks since 2012, +and are published via the *Python Package Index (PyPI)*, +see [https://pypi.org/project/easybuild](https://pypi.org/project/easybuild/). + +**Comprehensive testing** practices are applied throughout the +development cycle, with extensive suites of unit and integration tests being run in a CI environment (GitHub Actions), +consistent testing of incoming contributions (pull requests), and thorough regression testing before every release. + +--- + +### *A worldwide community* + +EasyBuild has grown into a **community project** since it was first released publicy in 2012, +and is used and developed by dozens (hundreds?) of HPC centres and consortia worldwide. + +Since 2016 the EasyBuild community has gathered for a yearly [**EasyBuild User Meeting**](https://easybuild.io/eum), +which is an open and highly interactive event that provides a great opportunity to meet fellow EasyBuild users, +developers, and maintainers, and to discuss related topics, learn about new aspects of the tool, and how other HPC +sites leverage it, and integrate it with other tools like [Jenkins](https://www.youtube.com/watch?v=bzW45cfRUMU), +[Ansible](https://www.youtube.com/watch?v=Qr2udsAjkG0), [CernVM-FS](https://www.youtube.com/watch?v=_0j5Shuf2uE), +[ReFrame](https://www.youtube.com/watch?v=KN9QwQIrso4) etc. + +HPC sites, consortia, and companies that use EasyBuild include (but are not limited to): + +VSC logo +JSC logo +HPCNow! logo +Fred Hutch logo +Vienna BioCenter logo +CÉCI logo +Compute Canada logo +UiO logo +BEAR logo +SNIC logo +National University of Singapore logo +IT4Innovations logo +Texas A&M logo +TU Dresden logo +University of Melbourne logo +University of Luxembourg logo +EPCC logo +Cyprus Institute logo +University of Basel logo +Microsoft logo + +--- + +### *Getting help* + +#### *Documentation* + +If you have any questions about EasyBuild, make sure to check the EasyBuild documentation first, +which is available at [**https://docs.easybuild.io**](https://docs.easybuild.io). + +It is not unlikely that an answer to your question is available there already. +The [search box](https://docs.easybuild.io/en/latest/search.html) on the top left may be helpful. + +#### GitHub + + + +You can open an issue in one of the EasyBuild repositories in [the `easybuilders` GitHub organisation](https://github.com/easybuilders/) to +ask questions, report problems or share suggestions and ideas for additional +improvements. + +Please try to post the issue in the most suitable repository. If you are +not sure which repository is most suited just go ahead and open the issue anyway, +the EasyBuild maintainers can move your issue if deemed appropriate. + +Also here you can use the search box on the top left, to see if somebody else +has already asked a similar question or reported a similar problem. + +#### Mailing list + +Feel free to ask any questions or provide feedback via the EasyBuild mailing list +`easybuild@lists.ugent.be`. + +Keep in mind that **you must be subscribed to the mailing list in order to +post messages there!** You can subscribe via [this link](https://lists.ugent.be/wws/subscribe/easybuild). + +In 2016-2017 over 1,000 messages were posted to the EasyBuild mailing list. +More recently this has decreased to about 700 messages per year, mostly due +to the EasyBuild Slack channel. + +#### Slack + +Slack logo + +The [EasyBuild Slack](https://easybuild.slack.com/) is a quick and easy way +to get in touch with the EasyBuild community. +Over 400 people have joined it so far, and there is usually somebody awake in there +who may be able to help you out. + +**To join the EasyBuild Slack request an invitation via +[https://easybuild-slack.herokuapp.com](https://easybuild-slack.herokuapp.com/).** + +If you are not a big fan of Slack, you can also consider joining the `#easybuild` +IRC channel (available on the [Freenode IRC network](https://freenode.net/)). +A bot relays messages between this IRC channel and the `#general` channel in the EasyBuild Slack. + + +#### Conference calls + +Every other week on Wednesday an EasyBuild conference call is set up to discuss recent +developments, provide an outlook to the next EasyBuild release, and +provide answers to questions that can be raised by the people attending the call. + +These conference calls are open to anyone interested in the EasyBuild project, +and have been organised since November 2013. + +More information, including the dates and times of the next calls, is available +[on the EasyBuild Wiki](https://github.com/easybuilders/easybuild/wiki/Conference-calls). + +--- + +*[[next: Terminology]](terminology.md)* From e4381949da943c5c8662740f15a050256a866f6f Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 9 Mar 2022 10:58:10 +0100 Subject: [PATCH 123/159] Updated the structure, initial updates of terminology and a new page about the Cray PE and Lmod. --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 4 +- .../2022-CSC_and_LO/1_01_what_is_easybuild.md | 34 +- docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md | 71 ++++ docs/2022-CSC_and_LO/1_03_terminology.md | 348 ++++++++++++++++++ 4 files changed, 438 insertions(+), 19 deletions(-) create mode 100644 docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md create mode 100644 docs/2022-CSC_and_LO/1_03_terminology.md diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md index 0bfd0ad5..e381f45d 100644 --- a/docs/2022-CSC_and_LO/1_00_part1_intro.md +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -5,8 +5,8 @@ *(Tue March 4th 2021, 9am-12 CET)* - [What is EasyBuild?](1_01_what_is_easybuild.md) - -- [Terminology](terminology.md) +- [Terminology](1_02_terminology.md) + - [Installation](installation.md) *(hands-on)* - [Configuration](configuration.md) *(hands-on)* - [Basic usage](basic_usage.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md index 678c8c4a..5ccdb1db 100644 --- a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md @@ -15,7 +15,7 @@ and provides a **command-line interface** to facilitate the installation and man -### *Elevator pitch* +## Elevator pitch EasyBuild intends to relieve HPC support teams from manually managing software installations while at the same time **providing a consistent and well performing @@ -46,7 +46,7 @@ is still under development on LUMI it is not yet covered in this LUMI-specific v --- -### *Key features* +## Key features EasyBuild is capable of **fully autonomously installing (scientific) software**, including making sure that all necessary dependencies are installed, @@ -73,13 +73,13 @@ packaging tools ([FPM](https://fpm.readthedocs.io)), and so on. --- -### *Focus points* +## Focus points EasyBuild was created specifically for **installing scientific software on HPC systems**, which is reflected in some of the design choices that were made. -#### Performance +### Performance EasyBuild strongly prefers to **build software from source code**, whenever possible. @@ -99,7 +99,7 @@ idea to hard-code options such as ``-march=native`` or ``--xHost`` in a way that cannot easily overwrite those. -#### Reproducibility +### Reproducibility In addition to performance, **reproducibility of installations** is a core aspect of EasyBuild. @@ -120,7 +120,7 @@ As we shall see when discussing the HPE Cray Programming Environment, the reprod in that environment due to the way the programming environment selects shared libraries at runtime. -#### Community effort +### Community effort In a number of different ways, we try to encourage EasyBuild users to **collaborate** and help each other out. @@ -148,7 +148,7 @@ All together this leads to improved stability and consistency. --- -### *What EasyBuild is* ***not*** +## What EasyBuild is ***not*** EasyBuild is ***not*** **YABT (Yet Another Build Tool)**: it does *not* replace established build tools like CMake or ``make``, it wraps around them. @@ -167,7 +167,7 @@ or even with the AMD compilers. --- -### *Requirements* +## Requirements Since EasyBuild is primarily intended for installing (scientific) software on HPC systems, the main target is systems running some flavor of **GNU/Linux**. EasyBuild also works on *macOS*, @@ -187,7 +187,7 @@ On LUMI Lmod is used as the module tool. --- -### *Development* +## Development EasyBuild is **actively developed** by a worldwide community under the @@ -213,7 +213,7 @@ consistent testing of incoming contributions (pull requests), and thorough regre --- -### *A worldwide community* +## A worldwide community EasyBuild has grown into a **community project** since it was first released publicy in 2012, and is used and developed by dozens (hundreds?) of HPC centres and consortia worldwide. @@ -250,9 +250,9 @@ HPC sites, consortia, and companies that use EasyBuild include (but are not limi --- -### *Getting help* +## Getting help -#### *Documentation* +### *Documentation* If you have any questions about EasyBuild, make sure to check the EasyBuild documentation first, which is available at [**https://docs.easybuild.io**](https://docs.easybuild.io). @@ -260,7 +260,7 @@ which is available at [**https://docs.easybuild.io**](https://docs.easybuild.io) It is not unlikely that an answer to your question is available there already. The [search box](https://docs.easybuild.io/en/latest/search.html) on the top left may be helpful. -#### GitHub +### GitHub @@ -275,7 +275,7 @@ the EasyBuild maintainers can move your issue if deemed appropriate. Also here you can use the search box on the top left, to see if somebody else has already asked a similar question or reported a similar problem. -#### Mailing list +### Mailing list Feel free to ask any questions or provide feedback via the EasyBuild mailing list `easybuild@lists.ugent.be`. @@ -287,7 +287,7 @@ In 2016-2017 over 1,000 messages were posted to the EasyBuild mailing list. More recently this has decreased to about 700 messages per year, mostly due to the EasyBuild Slack channel. -#### Slack +### Slack Slack logo @@ -304,7 +304,7 @@ IRC channel (available on the [Freenode IRC network](https://freenode.net/)). A bot relays messages between this IRC channel and the `#general` channel in the EasyBuild Slack. -#### Conference calls +### Conference calls Every other week on Wednesday an EasyBuild conference call is set up to discuss recent developments, provide an outlook to the next EasyBuild release, and @@ -318,4 +318,4 @@ More information, including the dates and times of the next calls, is available --- -*[[next: Terminology]](terminology.md)* +*[[next: The HPE Cray Programming Environment and Lmod]](1_02_CPE_and_Lmod.md)* diff --git a/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md b/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md new file mode 100644 index 00000000..01812057 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md @@ -0,0 +1,71 @@ +# The HPE Cray Programming Environment and Lmod + +*[[back: What is EasyBuild?]](1_01_what_is_easybuild.md)* + +--- + +On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated +as Cray PE). The environment provides several tools, including compilers, communication libraries, +optmized math libraries and various other libraries, analyzers and debuggers. + +The Cray PE is made available through *environment modules* tha allow to select particular versions of +tools and to configure the environment in a flexible way. + +--- + +## Modules + +*Module* is a massively overloaded term in (scientific) software and IT in general +(kernel modules, Python modules, and so on). +In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. + +[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept +on HPC systems: it is a way to specify changes that should be made to one or more +[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a +[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file +is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or +[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, +and specifies which environment variables should be updated, and how (append, +prepend, (re)define, undefine, etc.) upon loading the environment module. +Unloading the environment module will restore the shell environment to its previous state. + +Environment module files are processed via a **modules tool**, of which there +are several conceptually similar yet slightly different implementations. +The oldest module tool still in use today is Environment Modules 3.2, implemented in C and +supporting module files written in Tcl. After a gap in developement, Xavier Delaruelle of CEA +developed [Environment Modules 4 and 5](https://sourceforge.net/projects/modules/) which is +fully implemented on Tcl. An alternative module tool is [Lmod](https://lmod.readthedocs.io), +developed by Robert McLay at TACC and implemented in LUA. This tool supports natively LUA +module files but also offers a high degree of compatibility with Tcl-based module files +developed for Environment Modules fia a translation layer and some API translation. + +The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no +packages or official support for Environment Modules 4 or 5. On LUMI, Lmod was selected as +the module tool. At the user level, Lmod and the various versions of Emvironment Modules +have very simmilar commands for managing the environment, but with different options. +The commands for searching for modules are very different though so if you are not familiar +with Lmod and its commands for users, it is worthwile to read the +[LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). + +--- + +## Lmod features + + +--- + +## Cray PE components + + +--- + +## Configuring the Cray PE through modules + +--- + +## Further reading + + +--- + +*[[next: Terminology]](1_03_terminology.md)* diff --git a/docs/2022-CSC_and_LO/1_03_terminology.md b/docs/2022-CSC_and_LO/1_03_terminology.md new file mode 100644 index 00000000..ec085b0c --- /dev/null +++ b/docs/2022-CSC_and_LO/1_03_terminology.md @@ -0,0 +1,348 @@ +# Terminology + +*[[back: The HPE Cray Programming Environment and Lmod]](1_02_CPE_and_Lmod.md)* + +--- + +Over the years, we have come up with some terminology specific to EasyBuild +to refer to particular components, which we use alongside established terminology relevant to the context +of building and installing software. + +It is important to be familiar with these terms, so we'll briefly cover them one by one. + +--- + +## *Toolchains* + +A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, +which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. + +We refer to the different parts of a toolchain as **toolchain components**. + +The *compiler component* typically consists of [C](https://en.wikipedia.org/wiki/C_(programming_language)), +[C++](https://en.wikipedia.org/wiki/C%2B%2B), and [Fortran](https://en.wikipedia.org/wiki/Fortran) +compilers in the context of HPC, but additional compilers (for example, +a [CUDA](https://developer.nvidia.com/cuda-zone) compiler for +[GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units) software) +can also be included. + +Additional toolchain components are usually special-purpose libraries: + +- an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/)); + +- libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), + [LAPACK](http://performance.netlib.org/lapack/)); + + These libraries are included because they tend to provide a common API, so mixing them in a single + executable would cause name conflicts when linking. + +- a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); + +A toolchain that includes all of these libraries is referred to as a **full toolchain**, while +a **subtoolchain** is a toolchain that is missing one or more of these libraries. +A **compiler-only toolchain** only consists of compilers (no additional libraries). + +Many EasBuild toolchains are organised in a hierarchy. Each toolchain can use components compiled +with itself or with one of the subtoolchains. + + +### System toolchain + +The **`system` toolchain** is a special case which corresponds to using the compilers and libraries +*provided by the operating system*, rather than using toolchain components that were installed using EasyBuild. + +It used sparingly, mostly to install software where no actual compilation is done or +to build a set of toolchain compilers and its dependencies, since the versions of the system compilers +and libraries are beyond the control of EasyBuild, which could affect the reproducibility of the installation. + +On LUMI however it is used a bit more and it takes some of the functions of the ``GCCcore`` toolchain in +other EasyBuild toolchain hierarchies. + + +### Common toolchains + +The `foss` and `intel` toolchains are also known as the *common toolchains*, +because they are widely adopted by the EasyBuild community. + +The `foss` toolchain consists of all open source components (hence the name: +"FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), +[ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). + +The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version +controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. + +Roughly every 6 months, a new version of these common toolchains is agreed upon +in the EasyBuild community, after extensive testing. + +More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). + +--- + +## *Modules* + +*Module* is a massively overloaded term in (scientific) software and IT in general +(kernel modules, Python modules, and so on). +In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. + +[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept +on HPC systems: it is a way to specify changes that should be made to one or more +[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a +[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file +is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or +[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, +and specifies which environment variables should be updated, and how (append, +prepend, (re)define, undefine, etc.) upon loading the environment module. +Unloading the environment module will restore the shell environment to its previous state. + +Environment module files are processed via a **modules tool**, of which there +are several conceptually similar yet slightly different implementations. +The Tcl-based [Environment Modules](https://sourceforge.net/projects/modules/) implementation, and +[Lmod](https://lmod.readthedocs.io), a more recent Lua-based implementation (which also supports module +files written in Tcl syntax), are the most commonly used ones. + +**Environment module files are automatically generated for each software installation** by EasyBuild, +and *loading* a module results in changes being made to the environment of the current shell +session such that the corresponding software installation can be used. + +--- + +## *EasyBuild framework* + +The EasyBuild *framework* consists of a set of Python modules organised in packages (``easybuild.framework``, +``easybuild.toolchains``, ``easybuild.tools``, etc.) that collectively form **the core of EasyBuild**, +and is developed in the [``easybuild-framework`` repository on GitHub](https://github.com/easybuilders/easybuild-framework). + +It implements the **common functionality that you need when building software from source**, +providing functions for unpacking source files, applying patch files, collecting the output produced +by shell commands that are being run and checking their exit code, generating environment module files, etc. + +The EasyBuild framework does *not* implement any specific installation procedure, it only provides +the necessary functionality to facilitate this. + + +--- + +## *Easyblocks* + +An *easyblock* is **a Python module that implements a specific software installation procedure**, +and can be viewed as a plugin to the EasyBuild framework. Easyblocks can be either *generic* or *software-specific*. + +A **generic easyblock** implements an installation procedure that can be used for +multiple different software packages. Commonly used examples include the ``ConfigureMake`` easyblock +which implements the ubiquitous ``configure``-``make``-``make install`` procedure, and the +``PythonPackage`` easyblock that can be used to install a Python package. + +A **software-specific easyblock** implements an installation procedure that is specific to a particular software packages. +Infamous examples include the easyblocks we have for ``GCC``, ``OpenFOAM``, ``TensorFlow``, ``WRF``, ... + +The installation procedure performed by an easyblock can be controlled by defining +**easyconfig parameters** in an easyconfig file. + +A collection of (generic and software-specific) easyblocks is developed by the EasyBuild community +in the [``easybuild-easyblocks`` repository on GitHub](https://github.com/easybuilders/easybuild-easyblocks). + +--- + +## *Easyconfig parameters* + +An **easyconfig parameter** specifies a particular aspect of a software installation that should be performed by +EasyBuild. + +Some easyconfig parameters are **mandatory**. The following parameters *must* be defined in *every* easyconfig file: + +* ``name`` and ``version``, which specify the name and version of the software to install; +* ``homepage`` and ``description``, which provide key metadata for the software; +* ``toolchain``, which specifies the compiler toolchain to use to install the software (see + ``toolchains`` tab); + +Other easyconfig parameters are **optional**: they can be used to provide required information, +or to control specific aspects of the installation procedure performed by the easyblock. + +Some commonly used optional easyconfig parameters include: + +* ``easyblock``, which specifies which (generic) easyblock should be used for the installation; +* ``sources`` and ``source_urls``, which specify the list of source files and where to download them; +* ``dependencies`` and ``builddependencies``, which specify the list of (build) dependencies; +* ``configopts``, ``buildopts``, and ``installopts``, which specify options for the configuration/build/install commands, respectively; + +If no value is specified for an optional easyconfig parameter, the corresponding default value will be used. + +There are two groups of easyconfig parameters. *General* easyconfig parameters can be defined for any software +package, and (usually) control a specific aspect of the installation. *Custom* easyconfig parameters are +only supported by certain easyblocks, and only make sense for particular (types of) software. + +--- + +## *Easyconfig files* + +*Easyconfig files* (or *easyconfigs* for short), are **simple text files written in Python syntax +that specify what EasyBuild should install**. +Each easyconfig file defines the set of **easyconfig parameters** that collectively form a complete specification +for a particular software installation. + +The **filename** of an easyconfig file usually ends with the ``.eb`` extension. +In some contexts the filename is expected to be determined by the value of a handful of key +easyconfig parameters: ``name``, ``version``, ``toolchain`` and ``versionsuffix``. The general format for +the filename of an easyconfig file is: ``--.eb``, +where the toolchain part is omitted when a ``system`` toolchain is used, and the `` can be empty. + +The filename of easyconfig files is particularly relevant when EasyBuild is searching for easyconfig files to resolve dependencies, +since it does this purely based on filenames: interpreting the contents of every (potential) easyconfig file +it encounters would be too expensive. + +In the [``easybuild-easyconfigs`` repository on GitHub](https://github.com/easybuilders/easybuild-easyconfigs), +the EasyBuild community maintains a large (and growing) collection of easyconfig files, for a wide range of +(scientific) software. + +--- + +## *Easystack files* + +[**Easystack files**](https://docs.easybuild.io/en/latest/Easystack-files.html) +are a new concept in EasyBuild, providing a way to define a *software stack* +that should be installed by EasyBuild. + +They are written in [YAML syntax](https://yaml.org/), and include a list of *software specifications* +which correspond to a list of easyconfig files, with support for providing specific EasyBuild +configuration options for particular software packages, and including or excluding specific software +packages based on labels. + +The support for using easystack files is currently marked as *experimental*, +which means it is subject to change in future EasyBuild releases, and may be prone to errors. + +--- + +## *Extensions* + +*Extensions* is the collective term we use for **additional software packages that can be installed +on top of another software package**. Common examples are *Python packages*, *R libraries*, and *Perl modules*. + +As you can tell the common terminology here is a bit messy, so we came up with a unifying term... + +Extensions can be installed in different ways: + +* *stand-alone*, as a separate installation on top of one or more other installations; +* as a part of a *bundle* of extensions that collectively form a separate installation; +* or as an actual *extension* to a specific installation to yield a "batteries included" + type of installation (for examples by adding a bunch of Python packages from PyPI into + a Python installation); + +--- + +## *Dependencies* + +A *dependency* is a common term in the context of software. It refers to **a software +package that is either strictly required by other software, or that can be leveraged to +enhance other software** (for example to support specific features). + +There are three main types of dependencies for computer software: + +* a **build dependency** is only required when building/installing a software package; + once the software package is installed, it is no longer needed to *use* that software + (examples: `CMake`, `pkg-config`); +* a **run-time dependency** (often referred to simply as *dependency*) is a software package that is + required to *use* (or *run*) another software package (example: `Python`); +* a **link-time dependency** is somewhere in between a build and runtime dependency: + it is only needed when *linking* a software package; it can become either a build or runtime + dependency, depending on exactly how the software is installed (example: `OpenBLAS`); + +The distinction between link-time and run-time dependencies is mostly irrelevant for this tutorial, +but we will discriminate build-only dependencies. + +--- + +## *Toolchains* + +A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, +which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. + +We refer to the different parts of a toolchain as **toolchain components**. + +The *compiler component* typically consists of [C](https://en.wikipedia.org/wiki/C_(programming_language)), +[C++](https://en.wikipedia.org/wiki/C%2B%2B), and [Fortran](https://en.wikipedia.org/wiki/Fortran) +compilers in the context of HPC, but additional compilers (for example, +a [CUDA](https://developer.nvidia.com/cuda-zone) compiler for +[GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units) software) +can also be included. + +Additional toolchain components are usually special-purpose libraries: + +* an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/)); +* libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), + [LAPACK](http://performance.netlib.org/lapack/)); +* a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); + +A toolchain that includes all of these libraries is referred to as a **full toolchain**, while +a **subtoolchain** is a toolchain that is missing one or more of these libraries. +A **compiler-only toolchain** only consists of compilers (no additional libraries). + +### System toolchain + +The **`system` toolchain** is a special case which corresponds to using the compilers and libraries +*provided by the operating system*, rather than using toolchain components that were installed using EasyBuild. + +It used sparingly, mostly to install software where no actual compilation is done or +to build a set of toolchain compilers and its dependencies, since the versions of the system compilers +and libraries are beyond the control of EasyBuild, which could affect the reproducibility of the installation. + +### Common toolchains + +The `foss` and `intel` toolchains are also known as the *common toolchains*, +because they are widely adopted by the EasyBuild community. + +The `foss` toolchain consists of all open source components (hence the name: +"FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), +[ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). + +The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version +controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. + +Roughly every 6 months, a new version of these common toolchains is agreed upon +in the EasyBuild community, after extensive testing. + +More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). + +--- + +## *Modules* + +*Module* is a massively overloaded term in (scientific) software and IT in general +(kernel modules, Python modules, and so on). +In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. + +[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept +on HPC systems: it is a way to specify changes that should be made to one or more +[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a +[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file +is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or +[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, +and specifies which environment variables should be updated, and how (append, +prepend, (re)define, undefine, etc.) upon loading the environment module. +Unloading the environment module will restore the shell environment to its previous state. + +Environment module files are processed via a **modules tool**, of which there +are several conceptually similar yet slightly different implementations. +The Tcl-based [Environment Modules](https://sourceforge.net/projects/modules/) implementation, and +[Lmod](https://lmod.readthedocs.io), a more recent Lua-based implementation (which also supports module +files written in Tcl syntax), are the most commonly used ones. + +**Environment module files are automatically generated for each software installation** by EasyBuild, +and *loading* a module results in changes being made to the environment of the current shell +session such that the corresponding software installation can be used. + +--- + +## *Bringing it all together* + +The EasyBuild **framework** leverages **easyblocks** to automatically build and install +(scientific) software, potentially including additional **extensions**, using a particular compiler **toolchain**, +as specified in **easyconfig files** which each define a set of **easyconfig parameters**. + +EasyBuild ensures that the specified **(build) dependencies** are in place, +and automatically generates a set of **(environment) modules** that facilitate access to the installed software. + +An **easystack file** can be used to specify a collection of software to install with EasyBuild. + +--- + +*[[next: Installation]](installation.md)* From 7ec3ad48e83ebc93ed41b51e28125ac21ad95ebe Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 10 Mar 2022 14:37:51 +0100 Subject: [PATCH 124/159] Additions to the Lmod section and some restructuring, and better information in the README for the repository. --- README.md | 8 +- .../2022-CSC_and_LO/1_01_what_is_easybuild.md | 2 +- docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md | 71 ------ docs/2022-CSC_and_LO/1_02_Lmod.md | 211 ++++++++++++++++++ docs/2022-CSC_and_LO/1_03_CPE.md | 30 +++ ..._03_terminology.md => 1_04_terminology.md} | 2 +- 6 files changed, 249 insertions(+), 75 deletions(-) delete mode 100644 docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md create mode 100644 docs/2022-CSC_and_LO/1_02_Lmod.md create mode 100644 docs/2022-CSC_and_LO/1_03_CPE.md rename docs/2022-CSC_and_LO/{1_03_terminology.md => 1_04_terminology.md} (99%) diff --git a/README.md b/README.md index 3d6d3571..15a32928 100644 --- a/README.md +++ b/README.md @@ -20,11 +20,15 @@ which makes it very easy to preview the result of the changes you make locally. * Start the MkDocs built-in dev-server to preview the tutorial as you work on it: - make preview + ```bash + make preview + ``` or - mkdocs serve + ```bash + mkdocs serve + ``` Visit http://127.0.0.1:8000 to see the local live preview of the changes you make. diff --git a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md index 5ccdb1db..979c6e90 100644 --- a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md @@ -318,4 +318,4 @@ More information, including the dates and times of the next calls, is available --- -*[[next: The HPE Cray Programming Environment and Lmod]](1_02_CPE_and_Lmod.md)* +*[[next: The Lmod module system]](1_02_Lmod.md)* diff --git a/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md b/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md deleted file mode 100644 index 01812057..00000000 --- a/docs/2022-CSC_and_LO/1_02_CPE_and_Lmod.md +++ /dev/null @@ -1,71 +0,0 @@ -# The HPE Cray Programming Environment and Lmod - -*[[back: What is EasyBuild?]](1_01_what_is_easybuild.md)* - ---- - -On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated -as Cray PE). The environment provides several tools, including compilers, communication libraries, -optmized math libraries and various other libraries, analyzers and debuggers. - -The Cray PE is made available through *environment modules* tha allow to select particular versions of -tools and to configure the environment in a flexible way. - ---- - -## Modules - -*Module* is a massively overloaded term in (scientific) software and IT in general -(kernel modules, Python modules, and so on). -In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. - -[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept -on HPC systems: it is a way to specify changes that should be made to one or more -[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a -[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file -is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or -[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, -and specifies which environment variables should be updated, and how (append, -prepend, (re)define, undefine, etc.) upon loading the environment module. -Unloading the environment module will restore the shell environment to its previous state. - -Environment module files are processed via a **modules tool**, of which there -are several conceptually similar yet slightly different implementations. -The oldest module tool still in use today is Environment Modules 3.2, implemented in C and -supporting module files written in Tcl. After a gap in developement, Xavier Delaruelle of CEA -developed [Environment Modules 4 and 5](https://sourceforge.net/projects/modules/) which is -fully implemented on Tcl. An alternative module tool is [Lmod](https://lmod.readthedocs.io), -developed by Robert McLay at TACC and implemented in LUA. This tool supports natively LUA -module files but also offers a high degree of compatibility with Tcl-based module files -developed for Environment Modules fia a translation layer and some API translation. - -The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no -packages or official support for Environment Modules 4 or 5. On LUMI, Lmod was selected as -the module tool. At the user level, Lmod and the various versions of Emvironment Modules -have very simmilar commands for managing the environment, but with different options. -The commands for searching for modules are very different though so if you are not familiar -with Lmod and its commands for users, it is worthwile to read the -[LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). - ---- - -## Lmod features - - ---- - -## Cray PE components - - ---- - -## Configuring the Cray PE through modules - ---- - -## Further reading - - ---- - -*[[next: Terminology]](1_03_terminology.md)* diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md new file mode 100644 index 00000000..91cb7571 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -0,0 +1,211 @@ +# The Lmod module system + +*[[back: What is EasyBuild?]](1_01_what_is_easybuild.md)* + +--- + +On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated +as Cray PE). The environment provides several tools, including compilers, communication libraries, +optmized math libraries and various other libraries, analyzers and debuggers. + +The Cray PE is made available through *environment modules* tha allow to select particular versions of +tools and to configure the environment in a flexible way. + +--- + +## Modules + +*Module* is a massively overloaded term in (scientific) software and IT in general +(kernel modules, Python modules, and so on). +In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. + +[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept +on HPC systems: it is a way to specify changes that should be made to one or more +[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a +[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file +is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or +[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, +and specifies which environment variables should be updated, and how (append, +prepend, (re)define, undefine, etc.) upon loading the environment module. +Unloading the environment module will restore the shell environment to its previous state. + +Environment module files are processed via a **modules tool**, of which there +are several conceptually similar yet slightly different implementations. +The oldest module tool still in use today is Environment Modules 3.2, implemented in C and +supporting module files written in Tcl. After a gap in developement, Xavier Delaruelle of CEA +developed [Environment Modules 4 and 5](https://sourceforge.net/projects/modules/) which is +fully implemented on Tcl. An alternative module tool is [Lmod](https://lmod.readthedocs.io), +developed by Robert McLay at TACC and implemented in LUA. This tool supports natively LUA +module files but also offers a high degree of compatibility with Tcl-based module files +developed for Environment Modules fia a translation layer and some API translation. + +The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no +packages or official support for Environment Modules 4 or 5. On LUMI, Lmod was selected as +the module tool. At the user level, Lmod and the various versions of Emvironment Modules +have very simmilar commands for managing the environment, but with different options. +The commands for searching for modules are very different though so if you are not familiar +with Lmod and its commands for users, it is worthwile to read the +[LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). + +--- + +## Lmod hierarchy + +### User view + +Lmod supports a module hierarchy. In a hierarchy, there is a distinction between the *installed +modules* and the *available modules*. Available modules are those that can be loaded directly +without first loading any other module, while the installed modules is the complete set of +modules that one could load one way or another. A typical use case +is a hierarchy to deal with different compilers on a system and different MPI implementations. +After all, it is a common practice to only link libraries and application code compiled with the +same compiler to avoid compatibility problems between compilers (and to be able to use advanced +features such as link time optimization). This is even more important for MPI, as Open MPI and +MPCIH-derived MPI implementations have incompatible Application Binary Interfaces. This would lead +to a hierarchy with 3 levels: + +1. The ``Core`` level containing the modules for the compilers themselves, e.g., one or more versions + of the GNU compiler suite and one or more versions of LLVM-based compilers. + + Loading a compiler module would then make the next level available: + +2. The ``Compiler`` level, containing modules for libraries and packages that only rely on the compilers + but do not use MPI, as well as the MPI modules, e.g., a version of Open MPI and a version of MPICH. + + Loading one of the MPI modules would then make the next level available: + +3. The ``MPI`` level, containing libraries and applications that depend on the compiler used and the MPI + implementation. + +Now assume that we have two compilers in the hierarchy, Compiler_A and Compiler_B. Their modules would reside +at the ``Core`` level. Both copilers provide the same MPI implementation, MPI_C. So there would be two modules +for ``MPI_C`` in two different subdirectories at the ``Compiler`` level. And further assume that we have an +application, Appl_E, compiled with both Compiler_A and Compiler_B and using MPI_C. For that application there would +also be two module files at the ``MPI`` level, one in a subdirectory corresponding ao Compiler_A and MPI_C and one +in a subdirectory corresponding to Compiler_B and MPI_C. To be able to load the module for Appl_E, a user should +first load Compiler_A, then load MPI_C and only then is it possible to load the module for Appl_E: + +```bash +module load Compiler_A MPI_C Appl_E +``` + +What is interesting is what happens if the user now loads Compiler_B: + +```bash +module load Compiler_B +``` + +In a properly designed and implemented hierarchy, Lmod will unload Compiler_A which will also trigger the unloading/deactivation +of MPI_C and Appl_E. It will then load the module for Compiler_B and proceed with looking if it can find another module for +MPI_C. That will then be loaded which now makes a different module for Appl_E available, which Lmod will proceed to load. If it +cannot find an exact match for the version, Lmod will even try to locate a different version. Hence the situation after loading +Compiler_B is that now modules are loaded for Compiler_B, MPI_C for Compiler_B and Appl_E for Compiler_A with MPI_C. +All this requires very little effort from the module file programmer and very little logic in the module files. E.g., rather +then implementing a single module file for Appl_E that would require logic to see which compiler and MPI implementation is loaded +and depending on those adapt the path to the binaries, several very simple modules need to be written with very little +logic, and one could add an Appl_E module for a different compiler or MPI implementation without touching any of the already +existing module files for that application. + + +### Building blocks + +Some mechanisms in Lmod make implementing a hierarchy fairly easy (though there are a lot of hidden pitfalls) + +- The *MODULEPATH* environment variable determines which modules are available. MODULEPATH is different from any other + path-style variable in Lmod in that any change will immediately trigger a re-evaluation of which modules are available + and trigger deactivating modules that are no longer available when a directory is removed from the MODULEPATH or + looking for alternatives for deactivated modules when a directory is added to the MODULEPATH. + +- The *"one name rule"*: Lmod cannot have two modules loaded with the same name (but a different version). By default, when loading + a module with the name of an already loaded module, Lmod will automatically swap the old one with the new one, i.e., unload the + already loaded module and load the new one. + +- The *family* concept: It is possible to declare a module to be part of a family using a command in the module file. No two modules + of the same family can be loaded at the same time, and Lmod will again by default auto-swap the already loaded one with the one + being loaded. The procedure is different though as Lmod now first has to read the new module file to discover the family, and this + may lead to more side effects. But that discussion is outside the scope of this tutorial. + + The family concept was for a long time a unique feature of Lmod, but it has been added now also to Environment Modules version 5.1. + + +### Implementation details + +The above example could be implemented using 6 module files: One for each compiler, two for the MPI module and +two for the application module. + +``` +moduleroot +├── Core +│ ├── Compiler_A +│ │ └── version_A.lua +│ └── Compiler_B +│ └── version_B.lua +├── Compiler +│ ├── Compiler_A +│ │ └── version_A +│ │ └── MPI_C +│ │ └── version_C.lua +│ └── Compiler_B +│ └── version_B +│ └── MPI_C +│ └── version_C.lua +└── MPI + ├── Compiler_A + │ └── version_A + │ └── MPI_C + │ └── version_C + │ └── Appl_E + │ └── version_E.lua + └── Compiler_B + └── version_B + └── MPI_C + └── version_C + └── Appl_E + └── version_E.lua +``` + +Besides the module functions needed to create the environment needed to run the compiler, the module file for +Compiler_A would need only two lines to implement the hierarchy: + +```Lua +family('Compiler') +prepend_path('MODULEPATH', 'moduleroot/Compiler/Compiler_A/version_A') +``` + +There are now two different ``version_C.lua`` files. One contains the necessary calls to module functions to +initialise the environment to use the version compiled with Compiler_A/version_A while the other contains the +necessary functions to do that for Compiler_B/version_B. Again, two more lines are needed to implement the hierarchy. +E.g., for ``moduleroot/Compiler/Compiler_A/version_A/MPI_C/version_C.lua``: + +```Lua +family('MPI') +prepend_path('MODULEPATH', 'moduleroot/MPI/Compiler_A/version_A/MPI_C/version_C') +``` + +Finally two vesions of the ``version_E.lua`` file are needed, one to prepare the environment for using the +package with Compiler_A anmd MPI_C and one for using the package with Compiler_B and MPI_C. However, these +are just regular modules and no additions are needed to work for the hierarchy. + +Both EasyBuild and Spack support Lmod hierarchies and with these tools it is also fairly automatic to create +different versions of the module files for each compiler and MPI library used to build the application. When +hand-writing modules it may be more interesting to have a generic module which would work for all those cases +and that is also possible with Lmod. Lmod does have a range of *introspection functions* that a module can use +to figure out its name, version and place in the module tree. All that would be needed is that the various +instances of the module file are at the correct location in the module tree and link to the generic file which +can be outside the module tree. In fact, this feature is used on LUMI to implement the modules that load a +particular version of the hardware for a particular section of LUMI. + +--- + +## Finding modules + +TODO: module spider, module help, module whatis, module keyword and where do they get their information. + +--- + +## Further reading + + +--- + +*[[next: The Cray Programming Environment]](1_03_CPE.md)* diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md new file mode 100644 index 00000000..ff054066 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -0,0 +1,30 @@ +# The HPE Cray Programming Environment + +*[[back: The Lmod module system]](1_02_Lmod.md)* + +--- + +On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated +as Cray PE). The environment provides several tools, including compilers, communication libraries, +optmized math libraries and various other libraries, analyzers and debuggers. + +The Cray PE is made available through *environment modules* tha allow to select particular versions of +tools and to configure the environment in a flexible way. + +--- + +## Cray PE components + + +--- + +## Configuring the Cray PE through modules + +--- + +## Further reading + + +--- + +*[[next: Terminology]](1_04_terminology.md)* diff --git a/docs/2022-CSC_and_LO/1_03_terminology.md b/docs/2022-CSC_and_LO/1_04_terminology.md similarity index 99% rename from docs/2022-CSC_and_LO/1_03_terminology.md rename to docs/2022-CSC_and_LO/1_04_terminology.md index ec085b0c..3450c8f0 100644 --- a/docs/2022-CSC_and_LO/1_03_terminology.md +++ b/docs/2022-CSC_and_LO/1_04_terminology.md @@ -1,6 +1,6 @@ # Terminology -*[[back: The HPE Cray Programming Environment and Lmod]](1_02_CPE_and_Lmod.md)* +*[[back: The HPE Cray Programming Environment]](1_03_CPE.md)* --- From feae23bbf068266f8d8ef254ca9b6a534ce12849 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 10 Mar 2022 14:42:32 +0100 Subject: [PATCH 125/159] Corrections to the overview for part I. --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md index e381f45d..bdfddfc5 100644 --- a/docs/2022-CSC_and_LO/1_00_part1_intro.md +++ b/docs/2022-CSC_and_LO/1_00_part1_intro.md @@ -2,10 +2,10 @@ *[[back to start page]](index.md)* -*(Tue March 4th 2021, 9am-12 CET)* - - [What is EasyBuild?](1_01_what_is_easybuild.md) -- [Terminology](1_02_terminology.md) +- [The Lmod module system](1_02_Lmod.md) +- [The HPE Cray Programming Environment and Lmod](1_03_CPE.md) +- [Terminology](1_04_terminology.md) - [Installation](installation.md) *(hands-on)* - [Configuration](configuration.md) *(hands-on)* From 6513998921471af43ce4ffda3973f762a3c3c36a Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 11 Mar 2022 14:16:34 +0100 Subject: [PATCH 126/159] Further work on the Lmod section. --- docs/2022-CSC_and_LO/1_02_Lmod.md | 314 ++++++++++++++++++++++++++++-- 1 file changed, 299 insertions(+), 15 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md index 91cb7571..04ccff8e 100644 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -4,15 +4,6 @@ --- -On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated -as Cray PE). The environment provides several tools, including compilers, communication libraries, -optmized math libraries and various other libraries, analyzers and debuggers. - -The Cray PE is made available through *environment modules* tha allow to select particular versions of -tools and to configure the environment in a flexible way. - ---- - ## Modules *Module* is a massively overloaded term in (scientific) software and IT in general @@ -40,12 +31,16 @@ module files but also offers a high degree of compatibility with Tcl-based modul developed for Environment Modules fia a translation layer and some API translation. The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no -packages or official support for Environment Modules 4 or 5. On LUMI, Lmod was selected as -the module tool. At the user level, Lmod and the various versions of Emvironment Modules -have very simmilar commands for managing the environment, but with different options. -The commands for searching for modules are very different though so if you are not familiar -with Lmod and its commands for users, it is worthwile to read the -[LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). +packages or official support for Environment Modules 4 or 5. At the user level, +Environment Modules 3.2 and Lmod have many commands in common, but with different options. +Lmod also has some powerful features that are lacking in Environment Modules 3.2. + +!!! Note "The Cray PE on LUMI" + On LUMI, Lmod was selected as the module tool. One area where there are significant + differences between Environment Modules 3.2 (and also the newer versions) and Lmod is + in the commands for discovering modules on the system. If you are not familiar with Lmod + and its commands for users, it is worthwile to read the + [LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). --- @@ -199,8 +194,297 @@ particular version of the hardware for a particular section of LUMI. ## Finding modules +In a hierarchical setup, not all modules are available at login. This implies that a user cannot use +``module avail`` to discover which software is available on the system. To this end Lmod has powerful +search commands. It is important to understand how these commands work to ensure that the proper information +is included in the module files to improve discoverability of software. + +!!! Note "Documentation in the LUMI documentation" + Extensive information on search commands with examples of how to use them on LUMI can be found + in the [LUMI documentation]([https://)](https://docs.lumi-supercomputer.eu/), in + [the computing section, "Module environment page", "Finding modules" section](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/#finding-modules). + TODO: module spider, module help, module whatis, module keyword and where do they get their information. +### module spider command + +The available modules at any point in time are often only a subset of all installed modules on a +system. However, Lmod provides the ``module spider`` command to search for a module with a given name +among all installed modules and to tell you how this module can be loaded (i.e., which other modules +need to be loaded to make the module available). + +The ``module spider`` command has three levels, producing different outputs: + + 1. ``module spider`` without further arguments will produce a list of all + installed software and show some basic information about those packages. + Some packages may have an ``(E)`` behind their name and will appear in blue + (in the default colour scheme) which means that they are part of a different + package. These are called *extensions* of packages or modules. + This is explained a little further in this page. + + Note that ``module spider`` will also search in packages that are hidden from + being displayed. These packages can be loaded and used. However administrators + may have decided to hide them + either because they are not useful to regular users or because they think that + they will rarely or never be directly loaded by a user and want to avoid + overloading the module display. + + 2. ``module spider `` will search for the specific package. This + can be the name of a module, but it will also search some other information + that can be included in the modules. The search is also case-insensitive. + E.g., on LUMI + ```bash + module spider GNUplot + ``` + will show something along the lines of + ``` + ------------------------------------------------------------------ + gnuplot: + ------------------------------------------------------------------ + Description: + Gnuplot is a portable command-line driven graphing utility + + Versions: + gnuplot/5.4.2-cpeCray-21.08 + gnuplot/5.4.2-cpeGNU-21.08 + ``` + so even though the capitalisation of the name was wrong, it can tell us that + there are two versions of gnuplot. The ``cpeGNU-21.08`` and ``cpeCray-21.08`` + tell that the difference is the compiler that was used to install gnuplot, + being the GNU compiler (PrgEnv-gnu) and the Cray compiler (PrgEnv-cray) + respectively. + + In some cases, if there is no ambiguity, `module spider` will actually + already produce help about the package, which is the next level. + + 3. `module spider /` will show more help information + about the package, including information on which other modules need to be + loaded to be able to load the package. E.g., + ```bash + module spider git/2.35.1 + ``` + will return something along the lines of + ``` + ------------------------------------------------------------------- + git: git/2.35.1 + ------------------------------------------------------------------- + Description: + Git is a free and open source distributed version control + system + + You will need to load all module(s) on any one of the lines below + before the "git/2.35.1" module is available to load. + + CrayEnv + LUMI/21.12 partition/C + LUMI/21.12 partition/D + LUMI/21.12 partition/G + LUMI/21.12 partition/L + + Help: + ``` + (abbreviated output). Note that it also tells you which other modules need + to be loaded. You need to choose the line which is appropriate for you and + load all modules on that line, not the whole list of in this case 9 + modules. + +!!! failure "Known issue" + The Cray PE uses Lmod in an unconventional manner with the hierarchy not + build fully in the way Lmod expects. As a consequence Lmod is not always + able to generate the correct list of modules that need to be loaded to make + a package available, and the list of ways to make a module available may + also be incomplete. + + The problem is somewhat aggrevated on LUMI because the Cray PE hierarchy sits + next to the hierarchy of the software stack as the Cray PE is installed + separately and hence cannot be integrated in the way the Lmod developer had + in mind. + + +#### Module extensions + +Certain packages, e.g., Python, Perl or R, get a lot of their functionality through +other packages that are installed together with them and extend the functionity, +e.g., NumPy and SciPy for Python. Installing all those packages as separate modules +to make it easy to see if they are installed or not on a system would lead to an +overload of modules on the system. + +Similary, admins of a software stack may chose to bundle several libraries or tools +that are often used together in a single module (and single installation directory), +e.g., to reduce module clutter but also to reduce the length of the search paths for +binaries, libraries or manual pages to speed up loading of applications. + +Lmod offers a way to make those individual packages installed in a module discoverable +by declaring them as *extensions* of the module. The ``module spider`` command will +search for those too. + + 1. ``module spider`` without further arguments: The output may contain lines similar + to + ``` + ----------------------------------------------------------------------- + The following is a list of the modules and extensions currently available: + ----------------------------------------------------------------------- + Autoconf: Autoconf/2.71 (E) + + CMake: CMake/3.21.2 (E), CMake/3.22.2 (E) + ``` + which tells that ``Autoconf`` and ``CMake`` are not available as modules themselves + but as extensions of another module, and it also tells the versions that are available, + though that list may not be complete (and is not always complete for modules either + as it is limited to one line of output). + +2. ``module spider `` will search for extensions also. E.g., + ``` + module spider CMake + ``` + on LUMI will return something along the lines of + ``` + ----------------------------------------------------------------------- + CMake: + ----------------------------------------------------------------------- + Versions: + CMake/3.21.2 (E) + CMake/3.22.2 (E) + ``` + (output abbreviated). + This tells that there is no ``CMake`` module on the system but that two versions + of ``CMake`` are provided in another module. + + 3. `module spider /` will show more information on the + extension, including which module provides the extension and which other modules + have to be loaded to make that module available. E.g., on LUMI, + ``` + module spider CMake/3.22.2 + ``` + will output something along the lines of + ``` + ----------------------------------------------------------------------- + CMake: CMake/3.22.2 (E) + ----------------------------------------------------------------------- + This extension is provided by the following modules. To access the + extension you must load one of the following modules. Note that any + module names in parentheses show the module location in the software + hierarchy. + + buildtools/21.12 (LUMI/21.12 partition/L) + buildtools/21.12 (LUMI/21.12 partition/G) + buildtools/21.12 (LUMI/21.12 partition/D) + buildtools/21.12 (LUMI/21.12 partition/C) + buildtools/21.12 (CrayEnv) + ``` + (output abbreviated and slightly reformatted for readability). This tells that + ``CMake/3.22.2`` is provided by the ``bvuildtools/21.12`` module and that there + are 5 different ways to make that package available. + +??? Bug "Restrictions with older Lmod versions" + At the time of development of this tutorial, Cray is still using the pretty old + 8.3.1 version of Lmod. Even though extensions were supported since Lmod version 8.2.5, + Lmod 8.3.1 has several problems: + + - It is not possible to hide extensions in the output of ``module avail``, a feature + that only became available in version 8.5. This may be annoying to many users as + the extension list of packages such as Python, R and Perl can be very long (the + default EasyBuild installation of R contains on the order of 600 packages). + + For that reason on LUMI extensions are only used for some modules. + + - ``module avail`` also shows extensions for modules that are not available which + makes no sense. This bug was only corrected in Lmod 8.6.13 and 8.6.14. + + +### module keyword + +Another search command that is sometimes useful is `module keyword`. It really +just searches for the given word in the short descriptions that are included in +most module files and in the name of the module. The output is not always +complete since not all modules may have a complete enough short description. + +Consider we are looking for a library or package that supports MP3 audio +encoding. +```bash +module keyword mp3 +``` +will return something along the lines of +``` +---------------------------------------------------------------- + +The following modules match your search criteria: "mp3" +---------------------------------------------------------------- + + LAME: LAME/3.100-cpeCray-21.08, LAME/3.100-cpeGNU-21.08 + LAME is a high quality MPEG Audio Layer III (mp3) encoder +``` +though the output will depend on the version of Lmod. This may not be the most +useful example on a supercomputer, but the library is in fact needed to be able +to install some other packages even though the sound function is not immediately +useful. + +??? bug "Know issue: Irrelevant output" + At the moment of the development of this tutorial, this command actually + returns a lot more output, referring to completely irrelevant extensions. + This is a bug in the HPE-Cray-provided version of Lmod (8.3.1 at the time + of development of this tutorial) that was only solved in more recent versions. + + +### module avail + +The `module avail` command is used to show only available modules, i.e., modules +that can be loaded directly without first loading other modules. It can be used +in two ways: + + 1. Without a further argument it will show an often lengthy list of all + available modules. Some modules will be marked with `(D)` which means that + they are the default module that would be loaded should you load the module + using only its name. + + 2. With the name of a module (or a part of the name) it will show all modules + that match that (part of) a name. E.g., + ```bash + module avail gnuplot + ``` + will show something along the lines of + ``` + ------ EasyBuild managed software for software stack LUMI/21.08 on LUMI-L ------ + gnuplot/5.4.2-cpeCray-21.08 gnuplot/5.4.2-cpeGNU-21.08 (D) + + Where: + D: Default Module + (output abbreviated). + ``` + but + ```bash + module avail gnu + ``` + will show you an often lengthy list that contains all packages with gnu + (case insensitive) in their name or version. + + +### Getting help + +One way to get help on a particular module has already been discussed on this +page: ``module spider /`` will produce help about the package as +soon as it can unambiguously determine the package. It is the only command that +can produce help for all installed packages. The next two commands can only +produce help about available packages. + +A second command is ``module whatis`` with the name or name and version of a +module. It will show the brief description of the module that is included in +most modules on the system. If the full version of the module is not given, it +will display the information for the default version of that module. + +The third command is ``module help``. Without any further argument it will display +some brief help about the module command. However, when used as +``module help `` or ``module help /`` it will produce help for either the +default version of the package (if the version is not specified) or the +indicated version. + + +### Implementation details + +**TODO** Tell about the extensions, whatis and help functions in module file and maybe a +little bit about how module spider works. + + --- ## Further reading From f41da0037c69da97c4fee01cef75f5adc782fd44 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 14 Mar 2022 14:38:34 +0100 Subject: [PATCH 127/159] Finished the Lmod section. --- docs/2022-CSC_and_LO/1_02_Lmod.md | 74 +++++++++++++++++++++++++++++-- 1 file changed, 71 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md index 04ccff8e..2d1d5121 100644 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -41,6 +41,7 @@ Lmod also has some powerful features that are lacking in Environment Modules 3.2 in the commands for discovering modules on the system. If you are not familiar with Lmod and its commands for users, it is worthwile to read the [LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). + Some of those commands are also discussed on this page. --- @@ -204,7 +205,6 @@ is included in the module files to improve discoverability of software. in the [LUMI documentation]([https://)](https://docs.lumi-supercomputer.eu/), in [the computing section, "Module environment page", "Finding modules" section](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/#finding-modules). -TODO: module spider, module help, module whatis, module keyword and where do they get their information. ### module spider command @@ -481,14 +481,82 @@ indicated version. ### Implementation details -**TODO** Tell about the extensions, whatis and help functions in module file and maybe a -little bit about how module spider works. +Lmod works by executing the module file. However, the actions of all Lmod-defined +functions will depend upon the mode in which Lmod is executing the module function, +and the module file can also detect in which mode it is executing. +Modes include "load", "unload"but also "spider". E.g., when te mode is "load", the +``setenv`` function will set an environment variable to the indicated value while in +"unload" mode that environment variable will be unset, and in "spider" mode the +environment variable is left untouched. The working of ``prepend_path``, a function +that modifies PATH-style variables, depends a bit on how Lmod is configured (as it is +possible to work with reference counts), but in its most basic mode, ``prepend_path`` +will add a given directory to a given PATH-style environment variable (or move it to +the front of the PATH-style variable if the directory is already in there), while in +"unload" mode that specific directory will be removed from the PATH (but no error will +be generated should the directory that is used as the argument not be part of the path +in that PATH-style variable). When the mode is "spider", the function has special behaviour +if it is used to change the ``MODULEPATH``. It will then note the change and add that +directory to the list of directories that has to be searched for module files. +This makes ``module spider`` a very expensive command as it may have to traverse a lot +of directories and has to execute all module files in there. Therefor Lmod will build +a so-called spider cache which can be pre-built in the system for certain directories +and otherwise will be build in the user's home directory (in the ``.lmod.d/.cache`` +subdirectory). Our experience is that this cache tends to be rather fragile, +in particular on Cray systems (and that has been confirmed in discussions with +people with access to some other Cray systems) so from time to time Lmod fails to +note changes to the modules, at least when using commands such as ``module spider``. +The actual loading and unloading of the module is not based on cached information. + +Lmod has several functions that can be used in module files to provide the information +that Lmod needs for the search-related and help commands. + +The ``help`` function defines the long help text used by ``module help`` and by +``module spider`` as soon as there is no ambiguity anymore about which module is +being searched for. + +The ``whatis`` function is used to provide short information about a module. That +information is then used by ``module whatis`` and ``module keyword`` , but also +for brief information shown by ``module spider`` when multiple modules or versions +of modules are found by the command. A module file can contain multiple ``whatis`` +commands and the Lmod manuel suggests to use those lines as a kind of database +record. See, e.g., +[the Lmod manual page with module file examples](https://lmod.readthedocs.io/en/latest/100_modulefile_examples.html?highlight=whatis). +One such example is +```Lua +whatis("Name: valgrind") +whatis("Version: 3.7.0") +whatis("Category: tools") +whatis("URL: http://www.valgrind.org") +whatis("Description: memory usage tester") +``` +It is not all that important to include all those lines in a module file, but some of +those lines get a special treatment from Lmod. The line starting with ``Description`` +is used by ``module spider`` to provide some brief information about the module if it +is not totally resolved. This comes with a limitation though: It is not show for each +version of the module, so ideally all "GROMACS" modules should contain the same +description line and use other lines to provide further information about what +distinguished a particular version. +Likewise the ``Category:`` line is used by the ``spider_decoration`` hook that can be +used to add decoration to the spider level 1 output. +All in all the ``whatis`` function if often overlooked in Lmod-based module functionx +but it is a very useful function to include in the proper way in module files. + +A third function that provides information to the search commands is ``extensions``. +It can be used to list up the extensions supported by the module. The argument list +may seem strange as it takes only a single argument, a string of comma-separated ``extension/version`` +elements, but that is because the number of arguments to a function is limited in +Lua and that limit can actually be met easily by modules for Python, Perl or R packages. --- ## Further reading +- [Lmod documentation](https://lmod.readthedocs.io/en/latest/index.html) +- [Lmod on LUMI in the LUMI documentation](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/) +- [Documentation of Environment Modules 5](https://modules.readthedocs.io/en/latest/), + an alternative to Lmod (though not currently supported by HPE Cray) + --- From 5415f720bce76cf2afba36495d4b058aacd375be Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 14 Mar 2022 18:27:32 +0100 Subject: [PATCH 128/159] Tutorial page about the Cray PE, and correction of a typo. --- docs/2022-CSC_and_LO/1_03_CPE.md | 251 +++++++++++++++++++++++++++++++ 1 file changed, 251 insertions(+) diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md index ff054066..8de332a4 100644 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -15,16 +15,267 @@ tools and to configure the environment in a flexible way. ## Cray PE components +- **Cray Compiling Environment (CCE)**: + + CCE consists of Cray compilers performing code analysis during compilation to generate highly optimized code. + Supported languages include Fortran, C and C++, and UPC (Unified Parallel C). + The Cray C/C++ compiler is basically the Clang/LLVM compiler with a back-end configured by HPE Cray that also + adds some Cray-specific options while the Cray Fortran compiler uses an HPE-Cray specific front-end + with a back-end based on LLVM. The Fortran compiler supports most of Fortran 2018 and tends to be + considerably stricter than the GNU or Intel Fortran compilers. + + The CCE compilers will also support OpenMP offload to NVIDIA and AMD GPUs but that is still work in progress + at the time of the development of this tutorial. The Fortran compiler (but not the C/C++ compiler) also supports + OpenACC for offloading to GPU. + + The classic pre-LLVM Cray C/C++ compiler is not available on Cray EX systems. + +- **Cray Scientific and Mathematical Libraries (CSML)**: + + A set of high performance libraries providing portability for scientific applications, sparse and dense linear + algebra (BLAS, LAPACK, ScaLAPACK) and fast Fourier transforms (FFTW). + + There is a special (more limited) version of the library with GPU-optimized routines. + + CSML also includes the HDF5 and netCDF libraries, in versions with and without MPI. + +- **Cray Message Passing Toolkit**: + + A collection of software libraries used to perform data transfers between nodes running in parallel applications. + It includes the Message Passing Interface (MPI) and OpenSHMEM parallel programming models. + + The HPE Cray MPI implementation is based on the MPICH3 sources from ANL. However, some algorithms are tweaked + specifically for HPE Cray hardware and the Cray interconnects. + +- **Cray Environment Setup and Compiling Support**: + + Libraries supporting code compilation and setting up the development environment, including compiler drivers. + +- **Cray Performance and Analysis Tools**: + + Tools to analyze the performance and behavior of programs that are run on Cray systems, and a Performance API (PAPI). + +- **Cray Debugging Support Tools**: + + Debugging tools, including `gdb4hpc` and `Valgrind4hpc`. + +- **Third-party compilers**: + + The Cray PE also supports several third-party compilers. Not all compilers are supported on all systems: + + - The GNU C/C++ and Fortran compilers will be present on most systems with the Cray PE. + + - The AMD AOCC C/C++ and Fortran compilers are supported on systems with AMD CPUs. + + - The AMD ROCm compilers (sometimes known as AOMP) are supported on systems with AMD GPU technology. + + - The Intel C/C++ and Fortran compilers (also the oneAPI versions) are supported on systems with Intel + processors. HPE Cray does not include them in licenses for AMD CPUs. + + - The NVIDIA HPC Toolkit compilers are supported on systems with NVIDIA GPUs. + --- ## Configuring the Cray PE through modules +Compilers in the Cray Programming Environment are almost always used through the Cray compiler wrappers. In fact, +until recently, some compilers could not be used without the wrappers as they failed to even detect their own +include files and some components. The behaviour of the wrapper will depend on the other modules that are +loaded in the environment. E.g., there is no separate wrapper for MPI as the MPI header files and libraries are +automatically activated through the regular wrappers as soon as a Cray MPI module is loaded. + +Multiple releases of the HPE Cray PE can be installed simultaneously on the system and users can mix-and-match +components. However, it is also possible to load only components for a specific release of the PE. +Cray PE versions have version numbers of the form ``yy.mm``, e.g., ``22.02`` for the version released in +February of 2022. However, each of the components have their own version number and it is not easy to see +which version of a component came with which version(s) of the Cray PE. + +Below we only discuss those modules that are important when building software with EasyBuild. Debuggers, +profilers, etc., are not included in the list. + + +### The PrgEnv-* modules + +The usual way to initialise the Cray PE is by loading one of the ``PrgEnv-*`` modules. This module will then +load the appropriate compiler, compiler wrappers and other libraries. Some of the components that will be +loaded are configured through the ``/etc/cray-pe.d/cray-pe-configuration.sh`` file, so the list of modules +may depend on the actual system that you are using. + +The Cray PE supports the following ``PrgEnv-*`` modules. On LUMI, only the first three are currently available: + +| Module | Compiler module | What? | +|:---------------|-----------------|:--------------------------------------------| +| PrgEnv-cray | cce | The Cray Compiling Environment compilers | +| PrgEnv-gnu | gcc | The GNU compilers | +| PrgEnv-aocc | aocc | AMD compilers for CPU-only systems | +| PrgEnv-amd | rocm | AMD ROCm compilers for GPU systems | +| PrgEnv-intel | intel | The Intel compilers | +| PrgEnv-nvidia | nvidia | NVIDIA HPC toolkit compilers (formerly PGI) | + +All ``PrgEnv-*`` modules belong to the same module family. Hence Lmod will automatically unload any +already loaded ``PrgEnv-*`` module when you load a different one. + + +### Selecting the version through the cpe module + +Loading one of the ``cpe/yy.mm`` modules (e.g., ``cpe/22.02``) has the following effects: + +- It sets the default versions of each of the Cray PE modules to the version that comes with that + particular release of the Cray PE. E.g., + + ``` bash + module load cpe/22.02 + module load cce + ``` + + would load that version of the ``cce`` compiler that comes with the 22.02 release of the Cray PE. + +- It will reload all already loaded Cray PE modules and switch them over to the version corresponding + to that particular release of the Cray PE. + +???+ Bug "Limitations and bugs" + Due to the way Lmod works and implementation bugs in the ``cpe`` modules, loading the ``cpe`` module + does not always have the desired effect. + + - The Cray PE sets the default version of each module by adding a file to the list of files in the + ``LMOD_MODULERCFILE`` environment variable. This is because Lmod does not re-evaluate the visibility + of modules and the internal list of default version immediately when the value of ``LMOD_MODULERCFILE`` + is changed but only the next time the ``module`` command is executed. Hence + ``` bash + module load cpe/22.02 ; module load cce + ``` + and + ``` + module load cpe/22.02 cce + ``` + do not have the same effect. In the first version, the version of ``cce`` loaded is the version that + corresponds to the 22.02 release of the Cray PE. In the second case however the default version of the + ``cce`` module is determined by whatever list of default modules was used when calling the ``module`` + command so may or may not be the one of the 22.02 release. + + - Loading the ``cpe`` module after loading the other Cray PE modules also does not always have the + desired effect in many versions of the Cray PE. This is because of a bug in the ``cpe `` module that + reloads the modules in the wrong order which may trigger the reload of a module with whatever version + was the default when the ``module`` command was called rather than the version the the ``cpe`` module + intends to (re-)load. + + +### The compiler wrapper module craype + +The ``craype`` module loads the compiler wrappers. There is only one set of compiler wrappers for all compilers. +Which compiler will be called, which libraries will be included, but also processor and GPU target options will +be used, is all determined by other modules. Hence it is in principle possible to use a single Makefile for +a project and still reconfigure the build by loading certain modules. + + +### Target modules + +The targets for CPU and GPU optimization, the network library for MPI bt also some other compiler options, +can be set through target modules: + +- ``craype-x86-*`` (and similar options can be expected on ARM-based systems) set the target for CPU + optimisations. For LUMI, the ``crape-x86-rome``, ``craype-x86-milan`` and ``craype-x86-trento`` modules + are relevant. + + This can also be used to cross-compile to a different target architecture unless the compiler target + gets overwritten by a compiler flag added to the command line through the Makefile or other means, + something that unfortunately happens more and more often in faulty software installation procedures. + +- ``creype-accel-*`` sets the target for OpenMP offload (and likely other technologies in the future). E.g., + loading ``craype-accel-amd-gfx90a`` tells the compilers to target AMD MI200 family GPUs, while loading + ``craype-accel-host`` tells the compiler to use the CPU instead (according to the documentation, the latter + is for ``PrgEnv-cray`` only). + +- ``craype-network-*`` selects the communication library to be used by Cray MPICH. On Slingshot 11 EX systems, + only ``craype-network-ofi`` is supported, but Slingshot 10 EX systems also offer support for UCX through the + ``craype-network-ucx`` module. + +- The ``craype-hugepages*`` modules enable Cray Huge Pages support. To fully enable this support they have to + be used at link-time and at run-time. At link time, support is compiled into the binary wile at run-time they + are used to set the actual size of the huge pages. + + The ``craype-hugepages*`` modules are not supported by all compilers. E.g., the AOCC compiler does not support + huge pages at the moment, and loading the module at link time will cause an error message from the linker. + + +### The compiler modules + +The compiler modules have already been discussed with the ``PrgEnv-*`` modules above. The different regular +compiler modules also all belong to the same family so no two different compilers can be loaded simultaneously +and Lmod will automatically unload the other compiler when a new one is loaded. + + +### The MPI modules + +To load the Cray MPI libraries, both one of the ``craype-network-*`` modules and a compiler module has to be +loaded as the MPI libraries are both network- and compiler specific. + +For some unknown reason, the MPI module for the libfabric (``craype-network-ofi``) transport is called ``cray-mpich`` +while the library for the UCX transport (``craype-network-ucx``) is called ``craype-network-ucx``. As a result, +the MPI module fails to reload automatically when switching between both transports, but it does reload automatically +when switching compilers. + +Loading an MPI module will also automatically configure the regular compiler wrappers to compile with support for +that MPI module. However, the libfabric and UCX versions of the MPI library have compatible interfaces, so it is +always possible to swap between those versions at runtime. + + +### The Cray Scientific libraries + +The Cray Scientific Libraries are loaded through the ``cray-libsci`` module (or ``cray-libsci_acc`` for the GPU +versions). Loading this module makes the BLAS, LAPACK, and ScaLAPACK libraries available, and also the Cray IRT +(Iterative Refinement Toolkit), a Cray-specific library. It will also configure the compiler wrappers to link +with these libraries, so no additional include or link options are needed. + + +### The ``fftw`` module + +As FFTW is a third party library, it gets its own module (``cray-fftw``) and is not included in the ``cray-libsci`` module. + + +--- + +## Some unexpected behaviour of the Cray PE + +On Cray EX systems, dynamic linking is the preferred way of linking applications. Whith that comes some +unexpected behaviour of the Cray modules. EasyBuild users expect that at run time the versions of the +libraries that are used, are the ones from the modules that are loaded. This is not always the case +for the runtime libraries of the Cray PE. By default the Cray PE will use a default set of libraries +that is determined by the system default version of the Cray PE (which is set by the sysadmins, not +determined by the ``cpe`` module). This can only be avoided by either using\ +rpath-linking (which is also special in the Cray PE as the wrappers activate rpath linking if the +environment variable ``CRAY_ADD_RPATH`` is defined and set to ``yes``) or by manually adjusting +the library path after loading the modules: +``` bash +export LD_LIBRARY_PATH=${CRAY_LD_LIBRARY_PATH}:$LD_LIBRARY_PATH +``` +The latter cannot be easily automated in modulefiles. Any technique that can be used (without actually +reworking the Cray PE modules) has nasty side effects in some scenarios. + +The net result of this feature is that some reproducibility of the results is lost. Programs will +react differently if the system default version of the Cray PE is changed as that will change the +set of default libraries that will be used at runtime unless rpath-linking is used or users +redfine ``LD_LIBRARY_PATH``. + + --- ## Further reading +- [LUMI documentation: "Developing" section](https://docs.lumi-supercomputer.eu/development/) +- The Cray PE is mostly documented through man pages. There used to be some documentation on the + Cray web site also but the documentation system got reworked after the merger with HPE. + The documentation is now in the [HPE Support Centre](https://support.hpe.com/) where it + is very difficult to find the right version of the documents. +- The [PE-Cray](https://github.com/PE-Cray) GitHub project also provides some additional + documentation, including + - [some whitepapers](https://github.com/PE-Cray/whitepapers), + - [documentation for Cray OpenSMHEMX](https://github.com/PE-Cray/cray-openshmemx) and + - [information about Cray DSMML](https://github.com/PE-Cray/cray-dsmml) + though much of that information ia also not updated regularly anymore. + --- *[[next: Terminology]](1_04_terminology.md)* From 0f198437b25e16d3033ab5d768cd4602e39467cb Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 16 Mar 2022 15:24:08 +0100 Subject: [PATCH 129/159] Adding an example of implementing a hierarchy to the Lmod section of the tutorial. --- docs/2022-CSC_and_LO/1_02_Lmod.md | 191 ++++++++++++++++++++++++++++++ 1 file changed, 191 insertions(+) diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md index 2d1d5121..0caa8a47 100644 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -548,6 +548,197 @@ elements, but that is because the number of arguments to a function is limited i Lua and that limit can actually be met easily by modules for Python, Perl or R packages. +--- + +## Some warnings about writing modulefiles + +***This section is very technical and only useful if you want to manually implement +modules that depend on each other one way or another.*** + +Lmod cannot guarantee that the order of unloading the modules will be the inverse of +the order in which they were loaded. Moreover, unloading a module is not done by reverting +stored actions done when loading the module, but by executing the modulefile again +in a mode that reverts certain actions. This can lead to subtle problems when modulefiles +communicate with each other through environment variables or by detecting which other +modules are loaded. These problems are usually solved by using a proper hierarchy +and basing actions of modulefiles on their position in the hierarchy. + +One case where passing information between modules through environment variables will +go wrong is when that environment variable is subsequently used to compute a directory +name that should be added to a PATH-like variable. Assume we have two versions of +a ``MyPython`` module, e.g., ``MyPython/2.7.18`` and ``MyPython/3.6.10``. That module then +sets an environment variable ``PYTHON_API_VERSION`` to either ``2.7`` or ``3.6``. +Next we have a module ``MyPythonPackage`` that makes a number of Python packages available +for both Python modules. However, as some Python packages have to be installed separately +for each Python version, it does so by adding a directory to the environment variable +``PYTHON_PATH`` that contains the version which it gets by using the Lua function +``os.getenv`` to request the value of ``PYTHON_API_VERSION``. + +One problem becomes clear in the following scenario: +``` bash +module load MyPython/2.7.18 +module load MyPythonPackage/1.0 +module load MyPython/3.6.10 +``` +The ``module load MyPythonPackage`` will find the environment variable ``PYTHON_PACKAGE_API`` +with the value ``2.7`` as set by ``module load MyPython/2.7.18`` and hence add the directory +for the packages for version 2.7 to ``PYTHON_PATH``. The ``module load MyPython/3.6.10`` +command will trigger two operations because of the *"one name rule"*: First it will +automatically unload ``MyPython/2.7.18`` (which will unset ``PYTHON_API_VERSIUON``) and +next it will load ``MyPython/3.6.10`` which will set ``PYTHON_API_VERSION`` to ``3.6``. +However, ``MyPythonPackage`` is not reloaded so the ``PYTHON_PATH`` variable will now point +to the wrong directory. One would be tempted to think that the easy fix for the user would +be to reload ``MyPythonPackage/1.0``: +``` bash +module load MyPythonPackage/1.0 +``` +Because of the *"one name rule"* this will again trigger an unload followed by a load +of the module. The problem is in the unload. One would expect that first unloading +``MyPythonPackage`` would remove the 2.7 directory from the ``PYTHON_PATH`` but it +will not. Lmod does not remeber that last time it loaded ``MyPythonPackage`` it added +the 2.7 directory to ``PythonPath``. Instead it will execute the commands in the +modulefile and reverse certain commands. Since ``PYTHON_API_VERSION`` has now the value +``3.6``, it will try to remove the directory for version ``3.6`` which is not in the +``PYTHON_PATH``. The subsequent load will then add the 3.6 directory to ``PYTHON_PATH`` +so the environment variable now contains both directories. + +In this simple case, a ``module purge`` after the first two ``module load`` commands would +still work as Lmod is able to figure out the right order to unload modules, but in more +complicated examples this may also go wrong. However, a ``module purge`` command after +the load of ``MyPython/3.6.10`` would also fail to clean up the environment as it would +still fail to remove the 2.7 directory from ``PYTHONPATH``. + +??? Note "Running the example" + To test this example for yourself, create a directory and add that directory to + the ``MODULEPATH`` using ``module use``. In that directory, create the following + subdirectories and files: + 1. ``MyPython/2.7.18.lua`` with content: + ``` lua + LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) + setenv( 'PYTHON_API_VERSION', '2.7' ) + ``` + 2. ``MyPython/3.6.10.lua`` with content: + ``` lua + LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) + setenv( 'PYTHON_API_VERSION', '3.6' ) + ``` + 3. ``MyPythonPackage/1.0.lua`` with content: + ``` lua + LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) + LmodMessage( 'PYTHON_API_VERSION = ' .. ( os.getenv( 'PYTHON_API_VERSION' ) or '') ) + prepend_path( 'PYTHON_PATH', 'someroot/python' .. + ( os.getenv( 'PYTHON_API_VERSION' ) or 'TTT' ) .. '/packages' ) + LmodMessage( 'PYTHON_PATH = ' .. ( os.getenv( 'PYTHON_PATH' ) or '') ) + ``` +??? Note "Solution with a hierarchy" + The better way in Lmod to implement the above scenario would be in a module hierarchy. + + Just to show the power of Lmod introspection functions combined with a proper hierarchy + we present a solution using only one version of the code for ``MyPython`` and one version + of the code for ``MyPythonPackages``. + + It is best to start from a clean directory. In that directory, create: + + 1. The files ``level1/MyPython/2.7.18.lua`` and ``level1/MyPython/3.6.10.lua``, + both with the same contents: + ``` lua + LmodMessage( '\nIn ' .. myModuleFullName() .. ' in mode ' .. mode() ) + + local api_version = myModuleVersion():match( '(%d+%.%d+)%..*' ) + + -- Set the variable PYTHON_API_VERSION but not for internal use in the modules. + setenv( 'PYTHON_API_VERSION', api_version ) + + local module_root = myFileName():match( '(.*)/level1/' .. myModuleFullName() ) + prepend_path( 'MODULEPATH', pathJoin( module_root, 'level2/PythonAPI', api_version ) ) + LmodMessage( 'MODULEPATH is now\n ' .. + os.getenv( 'MODULEPATH' ):gsub( ':', '\n ' ) ) + ``` + + 2. The files ``level2/PythonAPI/2.7/MyPythonPackage/1.0.lua`` and + ``level2/PythonAPI/3.6/MyPythonPackage/1.0.lua``, both with the contents: + ``` lua + LmodMessage( '\nIn ' .. myFileName() .. ' in mode ' .. mode() ) + + local python_api_version = myFileName():match( '.*/level2/PythonAPI/([^/]+)/.*' ) + LmodMessage( 'Detected Python API version from hierarchy: ' .. python_api_version ) + LmodMessage( 'Detected Python API version from environment: ' .. + ( os.getenv( 'PYTHON_API_VERSION' ) or '' ) ) + + prepend_path( 'PYTHON_PATH', 'someroot/python' .. python_api_version .. '/packages' ) + + LmodMessage( 'PYTHON_PATH = ' .. (os.getenv( 'PYTHON_PATH' ) or '') ) + ``` + + Now add the ``level1`` subdirectory to ``MODULEPATH``, e.g., if you're in the directory + containing the ``level1`` and ``level2`` subdirectories: + ``` bash + module use $PWD/level1 + ``` + and then try the following commands: + ``` bash + module avail + module load MyPython/2.7.18 + module avail + module load MyPythonPackage/1.0 + module load MyPython/3.6.10 + ``` + and pay attention to the output. + + Initially ``module avail`` will show none of the ``MyPythonPackage`` modules. These are + installed modules but not available modules. ``module load MyPython/2.7.18`` will set the + environment variable ``PYTHON_API_VERSION`` to ``2.7`` and also add a directory to the front + of the ``MODULEPATH`` with the directory name ending on ``level2/PythonAPI/2.7``. Now + ``module avail`` will show the ``MyPythonPackage/1.0`` module. + + The ``MyPythonPackage`` shows two ways to get the version of the Python API to use for + determining the right directory to add to ``PYTHON_PATH``. The fragile way is to enquire + the value of the environment variable ``PYTHON_API_VERSION`` set by loading ``MyPython/2.7.18``. + The more robust way is to use the Lmod introspection function ``myFileName()`` which returns + the full path and file name of the module file that is executing, and extracting the version + from the path with a pattern matching function. In this particular situation both computed + values are the same so both would have worked to correctly add + ``somedir/python2.7/packages`` to the front of ``PYTHON_PATH``. + + The next command, ``module load MyPython/3.6.10`` triggers a chain of events. + + First, Lmod notices that there is already a module loaded with the same name, so it will + unload ``MyPython/2.7.18``. This will unset the environment variable ``PYTHON_API_VERSION`` + (the inverse operation of ``setenv``) and will remove the ``.../level2/PythonAPI/2.7`` + subdirectory from the ``MODULEPATH`` (the inverse action of ``prepend_path``). + + Now due to + the change of the ``MODULEPATH`` the ``MyPythonPackage/1.0`` module which was loaded from + ``.../level2/PythonAPI/2.7`` is no longer available so Lmod will continue with unloading + that module. The interesting bit now is that ``PYTHON_API_VERSION`` is unset. So had we + computed the name of the directory to add to ``PYTHON_PATH`` using the value of that + environment variable, the module would have failed to compute the correct directory name + to remove so ``prepend_path`` would have left the ``PYTHON_PATH`` environment variable + untouched. However, by computing that value from the directory of the modulefile, we get + the right value and can correctly remove ``somedir/python2.7/packages`` from ``PYTHON_PATH``. + Lmod will also remember that the module was only unloaded due to a change in the + ``MODULEPATH`` and not because a user explicitly unloaded the module. I.e., it considers + the module as deactivated but not as unloaded. + + Lmod proceeds with loading the ``MyPython/3.6.10`` module. This will now set + ``PTHON_API_VERSION`` to ``3.6`` and add a directory with name ending on + ``level2/PythonAPI/3.6`` to ``MODULEPATH``. + + Things are not done yet though. As the ``MODULEPATH`` has changed, Lmod looks at its list + of deactivated modules and notices that a different version of ``MyPythonPackage/1.0`` is + now available. Hence it will now automatically load that module from the + ``.../level2/PythonAPI/3.6`` subdirectory so that that module now correctly detects + that ``somedir/python3.6/package`` should be added to ``PYTHON_PATH``. + + Hence at the end of the cycle we have again a correctly configured environment with no + trace of the ``2.7`` version that was loaded initially and with no action required from + the user to ensure that ``MyPythonPackage`` is unloaded and reloaded to ensure the + correct configuration. + + This idea is used on LUMI to implement the various versions of the software stack with + for each software stack also optimised binaries for each of the node types. + + --- ## Further reading From 70f23f1e9d5ee5fbeb9e0df0375e212344534606 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 10:08:23 +0100 Subject: [PATCH 130/159] More information about hierarchy in the Cray PE, and definition of toolchain in Terminology. --- docs/2022-CSC_and_LO/1_03_CPE.md | 3 +++ docs/2022-CSC_and_LO/1_04_terminology.md | 21 ++++++++++++++++++--- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md index 8de332a4..42eb197e 100644 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -233,6 +233,9 @@ with these libraries, so no additional include or link options are needed. As FFTW is a third party library, it gets its own module (``cray-fftw``) and is not included in the ``cray-libsci`` module. +The ``cray-fftw`` module can only be loaded if one of the processor target modules (the ``craype-x86-*`` modules) +is loaded first. + --- diff --git a/docs/2022-CSC_and_LO/1_04_terminology.md b/docs/2022-CSC_and_LO/1_04_terminology.md index 3450c8f0..19f1e01f 100644 --- a/docs/2022-CSC_and_LO/1_04_terminology.md +++ b/docs/2022-CSC_and_LO/1_04_terminology.md @@ -28,22 +28,37 @@ can also be included. Additional toolchain components are usually special-purpose libraries: -- an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/)); +- an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/) + or vendor-specific MPI implementations like Intel MPI in a toolchain based on the Intel compilers); - libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), - [LAPACK](http://performance.netlib.org/lapack/)); + [LAPACK](http://performance.netlib.org/lapack/), [ScaLAPACK](http://www.netlib.org/scalapack/)); These libraries are included because they tend to provide a common API, so mixing them in a single executable would cause name conflicts when linking. - a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); +These libraries are included because they are some of the most popular libraries in scientific software, +but also because they provide an API that is available through different implementations. Mixing +implementations in a single build is a recipe for problems as the various implementations might +define the same symbols. + A toolchain that includes all of these libraries is referred to as a **full toolchain**, while a **subtoolchain** is a toolchain that is missing one or more of these libraries. A **compiler-only toolchain** only consists of compilers (no additional libraries). +The toolchain concept also maps nicely on the Programming Environment concept in the Cray +Programming Environment. Each ``PrgEnv-*`` module in fact provides a full toolchain in a +typical Cray PE installation, except for the FFTW library. Note that systems could configure +the contents of a ``PrgEnv-*`` module differently as the MPI library and scientific library are +optional. In the Cray PE, the MPI library is provided by the ``cray-mpic`` module, +the BLAS, LAPACK and ScaLAPACK libraries by the ``cray-libsci`` module (for CPU-only nodes) +and the FFTW library through the ``cray-fftw`` module. + Many EasBuild toolchains are organised in a hierarchy. Each toolchain can use components compiled -with itself or with one of the subtoolchains. +with itself or with one of the subtoolchains. The Cray toolchains however are not currently +organised in a hierarchy. ### System toolchain From ad10f45b17e2291baebc99e885ecc3892898ae52 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 10:33:37 +0100 Subject: [PATCH 131/159] Add a mermaid.js diagram with the toolchain hierarchy. --- docs/2022-CSC_and_LO/1_04_terminology.md | 42 +++++++++++++++++------- 1 file changed, 30 insertions(+), 12 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_04_terminology.md b/docs/2022-CSC_and_LO/1_04_terminology.md index 19f1e01f..ebb99963 100644 --- a/docs/2022-CSC_and_LO/1_04_terminology.md +++ b/docs/2022-CSC_and_LO/1_04_terminology.md @@ -12,7 +12,7 @@ It is important to be familiar with these terms, so we'll briefly cover them one --- -## *Toolchains* +## Toolchains A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. @@ -82,6 +82,8 @@ because they are widely adopted by the EasyBuild community. The `foss` toolchain consists of all open source components (hence the name: "FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), [ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). +In recent versions (since 2021a), [FlexiBLAS](https://www.mpi-magdeburg.mpg.de/projects/flexiblas) +is used as the BLAS library, with OpenBLAS and LAPACK as the backend. The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. @@ -89,11 +91,27 @@ controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.c Roughly every 6 months, a new version of these common toolchains is agreed upon in the EasyBuild community, after extensive testing. +The common toolchains are also organised in a toolchain hierarchy: + +```mermaid +graph TD; +A[SYSTEM] --> B[GCCcore]; +B --> C[GCC]; +B --> D[iccifort]; +C --> E[gompi: Adds OpenMPI]; +D --> F[iimpi: Adds Intel MPI]; +D --> G[imkl: Adds Intel MKL]; +E --> H[foss: Adds OpenBLAS, LAPACK, ScaLAPACK, FFTW]; +F --> I[intel]; +G --> I; +``` + More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). + --- -## *Modules* +## Modules *Module* is a massively overloaded term in (scientific) software and IT in general (kernel modules, Python modules, and so on). @@ -121,7 +139,7 @@ session such that the corresponding software installation can be used. --- -## *EasyBuild framework* +## EasyBuild framework The EasyBuild *framework* consists of a set of Python modules organised in packages (``easybuild.framework``, ``easybuild.toolchains``, ``easybuild.tools``, etc.) that collectively form **the core of EasyBuild**, @@ -137,7 +155,7 @@ the necessary functionality to facilitate this. --- -## *Easyblocks* +## Easyblocks An *easyblock* is **a Python module that implements a specific software installation procedure**, and can be viewed as a plugin to the EasyBuild framework. Easyblocks can be either *generic* or *software-specific*. @@ -158,7 +176,7 @@ in the [``easybuild-easyblocks`` repository on GitHub](https://github.com/easybu --- -## *Easyconfig parameters* +## Easyconfig parameters An **easyconfig parameter** specifies a particular aspect of a software installation that should be performed by EasyBuild. @@ -188,7 +206,7 @@ only supported by certain easyblocks, and only make sense for particular (types --- -## *Easyconfig files* +## Easyconfig files *Easyconfig files* (or *easyconfigs* for short), are **simple text files written in Python syntax that specify what EasyBuild should install**. @@ -211,7 +229,7 @@ the EasyBuild community maintains a large (and growing) collection of easyconfig --- -## *Easystack files* +## Easystack files [**Easystack files**](https://docs.easybuild.io/en/latest/Easystack-files.html) are a new concept in EasyBuild, providing a way to define a *software stack* @@ -227,7 +245,7 @@ which means it is subject to change in future EasyBuild releases, and may be pro --- -## *Extensions* +## Extensions *Extensions* is the collective term we use for **additional software packages that can be installed on top of another software package**. Common examples are *Python packages*, *R libraries*, and *Perl modules*. @@ -244,7 +262,7 @@ Extensions can be installed in different ways: --- -## *Dependencies* +## Dependencies A *dependency* is a common term in the context of software. It refers to **a software package that is either strictly required by other software, or that can be leveraged to @@ -266,7 +284,7 @@ but we will discriminate build-only dependencies. --- -## *Toolchains* +## Toolchains A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. @@ -319,7 +337,7 @@ More information on these toolchains is available [in the EasyBuild documentatio --- -## *Modules* +## Modules *Module* is a massively overloaded term in (scientific) software and IT in general (kernel modules, Python modules, and so on). @@ -347,7 +365,7 @@ session such that the corresponding software installation can be used. --- -## *Bringing it all together* +## Bringing it all together The EasyBuild **framework** leverages **easyblocks** to automatically build and install (scientific) software, potentially including additional **extensions**, using a particular compiler **toolchain**, From 36439fc75f92fc869db91edd2d4024d5a7a73f13 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 15:02:25 +0100 Subject: [PATCH 132/159] Finished terminology session, extended the installation section with LUMI-specific material. --- docs/2022-CSC_and_LO/1_04_terminology.md | 141 ++---- docs/2022-CSC_and_LO/1_05_installation.md | 543 ++++++++++++++++++++++ 2 files changed, 570 insertions(+), 114 deletions(-) create mode 100644 docs/2022-CSC_and_LO/1_05_installation.md diff --git a/docs/2022-CSC_and_LO/1_04_terminology.md b/docs/2022-CSC_and_LO/1_04_terminology.md index ebb99963..9c643579 100644 --- a/docs/2022-CSC_and_LO/1_04_terminology.md +++ b/docs/2022-CSC_and_LO/1_04_terminology.md @@ -109,38 +109,13 @@ G --> I; More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). ---- - -## Modules - -*Module* is a massively overloaded term in (scientific) software and IT in general -(kernel modules, Python modules, and so on). -In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. - -[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept -on HPC systems: it is a way to specify changes that should be made to one or more -[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a -[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file -is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or -[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, -and specifies which environment variables should be updated, and how (append, -prepend, (re)define, undefine, etc.) upon loading the environment module. -Unloading the environment module will restore the shell environment to its previous state. - -Environment module files are processed via a **modules tool**, of which there -are several conceptually similar yet slightly different implementations. -The Tcl-based [Environment Modules](https://sourceforge.net/projects/modules/) implementation, and -[Lmod](https://lmod.readthedocs.io), a more recent Lua-based implementation (which also supports module -files written in Tcl syntax), are the most commonly used ones. - -**Environment module files are automatically generated for each software installation** by EasyBuild, -and *loading* a module results in changes being made to the environment of the current shell -session such that the corresponding software installation can be used. - --- ## EasyBuild framework +EasyBuild is written in Python and organised in three layers, represented by the three blocks in the logo: +the EasyBuild framework, easyblocks and Easyconfig files. + The EasyBuild *framework* consists of a set of Python modules organised in packages (``easybuild.framework``, ``easybuild.toolchains``, ``easybuild.tools``, etc.) that collectively form **the core of EasyBuild**, and is developed in the [``easybuild-framework`` repository on GitHub](https://github.com/easybuilders/easybuild-framework). @@ -165,12 +140,20 @@ multiple different software packages. Commonly used examples include the ``Confi which implements the ubiquitous ``configure``-``make``-``make install`` procedure, and the ``PythonPackage`` easyblock that can be used to install a Python package. +*In Spack this functionality is provided in the core of the package.* + A **software-specific easyblock** implements an installation procedure that is specific to a particular software packages. Infamous examples include the easyblocks we have for ``GCC``, ``OpenFOAM``, ``TensorFlow``, ``WRF``, ... The installation procedure performed by an easyblock can be controlled by defining **easyconfig parameters** in an easyconfig file. +*In Spack, this functionality is provided by the ``package.py`` files for the various packages. However, EasyBuild +does not need a software-specific easyblock for every package as for many packages the combination of a +generic easyblock with the much simpler description in the easyconfig file is sufficient. +The software-specific easyblocks can be a bit troublesome on Cray PE systems. Some easyblocks contain code that +is specific for certain compilers or toolchains and do not support all compilers or all toolchains.* + A collection of (generic and software-specific) easyblocks is developed by the EasyBuild community in the [``easybuild-easyblocks`` repository on GitHub](https://github.com/easybuilders/easybuild-easyblocks). @@ -227,6 +210,17 @@ In the [``easybuild-easyconfigs`` repository on GitHub](https://github.com/easyb the EasyBuild community maintains a large (and growing) collection of easyconfig files, for a wide range of (scientific) software. +*EasyBuild does not have an equivalent of the concretiser of Spack. EasyBuild always starts from a complete, +precise configuration of a package to install, determined by the easyconfig parameters and possibly a +custom easyblock. Both approaches have their pros and cons. The concretiser in Spack combined with the +configuration of a software package via command line parameters makes it easy to quickly generate a +number of different configurations of a package to test performance or to check which combinations work +well. But one also has to be careful to not cause an explosion of modules and software installations, and +take care of using the appropriate features to ensure that one can generate the same configurations again. +The precise configurations of EayBuild can sometimes slow down things as they may need a lot of file editing, +but when it comes to building a reproducible software stack and sharing precise building instructions, this +approach may be preferred.* + --- ## Easystack files @@ -273,95 +267,14 @@ There are three main types of dependencies for computer software: * a **build dependency** is only required when building/installing a software package; once the software package is installed, it is no longer needed to *use* that software (examples: `CMake`, `pkg-config`); -* a **run-time dependency** (often referred to simply as *dependency*) is a software package that is - required to *use* (or *run*) another software package (example: `Python`); * a **link-time dependency** is somewhere in between a build and runtime dependency: it is only needed when *linking* a software package; it can become either a build or runtime dependency, depending on exactly how the software is installed (example: `OpenBLAS`); +* a **run-time dependency** (often referred to simply as *dependency*) is a software package that is + required to *use* (or *run*) another software package (example: `Python`); -The distinction between link-time and run-time dependencies is mostly irrelevant for this tutorial, -but we will discriminate build-only dependencies. - ---- - -## Toolchains - -A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, -which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. - -We refer to the different parts of a toolchain as **toolchain components**. - -The *compiler component* typically consists of [C](https://en.wikipedia.org/wiki/C_(programming_language)), -[C++](https://en.wikipedia.org/wiki/C%2B%2B), and [Fortran](https://en.wikipedia.org/wiki/Fortran) -compilers in the context of HPC, but additional compilers (for example, -a [CUDA](https://developer.nvidia.com/cuda-zone) compiler for -[GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units) software) -can also be included. - -Additional toolchain components are usually special-purpose libraries: - -* an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/)); -* libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), - [LAPACK](http://performance.netlib.org/lapack/)); -* a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); - -A toolchain that includes all of these libraries is referred to as a **full toolchain**, while -a **subtoolchain** is a toolchain that is missing one or more of these libraries. -A **compiler-only toolchain** only consists of compilers (no additional libraries). - -### System toolchain - -The **`system` toolchain** is a special case which corresponds to using the compilers and libraries -*provided by the operating system*, rather than using toolchain components that were installed using EasyBuild. - -It used sparingly, mostly to install software where no actual compilation is done or -to build a set of toolchain compilers and its dependencies, since the versions of the system compilers -and libraries are beyond the control of EasyBuild, which could affect the reproducibility of the installation. - -### Common toolchains - -The `foss` and `intel` toolchains are also known as the *common toolchains*, -because they are widely adopted by the EasyBuild community. - -The `foss` toolchain consists of all open source components (hence the name: -"FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), -[ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). - -The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version -controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. - -Roughly every 6 months, a new version of these common toolchains is agreed upon -in the EasyBuild community, after extensive testing. - -More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). - ---- - -## Modules - -*Module* is a massively overloaded term in (scientific) software and IT in general -(kernel modules, Python modules, and so on). -In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. - -[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept -on HPC systems: it is a way to specify changes that should be made to one or more -[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a -[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file -is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or -[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, -and specifies which environment variables should be updated, and how (append, -prepend, (re)define, undefine, etc.) upon loading the environment module. -Unloading the environment module will restore the shell environment to its previous state. - -Environment module files are processed via a **modules tool**, of which there -are several conceptually similar yet slightly different implementations. -The Tcl-based [Environment Modules](https://sourceforge.net/projects/modules/) implementation, and -[Lmod](https://lmod.readthedocs.io), a more recent Lua-based implementation (which also supports module -files written in Tcl syntax), are the most commonly used ones. - -**Environment module files are automatically generated for each software installation** by EasyBuild, -and *loading* a module results in changes being made to the environment of the current shell -session such that the corresponding software installation can be used. +EasyBuild currently doesn't treat link-time dependencies differently from run-time dependencies +and hence has no specific easyconfig parameter for them. --- @@ -378,4 +291,4 @@ An **easystack file** can be used to specify a collection of software to install --- -*[[next: Installation]](installation.md)* +*[[next: Installation]](1_05_installation.md)* diff --git a/docs/2022-CSC_and_LO/1_05_installation.md b/docs/2022-CSC_and_LO/1_05_installation.md new file mode 100644 index 00000000..163262ea --- /dev/null +++ b/docs/2022-CSC_and_LO/1_05_installation.md @@ -0,0 +1,543 @@ +# Installing EasyBuild + +*[[back: Terminology]](1_04_terminology.md)* + +--- + +Before deep diving into the wonderful world of EasyBuild and getting your hands dirty with +the hands on exercises coming up in this tutorial, you will need to install EasyBuild. + +In this section we outline a couple of different ways of doing this, and also the things you should pay attention to. +By the end, you will have a functional EasyBuild installation that you can use +for the remainder of this tutorial. + +!!! Remark "EasyBuild on LUMI" + + EasyBuild is already installed for each of the LUMI software stacks. The LUMI software + stacks are based on the Cray PE (using the Cray-specific toolchains). + Moreover, in case you want to install a full shadow stack rather than build on top + of one of the LUMI software stacks, scripts are provided to initialise a new version + of the LUMI software stack based on a specific version of the Cray PE, and those + scripts also take care of the installation of EasyBuild. + + Hence this section of the tutorial is only useful for people who want more background + knowledge or who want to experiment with other EasyBuild toolchains. + + +!!! summary + + * *Requirements*: Linux, Python 2.7 or 3.5+, environment modules tool (Lmod is recommended). + * *Installation methods*: + * ``pip install easybuild`` + * installing EasyBuild with EasyBuild + * development setup + * Verify EasyBuild installation using `eb --version` and `eb --help`. + + +--- + +## Requirements + +### Linux + +The main target platform for EasyBuild is *Linux*, since it is heavily focused on installing software +on HPC system where Linux is dominant operating system (to the point where 100% of the current Top500 list of supercomputers are running Linux). + +EasyBuild is also compatible with *macOS*, but the included easyconfig files are heavily focused +on Linux so most software installations supported by EasyBuild won't work out-of-the-box on macOS. +You can still use the EasyBuild command line interface on macOS for other tasks though, +like development or contributing back to the project. + +### Python + +EasyBuild is implemented in Python, and is **compatible with both Python 2.7 and Python 3.5+** +(that is, Python 3.5 or a newer version of Python 3). + +To check which Python version you have, use: + +```shell +python -V +``` + +*No additional Python packages are required by EasyBuild*, the ones that come with the standard +Python distribution are sufficient. Some additional Python packages *can* be leveraged for specific features. +More on that later. + +### Environment modules tool + +**An *environment modules tool* is required for using EasyBuild.** + +We strongly recommend using [Lmod](https://lmod.readthedocs.io), a Lua-based modern environment +modules implementation and the most commonly used modules tool in the EasyBuild community. +Other implementations, like the original Tcl-based one, are also supported. + +To check if you have a modules tool installed, use: + +``` +module --version +``` + +If this produces output that starts with something like "`Modules based on Lua: Version 8.3.1`" you have Lmod installed, +which is the default modules tool used by EasyBuild, and you are all set for installing and using EasyBuild. +Any sufficiently recent Lmod version (8.x or even 7.x) should be fine. + +If you see output that starts with a line like "`VERSION=3.2.10`" or "`Modules Release 4.5.0`", +you have the original Tcl-based environment modules tool installed, either in its original +C-based imnplementation (``VERSION=3.2.10``) or the new Tcl-based implementation developed at CEA +(``Modules Release 4.5.0``) +and *EasyBuild will need to be configured to use it after installation*. + +If the `module` function is not defined either you do not have a modules tool installed +or your environment is not properly set up to use it. In this case, +please refer to the EasyBuild documentation here for more information. + +HPE Cray supports both the old-style Environments Modules (based on version 3.2.10) and Lmod. On LUMI, +we use Lmod as the default module system for users, and EasyBuild is configured to use that one. + +--- + +## EasyBuild as a Python package + +EasyBuild consists of a number of interdependent Python packages, +and is available via both GitHub at https://github.com/easybuilders, +as well as via the standard Python Package Index (PyPI) at +https://pypi.org/project/easybuild. + +As you may be aware the Python packaging ecosystem is bit convoluted, +which is reflected in the many different ways in which you can install a Python package. +In addition, EasyBuild is packaged in 3 components (framework, easyblocks, easyconfigs) +which slightly complicates the installation. + +
+ +Nevertheless, you don't need to be a rocket scientist to install EasyBuild (and even if you are, +that's OK too), so don't worry. + +You can install EasyBuild just like you can install any other Python software that is released +via the standard *Python Package Index* (PyPI), through one of the standard Python installation tools +(like `pip`, `virtualenv`, `pipenv`, ...). +And since EasyBuild is a software installation tool in its own right, we actually have a couple +of additional tricks up our sleeve! + +### Python 2 or Python 3? + +For EasyBuild it does not matter much whether you install it on top of Python 2 or Python 3. +Since version 4.5, Python 3 does offer some optional features (requiring additional packages) +that are not available with Python 2. +Since Python 2 is end-of-life, +we strongly recommend using Python 3 if you have the choice. + +By default EasyBuild will use the `python` command to run, +but you can control this if needed via ``$EB_PYTHON``. + + +## Installing EasyBuild + +We present three methods for installing EasyBuild. +It is up to you which one you prefer, both result a fully functional EasyBuild installation. + +Time to get your hands dirty! + +* Method 1: Using ``pip`` +* Method 2: installing EasyBuild with EasyBuild +* Method 3: Development setup + +--- + +### Method 1: Using `pip` + +Since EasyBuild is released as a [Python package on PyPI](https://pypi.org/project/easybuild) +you can install it using `pip`, the most commonly used tool for installing Python packages. + +You may need to take additional steps after the installation, depending on the exact installation command. + +!!! note + There are various other ways of installing Python packages, which we won't cover here. + If you are familiar with other tools like `virtualenv` or `pipenv`, feel free to use those + instead to install EasyBuild. + +#### Running `pip install` + +Installing EasyBuild with `pip` is as simple as running the following command: + +```shell +pip install easybuild +``` + +However, you may need to slightly change this command depending on the context and your personal preferences: + +* To install EasyBuild *system-wide*, you can use `sudo` (if you have admin privileges): + ```shell + sudo pip install easybuild + ``` + +* To install EasyBuild *in your personal home directory*, you can use the `--user` option: + ```shell + pip install --user easybuild + ``` + This will result in an EasyBuild installation in `$HOME/.local/`. + +* To install EasyBuild in a *specific directory* you can use the `--prefix` option: + ```shell + pip install --prefix _PREFIX_ easybuild + ``` + In this command, you should replace '`_PREFIX_`' with the location where you want to have EasyBuild installed + (for example, `$HOME/tools` or `/tmp/$USER`). + +#### `pip` vs `pip3` + +On systems where both Python 2 and Python 3 are installed you may also have different `pip` commands +available. Or maybe `pip` is not available at all, and only "versioned" `pip` commands like `pip3` are +available. + +If you (only) have `pip3` available, you can replace `pip` with `pip3` in any of the `pip install` commands +above. + +If you want to ensure that you are using the ``pip`` installation that corresponds to the Python 3 installation +that you intend to use, you can use ``python3 -m pip`` rather than ``pip3``. + +#### Updating your environment + +If you used the `--user` or `--prefix` option in the `pip install` command, +or if you installed EasyBuild with a `pip` version that does not correspond +to your default Python installation, +you will need to update your environment to make EasyBuild ready for use. +This is not required if you did a system-wide installation in a standard location with the default Python version. + +!!! note + Keep in mind that you will have to make these environment changes again if you start a new shell session. + To avoid this, you can update one of the shell startup scripts in your home directory (`.bashrc` for example). + +#### Updating ``$PATH`` + +Update the `$PATH` environment variable to make sure the `eb` command is available: +```shell + +export PATH=_PREFIX_/bin:$PATH +``` +**Replace '`_PREFIX_`' in this command** with the directory path where EasyBuild was installed into +(use `$HOME/.local` if you used `pip install --user`). + +This is not required if you installing EasyBuild in a standard system location. + +You can check with the `which eb` command to determine whether or not you need to update the ``$PATH`` environment variable. + +#### Updating ``$PYTHONPATH`` + +If you installed EasyBuild to a non-standard location using `pip install --prefix`, +you also need to update the Python search path environment variable +[`$PYTHONPATH`](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH) to instruct Python where +it can find the EasyBuild Python packages. + +This is not required if you used the `--user` option, since Python will automatically consider +`$HOME/.local` when searching for installed Python packages, or if you installed EasyBuild in a standard +system-wide location. + +Update `$PYTHONPATH` by running a command like: + +```shell + +export PYTHONPATH=_PREFIX_/lib/pythonX.Y/site-packages:$PYTHONPATH +``` + +Here, you need to replace the `X` and `Y` with the major and minor version of your Python installation, +which you can determine by running `python -V`. +For example, if you are using Python 3.6, make sure you are using `/python3.6/` in the command to update `$PYTHONPATH`. + +And of course, you again need to **replace '`_PREFIX_`'** with the installation prefix where EasyBuild was installed +into. + +For example: + +```shell +# update $PYTHONPATH if EasyBuild was installed in $HOME/tools with Python 3.6 +export PYTHONPATH=$HOME/tools/lib/python3.6/site-packages:$PYTHONPATH +``` + +#### Setting ``$EB_PYTHON`` + +If you want to control which Python version is used to run EasyBuild, +you can specify the name or the full path to the `python` command that should be used by the `eb` command +via the `$EB_PYTHON` environment variable. + +This may be required when you installing EasyBuild with a version of `pip` that does not correspond +with the default Python version. + +For example, to ensure that `eb` uses `python3.6`: + +```shell +export EB_PYTHON=python3.6 +``` + +#### Setting ``$EB_VERBOSE`` + +To determine which `python` commands are being considered by the `eb` command, +you can define the `$EB_VERBOSE` environment variable. For example: + +```shell +$ EB_VERBOSE=1 eb --version +>> Considering 'python3.6'... +>> 'python3' version: 3.6.8, which matches Python 3 version requirement (>= 3.5) +>> Selected Python command: python3 (/usr/bin/python3.6) +>> python3.6 -m easybuild.main --version +This is EasyBuild 4.3.3 (framework: 4.3.3, easyblocks: 4.3.3) on host example +``` + +!!! Tip "Debugging startup problems" + + Using `ÈB_VERBOSE` is useful if EasyBuild fails to start up and complains it + cannot find a suitable Python executable. Rather often the error message is + caused by a failure elsewhere in EasyBuild. + + +--- + +### Method 2: Installing EasyBuild with EasyBuild + +!!! note + This section covers an alternative method for installing EasyBuild. + + If you already have EasyBuild installed through ``pip``, + you can skip ahead to the next section. + +If you prefer having EasyBuild available through an environment module file, +you can consider installing EasyBuild with EasyBuild. This can be done in 3 steps: + +* Step 1: Installing EasyBuild with ``pip`` into a temporary location (only needed if EasyBuild is not installed yet) +* Step 2: Using EasyBuild to install EasyBuild as a module +* Step 3: Loading the EasyBuild module + +!!! note + A [bootstrap script](https://docs.easybuild.io/en/latest/Installation.html#bootstrapping-easybuild) + is available that automates this procedure, + but is known to be problematic in some contexts, and is not being actively + maintained anymore. + + As a result, we do not recommend using the bootstrap script anymore. + + +#### Step 1: Installing EasyBuild into a temporary location + +If you don't have EasyBuild installed yet, you need to install it in a temporary location first. +The recommended way of doing this is using ``pip``. + +For example, to install EasyBuild into a subdirectory `/tmp/$USER` using the default Python 3 version: + +```shell +# pick installation prefix, and install EasyBuild into it +export EB_TMPDIR=/tmp/$USER/eb_tmp +python3 -m pip install --ignore-installed --prefix $EB_TMPDIR easybuild +``` + +```shell +# update environment to use this temporary EasyBuild installation +export PATH=$EB_TMPDIR/bin:$PATH +export PYTHONPATH=$(/bin/ls -rtd -1 $EB_TMPDIR/lib*/python*/site-packages | tail -1):$PYTHONPATH +export EB_PYTHON=python3 +``` + +#### Step 2: Using EasyBuild to install EasyBuild + +Once you have a working (recent) temporary EasyBuild installation, you can use it to +install EasyBuild as a module. Usually this is done in the location where you would +like to install other software too. + +You can use the ``eb --install-latest-eb-release`` command for this, +combined with the ``--prefix`` option to control which directories are used by EasyBuild for the installation. + +For example, to install the latest version of EasyBuild as a module into ``$HOME/easybuild``: + +```shell +eb --install-latest-eb-release --prefix $HOME/easybuild +``` + +!!! note + You may see a harmless deprecation warning popping up when performing this installation, just ignore it. + +#### Step 3: Loading the EasyBuild module + +Once step 2 is completed, you should be able to load the module that was generated alongside +the EasyBuild installation. You will need to do this every time you start a new shell session. + +First, make the module available by running the following command (which will update the module search path +environment variable `$MODULEPATH`): + +```shell + +module use _PREFIX_/modules/all +``` + +**Replace '`_PREFIX_`'** with the path to the directory that you used when running step 2 +(for example, ``$HOME/easybuild``). + +Then, load the `EasyBuild` module to update your environment and make EasyBuild available for use: + +```shell + +module load EasyBuild +``` + +Note that in this case, we don't need to make any changes to our environment for EasyBuild to work correctly. +The environment module file that was generated by EasyBuild specifies all changes that need to be made. + +### Method 3: Development setup + +If you are planning to make changes to EasyBuild, or if you prefer using the latest *bleeding edge* +version of EasyBuild that is being developed, you can consider *cloning* the 3 main EasyBuild repositories +from GitHub, and updating your environment to run EasyBuild from there. + +This can be done as follows (into ``$HOME/easybuild``): + +```shell +mkdir -p $HOME/easybuild +cd $HOME/easybuild +``` + +```shell +# clone EasyBuild repositories from GitHub +git clone https://github.com/easybuilders/easybuild-framework.git +git clone https://github.com/easybuilders/easybuild-easyblocks.git +git clone https://github.com/easybuilders/easybuild-easyconfigs.git +``` +```shell +# update environment for running EasyBuild from there +export PATH=$HOME/easybuild/easybuild-framework:$PATH +export PYTHONPATH=$HOME/easybuild/easybuild-framework:$HOME/easybuild/easybuild-easyblocks:$HOME/easybuild/easybuild-easyconfigs:$PYTHONPATH + +# control which Python command is used to run EasyBuild +export EB_PYTHON=python3 +``` + + +### Approach on LUMI + +*Documentation on the inner workings of the LUMI software stack can be found in the +[LUMI-SoftwareStack GitHub, docs subdirectory](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/docs).* + +To keep the different versions of the LUMI software stack as independent from one another as possible, +EasyBuild is bootstrapped for each software stack by the ``prepare_LUMI_stack.sh`` script. We use the +system Python 3 for running EasyBuild. That Python version currently doesn't have ``pip`` installed. +EasyBuild is first installed in a temporary work directory from files downloaded from PyPi. These +are each untarred and then installed into their location by running +``python3 setup.py install --prefix=...`` in their unpacked directory. +We do so only for the framework and easyblocks files as the easconfig files are not used to +install EasyBuild. Instead we create our own EasyConfig file for EasyBuild which contains +some additional packages that enable extra features in EasyBuild and also provide more information +to Lmod. Next the configuration module for EasyBuild (see the next section, +["Configuring EasyBuild"](1_06_configuring.md), for more information) and use the temporary +installation of EasyBuild with our own EasyConfig file to do a proper installation of EasyBuild +with module in the final location. + + +TODO: follow development setup but with downloading rather than cloning, then use that one with the +EasyBuild configuration module to install EasyBuild with itself. + + + +## Verifying the installation + +Regardless of how EasyBuild was installed, you can now run a couple of basic commands to verify the installation: + +#### Checking the version + +To check which EasyBuild version you have installed, run: + +```shell + +eb --version +``` + +The output should match with the latest EasyBuild version. + + +#### Consulting the help output + +You can consult the help output of the `eb` command, which produces a long list of available options +along with a short informative message. + +```shell +eb --help +``` + +#### Showing the default EasyBuild configuration + +To inspect the current EasyBuild configuration, you can use this command: + +```shell +eb --show-config +``` + +This should tell you that EasyBuild (ab)uses `$HOME/.local/easybuild` as a default location. +More on configuring EasyBuild in the [next part of the tutorial](configuration.md). + +#### System information + +You ask EasyBuild to collect and print some information about the +system you are using it on (OS, CPU, Python, etc.) using this command: + +```shell +eb --show-system-info +``` + +## Updating EasyBuild + +Before we wrap up here, a brief word about updating EasyBuild. + +Once you have EasyBuild installed, the easiest way to update to a newer version is by instructing EasyBuild +to install the latest available version as a module: + +``` + +eb --install-latest-eb-release +``` + +This will result in a *new* EasyBuild installation, which is entirely separate from the EasyBuild installation +you are currently using (so it is *not* an in-place update). +The location where this new EasyBuild version will be installed is determined by the active +EasyBuild configuration. + +If you have installed EasyBuild through ``pip``, and you prefer updating that installation, +you can use ``pip install --upgrade easybuild`` (perhaps with additional options like ``--user`` or ``--prefix``). + +!!! Note "EasyBuild upgrade policy on LUMI" + + Even minor version or patch level updates of EasyBuild may break things (and this has happned + in the past on other systems managed previously by LUMI User Support Team members). Hence a very + conservative upgrade policy is used on LUMI. + + In general we fix the version of EasyBuild within a particular version of the LUMI software stack + and only transition to the latest version when starting a new software stack. The reason is that we + want to be able to rebuild an existing software stack with as little unexpected problems as possible. + A move to a newer version of EasyBuild is acceptable should we run into major problems with an existing + version that cannot be solved in an easyconfig file or LUMI-specific custom easyblock, but in those + cases we will first rebuild the whole software stack on a test setup to ensure that no problems are + introduced by upgrading EasyBuild. + + +--- + +## Exercise (optional) + +*This exercise is not needed to proceed the course when working on LUMI as we will +use the installation that is already present.* + +Install EasyBuild in your home directory. + +Make sure that the EasyBuild installation uses the `python3` command to run, +rather than the standard `python` command. + +Choose your own adventure, or try all these installation methods! + +* install EasyBuild with `pip` (or another very similar command...) using either the `--user` or `--prefix` option; +* perform a (manual) "bootstrap" installation into `$HOME/easybuild`, as outlined in installation method 2; +* set up a development installation; + +Check that the installation works by running the verification commands outlined +[above](#verifying-the-installation). + +--- + +**Make sure you have a working EasyBuild installation before proceeding +with the rest of the tutorial!** + +*[[next: Configuring EasyBuild]](1_06_configuration.md)* From 0e2a846b4e286b04931d9fe4c657005fc086a794 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 17:31:43 +0100 Subject: [PATCH 133/159] Tutorial on EasyBuild configuration added, and some restructuring. --- .../1_04_LUMI_software_stack.md | 17 + ..._04_terminology.md => 1_05_terminology.md} | 0 ...5_installation.md => 1_06_installation.md} | 0 docs/2022-CSC_and_LO/1_07_configuration.md | 610 +++++++++++ docs/2022-CSC_and_LO/1_08_basic_usage.md | 976 ++++++++++++++++++ 5 files changed, 1603 insertions(+) create mode 100644 docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md rename docs/2022-CSC_and_LO/{1_04_terminology.md => 1_05_terminology.md} (100%) rename docs/2022-CSC_and_LO/{1_05_installation.md => 1_06_installation.md} (100%) create mode 100644 docs/2022-CSC_and_LO/1_07_configuration.md create mode 100644 docs/2022-CSC_and_LO/1_08_basic_usage.md diff --git a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md new file mode 100644 index 00000000..0e578a59 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md @@ -0,0 +1,17 @@ +# LUMI software stacks + +*[[back: The Cray Programming Environment]](1_03_CPE.md)* + +--- + + + + + + + + +--- + +*[[next: Terminology]](1_05_terminology.md)* + diff --git a/docs/2022-CSC_and_LO/1_04_terminology.md b/docs/2022-CSC_and_LO/1_05_terminology.md similarity index 100% rename from docs/2022-CSC_and_LO/1_04_terminology.md rename to docs/2022-CSC_and_LO/1_05_terminology.md diff --git a/docs/2022-CSC_and_LO/1_05_installation.md b/docs/2022-CSC_and_LO/1_06_installation.md similarity index 100% rename from docs/2022-CSC_and_LO/1_05_installation.md rename to docs/2022-CSC_and_LO/1_06_installation.md diff --git a/docs/2022-CSC_and_LO/1_07_configuration.md b/docs/2022-CSC_and_LO/1_07_configuration.md new file mode 100644 index 00000000..17e60afe --- /dev/null +++ b/docs/2022-CSC_and_LO/1_07_configuration.md @@ -0,0 +1,610 @@ +# Configuring EasyBuild + +*[[back: Installation]](1_06_installation.md)* + +--- + +After installing EasyBuild, you should **configure** it. + +EasyBuild should work fine out-of-the-box if you use Lmod as your modules tool. +If you are not using Lmod, please see here for more information. + +Nevertheless, we strongly recommend you to inspect the default configuration, +and to configure EasyBuild according to your preferences and the system on which you will use it. + +!!! Note "EasyBuild configuration on LUMI" + + On LUMI serveral configurations of EasyBuild are already available. + + The ``EasyBuild-user`` module is the most important one. It will configure EasyBuild + to install software in either a default location in the user's home directory + (``$HOME/EasyBuild``) or the location pointed to by the environment variable + ``EBU_USER_PREFIX``. The latter can be used to install software in the project directory + so that it is available to all users of the project. It enables the creation of a custom + software environment build on top of the LUMI software stack. Software in this environment is + build in exactly the same way as it would if it were installed in the central repository, but + one can now also easily create custom configurations without confusing other projects with + different or non-standard configurations of software. + + There are two more admin-only configuration modules for EasyBuild. The main one is + ``EasyBuild-production`` which is used for software installations in the central software stack. + The other one is ``EasyBuild-infrastructure`` which is used to install infrastructure modules + that need to be installed in a pure Lmod hierarchy, e.g., the modules that load the toolchains. + + +--- + +## Available configuration settings + +One of the central policies in the EasyBuild project is to avoid hardcoded settings in the codebase. +While this significantly increases the ability to configure EasyBuild to your liking, it also results +in a large amount of available configuration settings. + +The full list of configuration settings can be consulted via `eb --help`, +which shows the corresponding command line option accompanied by a short description. +At the time of writing, *over 240 different configuration settings* are supported by EasyBuild. + +For the sake of this tutorial we will focus on a specific subset of configuration settings, +and cover only the most prominent and important ones. + +We will refer to EasyBuild configuration settings using the names as they appears in the output of `eb --help`, +and omit the leading dashes (`--`) for the sake of clarity. + +Keep in mind that *every* configuration setting can be defined in 3 different ways, +see below for more details. + +--- + +### Overall prefix + +*(default: `HOME/.local/easybuild`)* + +The `prefix` configuration setting specifies the overall prefix that EasyBuild should use, +which **determines the default value for various other configuration settings**: + +* `installpath`: `` +* `buildpath`: `/build` +* `sourcepath`: `/sources` +* `repositorypath` (easyconfigs archive): `/ebfiles_repo` +* `containerpath`: `/containers` + +Here, `` represents the value of the `prefix` configuration setting. + +If one of the configuration settings affected by `prefix` is defined specifically, +the `prefix` value becomes irrelevant for that specific configuration setting. + +--- + +### Install path + +*(default: ``)* + +The **location for both the software installation directories and generated module files** +can be controlled via the `installpath` configuration setting. +Software installation directories will be placed in `/software`, while +`/modules/all` will be used for generated module files. + +The `installpath` location is usually set to a directory on a *shared filesystem* when installing +software for an HPC cluster. Of course, software can also be installed on a local filesystem, +which is particularly useful to test and evaluate software installations. + +Separate configuration settings are available for both software and modules locations, +as well as for controlling the name of the `software` and `modules/all` subdirectories. + +*The EasyBuild community recommends to only change the `installpath` configuration setting to control the location +of software installations and accompanying module files, +such that the software and modules directories are located in the same parent directory, +and the default `software` and `modules/all` names for the subdirectories are used.* + +!!! Note "Path for software and for modules on LUMI" + + On LUMI we do not follow that recommendation. There is a three for the software installations + themselves with subdirectories based on the version of the software stack and LUMI hardware + partition, and a separate tree for the modules organised in a similar way. + + This makes it slightly easier to organise the module tree with user-friendly labeling, but above + all also makes the synchronisation process of the 4 instances of the software directory more robust + as it is now easy to synchonise all modules in the last step, which is a much quicker process than + syncrhonising the software installations. + + We also use short paths for software installations (to avoid overrunning the maximum length of a + shebang line in scripts) while we use longer, more descriptive names for subdirectories in the + module tree. + + +--- + +### Build path + +*(default: `/build`)* + +For each installation it performs, EasyBuild creates a **separate build directory** where software will be compiled +before installing it. This directory is cleaned up automatically when the installation is successfully completed. +To control the location where these build directories are created, you can use the `buildpath` configuration setting. + +Keep in mind that build directories may grow out to several GBs in size during an installation, +and that the commands that run in there can be fairly I/O-intensive since they may involve +manipulating lots of small files. In addition, a build directory that corresponds to a failing installation +is *not* cleaned up automatically, but it will be cleaned up and recycled when the same installation is re-attempted. +Running out of disk space in the location where build directories are created will result in failing +installations. + +It is strongly recommend to use the path to a directory on a *local filesystem* for the value of the +`buildpath` configuration setting, since using a shared filesystem like Lustre or GPFS is known to cause +problems when building certain software packages. Using an in-memory location (like `/dev/shm/$USER`) can +significantly speed up the build process, but may also lead to problems (due to space limitations, +or specific mount options like `noexec`). + +!!! Note "buildpath on LUMI" + + The configuration modules on LUMI will use a RAM disk for the build path. On the login nodes, + ``$XDG_RUNTIME_DIR`` is used as that space is automatically cleared when the last session of a user + ends. However, on the compute nodes a job- or user-specific subdirectory of ``/dev/shm`` is currently used + as ``$XDG_RUNTIME_DIR`` does not exist. + + +--- + +### Source path + +*(default: `/sources`)* + +For most supported software, EasyBuild can **automatically download the source files** required for the installation. +Before trying to download a source file, EasyBuild will first check if it is already present in the source path. + +The locations considered by EasyBuild when checking for available source files, as well as the location to +store downloaded source files, can be controlled via the ``sourcepath`` configuration setting. + +The `sourcepath` value is a colon (`:`) separated list of directory paths. +Each of these paths will be considered in turn when checking for available source files, +until one of them provides the desired source file. Searching for source files is done +based on filename, and a couple of subdirectories are considered. +For example, for a software package named '`Example'`, EasyBuild will consider locations +like `/e/Example/`, `/Example/`, and so on. + +The first path listed in `sourcepath` is the location where EasyBuild will store downloaded source files, +organised by software name through subdirectories, so EasyBuild expects to have write permissions to this path. +For the other paths listed in `sourcepath` only read permissions are required. + +*Make sure you have write permissions to the first path listed in `sourcepath`, so EasyBuild is able +to store downloaded files there. Feel free to list additional paths if you already have a cache of downloaded +files available somewhere.* + +*Storing the downloaded files not only reduces the amount of downloads while developing new easyconfig files +but also greatly helps when recompiling a software stack, as it is not that uncommon that download sites change +of files become unavailable.* + + +--- + +### Easyconfigs archive + +*(default: `/ebfiles_repo`)* + +EasyBuild keeps track of the easyconfig files that were used for installations in the easyconfigs +archive, the location of which is specified by the `repositorypath` configuration setting. + +By default the specified path is assumed to be a regular directory, but using a Git repository +as easyconfigs archive is also supported (for more details, see +[the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Configuration.html#easyconfigs-repository-repository-repositorypath)). + + +--- + +### Modules tool & module syntax + +*(default: `Lmod` as modules tool, and `Lua` as module syntax)* + +By default, EasyBuild assumes you are using **[Lmod](https://lmod.readthedocs.io) as modules tool**. +In addition, it will generate module files in [Lua](https://www.lua.org/) syntax, as supported by Lmod +(next to `Tcl` syntax). + +To diverge from this, you can define the `modules-tool` configuration setting to indicate you +are using a different modules tool; see the output of `eb --avail-modules-tools` for a list of supported +modules tools. +Note that for anything other than Lmod, you *must* make +sure that the actual modules tool binary command is available through `$PATH` (more information +on this [in the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Configuration.html#modules-tool-modules-tool)). + +If you prefer that EasyBuild generates module files in [Tcl](https://tcl.tk/) syntax, you can instruct +it to do so via the `module-syntax` configuration setting. Note that this is *required* if you are using +a modules tool other than Lmod, since only Lmod supports module files in `Lua` syntax. + +*We recommend using EasyBuild with the defaults settings: Lmod as modules tool, and Lua as module syntax.* + +--- + +### Robot search path + +*(default: `robot-paths` specifies the location to the easyconfig files included with EasyBuild installation; +dependency resolution is* ***not*** *enabled)* + +When EasyBuild needs to locate one or more easyconfig files, it will do so via the **robot search path**. +This applies to both easyconfig files that were specified using only their filename as an argument to the +`eb` command, as well as to easyconfigs required to resolve dependencies (more on that later). + +To control the robot search path, two configuration settings are available: `robot` and `robot-paths`. +Both accept a colon-separated list of locations to consider when looking for easyconfig files, +with `robot` having a higher priority than `robot-paths` (i.e., the directories specified with `robot` +appear before those specified by `robot-paths` in the robot search path). + +The key difference between these two configuration settings is that defining `robot` also enables dependency resolution, +and hence makes EasyBuild install missing dependencies, alongside +specifying a list of paths to consider when searching for easyconfig files. +On the other hand, defining `robot-paths` does not have the side effect of enabling dependency resolution. + +In addition, you can use the `--robot` command line option without specifying any paths to it, +to only enable dependency resolution. ***This option is needed on LUMI if you want automatic +dependency resolution as it is disabled on LUMI to prevent accidental mistakes when installing +software.*** + +!!! note "Easyconfig files included with EasyBuild and the robot search path" + Keep in mind that when either of the `robot` or `robot-paths` configuration settings are + defined, the default value corresponding to the location of easyconfigs included with the + EasyBuild is *no longer considered*. + + There are ways around this however, which are outside of the scope of this tutorial. + + For more information, + [see the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Using_the_EasyBuild_command_line.html#controlling-robot-search-path). + + *On LUMI the included easyconfig files cannot be used without changes as they are for the + common toolchains which are currently not supported in the LUMI software stacks.* + + +!!! Note "Easyconfigs archive on LUMI and the robot search path" + + On LUMI we also include the easyconfigs archive at the front of the search path for easyconfig + files. This has the advantage that EasyBuild when checking dependencies will always find the + configuration that is actually installed on the system, even if the easyconfig file in one of + the regular repositories has changed. It also has the disadvantage that one may accidentally + be re-installing with the already installed configuration while one really wants to re-install + the module in a different configuration. + + + +--- + +### Module naming scheme + +*(default: `EasyBuildMNS`)* + +EasyBuild will use a specific naming scheme for the module files it generates for each of the software installations. +This ensures consistency, regardless of who uses EasyBuild to perform the installation. + +Different types of module naming schemes are supported (flat, hierarchical, ...) and you can provide an implementation of +your own custom module naming scheme if desired. A number of different naming schemes are included +with EasyBuild, which you can consult via `eb --avail-module-naming-schemes`. + +The default `EasyBuildMNS` module naming scheme roughly corresponds to the filename of easyconfig files, +and consists of the software name followed by a combination of the software version, toolchain and +an optional label (which corresponds to the value of the `versionsuffix` easyconfig parameter): +`/<-toolchain>`. Just like with [names of easyconfig +files](terminology.md#easyconfig-files), the `<-toolchain>` part is omitted when the `system` toolchain is used, +and the `` value is empty by default. + +!!! Note "Module naming scheme on LUMI" + + On LUMI we use a custom variant of the standard EasyBuild flat naming scheme. The EasyBuild default + naming scheme creates an ``all`` subdirectory in the module directory containing all modules, but also + a directory per category, with the categories determined from the ``moduleclass`` easyconfig parameter. + However, that choice is often rather arbitrary as modules can belong to different categories depending + on who you ask. E.g., GROMACS is used in physics, chemistry and life sciences while EasyBuild can only + assign a single category. + + Hence we do not generate the categories and drop the ``all`` level in the module subdirectory. + + +--- + +## Configuration levels + +Configuring EasyBuild can be done in different ways: + +* through one or more configuration files; +* via `$EASYBUILD_*` environment variables; +* using `eb` command line options; + +Each of the methods corresponds to a *configuration level*. + +*Every* configuration setting can be defined via one of these +mechanisms, without exception! + +--- + +### Configuration level hierarchy + +There is a strict **hierarchy between the different configuration levels** supported by EasyBuild. + +Settings defined via a *configuration file* only override default values. + +A configuration setting that is defined via the corresponding *`$EASYBUILD_*` environment variable* +takes precedence over the value specified in a configuration file (if any). + +Finally, values specified through `eb` command line options **always** win, +regardless of whether the corresponding +configuration setting was already defined some other way, be it via a configuration file or +an environment variable. + +For example, let us consider a fictional configuration setting named `magic`: + +* If a value for `magic` is specified in an EasyBuild configuration file, + then this value will only be used if the corresponding environment variable + (`$EASYBUILD_MAGIC`) is *not* defined **and** if the `--magic` command line option is *not* used; +* If the `$EASYBUILD_MAGIC` environment *is* defined however, then its value + will be used for the `this-is-magic` configuration setting; +* *Unless* the `--magic` command line option is used, + since in that case the value provided as an argument there will be used instead. + +--- + +### Configuration files + +*Configuration files* are the most basic way of configuring EasyBuild. +Two types of are supported by EasyBuild: *user-level* and *system-level* +configuration files. The output of `eb --show-default-configfiles` tells you which locations +are considered, and whether any configuration files were found. + +EasyBuild configuration files are written in the standard [INI format](https://en.wikipedia.org/wiki/INI_file), +and the configuration settings are grouped into different *sections*. + +To create an EasyBuild configuration file, the output of `eb --confighelp` is very useful: +it produces the list of all supported configuration settings which are readily grouped in sections +and with every entry commented out, along with accompanying descriptive comments mentioning the default values, +and in the correct syntax. + +*Configuration files are the recommended mechanism to define configuration settings +that should always be in place, regardless of the software you are installing.* + +#### EasyBuild configuration files vs easyconfig files + +EasyBuild configuration files are sometimes confused with easyconfig files, +due to the similar name. However, these are two entirely different concepts! + +EasyBuild configuration files (usually `*.cfg`) are a way of *configuring the general behaviour of EasyBuild* +across different software installations. They define *configuration settings*, +such as the location where software should be installed, or the syntax that should +be used when generating module files. + +An [easyconfig file](terminology.md#easyconfig-files) (`*.eb`) on the other hand *specifies the details for one particular software installation*. +It does this by defining a set of *easyconfig parameters*, which tell EasyBuild the name and version +of the software to install, which [toolchain](terminology.md#toolchains) and [easyblock](terminology.md#easyblocks) to use, etc. + +For each software installation performed by EasyBuild, there is a corresponding easyconfig file. +There typically are only a handful of configuration files used however, for example a system-level +configuration file, perhaps combined with a user-level one. Or there may be no configuration files involved +at all, since EasyBuild can also be configured through other mechanisms: environment variables and command line +options. + +--- + +### `$EASYBUILD_*` environment variables + +A particularly easy way to configure EasyBuild is through *environment variables*. + +At startup, EasyBuild will pick up any environment variable of which the name starts with '`EASYBUILD_`'. +For each of these, it will determine the corresponding configuration setting (or exit with an error if +none was found). + +Mapping the name of a configuration setting to the name of the corresponding environment variable is straightforward: +use capital letters, replace dashes (`-`) with underscores (`_`), and prefix with `EASYBUILD_`. + +For example: the `module-syntax` configuration setting can be specified by defining +the `$EASYBUILD_MODULE_SYNTAX` environment variable: + +```shell +export EASYBUILD_MODULE_SYNTAX=Tcl +``` + +*Configuring via environment variables is especially practical for controlling the EasyBuild configuration +in a more dynamic way. For example, you can implement a simple shell script that defines `$EASYBUILD_*` +environment variables based on the current context (user, hostname, other environment variables), and +configure EasyBuild through [sourcing](https://bash.cyberciti.biz/guide/Source_command) it.* + +!!! note + Keep in mind that environment variables are only defined for the shell session you are currently working in. + If you want to configure EasyBuild through environment variables in a more persistent way, + you can leverage one of the [shell startup scripts](https://bash.cyberciti.biz/guide/Startup_scripts) (for example `$HOME/.bash_profile` or `$HOME/.bashrc`). + +--- + +### `eb` command line options + +Finally, you can also configure EasyBuild by specifying one or options to the `eb` command. + +As mentioned earlier, the values for configuration settings defined this way override the value that +is specified through any other means. So if you want to be sure that a particular configuration setting +is defined the way you want it to be, you can use the corresponding command line option. + +There are various configuration settings for which it only makes sense to use the command line option. +An example of this is letting the `eb` command print the EasyBuild version (via `eb --version`). +Although you could configure EasyBuild to always print its version and then exit whenever the `eb` command is +run, that would not be very useful... + +*Command line options are typically used to define configuration settings that are only relevant to +that particular EasyBuild session. One example is doing a test installation into a temporary directory:* + +```shell +eb --installpath /tmp/$USER example.eb +``` + +## Inspecting the current configuration (`--show-config`) + +Given the large amount of available configuration settings in EasyBuild and the different configuration levels, +you can easily lose track of exactly how EasyBuild is configured. + +Through the `--show-config` command line option you can +easily inspect the currently active EasyBuild configuration. + +The output of `--show-config` includes a sorted list of all configuration settings that are defined to a +*non-default* value, +along with a couple of important ones that are always shown (like `buildpath`, `installpath`, `sourcepath`, and so on). +In addition, it also indicates at which configuration level each setting was defined, +so you can trace down *where* it was defined if needed. + +This is the output produced by `eb --show-config` for the default EasyBuild configuration, +where EasyBuild was installed via `pip install --user` (which results in the value shown for the +`robot-paths` configuration setting): + +```shell +# +# Current EasyBuild configuration +# (C: command line argument, D: default value, E: environment variable, F: configuration file) +# +buildpath (D) = /home/example/.local/easybuild/build +containerpath (D) = /home/example/.local/easybuild/containers +installpath (D) = /home/example/.local/easybuild +repositorypath (D) = /home/example/.local/easybuild/ebfiles_repo +robot-paths (D) = /home/example/.local/easybuild/easyconfigs +sourcepath (D) = /home/example/.local/easybuild/sources +``` + +As shown here, all configuration settings shown follow the default `prefix` value (`$HOME/.local/easybuild`), +and none of the values diverge from the default value, since all entries are marked with `(D)` for "default value"). + +**Example** + +Now let us do some basic configuring and inspect the resulting output of `--show-config`. + +First, create a user-level EasyBuild configuration file to define the `prefix` configuration setting: + +```shell +mkdir -p $HOME/.config/easybuild +echo '[config]' > $HOME/.config/easybuild/config.cfg +echo 'prefix=/apps' >> $HOME/.config/easybuild/config.cfg +``` + +In addition, define the `buildpath` configuration setting using the corresponding +environment variable: + +```shell +export EASYBUILD_BUILDPATH=/tmp/$USER +``` + +Then run `--show-config` while you specify that the `installpath` configuration +setting should be defined as `/tmp/$USER`: + +```shell +$ eb --installpath=/tmp/$USER --show-config +# +# Current EasyBuild configuration +# (C: command line argument, D: default value, E: environment variable, F: configuration file) +# +buildpath (E) = /tmp/easybuild +containerpath (F) = /apps/containers +installpath (C) = /tmp/easybuild +packagepath (F) = /apps/packages +prefix (F) = /apps +repositorypath (F) = /apps/ebfiles_repo +robot-paths (D) = /home/example/.local/easybuild/easyconfigs +sourcepath (F) = /apps/sources +``` + +The output indicates that the `installpath` setting was specified through a command line option (indicated +with `(C)`), that the `buildpath` setting was defined via an environment +variable (indicated with `(E)`), that the `robot-paths` setting still has the default value (indicated with `(D)`), and that all other configuration +settings were specified via a configuration file, some of which indirectly through the `prefix` value (indicated with +`(F)`). + +--- + +## Exercises (optional) + +*These exercises are not very relevant for LUMI as LUMI already offers a complete configuration +also for user installations of software. However, if you are a very advanced user, you may still +want to make changes to that configuration, and all three options (configuration files, +environment variables and command line parameters) are available to users in the LUMI setup.* + +***Exercise C.1* - Configure EasyBuild** + +Configure EasyBuild to use the `easybuild` subdirectory in your home directory for everything, except for: + +* The location of the build directories: use `/tmp/$USER` for this; +* The locations that should be considered when searching for source files: + include both `$HOME/easybuild/sources` and `/easybuild/sources`, but make + sure that source files that are downloaded by EasyBuild are stored in + `$HOME/easybuild/sources`. + +Leave other configuration settings set to their default value. + +??? success "(click to show solution)" + + This is pretty straightforward. + + Here we just define the corresponding environment variables: + + ```shell + export EASYBUILD_PREFIX=$HOME/easybuild + export EASYBUILD_BUILDPATH=/tmp/$USER + export EASYBUILD_SOURCEPATH=$HOME/easybuild/sources:/easybuild/sources + ``` + + The location where EasyBuild should download source files to + must be listed first in the `sourcepath` configuration setting. + + The output of `--show-config` should look like this: + + ```shell + buildpath (E) = /tmp/example + containerpath (E) = /home/example/easybuild/containers + installpath (E) = /home/example/easybuild + packagepath (E) = /home/example/easybuild/packages + prefix (E) = /home/example/easybuild + repositorypath (E) = /home/example/easybuild/ebfiles_repo + robot-paths (D) = /home/example/easybuild/easyconfigs + sourcepath (E) = /home/example/easybuild/sources:/easybuild/sources + ``` + +--- + +***Exercise C.2* - Install a trivial software package with EasyBuild** + +Try running the following command: + +```shell +eb bzip2-1.0.6.eb +``` + +Where do you expect to find the installation? + +??? success "(click to show solution)" + + The software was installed in `$HOME/easybuild`, + since that's how we configured EasyBuild in *Exercise 3.1*: + + ```shell + $ ls $HOME/easybuild + ebfiles_repo modules software sources + ``` + + The actual installation is in `$HOME/easybuild/software`, + while the module file was generated in `$HOME/easybuild/modules/all`: + + ```shell + $ ls $HOME/easybuild/software + bzip2 + $ ls $HOME/easybuild/software/bzip2 + 1.0.6 + $ ls $HOME/easybuild/software/bzip2/1.0.6 + bin easybuild include lib man + ``` + + ```shell + $ ls $HOME/easybuild/modules/all + bzip2 + $ ls $HOME/easybuild/modules/all/bzip2 + 1.0.6.lua + ``` + + The source file for bzip2 1.0.6 was downloaded to `$HOME/easybuild/sources`: + + ```shell + $ ls $HOME/easybuild/sources/b/bzip2 + bzip2-1.0.6.tar.gz + ``` + + We will discuss this in more detail in the next part of the tutorial. + +--- + +*[[next: Basic usage]](1_08_basic_usage.md)* diff --git a/docs/2022-CSC_and_LO/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_08_basic_usage.md new file mode 100644 index 00000000..5566d376 --- /dev/null +++ b/docs/2022-CSC_and_LO/1_08_basic_usage.md @@ -0,0 +1,976 @@ +# Basic usage + +*[[back: Configuring EasyBuild]](1_07_configuration.md)* + +--- + +Now that we have installed and configured EasyBuild, we can start using it for what it is intended for: +getting scientific software installed without breaking a sweat, or having to resist the urge to +shout out four-letter words. + +We will look at the high-level workflow first, and then cover each aspect in more detail. + +A couple of exercises are available at the end to help to make you more familiar with the +EasyBuild command line interface, so pay attention! + +--- + +## Overall workflow + +Installing software with EasyBuild is as easy as specifying to the **`eb` command** what we +want to install, and then sitting back to enjoy a coffee or tea (or whatever beverage you prefer). + +This is typically done by **specifying the name of one or more easyconfig files**, often combined +with the `--robot` option to let EasyBuild also install missing dependencies. + +It is recommended to first assess the current situation before letting EasyBuild install the software, +and to check which **dependencies** are already installed and which are still missing. In addition, +you may want to inspect the specifics of the **installation procedure** that will be performed by EasyBuild, +and ensure that the active EasyBuild configuration is what it should be. + +--- + +## Specifying easyconfigs + +Letting EasyBuild know what should be installed can be done by specifying one or more easyconfig files, +which is also the most common way. Alternative methods like using the `--software-name` option won't be +covered in this tutorial, since they are not commonly used. + +Arguments passed to the `eb` command, being anything that is *not* an option (which starts with `-` or `--`) or +is a value for a preceding configuration option, are assumed to refer to easyconfig files (with some exceptions). +These could be: + +* the *(absolute or relative) path* to an easyconfig file; +* the *name* of an easyconfig file; +* the path to a *directory* containing easyconfig files; + +Specified paths to files must of course point to existing files; if not, EasyBuild will print an appropriate error message: + +```shell +$ eb /tmp/does_not_exist.eb +ERROR: Can't find path /tmp/does_not_exist.eb +``` + +When only the *name* of an easyconfig file is specified, EasyBuild will automatically try and locate it. +First, it will consider the *current directory*. If no file with the specified name is found there, +EasyBuild will search for the easyconfig file in the [robot search path](../configuration/#robot-search-path). + +If the path to an existing *directory* is provided, EasyBuild will walk through the entire directory +(including all subdirectories), retain all files of which the name ends with `.eb`, and (try to) use these +as easyconfig files. + + +#### Example command + +Suppose we have the current situation in our home directory: + +* two (easyconfig) files named `example1.eb` and `example2.eb`; +* a subdirectory named `some_deps`, which has two easyconfig files `dep1.eb` and `dep2.eb` + alongside a text file named `list.txt`; +* a subdirectory named `more_deps` located *in* the `some_deps` subdirectory, + which contains another easyconfig file `dep3.eb`; + +Or, visually represented: + +```shell +example1.eb +example2.eb +some_deps/ +|-- dep1.eb +|-- dep2.eb +|-- list.txt +|-- more_deps/ + |-- dep3.eb +``` + +In this context, we run the following EasyBuild command from our home directory: + +```shell +eb example1.eb bzip2-1.0.6.eb $HOME/example2.eb some_deps +``` + +EasyBuild will interpret each of these arguments as follows: + +* `example1.eb` is the name of a file in the current directory, so it can be used directly; +* `bzip2-1.0.6.eb` is the name of an easyconfig file to locate via the robot search path + (since it does not exist in the current directory); +* `$HOME/example2.eb` specifies the full path to an existing file, which can be used directly; +* `some_deps` is the relative path to an existing directory, so EasyBuild will scan it and find three + easyconfig files: `some_deps/dep1.eb`, `some_deps/dep2.eb` and `some_deps/more_deps/dep3.eb`, + The `list.txt` file will be ignored since its name does not end with `.eb`. + +--- + +### Easyconfig filenames + +Note that the `eb` command does not care how easyconfig files are named, at least to some extent: the `.eb` +file extension *does* matter when easyconfig files are being picked up in subdirectories. + +File names for easyconfigs being mostly irrelevant is only correct with respect to the arguments passed to +the `eb` command however. As we will learn soon, the name of easyconfig files *does* matter (a lot) +when EasyBuild needs to locate easyconfigs that can be used to resolve a specified dependency +(see [here](#enabling-dependency-resolution)). + +This explains why easyconfig files usually adher to a very specific naming scheme, +corresponding to `--.eb`, where: + +* `` represents the software name; +* `` represents the software version; +* `` represents the toolchain used in the easyconfig file, which consists of the toolchain name + and version separated with a dash (`-`), and which is omitted (including the preceding `-`) when the + [`system` toolchain](../introduction#system-toolchain) is used; +* `` represents the value of the `versionsuffix` easyconfig parameter, + which is sometimes used to distinguish multiple variants of particular software installations + (and is empty by default); + +--- + +### Searching for easyconfigs + +You will frequently need to determine the exact name of an easyconfig file you want to install, +or just check which easyconfigs are available for a given software package. +This can be done by searching for easyconfigs using **`eb --search`** or **`eb -S`**. + +By default all directories listed in the [robot search path](../configuration#robot-search-path) will be +searched. If you want to search in additional directories without changing the robot search path, +you can use the `search-paths` configuration setting, or you can change the robot search path via either +the `robot` or `robot-paths` configuration options. + +Both the `--search` and `-S` options trigger the same search operation, but yield different output: +`eb --search` will print the full path to each easyconfig file that matches the specified search pattern, +while `eb -S` produces a more concise output. + +For example, let's check which easyconfig files are available for OpenFOAM 8 with a `foss` toolchain: + +```shell +$ eb --search openfoam-8-foss + * /home/example/.local/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-8-foss-2020a.eb + * /home/example/.local/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-8-foss-2020b.eb +``` + +The output is a bit more condensed when using `eb -S`: + +```shell +$ eb -S openfoam-8-foss +CFGS1=/home/example/.local/easybuild/easyconfigs/o/OpenFOAM + * $CFGS1/OpenFOAM-8-foss-2020a.eb + * $CFGS1/OpenFOAM-8-foss-2020b.eb +``` + +Note that the search is performed *case-insensitive*. + +The search pattern can include wildcards like `.*` and/or character groups like `[0-9]`, +or other special characters like `^` or `$` to mark the start/end of the filename, +but you need to be careful that `bash` does not expand these before the `eb` command is started, +so it is recommended to wrap the search pattern in single quotes (`'...'`) when using wildcards. + +For example, to check which easyconfigs are available to install TensorFlow 2.4.1 with the `2020b` version of a toolchain: + +```shell +$ eb -S '^tensorflow-2.4.1.*2020b' +CFGS1=/home/example/.local/easybuild/easyconfigs/t/TensorFlow + * $CFGS1/TensorFlow-2.4.1-foss-2020b.eb + * $CFGS1/TensorFlow-2.4.1-fosscuda-2020b.eb +``` + +--- + +#### Search index + +When searching for easyconfig files, you may see a message like this pop up: + +``` +== found valid index for , so using it... +``` + +This indicates that a **search index** was used for this particular directory which significantly speeds +up the search procedure, especially when the easyconfig files are located on a shared filesystem where metadata +operations involving lots of (small) files can be slow. + +For the easyconfig files included with an EasyBuild release, a search index is readily provided. +For other directories, you can create a search index using `eb --create-index `. +[See the EasyBuild documentation for more information](https://easybuild.readthedocs.io/en/latest/Easyconfigs_index.html). + +--- + +## Inspecting easyconfigs + +Once you have determined the name of the easyconfig file that corresponds to the software you want to install, +you may want to take a closer look at its contents before employing it. + +Since easyconfig files are simple text files (in Python syntax), you could use the ubiquitous `cat` command +or your favorite text editor (`vim`, what else). To avoid that you need to locate the easyconfig file first +and copy-paste the full path to it, you can use **`eb --show-ec`**. + +For example, let's inspect the contents of the `bzip2-1.0.6.eb` easyconfig file: + +```shell +$ eb --show-ec bzip2-1.0.6.eb +== temporary log file in case of crash /tmp/eb-jnpzclhl/easybuild-e37cbrj1.log +== Contents of /home/example/.local/easybuild/easyconfigs/b/bzip2/bzip2-1.0.6.eb: +name = 'bzip2' +version = '1.0.6' + +homepage = 'https://sourceware.org/bzip2' +description = """bzip2 is a freely available, patent free, high-quality data compressor. It typically +compresses files to within 10% to 15% of the best available techniques (the PPM family of statistical +compressors), whilst being around twice as fast at compression and six times faster at decompression.""" + +toolchain = SYSTEM +toolchainopts = {'pic': True} + +source_urls = ['https://sourceware.org/pub/bzip2/'] +sources = [SOURCE_TAR_GZ] +patches = ['bzip2-%(version)s-pkgconfig.patch'] +checksums = [ + 'a2848f34fcd5d6cf47def00461fcb528a0484d8edef8208d6d2e2909dc61d9cd', # bzip2-1.0.6.tar.gz + '5a823e820b332eca3684416894f58edc125ac3dace9f46e62f98e45362aa8a6d', # bzip2-1.0.6-pkgconfig.patch +] + +buildopts = "CC=gcc CFLAGS='-Wall -Winline -O3 -fPIC -g $(BIGFILES)'" + +# building of shared libraries doesn't work on OS X (where 'gcc' is actually Clang...) +with_shared_libs = OS_TYPE == 'Linux' + +moduleclass = 'tools' + +== Temporary log file(s) /tmp/eb-jnpzclhl/easybuild-e37cbrj1.log* have been removed. +== Temporary directory /tmp/eb-jnpzclhl has been removed. +``` + +We'll get back to what all of this means later... + +--- + +## Checking dependencies + +!!! note + In some of the examples below, we assume that some software is already installed with EasyBuild. + + If you are following hands-on in a prepared environment, make sure these installations are visible + in the output of "`module avail`". + + +Before kicking off an installation, it is good practice to check which of the required dependencies +are already installed, and which ones are still missing. + +This can be helpful to ensure that your EasyBuild configuration is set up correctly, +and to prevent from accidentally installing an entirely new software stack from scratch +in an unintended location (like `$HOME/.local/easybuild`, the +[default installation path](../configuration/#install-path)). + +--- + +### Dry run + +To get a complete overview of all required dependencies, and see which ones are already installed +and which ones aren't yet, you can use **`eb --dry-run`**. + +Since `--dry-run` produces rather verbose output including the full path to each easyconfig file, +there is a more concise equivalent available as well: `eb --dry-run-short`, which is equivalent with **`eb -D`**. + +For example, to check which of the required dependencies for `SAMtools-1.11-GCC-10.2.0.eb` are already installed: + +```shell +$ eb SAMtools-1.11-GCC-10.2.0.eb -D +== temporary log file in case of crash /tmp/eb-x4qofiph/easybuild-ehhi9fb1.log +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +Dry run: printing build status of easyconfigs and dependencies +CFGS=/home/example/.local/easybuild/easyconfigs + ... + * [x] $CFGS/b/bzip2/bzip2-1.0.8-GCCcore-10.2.0.eb (module: bzip2/1.0.8-GCCcore-10.2.0) + * [x] $CFGS/x/XZ/XZ-5.2.5-GCCcore-10.2.0.eb (module: XZ/5.2.5-GCCcore-10.2.0) + * [x] $CFGS/c/cURL/cURL-7.72.0-GCCcore-10.2.0.eb (module: cURL/7.72.0-GCCcore-10.2.0) + * [x] $CFGS/g/GCC/GCC-10.2.0.eb (module: GCC/10.2.0) + * [x] $CFGS/n/ncurses/ncurses-6.2-GCCcore-10.2.0.eb (module: ncurses/6.2-GCCcore-10.2.0) + * [ ] $CFGS/s/SAMtools/SAMtools-1.11-GCC-10.2.0.eb (module: SAMtools/1.11-GCC-10.2.0) +``` + +(We've trimmed the output a bit here, for the sake of brevity.) + +This output tells us that all dependencies required by `SAMtools-1.11-GCC-10.2.0.eb` are already installed, +since they are all marked with `[x]`, whereas the easyconfig for `SAMtools` itself is not installed yet, +as indicated by lack of an `x` in `[ ]`. + +--- + +### Missing dependencies + +If you are only interested in which dependencies are still *missing*, +you can consult the output of **`eb --missing`**, or the equivalent **`eb -M`**. + +For example, let's see which dependencies are missing in order to get version 3.1.0 of the h5py +Python package installed using the `2020b` version of the `foss` toolchain: + +```shell +$ eb h5py-3.1.0-foss-2020b.eb -M + +2 out of 61 required modules missing: + +* pkgconfig/1.5.1-GCCcore-10.2.0-python (pkgconfig-1.5.1-GCCcore-10.2.0-python.eb) +* h5py/3.1.0-foss-2020b (h5py-3.1.0-foss-2020b.eb) +``` + +That should be pretty self-explanatory: out of the 63 required dependencies (which includes the `foss` toolchain +and everything needed to install it), only 2 dependencies are missing. Great! + +--- + +## Inspecting install procedures + +Finally, before actually installing something you may want to assess *how* exactly EasyBuild is going +to install the software. + +Perhaps you don't trust EasyBuild yet (you will eventually though, hopefully), +or maybe you just want to double check that you have made the right choice before going through with +the actual installation. + +Using **`eb --extended-dry-run`**, or just **`eb -x`** for short, +you can get a **detailed overview of the installation procedure that would be performed by EasyBuild**, +**in a matter of seconds**. + +By means of example, let's inspect some parts of the installation procedure for `Boost-1.74.0-GCC-10.2.0.eb`: + +```shell +$ eb Boost-1.74.0-GCC-10.2.0.eb -x +... + +preparing... [DRY RUN] + +[prepare_step method] +Defining build environment, based on toolchain (options) and specified dependencies... + +Loading toolchain module... + +module load GCC/10.2.0 + +Loading modules for dependencies... + +module load bzip2/1.0.8-GCCcore-10.2.0 +module load zlib/1.2.11-GCCcore-10.2.0 +module load XZ/5.2.5-GCCcore-10.2.0 + +... + +Defining build environment... + + ... + export CXX='mpicxx' + export CXXFLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno -fPIC' + ... + +configuring... [DRY RUN] + +[configure_step method] + running command "./bootstrap.sh --with-toolset=gcc --prefix=/tmp/example/Boost/1.74.0/GCC-10.2.0/obj --without-libraries=python,mpi" + (in /tmp/example/build/Boost/1.74.0/GCC-10.2.0/Boost-1.74.0) + +... + +[sanity_check_step method] +Sanity check paths - file ['files'] + * lib/libboost_system.so + * lib/libboost_thread-mt-x64.so +Sanity check paths - (non-empty) directory ['dirs'] + * include/boost +Sanity check commands + (none) + +... +``` + +We've obviously trimmed the generated output a bit, but it should be sufficient. + +An overview of the installation procedure is shown, following the installation steps as they would be +performed by EasyBuild. The output above shows: + +* how the build environment will be set up during the `prepare` step, by loading the module for both the + toolchains and the dependencies, and defining a set of environment variables like `$CXX`, `$CXXFLAGS`, etc. +* which command will be executed during the configuration step, and in which directory; +* the list of files and directories that will be checked during the sanity check step; + +If you were concerned about EasyBuild being too much of a black box, that is hopefully resolved now. + +!!! note + It is important to highlight here that the reported installation procedure *may* not be 100% correct, + since the [easyblock](../introduction/#easyblocks) can change its mind based on the output of shell commands + that were executed, or based on the contents of a file that was generated during the installation. + Since all "actions" that would be performed during the installation are actually skipped when using `eb -x`, + the reported installation procedure could be partially incorrect. + + In addition, the easyblock may trip over the fact that the installation procedure is not actually being + executed, which sometimes leads to an unexpected error. These situations are handled gracefully by `eb -x` + though, which will report any errors that occurred and then continue anyway with inspecting the remainder of the + installation procedure. Although this obviously limits the value of the generated output, + it doesn't make it completely useless. + +--- + +## Installing software + +You should now be able to make an informed decision on which easyconfig file you should use to +make EasyBuild install the software you require. + +As mentioned before, installing an easyconfig is as simple as passing it to the `eb` command. + +So, let's try to install SAMtools version 1.11: + +```shell +$ eb SAMtools-1.11-GCC-10.2.0.eb +== temporary log file in case of crash /tmp/eb-zh7_fyre/easybuild-4q_lo57b.log +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +== processing EasyBuild easyconfig /home/example/.local/easybuild/easyconfigs/s/SAMtools/SAMtools-1.11-GCC-10.2.0.eb +== building and installing SAMtools/1.11-GCC-10.2.0... +== fetching files... +== creating build dir, resetting environment... +== unpacking... +== patching... +== preparing... +== configuring... +== building... +== testing... +== installing... +== taking care of extensions... +== restore after iterating... +== postprocessing... +== sanity checking... +== cleaning up... +== creating module... +== permissions... +== packaging... +== COMPLETED: Installation ended successfully (took 17 sec) +== Results of the build can be found in the log file(s) /home/example/easybuild/software/SAMtools/1.11-GCC-10.2.0/easybuild/easybuild-SAMtools-1.11-20210309.105601.log +== Build succeeded for 1 out of 1 +== Temporary log file(s) /tmp/eb-zh7_fyre/easybuild-4q_lo57b.log* have been removed. +== Temporary directory /tmp/eb-zh7_fyre has been removed. +``` + +That was... easy. Is that really all there is to it? Well, almost... + +### Enabling dependency resolution + +The SAMtools installation worked like a charm, but remember that all required dependencies were already +available (see [above](#dry-run)). + +If we try this with the `BCFtools-1.11-GCC-10.2.0.eb`, for which the required `GSL` and `HTSlib` dependencies are not available yet, it's less successful: + +```shell +$ eb BCFtools-1.11-GCC-10.2.0.eb -M + +3 out of 23 required modules missing: + +* GSL/2.6-GCC-10.2.0 (GSL-2.6-GCC-10.2.0.eb) +* HTSlib/1.11-GCC-10.2.0 (HTSlib-1.11-GCC-10.2.0.eb) +* BCFtools/1.11-GCC-10.2.0 (BCFtools-1.11-GCC-10.2.0.eb) +``` + +```shell +$ eb BCFtools-1.11-GCC-10.2.0.eb +... +== preparing... +== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/build/BCFtools/1.11/GCC-10.2.0): build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0 (took 2 sec) +== Results of the build can be found in the log file(s) /tmp/eb-3v1dfvnk/easybuild-BCFtools-1.11-20210308.195024.FlxkH.log +ERROR: Build of /home/example/.local/easybuild/easyconfigs/b/BCFtools/BCFtools-1.11-GCC-10.2.0.eb failed (err: 'build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0') +``` + +Oh my, what's this all about? + +If we filter the output a bit and focus on the actual error, the problem is clear: + +``` +Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0 +``` + +The required dependencies `HTSlib/1.11-GCC-10.2.0` and `GSL/2.6-GCC-10.2.0` are not installed yet, +and EasyBuild does not automatically install missing dependencies unless it is told to do so. + +It helpfully suggests to use the `--robot` command line option, so let's try that: + +```shell +$ eb BCFtools-1.11-GCC-10.2.0.eb --robot +... +== resolving dependencies ... +... +== building and installing HTSlib/1.11-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 13 sec) +... +== building and installing GSL/2.6-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 1 min 10 sec) +... +== building and installing BCFtools/1.11-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 8 sec) +... +== Build succeeded for 3 out of 3 +``` + +With dependency resolution enabled the `HTSlib` and `GSL` modules get installed first, +before EasyBuild proceeds with installing `BCFtools`. Great! + +--- + +### Trace output + +As you may have noticed if you tried the previous example hands-on, +some installations take a while. +An installation can be spending quite a bit of time during the build step, but what is actually going on there? + +To provide some more feedback as the installation progresses, you can enable the "`trace`" configuration setting. +Let's do this by defining the `$EASYBUILD_TRACE` environment variable, just to avoid having to type `--trace` +over and over again. + +We will redo the installation of `BCFtools-1.11-GCC-10.2.0.eb` by passing the `--rebuild` +option to the `eb` command (try yourself what happens if you don't use the `--rebuild` option!): + +```shell +$ export EASYBUILD_TRACE=1 +$ eb BCFtools-1.11-GCC-10.2.0.eb --rebuild +... +== configuring... + >> running command: + [started at: 2021-03-08 19:54:53] + [working dir: /tmp/example/build/BCFtools/1.11/GCC-10.2.0/bcftools-1.11] + [output logged in /tmp/eb-9u_ac0nv/easybuild-run_cmd-17m_he2x.log] + ./configure --prefix=/home/example/easybuild/software/BCFtools/1.11-GCC-10.2.0 --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --with-htslib=$EBROOTHTSLIB --enable-libgsl +== building... + >> running command: + [started at: 2021-03-08 19:54:54] + [working dir: /tmp/example/BCFtools/1.11/GCC-10.2.0/bcftools-1.11] + [output logged in /tmp/example/eb-9u_ac0nv/easybuild-run_cmd-bhkgjxi7.log] + make -j 8 + >> command completed: exit 0, ran in 00h00m03s +``` + +That's a bit more comforting to stare at... + +During the *configure* step, the `./configure` command is run with option to +enable support for leveraging `HTSlib` and `GSL`. + +During the *build* step, the software is actually being compiled +by running the `make` command. EasyBuild automatically uses the available cores on the system (in this case 8). + +We even get a pointer to a log file that contains the output of the command being run, +so we can use `tail -f` to see in detail how it progresses. + +Once the `make` command completes, we get a message that the command completed with a exit code 0 +(implying success), and that it took 3 seconds to run. That's good to know. + +Later during the installation, we now also see this output during the sanity check step: + +``` +== sanity checking... + >> file 'bin/bcftools' found: OK + >> file 'bin/plot-vcfstats' found: OK + >> file 'bin/vcfutils.pl' found: OK + >> (non-empty) directory 'libexec/bcftools' found: OK +``` + +Thanks to enabling trace mode, EasyBuild tells us which files & directories it is checking for +in the installation, before declaring it a success. Nice! + +The extra output you get when trace mode is enabled is concise and hence not overwhelming, +while it gives a better insight into what is going on during the installation. +It may also help to spot unexpected actions being taken during the installation early on, +so you can interrupt the installation before it completes, if deemed necessary. + +--- + +## Using installed software + +So far, we have already installed 4 different software packages (SAMtools, HTSlib, GSL, and BCFtools); +we even installed BCFtools twice! + +A lot was going on underneath the covers: locating and unpacking +the source tarballs, setting up the build environment, configuring the build, compiling, +creating and populating the installation directory, performing a quick sanity check on the installation, +cleaning things up, and finally generated the environment module file corresponding to the installation. + +That's great, but how do we now actually *use* these installations? + +This is where the generated module files come into play: they form the access portal to the software +installations, and we'll use the ubiquitous `module` command to digest them. + +First, we need to make sure that the modules tool is aware of where the module files for +our installations are located. If you're unsure where EasyBuild is installing stuff at this point, +check the output of `eb --show-config`; the value of the `installpath` configuration setting is what we are interested in now: + +```shell +$ eb --show-config +... +installpath (E) = /home/example/easybuild +... +repositorypath (E) = /home/example/easybuild/ebfiles_repo +... +sourcepath (E) = /home/example/easybuild/sources +... +``` + +So, what's in this directory? + +```shell +$ ls -l $HOME/easybuild +total 16 +drwxrwxr-x 5 example example 4096 Jun 10 20:11 ebfiles_repo +drwxrwxr-x 5 example example 4096 Jun 10 20:10 modules +drwxrwxr-x 6 example example 4096 Jun 10 20:10 software +drwxrwxr-x 6 example example 4096 Jun 10 20:10 sources +``` + +The `ebfiles_repo` and `sources` directories correspond to the `repositorypath` and `sourcepath` configuration +settings, respectively. The `modules` and `software` directories are what we need now. + +The `modules` subdirectory consists of multiple subdirectories: + +```shell +$ ls $HOME/easybuild/modules +all bio devel numlib tools +``` + +Directories like `bio` and `numlib` correspond to different software categories, +and contain symbolic links to the module files in the `all` directory, +which contains all actual module files for software installed in this EasyBuild installation path. +We'll ignore these separate category directories for now. + +Let's inform the modules tool about the existence of these module files using `"module use"`: + +```shell +module use $HOME/easybuild/modules/all +``` + +This command does little more that updating the `$MODULEPATH` environment variable, +which contains a list of paths that the modules tool should consider when looking for module files. + +Now the modules tool should be aware of our brand new installations: + +```shell +$ module avail + +---------------------- /home/example/easybuild/modules/all ----------------------- + BCFtools/1.11-GCC-10.2.0 GSL/2.6-GCC-10.2.0 SAMtools/1.11-GCC-10.2.0 + EasyBuild/4.3.3 HTSlib/1.11-GCC-10.2.0 bzip2/1.0.6 + +---------------------------- /easybuild/modules/all ----------------------------- + ... +``` + +This output shows both the modules for our own installations as well as the "central" installations in `/easybuild` (which we omitted above for brevity). + +Now we can load these modules and start using these software installations. + +Let's test this for BCFtools. In our current environment, the `bcftools` command is not available yet: + +```shell +$ module list +No modules loaded + +$ bcftools +-bash: bcftools: command not found +``` + +Loading the module for BCFtools changes that: + +```shell +$ module load BCFtools/1.11-GCC-10.2.0 + +$ module list +Currently Loaded Modules: + 1) GCCcore/10.2.0 6) XZ/5.2.5-GCCcore-10.2.0 + 2) zlib/1.2.11-GCCcore-10.2.0 7) cURL/7.72.0-GCCcore-10.2.0 + 3) binutils/2.35-GCCcore-10.2.0 8) HTSlib/1.11-GCC-10.2.0 + 4) GCC/10.2.0 9) GSL/2.6-GCC-10.2.0 + 5) bzip2/1.0.8-GCCcore-10.2.0 10) BCFtools/1.11-GCC-10.2.0 + +$ bcftools --version +bcftools 1.11 +Using htslib 1.11 +... +``` + +Note that the modules for the required dependencies, including the compiler toolchain (which provides runtime libraries +like `libstdc++.so`), are loaded automatically. The "`module load`" command changes the active environment, +by updating environment variables like `$PATH` for example, to make the software available for use. + +##### Resetting your environment + +To restore your environment to a pristine state in which no modules are loaded, you can either +unload the loaded modules one by one using "`module unload`", or you can unload all of them at once using +"`module purge`". + +**If you are using an EasyBuild installation provided by a module, +don't forget to load the `EasyBuild` module again after running "`module purge`".** + +--- + +## Stacking software + +Maybe you have overlooked how the software we are playing around with was not only installed across multiple +different installation directories per software, we are also "stacking" our own installations (in `$HOME/easybuild`) +on top of installations that are provided in a totally different location (`/easybuild`). + +**EasyBuild doesn't care *where* software is installed: as long as the module file that provides access to it +is available, it is happy to pick it up and use it when required.** + +This implies that end users of an HPC system can easily install their +own small software stack on top of what is provided centrally by the HPC support team, +for example. They can even +"replace" a central software installation for their purposes if they need to, since the modules tool will +load the first module file that matches the request being made (there are some caveats with this, but we +won't go into those here). + +--- + +## Exercises + +***Guidelines*** + +Do yourself a favor: don't peek at the solution until you have made an attempt to solve the exercise yourself! + +Please do not spoil solutions to others before they have been discussed by the tutorial organisers. + +The exercises are based on the easyconfig files included with EasyBuild 4.3.3. + +--- + +***Exercise U.0**** - Making installed software available* + +Before working on the exercises for this part of the tutorial, +make sure that the software that is already installed in the prepared environment is available. + +We will assume that you have a small software stack installed using the `2020b` version of the `foss` toolchain. + +**Tip:** execute a "`module use`" command, and verify with "`module avail`" that a bunch of software modules +are available for loading. + +??? success "(click to show solution)" + + Use the following command to make the modules for the software available that is pre-installed + in the prepared environment: + ```shell + module use /easybuild/modules/all + ``` + + If software is installed in a different location than `/easybuild/` in your environment, + you should adjust the command accordingly. + + +--- + +***Exercise U.1**** - Searching easyconfigs* + +See if EasyBuild provides any easyconfig files for installing GROMACS version 2020/5. + +??? success "(click to show solution)" + To check for available easyconfig files, we can use `eb --search` or `eb -S`: + ```shell + $ eb -S gromacs-2020.5 + == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + CFGS1=/home/example/.local/easybuild/easyconfigs/g/GROMACS + * $CFGS1/GROMACS-2020.5-fosscuda-2020a-Python-3.8.2.eb + * $CFGS1/GROMACS-2020.5_fix_threads_gpu_Gmxapitests.patch + ``` + This actually shows one easyconfig file but also a patch file. We can also search specifically + for GROMACS 2020.5 in the `foss` and `fosscuda` toolchains using + ```shell + $ eb -S gromacs-2020.5-foss + == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + CFGS1=/home/example/.local/easybuild/easyconfigs/g/GROMACS + * $CFGS1/GROMACS-2020.5-fosscuda-2020a-Python-3.8.2.eb + ``` + and now we find a single easyconfig file. + +--- + +***Exercise U.2**** - Checking dependencies* + +Check which dependencies are missing to install QuantumESPRESSO version 6.6 with the `2020b` version of the `foss` toolchain. + +??? success "(click to show solution)" + First, we need to determine the name of the easyconfig file for QuantumESPRESSO version 6.6: + ```shell + $ eb -S 'QuantumESPRESSO-6.6.*foss-2020b' + == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + CFGS1=/home/example/.local/easybuild/easyconfigs/q/QuantumESPRESSO + * $CFGS1/QuantumESPRESSO-6.6-foss-2020b.eb + ``` + To determine which dependencies are missing to install this QuantumESPRESSO easyconfig file, we can use `--missing`: + ```shell + $ eb QuantumESPRESSO-6.6-foss-2020b.eb --missing + + 3 out of 58 required modules missing: + + * libxc/4.3.4-GCC-10.2.0 (libxc-4.3.4-GCC-10.2.0.eb) + * ELPA/2020.11.001-foss-2020b (ELPA-2020.11.001-foss-2020b.eb) + * QuantumESPRESSO/6.6-foss-2020b (QuantumESPRESSO-6.6-foss-2020b.eb) + ``` + (some nonessential output removed). + +--- + +***Exercise U.3**** - Performing a dry run* + +Figure out which command EasyBuild would use to compile +the software provided by the `Bowtie2-2.4.2-GCC-9.3.0.eb` easyconfig file, +without actually installing `Bowtie2`. + +Also, which binaries will EasyBuild check for to sanity check the installation? + +??? success "(click to show solution)" + To inspect the installation procedure, we can use `eb -x Bowtie2-2.4.2-GCC-9.3.0.eb`. + + The output for the build step shows the actual compilation command that would be performed (`make ...`): + + ```shell + [build_step method] + >> running command: + [started at: 2021-03-08 20:15:08] + [working dir: /local_scratch/hkenneth/eb-1wodfohg/__ROOT__/local_scratch/hkenneth/Bowtie2/2.4.2/GCC-9.3.0/Bowtie2-2.4.2] + [output logged in /local_scratch/hkenneth/eb-1wodfohg/easybuild-run_cmd-haojzisn.log] + make -j 48 CC="gcc" CPP="g++" CXX="g++" RELEASE_FLAGS="-O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -std=gnu++98" + (in /local_scratch/hkenneth/Bowtie2/2.4.2/GCC-9.3.0/Bowtie2-2.4.2) + ``` + + If the output you get is less detailed, you may not have set `export EASYBUILD_TRACE=1`. + + The output for the sanity check step shows which binaries are expected to be installed: + ``` + [sanity_check_step method] + Sanity check paths - file ['files'] + * bin/bowtie2 + * bin/bowtie2-align-l + * bin/bowtie2-align-s + * bin/bowtie2-build + * bin/bowtie2-build-l + * bin/bowtie2-build-s + * bin/bowtie2-inspect + * bin/bowtie2-inspect-l + * bin/bowtie2-inspect-s + ``` + +--- + +***Exercise U.4**** - Installing software* + +Install version 3.1.0 of the `h5py` Python package and all missing dependencies, +using the `foss/2020b` toolchain, into `/tmp/$USER/easybuild`, +while leveraging the already installed software available from `/easybuild`. + +Enable trace output so you can see which parts of the installation take a while. + +??? success "(click to show solution)" + First, determine the easyconfig file we can use for this: + ```shell + $ eb -S 'h5py-3.1.0.*foss-2020b' + CFGS1=/home/example/.local/easybuild/easyconfigs/h/h5py + * $CFGS1/h5py-3.1.0-foss-2020b.eb + ``` + + Make sure the pre-install software in `/easybuild/` is available: + ```shell + module use /easybuild/modules/all + ``` + + Check which dependencies are missing to install this `h5py` easyconfig: + ```shell + $ eb h5py-3.1.0-foss-2020b.eb --missing + + 2 out of 63 required modules missing: + + * pkgconfig/1.5.1-GCCcore-10.2.0-python (pkgconfig-1.5.1-GCCcore-10.2.0-python.eb) + * h5py/3.1.0-foss-2020b (h5py-3.1.0-foss-2020b.eb) + ``` + + Install `h5py` by specifying the easyconfig file and enabling dependency resolution via `--robot`, + while indicating that we want to install the software into `/tmp/$USER/easybuild` using the `--installpath` + option. Also make sure that trace mode is enabled by defining the `$EASYBUILD_TRACE` environment variable. + ```shell + $ export EASYBUILD_TRACE=1 + $ eb h5py-3.1.0-foss-2020b.eb --robot --installpath /tmp/$USER/easybuild + ... + == building and installing pkgconfig/1.5.1-GCCcore-10.2.0-python... + ... + == building and installing h5py/3.1.0-foss-2020b... + ... + == building... + >> running command: + [started at: 2020-06-10 21:47:32] + [working dir: /tmp/example/h5py/3.1.0/foss-2020b/h5py-3.1.0] + [output logged in /tmp/eb-rjjkbqe1/easybuild-run_cmd-d_dkc4iz.log] + python setup.py configure --mpi --hdf5=$EBROOTHDF5 && /easybuild/software/Python/3.8.6-GCCcore-10.2.0/bin/python setup.py build + >> command completed: exit 0, ran in 00h01m27s + ... + == COMPLETED: Installation ended successfully (took 2 min 46 sec) + ... + == Build succeeded for 2 out of 2 + ``` + + The trace output shows that most time is spent in the build command, + which runs both `python setup.py configure` and `python setup.py build`. + +--- + +***Exercise U.5**** - Using installed software* + +Using the `h5py` installation from the previous exercise to create an empty HDF5 file, +using the following Python statements: + +```python +import h5py +f = h5py.File("empty.hdf5", "w") +f.close() +``` + +Check the resulting file using the `h5stat` command. + +??? success "(click to show solution)" + First, we need to make the modules tool aware of the module files that were installed into `/tmp/$USER/easybuild`: + ```shell + module use /tmp/$USER/easybuild/modules/all + ``` + + Then we can check the `h5py` module is available, and load it: + ```shell + $ module avail h5py + ------------ /tmp/example/easybuild/modules/all ------------ + h5py/3.1.0-foss-2020b + ``` + + ```shell + module load h5py/3.1.0-foss-2020b + ``` + + The Python code snippet can be run directly on the command line using "`python -c '...'`", since it's tiny: + ```shell + python -c 'import h5py; f = h5py.File("empty.hdf5", "w"); f.close()' + ``` + Of course you can also copy the Python code snippet in a file named `test_h5py.py`, + and then run it with `python test_h5py.py`. + + Checking with the `h5stat` command shows that the resulting `empty.hdf5` is indeed a valid HDF5 file: + ```shell + $ ls -l empty.hdf5 + -rw-rw-r-- 1 example example 800 Jun 10 21:54 empty.hdf5 + + $ h5stat empty.hdf5 + Filename: empty.hdf5 + File information + # of unique groups: 1 + # of unique datasets: 0 + ... + ``` + +--- + +If you've made it through the hands-on exercises, congratulations! + +If not, don't worry too much about it. We covered a lot of ground here, +and it's a lot to take in at once, take your time... + +Feel free to ask question in the `#tutorial` channel in the [EasyBuild +Slack](https://docs.easybuild.io/en/latest/#getting-help), +we're happy to help! + +--- + +*[[next: Part 2: Using EasyBuild]](2_00_part2_using.md)* From b5ea26dc410ddab86a88703cea446905208b76e0 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Mon, 21 Mar 2022 17:33:23 +0100 Subject: [PATCH 134/159] Restructuring continued. --- docs/2022-CSC_and_LO/1_03_CPE.md | 2 +- docs/2022-CSC_and_LO/1_05_terminology.md | 4 ++-- docs/2022-CSC_and_LO/1_06_installation.md | 10 ++++++---- 3 files changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md index 42eb197e..077fbc36 100644 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -281,4 +281,4 @@ redfine ``LD_LIBRARY_PATH``. --- -*[[next: Terminology]](1_04_terminology.md)* +*[[next: LUMI software stacks]](1_04_LUMI_software_stack.md)* diff --git a/docs/2022-CSC_and_LO/1_05_terminology.md b/docs/2022-CSC_and_LO/1_05_terminology.md index 9c643579..8e05012f 100644 --- a/docs/2022-CSC_and_LO/1_05_terminology.md +++ b/docs/2022-CSC_and_LO/1_05_terminology.md @@ -1,6 +1,6 @@ # Terminology -*[[back: The HPE Cray Programming Environment]](1_03_CPE.md)* +*[[back: LUMI software stacks]](1_04_LUMI_software_stack.md)* --- @@ -291,4 +291,4 @@ An **easystack file** can be used to specify a collection of software to install --- -*[[next: Installation]](1_05_installation.md)* +*[[next: Installation]](1_06_installation.md)* diff --git a/docs/2022-CSC_and_LO/1_06_installation.md b/docs/2022-CSC_and_LO/1_06_installation.md index 163262ea..6e68b500 100644 --- a/docs/2022-CSC_and_LO/1_06_installation.md +++ b/docs/2022-CSC_and_LO/1_06_installation.md @@ -1,6 +1,6 @@ # Installing EasyBuild -*[[back: Terminology]](1_04_terminology.md)* +*[[back: Terminology]](1_05_terminology.md)* --- @@ -537,7 +537,9 @@ Check that the installation works by running the verification commands outlined --- -**Make sure you have a working EasyBuild installation before proceeding -with the rest of the tutorial!** +**You'll need a working custom EasyBuild installation for some of the +optional generic EasyBuild exercises, but in general you can use the +installation provided by LUMI.** -*[[next: Configuring EasyBuild]](1_06_configuration.md)* + +*[[next: Configuring EasyBuild]](1_07_configuration.md)* From 555afb715d74760e72c5cea1508c26413adf6c7e Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 30 Mar 2022 12:07:24 +0200 Subject: [PATCH 135/159] Finished reworking the basic usage section, except for the exercises. --- docs/2022-CSC_and_LO/1_08_basic_usage.md | 644 ++++++++++++++++------- 1 file changed, 442 insertions(+), 202 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_08_basic_usage.md index 5566d376..f5031e48 100644 --- a/docs/2022-CSC_and_LO/1_08_basic_usage.md +++ b/docs/2022-CSC_and_LO/1_08_basic_usage.md @@ -21,7 +21,7 @@ Installing software with EasyBuild is as easy as specifying to the **`eb` comman want to install, and then sitting back to enjoy a coffee or tea (or whatever beverage you prefer). This is typically done by **specifying the name of one or more easyconfig files**, often combined -with the `--robot` option to let EasyBuild also install missing dependencies. +with the `--robot` (or `-r`) option to let EasyBuild also install missing dependencies. It is recommended to first assess the current situation before letting EasyBuild install the software, and to check which **dependencies** are already installed and which are still missing. In addition, @@ -53,7 +53,7 @@ ERROR: Can't find path /tmp/does_not_exist.eb When only the *name* of an easyconfig file is specified, EasyBuild will automatically try and locate it. First, it will consider the *current directory*. If no file with the specified name is found there, -EasyBuild will search for the easyconfig file in the [robot search path](../configuration/#robot-search-path). +EasyBuild will search for the easyconfig file in the [robot search path](../1_07_configuration/#robot-search-path). If the path to an existing *directory* is provided, EasyBuild will walk through the entire directory (including all subdirectories), retain all files of which the name ends with `.eb`, and (try to) use these @@ -75,12 +75,12 @@ Or, visually represented: ```shell example1.eb example2.eb -some_deps/ -|-- dep1.eb -|-- dep2.eb -|-- list.txt -|-- more_deps/ - |-- dep3.eb +some_deps +├── deb2.eb +├── dep1.eb +├── list.txt +└── more_deps + └── dep3.eb ``` In this context, we run the following EasyBuild command from our home directory: @@ -118,7 +118,7 @@ corresponding to `--.eb`, where: * `` represents the software version; * `` represents the toolchain used in the easyconfig file, which consists of the toolchain name and version separated with a dash (`-`), and which is omitted (including the preceding `-`) when the - [`system` toolchain](../introduction#system-toolchain) is used; + [`system` toolchain](../1_05_terminology#system-toolchain) is used; * `` represents the value of the `versionsuffix` easyconfig parameter, which is sometimes used to distinguish multiple variants of particular software installations (and is empty by default); @@ -131,7 +131,7 @@ You will frequently need to determine the exact name of an easyconfig file you w or just check which easyconfigs are available for a given software package. This can be done by searching for easyconfigs using **`eb --search`** or **`eb -S`**. -By default all directories listed in the [robot search path](../configuration#robot-search-path) will be +By default all directories listed in the [robot search path](../1_07_configuration#robot-search-path) will be searched. If you want to search in additional directories without changing the robot search path, you can use the `search-paths` configuration setting, or you can change the robot search path via either the `robot` or `robot-paths` configuration options. @@ -202,41 +202,91 @@ Since easyconfig files are simple text files (in Python syntax), you could use t or your favorite text editor (`vim`, what else). To avoid that you need to locate the easyconfig file first and copy-paste the full path to it, you can use **`eb --show-ec`**. +!!! Hint + To follow the examples below on LUMI, load ``LUMI/21.12`` and ``EasyBuild-user`` + (though results may differ or the examples not work anymore as the software installation + on LUMI evolves). + For example, let's inspect the contents of the `bzip2-1.0.6.eb` easyconfig file: ```shell -$ eb --show-ec bzip2-1.0.6.eb -== temporary log file in case of crash /tmp/eb-jnpzclhl/easybuild-e37cbrj1.log -== Contents of /home/example/.local/easybuild/easyconfigs/b/bzip2/bzip2-1.0.6.eb: -name = 'bzip2' -version = '1.0.6' - -homepage = 'https://sourceware.org/bzip2' -description = """bzip2 is a freely available, patent free, high-quality data compressor. It typically -compresses files to within 10% to 15% of the best available techniques (the PPM family of statistical -compressors), whilst being around twice as fast at compression and six times faster at decompression.""" - -toolchain = SYSTEM +$ eb --show-ec bzip2-1.0.8-cpeCray-21.12.eb +== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-53o823qb/easybuild-xn6nmt61.log +== Contents of /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeCray-21.12.eb: +# Contributed by Kurt Lust, LUMI project & UAntwerpen + +local_bzip2_version = '1.0.8' # http://www.bzip.org/downloads.html + +name = 'bzip2' +version = local_bzip2_version + +homepage = 'https://www.sourceware.org/bzip2/' + +whatis = [ + 'Description: bzip2 is a freely available, patent free, high-quality data compressor.', + 'The module contains both executables and libraries.' + 'Keywords: BZ2', +] + +description = """ +bzip2 is a freely available, patent free, high-quality data compressor. It +typically compresses files to within 10% to 15% of the best available techniques +(the PPM family of statistical compressors), whilst being around twice as fast +at compression and six times faster at decompression. It is based on the +Burrows-Wheeler block-sorting text compression algorithm and Huffman coding. +""" + +usage = """ +Check the man pages for the available commands or the web-based documentation for the +library functions. +""" + +docurls = [ + 'Web-based documentation: http://www.bzip.org/docs.html', + 'Man pages available for bzcmp, bzdiff, bzegrep, bzfgrep, bzgrep, bzip2, bunzip2, bzless and bzmore', +] + +toolchain = {'name': 'cpeCray', 'version': '21.12'} toolchainopts = {'pic': True} -source_urls = ['https://sourceware.org/pub/bzip2/'] -sources = [SOURCE_TAR_GZ] -patches = ['bzip2-%(version)s-pkgconfig.patch'] +source_urls = ['https://sourceware.org/pub/%(name)s/'] +sources = [SOURCE_TAR_GZ] +patches = ['bzip2-%(version)s-pkgconfig-manpath.patch'] checksums = [ - 'a2848f34fcd5d6cf47def00461fcb528a0484d8edef8208d6d2e2909dc61d9cd', # bzip2-1.0.6.tar.gz - '5a823e820b332eca3684416894f58edc125ac3dace9f46e62f98e45362aa8a6d', # bzip2-1.0.6-pkgconfig.patch + 'ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269', # bzip2-1.0.8.tar.gz + 'de11269dc6e4917023af4cee9ff83b204953ad9cde561dbc9d6fc70d9b9578e3', # bzip2-1.0.8-pkgconfig-manpath.patch ] -buildopts = "CC=gcc CFLAGS='-Wall -Winline -O3 -fPIC -g $(BIGFILES)'" +builddependencies = [ # Create a reproducible build environment. + ('buildtools', '%(toolchain_version)s', '', True), +] -# building of shared libraries doesn't work on OS X (where 'gcc' is actually Clang...) -with_shared_libs = OS_TYPE == 'Linux' +local_bzip2_major_minor = '.'.join(local_bzip2_version.split('.')[:2]) + +sanity_check_paths = { + 'files': [ 'lib/pkgconfig/bzip2.pc', 'lib/libbz2.a', 'lib/libbz2.%s' % SHLIB_EXT, + 'lib/libbz2.%s.%s' % (SHLIB_EXT, local_bzip2_major_minor), + 'lib/libbz2.%s.%s' % (SHLIB_EXT, local_bzip2_version), + 'include/bzlib.h' ] + + [ 'bin/b%s' % x for x in ['unzip2', 'zcat', 'zdiff', 'zgrep', 'zip2', 'zip2recover', 'zmore'] ] + + [ 'share/man/man1/bz%s.1' % x for x in ['cmp', 'diff', 'egrep', 'fgrep', 'grep', 'ip2', 'less', 'more'] ], + 'dirs': [] +} + +sanity_check_commands = [ + 'bzip2 --help', + 'pkg-config --libs bzip2', +] moduleclass = 'tools' -== Temporary log file(s) /tmp/eb-jnpzclhl/easybuild-e37cbrj1.log* have been removed. -== Temporary directory /tmp/eb-jnpzclhl has been removed. +== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-53o823qb/easybuild-xn6nmt61.log* have been removed. +== Temporary directory /run/user/10012026/easybuild/tmp/eb-53o823qb has been removed. ``` +The output may actually be longer for an easyconfig file that is already installed on the system +as a new easyconfig file is generated in the repository with some information about the installation +added to it and as on LUMI these are at the front of the robot search path to ensure that the system +finds the right easyconfig file matching with a module on the system. We'll get back to what all of this means later... @@ -272,25 +322,30 @@ there is a more concise equivalent available as well: `eb --dry-run-short`, whic For example, to check which of the required dependencies for `SAMtools-1.11-GCC-10.2.0.eb` are already installed: ```shell -$ eb SAMtools-1.11-GCC-10.2.0.eb -D -== temporary log file in case of crash /tmp/eb-x4qofiph/easybuild-ehhi9fb1.log -== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +$ eb SAMtools-1.14-cpeGNU-21.12.eb -D +== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-oo0lj9lq/easybuild-2cyomy8v.log Dry run: printing build status of easyconfigs and dependencies -CFGS=/home/example/.local/easybuild/easyconfigs - ... - * [x] $CFGS/b/bzip2/bzip2-1.0.8-GCCcore-10.2.0.eb (module: bzip2/1.0.8-GCCcore-10.2.0) - * [x] $CFGS/x/XZ/XZ-5.2.5-GCCcore-10.2.0.eb (module: XZ/5.2.5-GCCcore-10.2.0) - * [x] $CFGS/c/cURL/cURL-7.72.0-GCCcore-10.2.0.eb (module: cURL/7.72.0-GCCcore-10.2.0) - * [x] $CFGS/g/GCC/GCC-10.2.0.eb (module: GCC/10.2.0) - * [x] $CFGS/n/ncurses/ncurses-6.2-GCCcore-10.2.0.eb (module: ncurses/6.2-GCCcore-10.2.0) - * [ ] $CFGS/s/SAMtools/SAMtools-1.11-GCC-10.2.0.eb (module: SAMtools/1.11-GCC-10.2.0) +CFGS=/appl/lumi + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeGNU/cpeGNU-21.12.eb (module: cpeGNU/21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ncurses/ncurses-6.2-cpeGNU-21.12.eb (module: ncurses/6.2-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/zlib/zlib-1.2.11-cpeGNU-21.12.eb (module: zlib/1.2.11-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeGNU-21.12.eb (module: bzip2/1.0.8-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12-minimal.eb (module: gettext/0.21-cpeGNU-21.12-minimal) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/XZ/XZ-5.2.5-cpeGNU-21.12.eb (module: XZ/5.2.5-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/Brotli/Brotli-1.0.9-cpeGNU-21.12.eb (module: Brotli/1.0.9-cpeGNU-21.12) + * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cURL/cURL-7.78.0-cpeGNU-21.12.eb (module: cURL/7.78.0-cpeGNU-21.12) + * [ ] $CFGS/LUMI-EasyBuild-contrib/easybuild/easyconfigs/h/HTSlib/HTSlib-1.14-cpeGNU-21.12.eb (module: HTSlib/1.14-cpeGNU-21.12) + * [ ] $CFGS/LUMI-EasyBuild-contrib/easybuild/easyconfigs/s/SAMtools/SAMtools-1.14-cpeGNU-21.12.eb (module: SAMtools/1.14-cpeGNU-21.12) +== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-oo0lj9lq/easybuild-2cyomy8v.log* have been removed. +== Temporary directory /run/user/10012026/easybuild/tmp/eb-oo0lj9lq has been removed. ``` -(We've trimmed the output a bit here, for the sake of brevity.) +This output tells you that most of the dependencies required by ``SAMtools-1.14-cpeGNU-21.12.eb`` are +already installed, since they are marked with ``[x]``. However, the easyconfig files for +``HTSLib-1.14-cpeGNU-21.12.eb`` and SAMtools itself are not installed yet, denoted by the +lack of an ``x`` in ``[ ]``. -This output tells us that all dependencies required by `SAMtools-1.11-GCC-10.2.0.eb` are already installed, -since they are all marked with `[x]`, whereas the easyconfig for `SAMtools` itself is not installed yet, -as indicated by lack of an `x` in `[ ]`. --- @@ -299,20 +354,19 @@ as indicated by lack of an `x` in `[ ]`. If you are only interested in which dependencies are still *missing*, you can consult the output of **`eb --missing`**, or the equivalent **`eb -M`**. -For example, let's see which dependencies are missing in order to get version 3.1.0 of the h5py -Python package installed using the `2020b` version of the `foss` toolchain: +For example, for the SAMtools easyconfig file used in the previous example we get (with +some lines removed from the output): ```shell -$ eb h5py-3.1.0-foss-2020b.eb -M - -2 out of 61 required modules missing: +$ eb SAMtools-1.14-cpeGNU-21.12.eb -M +2 out of 11 required modules missing: -* pkgconfig/1.5.1-GCCcore-10.2.0-python (pkgconfig-1.5.1-GCCcore-10.2.0-python.eb) -* h5py/3.1.0-foss-2020b (h5py-3.1.0-foss-2020b.eb) +* HTSlib/1.14-cpeGNU-21.12 (HTSlib-1.14-cpeGNU-21.12.eb) +* SAMtools/1.14-cpeGNU-21.12 (SAMtools-1.14-cpeGNU-21.12.eb) ``` -That should be pretty self-explanatory: out of the 63 required dependencies (which includes the `foss` toolchain -and everything needed to install it), only 2 dependencies are missing. Great! +That should be pretty self-explanatory: out of the 113 required dependencies (which includes the `cpeGNU` toolchain +and everything needed to install it), only 2 dependencies (including SAMtools itself) are missing. Great! --- @@ -329,52 +383,78 @@ Using **`eb --extended-dry-run`**, or just **`eb -x`** for short, you can get a **detailed overview of the installation procedure that would be performed by EasyBuild**, **in a matter of seconds**. -By means of example, let's inspect some parts of the installation procedure for `Boost-1.74.0-GCC-10.2.0.eb`: +By means of example, let's inspect some parts of the installation procedure for ``HTSLib-1.14-cpeGNU-21.12.eb``: ```shell -$ eb Boost-1.74.0-GCC-10.2.0.eb -x +$ eb HTSlib-1.14-cpeGNU-21.12.eb -x ... -preparing... [DRY RUN] +*** DRY RUN using 'ConfigureMake' easyblock (easybuild.easyblocks.generic.configuremake @ /appl/lumi/SW/LUMI-21.12/common/EB/EasyBuild/4.5.3/lib/python3.6/site-packages/easybuild/easyblocks/generic/configuremake.py) *** + +== building and installing HTSlib/1.14-cpeGNU-21.12... +fetching files... [DRY RUN] + +... [prepare_step method] Defining build environment, based on toolchain (options) and specified dependencies... Loading toolchain module... -module load GCC/10.2.0 +module load cpeGNU/21.12 Loading modules for dependencies... -module load bzip2/1.0.8-GCCcore-10.2.0 -module load zlib/1.2.11-GCCcore-10.2.0 -module load XZ/5.2.5-GCCcore-10.2.0 +module load buildtools/21.12 +module load zlib/1.2.11-cpeGNU-21.12 +module load bzip2/1.0.8-cpeGNU-21.12 +module load XZ/5.2.5-cpeGNU-21.12 +module load cURL/7.78.0-cpeGNU-21.12 ... Defining build environment... - ... - export CXX='mpicxx' - export CXXFLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno -fPIC' - ... +... + + export CC='cc' + export CFLAGS='-O2 -ftree-vectorize -fno-math-errno' + +... configuring... [DRY RUN] [configure_step method] - running command "./bootstrap.sh --with-toolset=gcc --prefix=/tmp/example/Boost/1.74.0/GCC-10.2.0/obj --without-libraries=python,mpi" - (in /tmp/example/build/Boost/1.74.0/GCC-10.2.0/Boost-1.74.0) + running command "./configure --prefix=/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/HTSlib/1.14-cpeGNU-21.12" + (in /run/user/10012026/easybuild/build/HTSlib/1.14/cpeGNU-21.12/HTSlib-1.14) + +building... [DRY RUN] + +[build_step method] + running command "make -j 256" + (in /run/user/10012026/easybuild/build/HTSlib/1.14/cpeGNU-21.12/HTSlib-1.14) + +testing... [DRY RUN] + +[test_step method] + +installing... [DRY RUN] ... +sanity checking... [DRY RUN] + [sanity_check_step method] Sanity check paths - file ['files'] - * lib/libboost_system.so - * lib/libboost_thread-mt-x64.so + * bin/bgzip + * bin/tabix + * lib/libhts.so Sanity check paths - (non-empty) directory ['dirs'] - * include/boost + * include Sanity check commands - (none) + * bgzip --version + * htsfile --version + * tabix --version ... ``` @@ -385,7 +465,7 @@ An overview of the installation procedure is shown, following the installation s performed by EasyBuild. The output above shows: * how the build environment will be set up during the `prepare` step, by loading the module for both the - toolchains and the dependencies, and defining a set of environment variables like `$CXX`, `$CXXFLAGS`, etc. + toolchains and the dependencies, and defining a set of environment variables like `$CC`, `$CFLAGS`, etc. * which command will be executed during the configuration step, and in which directory; * the list of files and directories that will be checked during the sanity check step; @@ -393,7 +473,7 @@ If you were concerned about EasyBuild being too much of a black box, that is hop !!! note It is important to highlight here that the reported installation procedure *may* not be 100% correct, - since the [easyblock](../introduction/#easyblocks) can change its mind based on the output of shell commands + since the [easyblock](1_05_terminology.md#easyblocks) can change its mind based on the output of shell commands that were executed, or based on the contents of a file that was generated during the installation. Since all "actions" that would be performed during the installation are actually skipped when using `eb -x`, the reported installation procedure could be partially incorrect. @@ -413,64 +493,105 @@ make EasyBuild install the software you require. As mentioned before, installing an easyconfig is as simple as passing it to the `eb` command. -So, let's try to install SAMtools version 1.11: +So, let's try to install libdap version 3.20.9 with the cpeGNU/21.12 toolchain. + +Let's first check if it has any dependencies that still need to be installed: ```shell -$ eb SAMtools-1.11-GCC-10.2.0.eb -== temporary log file in case of crash /tmp/eb-zh7_fyre/easybuild-4q_lo57b.log -== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... -== processing EasyBuild easyconfig /home/example/.local/easybuild/easyconfigs/s/SAMtools/SAMtools-1.11-GCC-10.2.0.eb -== building and installing SAMtools/1.11-GCC-10.2.0... +$ eb libdap-3.20.9-cpeGNU-21.12.eb -D +== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-wm_bk3j6/easybuild-puyu_559.log +Dry run: printing build status of easyconfigs and dependencies + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeGNU/cpeGNU-21.12.eb (module: cpeGNU/21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/zlib/zlib-1.2.11-cpeGNU-21.12.eb (module: zlib/1.2.11-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libtirpc/libtirpc-1.3.2-cpeGNU-21.12.eb (module: libtirpc/1.3.2-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/Brotli/Brotli-1.0.9-cpeGNU-21.12.eb (module: Brotli/1.0.9-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ICU/ICU-69.1-cpeGNU-21.12.eb (module: ICU/69.1-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cURL/cURL-7.78.0-cpeGNU-21.12.eb (module: cURL/7.78.0-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeGNU-21.12.eb (module: bzip2/1.0.8-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ncurses/ncurses-6.2-cpeGNU-21.12.eb (module: ncurses/6.2-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libreadline/libreadline-8.1-cpeGNU-21.12.eb (module: libreadline/8.1-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12-minimal.eb (module: gettext/0.21-cpeGNU-21.12-minimal) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/PCRE2/PCRE2-10.37-cpeGNU-21.12.eb (module: PCRE2/10.37-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/XZ/XZ-5.2.5-cpeGNU-21.12.eb (module: XZ/5.2.5-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libxml2/libxml2-2.9.12-cpeGNU-21.12.eb (module: libxml2/2.9.12-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libxslt/libxslt-1.1.34-cpeGNU-21.12.eb (module: libxslt/1.1.34-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12.eb (module: gettext/0.21-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/file/file-5.41-cpeGNU-21.12.eb (module: file/5.41-cpeGNU-21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/util-linux/util-linux-2.37.1-cpeGNU-21.12.eb (module: util-linux/2.37.1-cpeGNU-21.12) + * [ ] /pfs/lustrep3/users/kurtlust/LUMI/LUMI-EasyBuild-contrib/easybuild/easyconfigs/l/libdap/libdap-3.20.9-cpeGNU-21.12.eb (module: libdap/3.20.9-cpeGNU-21.12) +== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-wm_bk3j6/easybuild-puyu_559.log* have been removed. +== Temporary directory /run/user/10012026/easybuild/tmp/eb-wm_bk3j6 has been removed. +``` + +and now instal the library: + +```shell +$ eb libdap-3.20.9-cpeGNU-21.12.eb +== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-kfphjoi8/easybuild-kcs00ai5.log +== processing EasyBuild easyconfig +/pfs/lustrep3/users/kurtlust/LUMI/LUMI-EasyBuild-contrib/easybuild/easyconfigs/l/libdap/libdap-3.20.9-cpeGNU-21.12.eb +== building and installing libdap/3.20.9-cpeGNU-21.12... == fetching files... +== ... (took 2 secs) == creating build dir, resetting environment... == unpacking... == patching... == preparing... +== ... (took 6 secs) == configuring... +== ... (took 1 min 6 secs) == building... +== ... (took 53 secs) == testing... == installing... +== ... (took 5 secs) == taking care of extensions... == restore after iterating... == postprocessing... == sanity checking... +== ... (took 3 secs) == cleaning up... == creating module... +== ... (took 2 secs) == permissions... == packaging... -== COMPLETED: Installation ended successfully (took 17 sec) -== Results of the build can be found in the log file(s) /home/example/easybuild/software/SAMtools/1.11-GCC-10.2.0/easybuild/easybuild-SAMtools-1.11-20210309.105601.log +== COMPLETED: Installation ended successfully (took 2 mins 20 secs) +== Results of the build can be found in the log file(s) +/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/libdap/3.20.9-cpeGNU-21.12/easybuild/easybuild-libdap-3.20.9-20220329.154535.log == Build succeeded for 1 out of 1 -== Temporary log file(s) /tmp/eb-zh7_fyre/easybuild-4q_lo57b.log* have been removed. -== Temporary directory /tmp/eb-zh7_fyre has been removed. +== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-kfphjoi8/easybuild-kcs00ai5.log* have been removed. +== Temporary directory /run/user/10012026/easybuild/tmp/eb-kfphjoi8 has been removed. ``` That was... easy. Is that really all there is to it? Well, almost... ### Enabling dependency resolution -The SAMtools installation worked like a charm, but remember that all required dependencies were already +The libdap installation worked like a charm, but remember that all required dependencies were already available (see [above](#dry-run)). -If we try this with the `BCFtools-1.11-GCC-10.2.0.eb`, for which the required `GSL` and `HTSlib` dependencies are not available yet, it's less successful: +If we try this with the `SAMtools-1.14-cpeGNU-21.12.eb`, for which the required `HTSlib` dependencies is not available yet, it's less successful: ```shell -$ eb BCFtools-1.11-GCC-10.2.0.eb -M +$ eb SAMtools-1.14-cpeGNU-21.12.eb -M -3 out of 23 required modules missing: +2 out of 11 required modules missing: -* GSL/2.6-GCC-10.2.0 (GSL-2.6-GCC-10.2.0.eb) -* HTSlib/1.11-GCC-10.2.0 (HTSlib-1.11-GCC-10.2.0.eb) -* BCFtools/1.11-GCC-10.2.0 (BCFtools-1.11-GCC-10.2.0.eb) +* HTSlib/1.14-cpeGNU-21.12 (HTSlib-1.14-cpeGNU-21.12.eb) +* SAMtools/1.14-cpeGNU-21.12 (SAMtools-1.14-cpeGNU-21.12.eb) ``` ```shell -$ eb BCFtools-1.11-GCC-10.2.0.eb +$ eb SAMtools-1.14-cpeGNU-21.12.eb ... == preparing... -== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/build/BCFtools/1.11/GCC-10.2.0): build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0 (took 2 sec) -== Results of the build can be found in the log file(s) /tmp/eb-3v1dfvnk/easybuild-BCFtools-1.11-20210308.195024.FlxkH.log -ERROR: Build of /home/example/.local/easybuild/easyconfigs/b/BCFtools/BCFtools-1.11-GCC-10.2.0.eb failed (err: 'build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0') +== FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12): build failed +(first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12 (took 3 secs) +== Results of the build can be found in the log file(s) +/run/user/10012026/easybuild/tmp/eb-rgj1v43y/easybuild-SAMtools-1.14-20220329.155911.ZtDcX.log + +ERROR: Build of /appl/lumi/LUMI-EasyBuild-contrib/easybuild/easyconfigs/s/SAMtools/SAMtools-1.14-cpeGNU-21.12.eb failed (err: 'build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12') ``` Oh my, what's this all about? @@ -478,36 +599,33 @@ Oh my, what's this all about? If we filter the output a bit and focus on the actual error, the problem is clear: ``` -Missing modules for dependencies (use --robot?): HTSlib/1.11-GCC-10.2.0, GSL/2.6-GCC-10.2.0 +Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12') ``` -The required dependencies `HTSlib/1.11-GCC-10.2.0` and `GSL/2.6-GCC-10.2.0` are not installed yet, -and EasyBuild does not automatically install missing dependencies unless it is told to do so. +The required dependency `HTSlib/1.14-cpeGNU-21.12` is not installed yet, +and EasyBuild does not automatically install missing dependencies unless it is told to do so +(which we didn't do in the configuration for LUMI). -It helpfully suggests to use the `--robot` command line option, so let's try that: +It helpfully suggests to use the `--robot` (or '-r') command line option, so let's try that: ```shell -$ eb BCFtools-1.11-GCC-10.2.0.eb --robot +$ eb SAMtools-1.14-cpeGNU-21.12.eb --robot ... == resolving dependencies ... ... -== building and installing HTSlib/1.11-GCC-10.2.0... +== building and installing HTSlib/1.14-cpeGNU-21.12... ... == COMPLETED: Installation ended successfully (took 13 sec) ... -== building and installing GSL/2.6-GCC-10.2.0... -... -== COMPLETED: Installation ended successfully (took 1 min 10 sec) -... -== building and installing BCFtools/1.11-GCC-10.2.0... +== building and installing SAMtools/1.14-cpeGNU-21.12... ... == COMPLETED: Installation ended successfully (took 8 sec) ... -== Build succeeded for 3 out of 3 +== Build succeeded for 2 out of 2 ``` -With dependency resolution enabled the `HTSlib` and `GSL` modules get installed first, -before EasyBuild proceeds with installing `BCFtools`. Great! +With dependency resolution enabled the `HTSlib` module gets installed first, +before EasyBuild proceeds with installing `SAMtools`. Great! --- @@ -521,35 +639,50 @@ To provide some more feedback as the installation progresses, you can enable the Let's do this by defining the `$EASYBUILD_TRACE` environment variable, just to avoid having to type `--trace` over and over again. -We will redo the installation of `BCFtools-1.11-GCC-10.2.0.eb` by passing the `--rebuild` +We will redo the installation of `SAMtools-1.14-cpeGNU-21.12.eb` by passing the `--rebuild` option to the `eb` command (try yourself what happens if you don't use the `--rebuild` option!): ```shell $ export EASYBUILD_TRACE=1 -$ eb BCFtools-1.11-GCC-10.2.0.eb --rebuild +$ eb SAMtools-1.14-cpeGNU-21.12.eb --rebuild ... == configuring... >> running command: - [started at: 2021-03-08 19:54:53] - [working dir: /tmp/example/build/BCFtools/1.11/GCC-10.2.0/bcftools-1.11] - [output logged in /tmp/eb-9u_ac0nv/easybuild-run_cmd-17m_he2x.log] - ./configure --prefix=/home/example/easybuild/software/BCFtools/1.11-GCC-10.2.0 --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --with-htslib=$EBROOTHTSLIB --enable-libgsl -== building... + [started at: 2022-03-29 18:46:31] + [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] + [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-g7vd83qv.log] + /users/kurtlust/LUMI-user-appl/sources/generic/eb_v4.5.3/ConfigureMake/config.guess + >> command completed: exit 0, ran in < 1s >> running command: - [started at: 2021-03-08 19:54:54] - [working dir: /tmp/example/BCFtools/1.11/GCC-10.2.0/bcftools-1.11] - [output logged in /tmp/example/eb-9u_ac0nv/easybuild-run_cmd-bhkgjxi7.log] - make -j 8 + [started at: 2022-03-29 18:46:31] + [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] + [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-k0etfv8i.log] + ./configure --prefix=/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/SAMtools/1.14-cpeGNU-21.12 --build=x86_64-pc-linux-gnu +--host=x86_64-pc-linux-gnu --with-htslib=$EBROOTHTSLIB >> command completed: exit 0, ran in 00h00m03s +== ... (took 3 secs) +== building... + >> running command: + [started at: 2022-03-29 18:46:34] + [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] + [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-svcps0yj.log] + make -j 256 CC="cc" CXX="CC" CFLAGS="-O2 -ftree-vectorize -fno-math-errno -fPIC" CXXFLAGS="-O2 -ftree-vectorize -fno-math-errno +-fPIC" + >> command completed: exit 0, ran in 00h00m06s +== ... (took 6 secs) ``` That's a bit more comforting to stare at... -During the *configure* step, the `./configure` command is run with option to -enable support for leveraging `HTSlib` and `GSL`. +SAMtools uses a custom easyblock that is derived from the generic `ConfigureMake` easyblock. +During the *configure* step, the `./configure` command is run with `--build` and `--host` +options added by the generic ConfigureMake easyblock (and the other command, `config.guess` plays +a role in determining the value of those flags). The `--with-htslib=$EBROOTHTSLIB` flag is added +via our easyconfig file to tell SAMtools to use an already available version of HTSlib rather +than the built-in one. During the *build* step, the software is actually being compiled -by running the `make` command. EasyBuild automatically uses the available cores on the system (in this case 8). +by running the `make` command. EasyBuild automatically uses the available cores on the system (in this case 256). We even get a pointer to a log file that contains the output of the command being run, so we can use `tail -f` to see in detail how it progresses. @@ -561,14 +694,19 @@ Later during the installation, we now also see this output during the sanity che ``` == sanity checking... - >> file 'bin/bcftools' found: OK - >> file 'bin/plot-vcfstats' found: OK - >> file 'bin/vcfutils.pl' found: OK - >> (non-empty) directory 'libexec/bcftools' found: OK + >> file 'bin/blast2sam.pl' found: OK + >> file 'bin/bowtie2sam.pl' found: OK + >> file 'bin/export2sam.pl' found: OK + >> file 'bin/interpolate_sam.pl' found: OK +... + >> loading modules: SAMtools/1.14-cpeGNU-21.12... + >> running command 'samtools version' ... + >> result for command 'samtools version': OK ``` -Thanks to enabling trace mode, EasyBuild tells us which files & directories it is checking for -in the installation, before declaring it a success. Nice! +Thanks to enabling trace mode, EasyBuild tells us which files (& directories, but there are non in this case) +it is checking for +in the installation, and which command it is trying to run before declaring it a success. Nice! The extra output you get when trace mode is enabled is concise and hence not overwhelming, while it gives a better insight into what is going on during the installation. @@ -579,13 +717,13 @@ so you can interrupt the installation before it completes, if deemed necessary. ## Using installed software -So far, we have already installed 4 different software packages (SAMtools, HTSlib, GSL, and BCFtools); -we even installed BCFtools twice! +So far, we have already installed 4 different software packages (SAMtools, HTSlib, libdap); +we even installed SAMtools twice! A lot was going on underneath the covers: locating and unpacking the source tarballs, setting up the build environment, configuring the build, compiling, creating and populating the installation directory, performing a quick sanity check on the installation, -cleaning things up, and finally generated the environment module file corresponding to the installation. +cleaning things up, and finally generating the environment module file corresponding to the installation. That's great, but how do we now actually *use* these installations? @@ -593,98 +731,165 @@ This is where the generated module files come into play: they form the access po installations, and we'll use the ubiquitous `module` command to digest them. First, we need to make sure that the modules tool is aware of where the module files for -our installations are located. If you're unsure where EasyBuild is installing stuff at this point, -check the output of `eb --show-config`; the value of the `installpath` configuration setting is what we are interested in now: +our installations are located. On LUMI, when using the EasyBuild-user module to configure EasyBuild, +everything is taken care of for you and the LUMI modules will also automatically add the +suitable module directories for user-installed software to the search path for modules. +By default, EasyBuild-config will install in `$HOME/EasyBuild`, but it is possible to +build the installation in a different directory by pointing to it with the environment +variable `EBU_USER_PREFIX`. Of course this variable needs to be set before loading the `LUMI` +module. (Note that one reason why we don't load a software stack by default is that in +the current setup of LUMI this module would be loaded before the user gets the chance to set +that environment variable in `.bash_profile` or `.bashrc`.) + +When loading the `EasyBuild-user` module, the module command will show you were EasyBuild +will install the software and put the modules, and also put its repository of +processed easyconfig file. ```shell -$ eb --show-config -... -installpath (E) = /home/example/easybuild -... -repositorypath (E) = /home/example/easybuild/ebfiles_repo -... -sourcepath (E) = /home/example/easybuild/sources -... +ml EasyBuild-user + +EasyBuild configured to install software from the LUMI/21.12 software stack for the LUMI/L +partition in the user tree at /users/kurtlust/LUMI-user-appl. + * Software installation directory: /users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L + * Modules installation directory: /users/kurtlust/LUMI-user-appl/modules/LUMI/21.12/partition/L + * Repository: /users/kurtlust/LUMI-user-appl/ebrepo_files/LUMI-21.12/LUMI-L + * Work directory for builds and logs: /run/user/10012026/easybuild + Clear work directory with clear-eb ``` -So, what's in this directory? - -```shell -$ ls -l $HOME/easybuild -total 16 -drwxrwxr-x 5 example example 4096 Jun 10 20:11 ebfiles_repo -drwxrwxr-x 5 example example 4096 Jun 10 20:10 modules -drwxrwxr-x 6 example example 4096 Jun 10 20:10 software -drwxrwxr-x 6 example example 4096 Jun 10 20:10 sources -``` +EasyBuild will copy each easyconfig file it installs to +the repository and add some lines to it with information about the installation. It +also has some options that may edit the source easyconfig, e.g., when asking EasyBuild +to try to build with another toolchain. -The `ebfiles_repo` and `sources` directories correspond to the `repositorypath` and `sourcepath` configuration -settings, respectively. The `modules` and `software` directories are what we need now. +You can always check where EasyBuild is installing stuff by checking the output of +`eb --show-config`. -The `modules` subdirectory consists of multiple subdirectories: +If you're unsure where EasyBuild is installing stuff at this point, +check the output of `eb --show-config`; the value of the `installpath` configuration setting is what we are interested in now: ```shell -$ ls $HOME/easybuild/modules -all bio devel numlib tools +$ eb --show-config +... +buildpath (E) = /run/user/XXXXXXXX/easybuild/build +... +installpath-modules (E) = /users/XXXXXXXX/LUMI-user/modules/LUMI/21.12/partition/L +installpath-software (E) = /users/XXXXXXXX/LUMI-user/SW/LUMI-21.12/L... +... +repositorypath (E) = /users/XXXXXXXX/LUMI-user/ebrepo_files/LUMI-21.12/LUMI-L +... +sourcepath (E) = /users/XXXXXXXX/LUMI-user/sources:/appl/lumi/sources/easybuild +... ``` -Directories like `bio` and `numlib` correspond to different software categories, -and contain symbolic links to the module files in the `all` directory, -which contains all actual module files for software installed in this EasyBuild installation path. -We'll ignore these separate category directories for now. +This is sligthly different from the default EasyBuild setup, where the modules, software, +repository and sources would be installed in respectively the subdirectories `modules`, +`software`, `ebfiles_repo` and `sources` of the directory pointed to by the `installpath` +line. -Let's inform the modules tool about the existence of these module files using `"module use"`: +The modules directory is also a simplified one from the standard EasyBuild one as that also +provides a module categorisation besides a directory containing all modules. As this categorisation +is largely arbitrary and hard to use in the module system, we decided simply not to use it in +our installation and use a custom naming scheme. -```shell -module use $HOME/easybuild/modules/all -``` - -This command does little more that updating the `$MODULEPATH` environment variable, -which contains a list of paths that the modules tool should consider when looking for module files. +However, if you would be using EasyBuild on another system with its default configuration, the +above setup would be used. For more information, we refer to the generic EasyBuild tutorials on the +[EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). Now the modules tool should be aware of our brand new installations: ```shell $ module avail +... +-- EasyBuild managed user software for software stack LUMI/21.12 on LUMI-L --- + HTSlib/1.14-cpeGNU-21.12 libdap/3.20.9-cpeGNU-21.12 + SAMtools/1.14-cpeGNU-21.12 ----------------------- /home/example/easybuild/modules/all ----------------------- - BCFtools/1.11-GCC-10.2.0 GSL/2.6-GCC-10.2.0 SAMtools/1.11-GCC-10.2.0 - EasyBuild/4.3.3 HTSlib/1.11-GCC-10.2.0 bzip2/1.0.6 - ----------------------------- /easybuild/modules/all ----------------------------- - ... +----- EasyBuild managed software for software stack LUMI/21.12 on LUMI-L ----- +... ``` -This output shows both the modules for our own installations as well as the "central" installations in `/easybuild` (which we omitted above for brevity). +This output shows both the modules for our own installations as well as the "central" installations +(which we omitted for brevity). Now we can load these modules and start using these software installations. -Let's test this for BCFtools. In our current environment, the `bcftools` command is not available yet: +Let's test this for SAMtools. In our current environment, the `samtools` command is not available yet: ```shell $ module list -No modules loaded -$ bcftools --bash: bcftools: command not found +Currently Loaded Modules: + 1) perftools-base/21.12.0 + 2) cce/13.0.0 + 3) craype/2.7.13 + 4) cray-dsmml/0.2.2 + 5) cray-mpich/8.1.12 + 6) cray-libsci/21.08.1.2 + 7) PrgEnv-cray/8.2.0 + 8) ModuleLabel/label (S) + 9) init-lumi/0.1 (S) + 10) craype-x86-rome + 11) craype-accel-host + 12) libfabric/1.11.0.4.106 + 13) craype-network-ofi + 14) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta + 15) partition/L (S) + 16) LUMI/21.12 (S) + + Where: + S: Module is Sticky, requires --force to unload or purge +$ samtools +-bash: samtools: command not found ``` -Loading the module for BCFtools changes that: +Loading the module for SAMtools changes that: ```shell -$ module load BCFtools/1.11-GCC-10.2.0 +$ module load SAMtools/1.14-cpeGNU-21.12 + +Lmod is automatically replacing "cce/13.0.0" with "gcc/11.2.0". +Lmod is automatically replacing "PrgEnv-cray/8.2.0" with "cpeGNU/21.12". + +Due to MODULEPATH changes, the following have been reloaded: + 1) cray-mpich/8.1.12 $ module list + Currently Loaded Modules: - 1) GCCcore/10.2.0 6) XZ/5.2.5-GCCcore-10.2.0 - 2) zlib/1.2.11-GCCcore-10.2.0 7) cURL/7.72.0-GCCcore-10.2.0 - 3) binutils/2.35-GCCcore-10.2.0 8) HTSlib/1.11-GCC-10.2.0 - 4) GCC/10.2.0 9) GSL/2.6-GCC-10.2.0 - 5) bzip2/1.0.8-GCCcore-10.2.0 10) BCFtools/1.11-GCC-10.2.0 - -$ bcftools --version -bcftools 1.11 -Using htslib 1.11 + 1) perftools-base/21.12.0 + 2) ModuleLabel/label (S) + 3) init-lumi/0.1 (S) + 4) craype-x86-rome + 5) craype-accel-host + 6) libfabric/1.11.0.4.106 + 7) craype-network-ofi + 8) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta + 9) partition/L (S) + 10) LUMI/21.12 (S) + 11) gcc/11.2.0 + 12) craype/2.7.13 + 13) cray-mpich/8.1.12 + 14) cray-libsci/21.08.1.2 + 15) cray-dsmml/0.2.2 + 16) cpeGNU/21.12 + 17) ncurses/6.2-cpeGNU-21.12 + 18) zlib/1.2.11-cpeGNU-21.12 + 19) bzip2/1.0.8-cpeGNU-21.12 + 20) gettext/0.21-cpeGNU-21.12-minimal + 21) XZ/5.2.5-cpeGNU-21.12 + 22) Brotli/1.0.9-cpeGNU-21.12 + 23) cURL/7.78.0-cpeGNU-21.12 + 24) HTSlib/1.14-cpeGNU-21.12 + 25) SAMtools/1.14-cpeGNU-21.12 + + Where: + S: Module is Sticky, requires --force to unload or purge + +$ samtools --version +samtools 1.14 +Using htslib 1.14 +Copyright (C) 2021 Genome Research Ltd.... ... ``` @@ -696,24 +901,61 @@ by updating environment variables like `$PATH` for example, to make the software To restore your environment to a pristine state in which no modules are loaded, you can either unload the loaded modules one by one using "`module unload`", or you can unload all of them at once using -"`module purge`". +"`module purge`". On LUMI, `module purge` will unload all application modules but will not undo the selection +of the software stack. It will reset the software stack though to use those modules that fit best +with the hardware of the current node (i.e., you may find a different `partition` module). -**If you are using an EasyBuild installation provided by a module, -don't forget to load the `EasyBuild` module again after running "`module purge`".** +```shell +$ module purge +The following modules were not unloaded: + (Use "module --force purge" to unload all): + + 1) LUMI/21.12 6) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta + 2) craype-x86-rome 7) partition/L + 3) craype-accel-host 8) init-lumi/0.1 + 4) libfabric/1.11.0.4.106 9) ModuleLabel/label + 5) craype-network-ofi +$ module list + +Currently Loaded Modules: + 1) LUMI/21.12 (S) + 2) craype-x86-rome + 3) craype-accel-host + 4) libfabric/1.11.0.4.106 + 5) craype-network-ofi + 6) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta + 7) partition/L (S) + 8) init-lumi/0.1 (S) + 9) ModuleLabel/label (S) + + Where: + S: Module is Sticky, requires --force to unload or purge +``` + +Runnin `module --force purge` instead will remove all modules, including the `init-lumi` +module which does part of the initialisation. You will not be able to use the software +stacks completely as before without first loading `init-lumi` in its most recent (or default) +version again! + +```shell +$ module --force purge +$ module list +No modules loaded +``` --- ## Stacking software Maybe you have overlooked how the software we are playing around with was not only installed across multiple -different installation directories per software, we are also "stacking" our own installations (in `$HOME/easybuild`) -on top of installations that are provided in a totally different location (`/easybuild`). +different installation directories per software, we are also "stacking" our own installations (in `$HOME/EasyBuild` +or `$EBU_USER_PREFIX`) on top of installations that are provided in a totally different location (`/appl/lumi`). **EasyBuild doesn't care *where* software is installed: as long as the module file that provides access to it is available, it is happy to pick it up and use it when required.** -This implies that end users of an HPC system can easily install their -own small software stack on top of what is provided centrally by the HPC support team, +This implies that end users of LUMI can easily install their +own small software stack on top of what is provided centrally by the LUMI User Support, for example. They can even "replace" a central software installation for their purposes if they need to, since the modules tool will load the first module file that matches the request being made (there are some caveats with this, but we @@ -723,6 +965,8 @@ won't go into those here). ## Exercises +TODO: Rework for LUMI and provide the necessary EasyConfig files. + ***Guidelines*** Do yourself a favor: don't peek at the solution until you have made an attempt to solve the exercise yourself! @@ -967,10 +1211,6 @@ If you've made it through the hands-on exercises, congratulations! If not, don't worry too much about it. We covered a lot of ground here, and it's a lot to take in at once, take your time... -Feel free to ask question in the `#tutorial` channel in the [EasyBuild -Slack](https://docs.easybuild.io/en/latest/#getting-help), -we're happy to help! - --- *[[next: Part 2: Using EasyBuild]](2_00_part2_using.md)* From d24d31b20cbdfa64efa0c3f782abfaae0f8fc819 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 30 Mar 2022 12:27:23 +0200 Subject: [PATCH 136/159] Additions to the LUMI software stack page of the tutorial. --- .../1_04_LUMI_software_stack.md | 86 ++++++++++++++++++- 1 file changed, 85 insertions(+), 1 deletion(-) diff --git a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md index 0e578a59..3bfae70c 100644 --- a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md +++ b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md @@ -1,9 +1,93 @@ -# LUMI software stacks +# LUMI software stacks (technical) *[[back: The Cray Programming Environment]](1_03_CPE.md)* --- +The user-facing documentation on how to use the LUMI software stacks is +avialable in [the LUMI documentation](https://docs.lumi-supercomputer.eu/computing/softwarestacks/). +On this page we focus more on the technical implementation behind it. + +--- + +# An overview of LUMI + +LUMI has different node types providing compute resources: + +- LUMI has 16 login nodes, though many of those are reserved for special purposes and not + available to all users. TODO + + +--- + +## CrayEnv and LUMI modules + +On LUMI, two types of software stacks are currently offered: + + - ``CrayEnv`` (module name) offers the Cray PE and enables one to use + it completely in the way intended by HPE-Cray. The environment also offers a + limited selection of additional tools, often in updated versions compared to + what SUSE Linux, the basis of the Cray Linux environment, offers. Those tools + are installed and managed via EasyBuild. However, EasyBuild is not available + in that partition. + + It also rectifies a problem caused by the fact that there is only one + configuration file for the Cray PE on LUMI, so that starting a login shell + will not produce an optimal set of target modules for all node types. + The ``CrayEnv`` module recognizes on which node type it is running and + (re-)loading it will trigger a reload of the recommended set of target + modules for that node. + + - ``LUMI`` is an extensible software stack that is mostly managed through + [EasyBuild][easybuild]. Each version of the LUMI software stack is based on + the version of the Cray Programming Environment with the same version + number. + + A deliberate choice was made to only offer a limited number of software + packages in the globally installed stack as the setup of redundancy on LUMI + makes it difficult to update the stack in a way that is guaranteed to not + affect running jobs and as a large central stack is also hard to manage. + However, the EasyBuild setup is such that users can easily install + additional software in their home or project directory using EasyBuild build + recipes that we provide or they develop, and that software will fully + integrate in the central stack (even the corresponding modules will be made + available automatically). + + Each ``LUMI`` module will also automatically activate a set of application + modules tuned to the architecture on which the module load is executed. To + that purpose, the ``LUMI`` module will automatically load the ``partition`` + module that is the best fit for the node. After loading a version of the + ``LUMI`` module, users can always load a different version of the ``partition`` + module. + +Note that the ``partition`` modules are only used by the ``LUMI`` module. In the +``CrayEnv`` environment, users should overwrite the configuration by loading their +set of target modules after loading the ``CrayEnv`` module. + + +--- + +## The ``partition`` module + +The ``LUMI`` module currently supports four partition modules, but that number may +be reduced in the future: + +| Partition | CPU target | Accelerator | +|:----------------|-----------------------|:----------------------------| +| ``partition/L`` | ``craype-x86-rome`` | ``craype-accel-host`` | +| ``partition/C`` | ``craype-x86-milan`` | ``craype-accel-host`` | +| ``partition/G`` | ``craype-x86-trento`` | ``craype-accel-amd-gfx90a`` | +| ``partition/D`` | ``craype-x86-rome`` | ``craype-accel-nvidia80`` | + +All ``partition`` modules also load `craype-network-ofi``. + +``pattition/D`` may be dropped in the future as it seems we have no working CUDA setup +and can only use the GPU nodes in the LUMI-D partition for visualisation and not with CUDA. + +Furthermore if it would turn out that there is no advantage in optimizing for Milan +specifically, or that there are no problems at all in running Milan binaries on Rome +generation CPUs, ``partition/L`` and ``partition/C`` might also be united in a single +partition. From bbec7d0eeb8211c05fb57b7bded521e44d64e060 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 30 Mar 2022 12:28:17 +0200 Subject: [PATCH 137/159] Integrating parts of the old tutorial into the new one, texts still need to be rewritten. --- docs/2022-CSC_and_LO/2_00_part2_using.md | 7 + docs/2022-CSC_and_LO/2_01_troubleshooting.md | 488 +++++++++ .../2_02_creating_easyconfig_files.md | 998 ++++++++++++++++++ .../2_03_implementing_easyblocks.md | 665 ++++++++++++ docs/2022-CSC_and_LO/3_00_part3_advanced.md | 9 + .../2022-CSC_and_LO/3_01_easybuild_library.md | 293 +++++ docs/2022-CSC_and_LO/3_02_hooks.md | 190 ++++ docs/2022-CSC_and_LO/3_03_slurm_jobs.md | 304 ++++++ .../3_04_module_naming_schemes.md | 617 +++++++++++ .../3_05_github_integration.md | 442 ++++++++ 10 files changed, 4013 insertions(+) create mode 100644 docs/2022-CSC_and_LO/2_00_part2_using.md create mode 100644 docs/2022-CSC_and_LO/2_01_troubleshooting.md create mode 100644 docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md create mode 100644 docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md create mode 100644 docs/2022-CSC_and_LO/3_00_part3_advanced.md create mode 100644 docs/2022-CSC_and_LO/3_01_easybuild_library.md create mode 100644 docs/2022-CSC_and_LO/3_02_hooks.md create mode 100644 docs/2022-CSC_and_LO/3_03_slurm_jobs.md create mode 100644 docs/2022-CSC_and_LO/3_04_module_naming_schemes.md create mode 100644 docs/2022-CSC_and_LO/3_05_github_integration.md diff --git a/docs/2022-CSC_and_LO/2_00_part2_using.md b/docs/2022-CSC_and_LO/2_00_part2_using.md new file mode 100644 index 00000000..e033499c --- /dev/null +++ b/docs/2022-CSC_and_LO/2_00_part2_using.md @@ -0,0 +1,7 @@ +# Part II: Using EasyBuild + +*[[back to start page]](index.md)* + +* [Troubleshooting](2_01_troubleshooting.md) *(hands-on)* +* [Creating easyconfig files](2_02_creating_easyconfig_files.md) *(hands-on)* +* [Implementing easyblocks](2_03_implementing_easyblocks.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/2_01_troubleshooting.md b/docs/2022-CSC_and_LO/2_01_troubleshooting.md new file mode 100644 index 00000000..86a2f1d3 --- /dev/null +++ b/docs/2022-CSC_and_LO/2_01_troubleshooting.md @@ -0,0 +1,488 @@ +# Troubleshooting + +*[[back: Overview]](index.md)* + +--- + +When installing scientific software you are bound to run into problems +that make the installation fail sooner or later, even when using EasyBuild. + +In this part we take a look at how you can **troubleshoot a failing installation**, +and focus on a couple of EasyBuild aspects that can be helpful in that context, +like the **error messages produced by EasyBuild**, the detailed **log file** that is +available for the installation and the **build directory** where most software is +being compiled before it actually gets installed. + +At the end of this part an exercise is available in which you get +to fix a failing installation, so pay attention! + +## EasyBuild error messages + +When EasyBuild detects that something went wrong, it will usually produce a +short (and hopefully helpful) error message. + +Things that could go wrong during an installation include: + +* missing source or patch files; +* a checksum error on a downloaded source or patch file; +* required dependencies that are not specified in the easyconfig file; +* failing shell commands; +* running out of available memory or disk space; +* a segmentation fault caused by a flipped bit triggered by a cosmic ray ([really, it happens!](https://blogs.oracle.com/linux/attack-of-the-cosmic-rays-v2)); + +Unfortunately this is not an exhaustive list, there are plenty of other +potential problems that could result in a failing installation... + +For each of the shell commands that EasyBuild executes during an +installation, it will check the exit status. +If the exit status is zero, EasyBuild will usually assume that the shell command +ran correctly, and it will continue with the rest of the installation procedure. +If the exit status is anything but zero, EasyBuild will assume that a problem +has occurred, and the installation will be interrupted. + +### Example + +Here is an example of an EasyBuild error message (slightly reformatted for clarity): + +``` +$ eb example.eb +... +== building... +== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): +build failed (first 300 chars): cmd "make" exited with exit code 2 and output: +/usr/bin/g++ -O2 -ftree-vectorize -march=native -fno-math-errno -std=c++14 -c -o core.o core.cpp +g++: error: unrecognized command line option '-std=c++14' (took 1 sec) +== Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log +ERROR: Build of /home/easybuild/subread.eb failed (err: ...) +``` + +Let's break this down a bit: during the `build` step of the installation +procedure EasyBuild was running `make` as a shell command, which +failed (exit code 2, so not zero). +The `make` command tripped over the compilation of `core.cpp` that failed because +`-std=c++14` is not a known option to the `g++` command. + +OK fine, but now what? Can you spot something suspicious here? +Wait a minute... Why is `make` using `/usr/bin/g++` for the compilation?! +That's not where our toolchain compiler is installed, +that's somewhere under `/easybuild/software`. + +Let's see what `/usr/bin/g++` is: + +```shell +$ /usr/bin/g++ --version +g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) +``` + +Oh my that's pretty ancient (GCC 4.8.5 was released in June 2015). +That could definitely explain why it doesn't know about the C++14 standard yet... + +Your next step in this case should probably be figuring +out why `/usr/bin/g++` is being used rather than just `g++`, which would +result in using the right compiler version because EasyBuild sets up the build +environment carefully. + +This is a fictitious example of course, but hopefully it gives you a feeling +of how errors that occur during the installation are handled. + +## EasyBuild log files + +Finding the cause of a problem that made the installation fail is, unfortunately, not always that straightforward... + +EasyBuild includes the first 300 characters of the output produced by a failing +shell command in the error message, which is a simple way to try include +actionable information in the error message while not flooding your terminal +with the full output of the command that failed. + +In some cases there won't be any useful information in there however, +since the actual error message(s) could only appear way later, perhaps even after +the command was already running for several minutes. + +In that case, you will have the dive into the log file that is created by EasyBuild for +every installation, which is located in the unique temporary directory for the EasyBuild session. + +See for example this output line from our earlier example error message: + +``` +== Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log +``` + +You can open this file with your favorite text editor or a tool like `less` +to take a look at the information collected in the log file, which includes +things like: + +* informative messages produced by both the EasyBuild framework and the easyblock + describing how the installation is progressing; +* how the build environment was set up: which modules were loaded, which environment variables were set; +* the exact shell commands that were executed, and in which directory they were run; +* the full output produced by these commands, and their exit code; + +Note that the installation log is also copied into each software installation +directory for successful installation, into the `easybuild` subdirectory. +For example: + +``` +/easybuild/software/HDF5/1.10.7-gompi-2020b/easybuild/easybuild-HDF5-1.10.7-20210308.214453.log +``` + +### Last log + +The `eb` command supports a handy little option that prints the location +to the most recently updated build log. You can leverage this to quickly +open the build log of the last failed EasyBuild session in an editor: + +``` +vim $(eb --last-log) +``` + +### Navigating log files + +Usually you want to go to the end of the log file and then work your way up, +either by scrolling or by searching for specific patterns. Here are a couple +of suggestions of patterns you can use to locate errors: + +* `ERROR` +* `Error 1` +* `error: ` +* `failure` +* `not found` +* `No such file or directory` +* `bazel` +* `Segmentation fault` + +Using "`error`" as a search pattern is not very useful: you will hit a lot of log lines +that are not actually errors at all (like the compilation of an `error.c` file), +and you'll miss others that do include errors but mention `ERROR` or `Error` +rather than `error`. + +When using `less` to view a log file, you can navigate it by: + +* hitting '`$`' followed by '`G`' to go to the end of the log file; +* using your arrow keys to scroll up/down; +* typing '`?`' followed by some text and Enter to search backwards for a particular +pattern ('`/`' to search forwards, '`n`' for next match); +* hitting '`q`' to exit; + +It can also be helpful to zoom in on a specific step of the installation procedure, +which you can do by looking for step markers like these: + +``` +== 2020-06-13 01:34:48,816 example INFO configuring... +== 2020-06-13 01:34:48,817 example INFO Starting configure step +... +== 2020-06-13 01:34:48,823 main.EB_HPL INFO Running method configure_step part of step configure +``` + +If you want to look at the start of the output produced by a particular command, +you can look for the log message that looks like this (this is from the installation +log for `HDF5`): + +``` +== 2020-06-09 13:11:19,968 run.py:222 INFO running cmd: make install +== 2020-06-09 13:11:25,186 run.py:538 INFO cmd " make install " exited with exit code 0 and output: +Making install in src +make[1]: Entering directory `/tmp/example/HDF5/1.10.7/gompi-2020b/hdf5-1.10.7/src' +... +``` + +It can be useful to look for the *first* error that occurred in the output of a command, since subsequent errors are +often fallout from earlier errors. You can do this by first navigating +to the start of the output for a command using "`INFO running cmd`" as a search pattern, and then looking for patterns +like "`error:`" from there. + +## Inspecting the build directory + +When an installation fails the corresponding build directory is *not* cleaned up +automatically, that is only done for successful installations. +This allows you to dive in and check for clues in the files that are stored there. + +The location of the build directory is mentioned in the EasyBuild error message: + +``` +== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): ... +``` + +For software using a classic `configure` script, you may have to locate +and inspect the `config.log` file in the build directory to determine the underlying cause of an error. For software using CMake as a configuration tool you often have to check in +`CMakeOutput.log` or `CMakeError.log` for clues, which are sneakily hidden by CMake in a `CMakeFiles` subdirectory of the build directory. + +As a side note here: as EasyBuild does not clean out old and failed builds you will need to eventually manually remove these build directories from the `buildpath` directory. + +## Exercise + +Let's work our way through a less smooth software installation, +using the easyconfig file that is provided below. + +Don't worry if most of this is still unclear to you, we'll get +to writing easyconfig files from scratch [later in this tutorial](../creating_easyconfig_files). + +```python +easyblock = 'MakeCp' + +name = 'Subread' +version = '2.0.1' + +homepage = 'http://subread.sourceforge.net' +description = "High performance read alignment, quantification and mutation discovery" + +toolchain = {'name': 'GCC', 'version': '8.5.0'} + +# download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz +sources = ['subread-%(version)s-source.tar.gz'] +checksums = ['d808eb5b1823c572cb45a97c95a3c5acb3d8e29aa47ec74e3ca1eb345787c17b'] + +start_dir = 'src' + +# -fcommon is required to compile Subread 2.0.1 with GCC 10, +# which uses -fno-common by default (see https://www.gnu.org/software/gcc/gcc-10/porting_to.html) +buildopts = '-f Makefile.Linux CFLAGS="-fast -fcommon"' + +files_to_copy = ['bin'] + +sanity_check_paths = { + 'files': ['bin/featureCounts', 'bin/subread-align'], + 'dirs': ['bin/utilities'], +} + +sanity_check_commands = ["featureCounts --version"] + +moduleclass = 'bio' +``` + +Do you spot any potential problems yet with this easyconfig file? + +***Preparation*** + +Start by copying the text above in a file named `subread.eb`, +so you can gradually fix the problem you'll encounter. + +Also make sure that the pre-installed software stack is available, +and that the EasyBuild module is loaded (unless you installed EasyBuild +yourself): + +``` +module use /easybuild/modules/all +module load EasyBuild +``` + +For this exercise, make sure EasyBuild is configured to +use `$HOME/easybuild` as `prefix`, and to use `/tmp/$USER` as `buildpath`: + +```shell +export EASYBUILD_PREFIX=$HOME/easybuild +export EASYBUILD_BUILDPATH=/tmp/$USER +``` + +Check your configuration via `eb --show-config`. + +Strictly speaking the configuration doesn't matter much for the sake of this +exercise, but it may help with the step-wise approach we'll take and +grasping the solutions. + +Remember though: *no peeking* before you tried to solve each step yourself! + +--- + +***Exercise T.1**** - Sources* + +Try to install the `subread.eb` easyconfig file, see what happens. + +Can you fix the problem you run into, perhaps without even changing +the easyconfig file? + +??? success "(click to show solution)" + + The installation fails because the source file `subread-2.0.1-source.tar.gz` + is not found: + ``` + $ eb subread.eb + ... + == FAILED: Installation ended unsuccessfully (build directory: /tmp/example/Subread/2.0.1/GCC-8.5.0): build failed (first 300 chars): + Couldn't find file subread-2.0.1-source.tar.gz anywhere, and downloading it didn't work either... + Paths attempted (in order): ... + ``` + + In this case, the problem is that the easyconfig file does not specify + where the sources can be downloaded from. Not automatically at least, + but there is a helpful comment included: + ```python + # download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz + sources = ['subread-%(version)s-source.tar.gz'] + ``` + + We can download the source tarball ourselves, + and move it to the location where EasyBuild expects to find it + (in the `sourcepath` directory): + ``` + curl -OL https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz + mv subread-2.0.1-source.tar.gz $HOME/easybuild/sources/s/Subread/ + ``` + + If downloading is problematic for some reason, the source tarball is also available + in `/easybuild/sources/s/Subread`. + + Or, we can change the easyconfig file to specify the location where + the easyconfig file can be downloaded from: + ```python + source_urls = ['https://download.sourceforge.net/subread/'] + sources = ['subread-%(version)s-source.tar.gz'] + ``` + Note that the `source_urls` value is a *list* of candidate download URLs, + *without* the filename of the source file itself. + + This way, EasyBuild will download the source file when running `eb subread.eb`. + + The source tarball is fairly large (23MB), so don't be alarmed if the download takes a little while. + + ```shell + $ ls -lh $HOME/easybuild/sources/s/Subread + total 23M + -rw-rw-r-- 1 easybuild easybuild 23M Jun 13 17:42 subread-2.0.1-source.tar.gz + ``` + +--- + +***Exercise T.2**** - Toolchain* + +After fixing the problem with missing source file, try the installation again. + +What's wrong now? How can you fix it quickly? + +Take into account that we just want to get this software package installed, +we don't care too much about details like the version of the dependencies or +the *toolchain* here... + + +??? success "(click to show solution)" + + The installation fails because the easyconfig specifies that GCC 8.5.0 + should be used as toolchain: + ``` + $ eb subread.eb + ... + == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-8.5.0): build failed (first 300 chars): + No module found for toolchain: GCC/8.5.0 (took 1 sec) + ``` + + We don't have this GCC version installed, but we do have GCC 10.2.0: + + ```shell + $ module avail GCC/ + ----------------- /easybuild/modules/all ------------------ + GCC/10.2.0 + ``` + + So let's try using that instead. + + Edit the easyconfig file so it contains this: + + ```python + toolchain = {'name': 'GCC', 'version': '10.2.0'} + ``` +--- + +***Exercise T.3**** - Build step* + +With the first two problems fixed, now we can actually try to build the software. + +Can you fix the next problem you run into? + +??? success "(click to show solution)" + + The compilation fails, but the error message we see is incomplete due to + EasyBuild truncating the command output (only the 300 first characters of the output are shown): + ``` + == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-10.2.0): build failed (first 300 chars): + cmd " make -j 1 -f Makefile.Linux CFLAGS="-fast -fcommon"" exited with exit code 2 and output: + gcc -mtune=core2 -O3 -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\""2.0.1"\" -D_FILE_OFFSET_BITS=64 -fmessage-length=0 -ggdb -fast -fcommon -c -o core.o core.c + gcc: error: unrecognized command line opti (took 1 sec) + ``` + + If you open the log file and scroll to the end, + the error is pretty obvious: + ``` + gcc: error: unrecognized command line option -fast; did you mean -Ofast? + make: *** [core.o] Error 1 + ``` + + The easyconfig file hard specifies the `-fast` compiler flag via the `CFLAGS` argument to the build command: + ```python + # -fcommon is required to compile Subread 2.0.1 with GCC 10, + # which uses -fno-common by default (see https://www.gnu.org/software/gcc/gcc-10/porting_to.html) + buildopts = '-f Makefile.Linux CFLAGS="-fast -fcommon"' + ``` + + EasyBuild sets up the build environment, so there should be no need + to hard specify compiler flags (certainly not incorrect ones). + The comment above the `buildopts` definition makes it clear that the `-fcommon` + flag *is* required though, because GCC 10 became a bit stricter by + using `-fno-common` by default. Note that we are using `-fcommon` + as an escape mechanism here: it would be better to fix the source code + and create a patch file instead. + + An easy way to fix this problem is to replace the `-fast` with `-Ofast`, + as the compiler error suggests. + + In this case it is advised to change the `CFLAGS` argument that is added + to be build command to replace the `-fast` with `$CFLAGS`, + which is defined in the build environment by EasyBuild. + + ```python + buildopts = '-f Makefile.Linux CFLAGS="$CFLAGS -fcommon"' + ``` + + Note that we need to be careful with quotes here: we use inner double quotes + to ensure that `$CFLAGS` will be expanded to its value when the build command is run. + +--- + +***Exercise T.4**** - Sanity check* + +After fixing the compilation issue, you're really close to getting the installation working, we promise! + +Don't give up now, try one last time and fix the last problem that occurs... + +??? success "(click to show solution)" + + Now the installation itself works but the sanity check fails, + and hence the module file does not get generated: + ``` + $ eb subread.eb + ... + == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-10.2.0): build failed (first 300 chars): + Sanity check failed: sanity check command featureCounts --version exited with code 255 + (output: featureCounts: unrecognized option '--version' + ... + ``` + + If you look at the full output in the log file you can see + that the correct option to check the version of the `featureCounts` command + is "`-v`" rather than "`--version`", so we need to fix this in the easyconfig file. + + Make the following change in the easyconfig file: + ```python + sanity_check_commands = ["featureCounts -v"] + ``` + + After doing so, **you don't have to redo the installation + from scratch**, you can use the `--module-only` option to only run the + sanity check and generate the module file again: + ``` + eb subread.eb --module-only + ``` + +--- + +In the end, you should be able to install Subread 2.0.1 with the GCC 10.2.0 toolchain by fixing the problems with the `subread.eb` easyconfig file. + +Check your work by manually loading the module and checking the version +via the `featureCounts` command, which should look like this: + +```shell +$ featureCounts -v +featureCounts v2.0.1 +``` + +--- + +*[[next: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md new file mode 100644 index 00000000..d46e003a --- /dev/null +++ b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md @@ -0,0 +1,998 @@ +# Creating easyconfig files + +*[[back: Troubleshooting]](2_02_troubleshooting.md)* + +--- + +We already know how to install easyconfig files that are included with EasyBuild, +but what about installing software for which no easyconfig is available yet? + +To do this we will need to **create additional easyconfig files**, +since every software installation performed by EasyBuild is done based on an easyconfig file. + +In this part of the tutorial we will look at the guts of easyconfig files and even create some ourselves! + +## Easyconfigs vs easyblocks + +Before we dive into writing [easyconfig files](../introduction/#easyconfig-files), +let us take a brief look at how they relate to [easyblocks](../introduction/#easyblocks). + +As we discussed [earlier](../introduction/#terminology), an easyconfig file (`*.eb`) is required +for each installation +performed by EasyBuild which specifies the details of the installation (which software +version, toolchain, etc.), while the installation procedure is implemented +in an easyblock (a Python module). + +When can we leverage a *generic easyblock*, perhaps via a "fat" easyconfig file that includes +a lot of carefully defined easyconfig parameters, and when should we use a minimal easyconfig file +together with a custom *software-specific* easyblock? + +This is not an easy question to answer in a general sense, since it depends on several factors: +the complexity of the software you want to get installed, how much flexibility you want, +how "intelligent" the installation procedure should be with respect to the compiler toolchain and dependencies +that are used for the installation, etc. + +In a nutshell, custom software-specific easyblocks are "do once and forget": they are central solution to peculiarities in the installation procedure of a particular software package. + +Reasons to consider implementing a software-specific easyblock rather than using a generic easyblock include: + +* 'critical' values for easyconfig parameters required to make installation succeed; +* toolchain-specific aspects of the build and installation procedure (e.g., configure options); +* interactive commands that need to be run; +* custom (configure) options for dependencies; +* having to create or adjust specific (configuration) files; +* 'hackish' usage of a generic easyblock; +* complex or very non-standard installation procedure; + +Implementing easyblocks is out of scope for this basic tutorial, for more information please consult +the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). + +## Writing easyconfig files + +Writing an easyconfig file boils down to defining a set of easyconfig parameters in a text file, +which we give a '`.eb`' extension by convention. The name of an easyconfig file doesn't matter +when using it directly to install software, but it does matter when EasyBuild needs to find it +to resolve a dependency for example (as we [discussed earlier](../basic_usage/#easyconfig-filenames)). + +The syntax for easyconfig files is *Python syntax*: you are basically defining a bunch of Python variables +that correspond to easyconfig parameters. + +The order in which the easyconfig parameters are defined doesn't matter, but we generally try to strick to a particular +order which roughly corresponds to the order in which the easyconfig parameters are used during the installation. +That is mostly helpful for humans staring at easyconfig files or comparing them. + +### Mandatory parameters + +A limited number of easyconfig parameters are *mandatory*, they must be defined in every easyconfig file: + +* `name`: the name of the software to install; +* `version`: the version of the software to install; +* `homepage`: a URL to the website of the software; +* `description`: a short description of the software; +* `toolchain`: the compiler toolchain to use for the installation; + +**`name`, `version`** + +It should be no surprise that specifying the name and version of the software you want to install is mandatory. +This information may influence the value of several other easyconfig parameters (like the name of the source file), and is also used to the determine the name of the module file to install. + +```python +name = 'example' +version = '1.0' +``` + +**`homepage`, `description`** + +The homepage and description are included in the generated module file for the installation. +That way the "`module show`" command provides some useful high-level information about the installation. + +```python +homepage = 'https://example.org' +description = "This is just an example." +``` + +Usually it does not matter whether you use single or double quotes to specify string values, +but you will often see that single quotes are used for values that don't have spaces (words) +and double quotes for values that do have spaces (sentences). There is no technical reason for +this, it just feels more natural to some people. There are cases where it is important to use +the right type of quotes however, we will get back to that later (keep it in mind for the exercises!). + +For multi-line descriptions, you will need to use "triple quoting" (which is standard Python syntax): + +```python +description = """This is an example + of a multi-line description. + It is spread across multiple lines.""" +``` + +**`toolchain`** + +EasyBuild also requires that the [compiler toolchain](../introduction/#toolchains) is specified, via the `toolchain` +easyconfig parameter. + +This can either be the [`system` toolchain](../introduction/#system-toolchain), for which a constant named `SYSTEM` is available: + +```python +toolchain = SYSTEM +``` + +Usually we specify a 'proper' toolchain like the compiler-only toolchain GCC 10.2.0 which we used before, +or the full toolchain `foss` 2020b. The name and version of the toolchain can be specified using a small Python dictionary, +for example: + +```python +toolchain = {'name': 'GCC', 'version': '10.2.0'} +``` + +### Commonly used parameters + +You will often need to specify additional easyconfig parameters to get something useful done. +We will cover the most commonly used ones here, but keep in mind that these are *not* mandatory. + +A full overview of all known easyconfig parameters can be obtained via "`eb --avail-easyconfig-params`" +or just "`eb -a`" for short, or can be consulted in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_parameters.html). + +#### Sources, patches, and checksums + +In most easyconfig files you will see that a list of source files is specified via the `sources` +easyconfig parameter, usually combined +with one or more URLs where these sources can be downloaded specified via `source_urls`. +There also may be patch files listed (specified via `patches`), +and checksums for both the source files and patches (specified via `checksums`). + +The `sources` easyconfig parameter is commonly defined but it is *not* mandatory, +because some easyconfig files only specify bundles of software packages and hence only +serve to generate a module file. + +Here is an example of how these easyconfig parameters can be specified: + +```python +source_urls = [ + 'https://example.org/download/', + 'https://example.org/download/archive/', +] +sources = ['example-1.0-src.tar.gz'] +patches = ['example-fix.patch'] +checksums = [ + '9febae18533d035ac688d977cb2ca050e6ca8379311d7a14490ad1ef948d45fa', + '864395d648ad9a5b75d1a745c8ef82b78421d571584037560a22a581ed7a261c', +] +``` + +Each of these require a *list* of values, so even if there is only a single source file or download URL +you must use square brackets as shown in the example. The default value for each of these is an empty list (`[]`). + +Some things worth pointing out here: + +* The download URLs specified via `source_urls` do *not* include the name of the file, that is added + automatically by EasyBuild when it tries to download the file (only if it's not available already.) +* If multiple download URLs are specified, they are each tried once in order until the download of the source file was + successful. This can be useful to include backup locations where source files can be downloaded from. +* Names of source files and patches should not include hardcoded software versions, they usually use a + template value like `%(version)s` instead: + ```python + sources = ['example-%(version)s-src.tar.gz'] + ``` + EasyBuild will use the value of the `version` easyconfig parameter to determine the actual name of the source + file. This way the software version is only specified in one place and the easyconfig file is easier to + update to other software versions. A list of template values can be consulted via the EasyBuild command + line via the `--avail-easyconfig-templates` option, or in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_templates.html). +* Source files can also be specified in ways other than just using a filename, see the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-param-sources-alt) for more information. +* Specified checksums are usually SHA256 checksum values, but [other types are also supported](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html?highlight=checksums#checksums). + + +#### Easyblock + +The easyblock that should be used for the installation can be specified via the `easyblock` easyconfig parameter. + +This is not mandatory however, because by default EasyBuild will determine the easyblock to use based on the +name of the software. If '`example`' is specified as software name, EasyBuild will try to locate a +software-specific easyblock named `EB_example` (in a Python module named `example.py`). Software-specific +easyblocks follow the convention that the class name starts with `'EB_`', followed by the software name +(where some characters are replaced, like '`-`' with '`_minus_`'). + +**Generic easyblocks** + +Usually the `easyblock` value is the name of a *generic* easyblock, if it is specified. The name of +a generic easyblock does *not* start with '`EB_`', so you can easily distinguish it from a software-specific +easyblock. + +Here are a couple of commonly used generic easyblocks: + +* `ConfigureMake`: implements the standard `./configure`, `make`, `make install` installation procedure; +* `CMakeMake`: same as `ConfigureMake`, but with `./configure` replaced with `cmake` for the configuration step; +* `PythonPackage`: implements the installation procedure for a single Python package, by default using + "`python setup.py install`" but other methods like using "`pip install`" are also supported; +* `Bundle`: a simple generic easyblock to bundle a set of software packages together in a single installation directory; +* `PythonBundle`: a customized version of the `Bundle` generic easyblock to install a bundle of Python packages + in a single installation directory; + +A full overview of the available generic easyblocks is available in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/generic_easyblocks.html). You can also consult the output of +`eb --list-easyblocks`, which gives an overview of *all* known easyblocks, and how they relate to each other. + +**Custom easyconfig parameters** + +Most generic easyblocks provide additional easyconfig parameters to steer their behaviour. +You can consult these via "`eb -a --easyblock`" or just "`eb -a -e`", which results in an +additional "`EASYBLOCK-SPECIFIC`" section to be added. See the (partial) output of this command for example: + +```shell +$ eb -a -e ConfigureMake +Available easyconfig parameters (* indicates specific to the ConfigureMake easyblock): +... +EASYBLOCK-SPECIFIC +------------------ +build_cmd* Build command to use [default: "make"] +build_type* Value to provide to --build option of configure script, e.g., x86_64-pc-linux-gnu (determined by config.guess shipped with EasyBuild if None, False implies to leave it up to the configure script) [default: None] +configure_cmd* Configure command to use [default: "./configure"] +configure_cmd_prefix* Prefix to be glued before ./configure [default: ""] +host_type* Value to provide to --host option of configure script, e.g., x86_64-pc-linux-gnu (determined by config.guess shipped with EasyBuild if None, False implies to leave it up to the configure script) [default: None] +install_cmd* Build command to use [default: "make install"] +prefix_opt* Prefix command line option for configure script ('--prefix=' if None) [default: None] +tar_config_opts* Override tar settings as determined by configure. [default: False] +``` + +#### Dependencies + +You will often need to list one or more [dependencies](../introduction/#dependencies) that are required +to install or run the software. +We distinguish between two main different types of dependencies: runtime dependencies and build dependencies. + +*Runtime dependencies* are required for using the installed software, and may also have to be available +during the installation. These dependencies can be specified via the `dependencies` easyconfig parameter. +EasyBuild will load the modules for these dependencies when setting up the build environment, +and will include load statements for them in the generated module file. + +*Build dependencies* are only required during the installation of the software, not for using the +software once it is installed. The modules for these dependencies will be loaded in the build environment +set up by EasyBuild during the installation, but they will *not* be loaded by the generated module file. +You can specify build dependencies via the `builddependencies` easyconfig parameter. +One typical example of a build dependency is `CMake`, which is only needed for configuring +the build. + +Here is a simple example of specifying dependencies: + +```python +builddependencies = [('CMake', '3.18.4')] + +dependencies = [ + ('Python', '3.8.2'), + ('HDF5', '1.10.6'), + ('SciPy-bundle', '2020.03', '-Python-%(pyver)s'), +] +``` + +Both `builddependencies` and `dependencies` require a list of tuples, +each of which specifying one dependency. +The name and version of a dependency is specified with a 2-tuple (a tuple with two string values). + +In some cases additional information may have to be provided, as is shown in the example above for the `SciPy-bundle` +dependency where a 3rd value is specified corresponding to the `versionsuffix` value of this dependency. +If this is not specified, it is assumed to be the empty string (`''`). + +Note how we use the '`%(pyver)s'` template value in the `SciPy-bundle` dependency +specification, to avoid hardcoding the Python version in different places. + +See also the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#dependencies) for additional options on specifying dependencies. + + +#### Version suffix + +In some cases you may want to build a particular software package in different configurations, or include a label in the module name to highlight a particular aspect +of the installation. + +The `versionsuffix` easyconfig parameter can be used for this purpose. The name of this parameter implies that this label will be added after the +software version (and toolchain label) in the standard module naming scheme. + +If you are configuring the software to build with a particular non-default value, +you can indicate this as follows for example: + +```python +versionsuffix = '-example-label' +``` + +This mechanism is used frequently to indicate that a software installation depends +on a particular version of Python, where the `%(pyver)s` template comes in useful again: + +```python +versionsuffix = '-Python-%(pyver)s' +... +dependencies = [('Python', '2.7.18')] +``` + +Even though Python 2 is officially dead and +buried some scientific software still requires it, and mixing modules where +some use Python 2 and other use Python 3 doesn't work well. +The `versionsuffix` label is helpful to inform the user that a particular Python version is required by the installation. + +#### Customizing configure, build, test, and install commands + +When using a generic easyblock like `ConfigureMake` or `CMakeMake`, you will often +find yourself having to specify options to the configure, build, test, or install commands, or to inject additional commands right before them. + +For this the following standard easyconfig parameters are available: + +* `preconfigopts`: string value that is glued *before* the configure command; +* `configopts`: string value that is added *after* the configure command, which can be used to specify configuration options; + +Equivalent easyconfig parameters are available for the `build`, `test` and `install` steps: `prebuildopts`, `buildopts`, `pretestopts`, `testopts`, `preinstallopts`, and `installopts`. + +Here is a fictitious example of how they can be used: + +```python +easyblock = 'ConfigureMake' +... +dependencies = [('HDF5', '1.10.6')] +... +configopts = '--enable-hdf5-support' + +prebuildopts = 'export HDF5_PREFIX="$EBROOTHDF5" && ' + +installopts = "PREFIX='%(installdir)s'" +``` + +Here we are: + +* Adding the `--enable-hdf5-support` configure option, to convince the `ConfigureMake` easyblock to run the following + command during the configure step: + ```shell + ./configure --prefix ... --enable-hdf5-support + ``` + (where the '`...`' represents the path to installation directory where the software should be installed). + +* Specifying that an additional command has to be run before running `make` in the build step. We use '`&&`' to glue the + command to the `make` command, so `make` will only be run if the command we specified ran correctly. So, the build step will run something like: + ```shell + export HDF5_PREFIX="$EBROOTHDF5" && make -j 4 + ``` + The '`4`' value passed to the `-j` option shown here, which specifies how many commands `make` can run in parallel, is automatically determined by EasyBuild based on the number of available cores (taking into account `ulimit` settings, and cpuset and cgroup restrictions). + +* Passing the location where the software should be installed via the `PREFIX` argument to the `make install` command during the installation step. This results in the + following command being run: + ```shell + make install PREFIX=... + ``` + (where the '`...`' again represents the path to installation directory). + Even though the + installation directory is already specified in the configure command, it is + apparently blatantly ignored by the software we are installing here, and we are expected to specify it + this way instead. How rude! + +The `$EBROOTHDF5` environment variable that we are using in `prebuildopts` corresponds to the path of +the installation directory of the HDF5 dependency. EasyBuild includes a statement +to define an `$EBROOT*` environment variable +like this in every environment module file it generates (see the output of "`module show HDF5`"). + +#### Sanity check + +One seemingly trivial yet important aspect of the installation procedure that EasyBuild performs +is the sanity check step. + +By default EasyBuild does a simple sanity check that verifies whether there is a non-empty `bin` subdirectory +in the installation, next to a non-empty `lib` or `lib64` directory (either is sufficient). + +It is recommended to customize the sanity check and check for something more specific, like a particular +binary or directory, or making sure that a trivial command (like `example -V` or `example --help`) +runs correctly. + +To specify a custom set of files and/or directories to check, +you can use the `sanity_check_paths` easyconfig parameter. The expected value is Python dictionary +with two keys: `files` and `dirs`. For example: + +```python +sanity_check_paths = { + 'files': ['bin/example'], + 'dirs': ['examples/one', 'examples/two'], +} +``` + +In addition, you can specify one or more commands that should be working without a problem (that is, have a zero exit status) via the `sanity_check_commands` easyconfig parameter. +These commands will be run just like a user would: after loading the module that was generated for this installation. +Here is an example: + +```python +sanity_check_commands = [ + "example --version", + "example --help", +] +``` + + +#### Module class + +Finally, you will usually see the `moduleclass` easyconfig parameter to be defined as well, for example: + +```python +moduleclass = 'lib' +``` + +This is done to categorize software, and it is used to group the generated module files into smaller sets ([remember what we saw when installing software earlier](../basic_usage/#using-installed-software)). + +## Generating tweaked easyconfigs + +Sometimes you may want to install software that differs only slightly from an +existing easyconfig file, like a newer software version or using a different +compiler toolchain. Do we need to create an easyconfig file for this too? + +We do, but EasyBuild does provide some help so you don't need to *manually* +create the easyconfig file. You can use one of the `--try-*` options provided +by the `eb` command to make EasyBuild *generate* a new easyconfig file based on +an existing one. + +For example, to try installing a different software version you can use the `--try-software-version` option: + +```shell +eb example-1.2.3.eb --try-software-version 1.2.4 +``` + +Or, to try using a different compiler toolchain you can use `--try-toolchain`: + +```shell +eb example-1.2.3-foss-2020b.eb --try-toolchain intel,2020b +``` + +It is important to keep in mind the *"try"* aspect here: while easyconfigs that +are generated by EasyBuild via a `--try-*` option often do work fine, there is +no strong guarantee they will. Newer software versions may come with changes to +the installation procedure, additional dependencies that are required, etc. +Using a different compiler toolchain may be as simple as just switching one for +another, but it may require additional changes to be made to configure options, for example. + +## Copying easyconfigs + +One additional handy command line option we want to highlight is `--copy-ec`, which can be used to +copy easyconfig files to a specific location. That may sound trivial, but +keep in mind that you can specify easyconfigs to the `eb` command using only +the filename, and letting the robot search mechanism locate them. + +So to copy an easyconfig file, we would have to use `eb --search` first to +get the full location to it, copy-paste that, and then use the `cp` command. + +It is a lot easier with `--copy-ec`: + +```shell +$ eb --copy-ec SAMtools-1.11-GCC-10.2.0.eb SAMtools.eb +... +SAMtools-1.10-GCC-10.2.0.eb copied to SAMtools.eb +``` + +If you omit the target location, the easyconfig file will simply be copied +to the current working directory, retaining the original filename. + +You can copy multiple easyconfig files at once, as long as the target location +is an existing directory. + +## Example + +By means of example, we are going to puzzle together an easyconfig file to install the +example software package `eb-tutorial`. + +The sources for `eb-tutorial` version 1.0.1 are available at: + +``` +https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.0.1.tar.gz +``` + +You can consult the unpacked sources at [https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1). + +### Preparation + +Make sure EasyBuild is properly configured before you start: + +```shell +export EASYBUILD_PREFIX=$HOME/easybuild +export EASYBUILD_BUILDPATH=/tmp/$USER +``` + +and that the installed software in `/easybuild` is available: + +```shell +module use /easybuild/modules/all +``` + +### Mandatory parameters + +Let's start by getting the mandatory easyconfig parameters defined in the easyconfig file: + +```python +name = 'eb-tutorial' +version = '1.0.1' + +homepage = 'https://easybuilders.github.io/easybuild-tutorial' +description = "EasyBuild tutorial example" +``` + +If we try using this (very) minimal easyconfig file, EasyBuild will inform us that we failed to specify one of the mandatory easyconfig parameters: `toolchain`: + +``` +$ eb example.eb +== Temporary log file in case of crash /tmp/eb-90j723rl/easybuild-q21plqvx.log +== found valid index for /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs, so using it... +ERROR: Failed to process easyconfig /home/example/example.eb: mandatory parameters not provided in pyheader: toolchain +``` + +We will use `GCC/10.2.0` as toolchain, since we know it is already installed in `/easybuild` in the prepared environment, so we also define the `toolchain` easyconfig parameter: + +```python +toolchain = {'name': 'GCC', 'version': '10.2.0'} +``` + +In addition, we'll also specify the `moduleclass`. +This is not required, but it is usually set to a sensible value: + +```python +moduleclass = 'tools' +``` + +The default value is '`base`', at least '`tools`' has *some* meaning. + +### Easyblock + +Let us see what happens if we take our current easyconfig file for a spin: + +```shell +$ eb example.eb +== temporary log file in case of crash /tmp/eb-8_vxjfn7/easybuild-k3aaoan2.log +ERROR: Failed to process easyconfig /home/example/example.eb: +No software-specific easyblock 'EB_eb_minus_tutorial' found for eb-tutorial +``` + +That didn't get us very far... + +The error shows that there is no software-specific easyblock available for installing the software with the name '`eb-tutorial`'. +Does that mean we have to implement an easyblock? + +In this simple case it doesn't, since we can leverage one of the available *generic easyblocks*. +But, which one? + +Build instructions are usually included in a `README` file, or in the documentation. +In this case, there's indeed a minimal [`README` +file](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1/README) available, +which tells us that we should use the `cmake` command to configure the installation, followed by `make` and `make install`. + +[We briefly discussed](#easyblock) a generic easyblock that does exactly this: `CMakeMake`. + +```python +easyblock = 'CMakeMake' +``` + +The "`easyblock =`" line is usually at the top of the easyconfig file, but strictly speaking +the order of the parameter definitions doesn't matter (unless one is defined in terms of another one). + +### CMake build dependency + +Does using the `CMakeMake` generic easyblock help at all? + +``` +$ eb example.eb +== temporary log file in case of crash /tmp/eb-yutbor1p/easybuild-4jc9v1u9.log +== found valid index for /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs, so using it... +== processing EasyBuild easyconfig /home/example/example.eb +== building and installing eb-tutorial/1.0.1-GCC-10.2.0... +== fetching files... +== creating build dir, resetting environment... +== unpacking... +== patching... +== preparing... +== configuring... +== FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/ebtutorial/1.0.1/GCC-10.2.0): +build failed (first 300 chars): cmd " cmake -DCMAKE_INSTALL_PREFIX=/home/example/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER='gcc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno' -DCMAKE_CXX_COMPILER='g++' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=native (took 0 sec) +``` + +It did help: EasyBuild made an attempt to configure the build using the `cmake` command, but that failed almost +instantly. We need to dive into the log file to see the actual reason. By starting at the end of the log file and +scrolling up, you should be able to locate the following error message: + +``` +/bin/bash: cmake: command not found +``` + +Ah, that explains it, `cmake` isn't even installed on this system. Or is it? + +```shell +$ module avail CMake + +--------------------------- /easybuild/modules/all ---------------------------- + CMake/3.18.4-GCCcore-10.2.0 +``` + +Since a module is available for `CMake` that is compatible with the toolchain we are using (GCC 10.2.0), +we can use it as a dependency for the installation. +It is only needed for building the software, not for running it, so it's only a *build* dependency: + +```python +builddependencies = [('CMake', '3.18.4')] +``` + +There is usually no need to specify toolchain for (build) dependencies, EasyBuild will automatically consider +[subtoolchains](../terminology/#toolchains) compatible with the specified toolchain to locate module for the dependencies. + +You can verify this via `eb -D` (equivalent with `eb --dry-run`): + +``` +$ eb example.eb -D + ... + * [x] /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs/g/GCC/GCC-10.2.0.eb (module: GCC/10.2.0) + * [x] /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs/c/CMake/CMake-3.18.4-GCCcore-10.2.0.eb (module: CMake/3.18.4-GCCcore-10.2.0) + * [ ] /home/example/example.eb (module: eb-tutorial/1.0.1-GCC-10.2.0) +``` + +### Sources + +If you try again after adding `CMake` as a build dependency, you will see the installation fail again in the +configuration step. Inspecting the log file reveals this: + +``` +CMake Error: The source directory "/tmp/example/ebtutorial/1.0.1/GCC-10.2.0" does not appear to contain CMakeLists.txt. +``` + +Wait, but there *is* a `CMakeLists.txt`, we can see it in the [unpacked sources](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1)! + +Let's inspect the build directory: + +``` +$ ls /tmp/$USER/ebtutorial/1.0.1/GCC-10.2.0 +easybuild_obj +$ ls /tmp/$USER/ebtutorial/1.0.1/GCC-10.2.0/easybuild_obj +$ +``` + +There's nothing there at all! And that's not strange because we didn't actually +specify any sources in our easyconfig file... + +The `sources` easyconfig parameter is commonly defined but it is *not* mandatory, +because some easyconfig files only specify bundles of software packages and hence only +serve to generate a module file. + +So we need to specify one or more source files that should be used, +via the `sources` easyconfig parameter which specifies a *list* of +names of source files: + +```python +sources = ['eb-tutorial-1.0.1.tar.gz'] +``` + +We can avoid hardcoding the version number here by using a *template value*: + +```python +sources = ['eb-tutorial-%(version)s.tar.gz'] +``` + +And since this is a standard way of naming software files, there's +even a constant available that we can use: + +```python +sources = [SOURCE_TAR_GZ] +``` + +That way, we only have the software version specified *once* in the easyconfig file, +via the `version` easyconfig parameter. That will come in useful later (see [Exercise 7.2](#exercises))... + +If now we try installing the easyconfig file again, EasyBuild complains +that it can't find the specified source file anywhere: + +``` +Couldn't find file eb-tutorial-1.0.1.tar.gz anywhere, and downloading it didn't work either... +``` + +To let EasyBuild automatically download the source file if it is not available yet, +we have to specify *where* it can be downloaded. This is done via `source_urls`: + +```python +source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] +sources = [SOURCE_TAR_GZ] +``` + +### Required configure option + +With `sources` and `source_urls` defined, we can try again. Yet again we see the configure step fail. +Is this a ruse to make you hate CMake with a passion? Maybe... + +Here's what we find in the log file: + +``` +CMake Error at CMakeLists.txt:7 (message): + EBTUTORIAL_MSG is not set! +``` + +Apparently the `eb-tutorial` software has a required configure option. It's almost as if that +was done on purpose, how silly! + +Options to the configure command can be specified by the `configopts` easyconfig parameter. +To define the value of a CMake option, we need to use `-DNAME_OF_OPTION`, so: + +```python +configopts = "-DEBTUTORIAL_MSG='Hello from the EasyBuild tutorial!' " +``` + +We need to be a little bit careful with quotes here. If we use outer double quotes, +we have to use single quotes to specify the actual value for the `EBTUTORIAL_MSG` configure option. +That works fine here, but that's not always the case! +In some cases we will have to use inner doubles quotes, for example to get environment variables +expanded when the configure command is run (see [Exercise 7.1](#exercises)). + +### Sanity check + +Hopefully that brings us closer to getting the installation to work... + +``` +$ eb example.eb +.... +== sanity checking... +== FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/ebtutorial/1.0.1/GCC-10.2.0): build failed (first 300 chars): +Sanity check failed: no (non-empty) directory found at 'lib' or 'lib64' in /home/easybuild/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 (took 2 sec) +``` + +It got all the way to the sanity check step, that's great! + +The sanity check failed because no '`lib`' or `'lib64'` directory was found. +Indeed: + +``` +$ ls $HOME/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 +bin +$ ls $HOME/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0/bin +eb-tutorial +``` + +There is only a binary named `eb-tutorial` in the `bin` subdirectory. +So we need to customize the standard sanity check: + +```python +sanity_check_paths = { + 'files': ['bin/eb-tutorial'], + 'dirs': [], +} +``` + +Since we want to obtain a *working* installation, we might as well try to run this `eb-tutorial` command as well: + +```python +sanity_check_commands = ['eb-tutorial'] +``` + +Let us now retry, but use `--module-only` rather than redoing the whole installation. +`--module-only` still sanity checks the installation, so if it creates +a module, we know it will work as expected. +By enabling trace mode via `--trace` we can get some more information too: + +```shell +$ eb example.eb --module-only --trace +... +== sanity checking... + >> file 'bin/eb-tutorial' found: OK + >> running command 'eb-tutorial' ... + >> result for command 'eb-tutorial': OK +... +== COMPLETED: Installation ended successfully (took 4 sec) +``` + +Yes, great success! + +To convince yourself that the installation works as intended, try to load the `eb-tutorial` module and +run the `eb-tutorial` command yourself: + +``` +$ module use $HOME/easybuild/modules/all +$ module load eb-tutorial +$ eb-tutorial +Hello from the EasyBuild tutorial! +``` + +### Complete easyconfig + +Here is the complete easyconfig we puzzled together for this example: + +```python +easyblock = 'CMakeMake' + +name = 'eb-tutorial' +version = '1.0.1' + +homepage = 'https://easybuilders.github.io/easybuild-tutorial' +description = "EasyBuild tutorial example" + +toolchain = {'name': 'GCC', 'version': '10.2.0'} + +source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] +sources = [SOURCE_TAR_GZ] +checksums = ['d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9'] + +builddependencies = [('CMake', '3.18.4')] + +configopts = "-DEBTUTORIAL_MSG='Hello from the EasyBuild tutorial!' " + +sanity_check_paths = { + 'files': ['bin/eb-tutorial'], + 'dirs': [], +} + +sanity_check_commands = ['eb-tutorial'] + +moduleclass = 'tools' +``` + +One additional easyconfig parameter we defined here is `checksums`, +which specifies SHA256 checksums for source (and patch) files, +so EasyBuild can verify them before performing an installation. + +You can let EasyBuild determine *and* inject these SHA256 checksums +automatically via `eb --inject-checksums`: + +``` +$ eb example.eb --inject-checksums +... +== injecting sha256 checksums for sources & patches in example.eb... +== * eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 +``` + +--- + +## Exercises + +--- + +***Exercise E.1**** - Making `eb-tutorial` a bit more personal* + +Change the easyconfig file for `eb-tutorial` to make the message printed by the `eb-tutorial` command +a bit more personal: include the username of the account that was used to install the software in it +(using the `$USER` environment variable). + +??? success "(click to show solution)" + For this we need to change the value that is passed to the `EBTUTORIAL_MSG` configure option: + ```python + configopts = '-DEBTUTORIAL_MSG="Hello from the EasyBuild tutorial! I was installed by $USER." ' + ``` + Here we have to use inner double quotes, to ensure that the `$USER` environment variable is expanded + by the shell when running the `cmake` configure command. + + When you run the `eb-tutorial` command yourself, you should get output like this (not a message that + includes a literal '`$USER`' string): + + ```shell + Hello from the EasyBuild tutorial! I was installed by example. + ``` + + To re-install the `example.eb` easyconfig, you will need to use `eb --rebuild` or `eb --force`. + +--- + +***Exercise E.2**** - Installing eb-tutorial version 1.1.0* + +Install version 1.1.0 of the `eb-tutorial` example software, +which is a trivial version bump compared to version 1.0.1. + +The sources are available via: + +``` +https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.1.0.tar.gz +``` + +You can leverage the `eb-tutorial` easyconfig file we have composed in the example above, +**but you should not make any manual changes to it!** + +??? success "(click to show solution)" + You can use the `--try-software-version` option for this: + ```shell + $ eb example.eb --try-software-version 1.1.0 + ... + == building and installing eb-tutorial/1.1.0-GCC-10.2.0... + ... + == COMPLETED: Installation ended successfully (took 4 sec) + ``` + To test: + ``` + $ module use $HOME/easybuild/modules/all + $ module load eb-tutorial/1.1.0-GCC-10.2.0 + $ eb-tutorial + I have a message for you: + Hello from the EasyBuild tutorial! + ``` + (`eb-tutorial` version 1.0.1 doesn't print "`I have a message for you:`") + +--- + +***Exercise E.3**** - Installing py-eb-tutorial 1.0.0* + +Try composing an easyconfig file for the `py-eb-tutorial` example software, which is a tiny Python package. +The source tarball can be downloaded from this link: [py-eb-tutorial-1.0.0.tar.gz](https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/py-eb-tutorial-1.0.0.tar.gz). + +A couple of tips: + +* There is a [generic easyblock](#easyblock) available for installing Python packages, which will come in useful here. + +* By default EasyBuild performs an `import` check when install Python packages, using a Python module name that is derived from the software name by default, which will be incorrect in this case. You can specify the correct name to use in the import check by specifying it via the `options` +easyconfig parameter in your easyconfig file: + ```python + options = {'modulename': 'example'} + ``` + (you will need to change '`example`' here, of course) + +* Leverage the software that is already pre-installed in `/easybuild` in the prepared environment. + Remember that some already installed modules may be a *bundle* of a couple of other software packages. + +Please also take this into account: + +* Unfortunately this software doesn't come with documentation. That is done to make it an example that + is representative for software that you may run into in the wild (it's *not* because + we were lazy when preparing the exercises, really!). + You can inspect the sources of this software [here](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/py-eb-tutorial-1.0.0). Definitely take a look at the `setup.py` file, it includes some clues + about the requirements to get this software installed. + +* Make sure the installation actually *works*, by checking that the `py-eb-tutorial` command runs correctly. + Maybe you will need to make sure other required software is available as well, for it to work correctly... + + +??? success "(click to show solution)" + + Here is a complete working easyconfig file for `py-eb-tutorial`: + ```python + + easyblock = 'PythonPackage' + + name = 'py-eb-tutorial' + version = '1.0.0' + versionsuffix = '-Python-%(pyver)s' + + homepage = 'https://easybuilders.github.io/easybuild-tutorial' + description = "EasyBuild tutorial Python example" + + source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] + sources = [SOURCE_TAR_GZ] + checksums = ['fcf73a9efc65527a210b993e8889d41ebf05977eef1f6a65ebac3188152cd496'] + + toolchain = {'name': 'foss', 'version': '2020b'} + + dependencies = [ + ('Python', '3.8.6'), + ('SciPy-bundle', '2020.11'), + ('eb-tutorial', '1.0.1'), + ] + + use_pip = True + + options = {'modulename': 'eb_tutorial'} + + sanity_check_paths = { + 'files': ['bin/py-eb-tutorial'], + 'dirs': ['lib/python%(pyshortver)s/site-packages'], + } + + sanity_check_commands = ["py-eb-tutorial"] + + moduleclass = 'tools' + ``` + + Some remarks: + + * We used the `PythonPackage` generic easyblock. There is also a `PythonBundle` easyblock for installing + bundles of Python packages, which is used for `SciPy-bundle` for example. But we don't need that here, + since we are only dealing with a single Python package. + + * The `versionsuffix` is not strictly needed, but it's common to tag Python packages with the Python version + for which they were installed. + * The SHA256 checksum for the source tarball was added automatically via `eb py-eb-tutorial.eb --inject-checksums`. + + * `py-eb-tutorial` only wants to be installed with `pip install`, so we had to set `use_pip = True`. + You can consult the custom easyconfig parameters supported by the `PythonPackage` easyblock via + "`eb -a -e PythonPackage`", see the `EASYBLOCK-SPECIFIC` part of the output. + Even when the default installation mechanism used by `PythonPackage` + (which consists of running `python setup.py install`) works fine, + it is recommended to instruct EasyBuild to use `pip install` instead. + + * By default EasyBuild will try to import `py_eb_tutorial`, while the actual name of the Python package + provided by `py-eb-tutorial` is just `eb_tutorial`. We fixed this by specifying the correct Python module name to + use via `options`. + + * Strictly speaking we don't need to specify a custom `sanity_check_paths`, since the default used + by Python package is already pretty decent (it will check for a non-empty `lib/python3.8/site-packages` + directory in the installation). We also want to make sure the `py-eb-tutorial` command is available in + the `bin` subdirectory however. Hardcoding to `python3.8` can be avoided using the `%(pyshortver)s` + template value. + + * A good way to check whether the `py-eb-tutorial` command works correctly is by running it as a sanity check + command. If the `eb-tutorial` command is not available the `py-eb-tutorial` command will fail, + since it basically just runs the `eb-tutorial` command. So we need to include `eb-tutorial` as a (runtime) + dependency in the `py-eb-tutorial` easyconfig file. + +*[[next: Implementing easyblocks]](2_03_implementing_easyblocks.md)* diff --git a/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md new file mode 100644 index 00000000..5355548e --- /dev/null +++ b/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md @@ -0,0 +1,665 @@ +# Implementing easyblocks + +*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* + +--- + +## The basics + +An *easyblock* is a Python module that implements a software build and installation procedure. + +This concept is essentially implemented as a Python script that plugs into the EasyBuild framework. + +EasyBuild will leverage easyblocks as needed, depending on which software packages it needs to install. Which easyblock is required is determined by the ``easyblock`` easyconfig parameter, if it is present, or by the software name. + + +## Generic vs software-specific easyblocks + +Easyblocks can either be *generic* or *software-specific*. + +*Generic* easyblocks implement a "standard" software build and installation procedure that is used by multiple different +software packages. +A commonly used example is the +[``ConfigureMake``](https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/generic/configuremake.py) +generic easyblock, which implements the standard ``configure`` - ``make`` - ``make install`` installation procedure used +by most GNU software packages. + +*Software-specific* easyblocks implement the build and installation procedure for a particular software package. +Typically this involves highly customised steps, for example specifying dedicated configuration options, creating +or adjusting specific files, executing non-standard shell commands, etc. Usually a custom implementation of the +sanity check is also included. + +Using a generic easyblock requires specifying the ``easyblock`` parameter in the easyconfig file. +If it is not specified, EasyBuild will try and find the software-specific easyblock derived from the software name. + +The distinction between generic and software-specific easyblocks can be made based on the naming scheme that is used +for an easyblock (see below). + + +## Naming + +Easyblocks need to follow a strict naming scheme, to ensure that EasyBuild can pick them up as needed. This involves two aspects: + +* the name of the Python class; +* the name and location of the Python module file. + +### Python class name + +The name of the Python class is determined by the *software name* for software-specific easyblocks. +It consists of a prefix '``EB_``', followed by the (encoded) software name. + +Because of limitations in Python on characters allowed in names of Python classes, +only alphanumeric characters and underscores (``_``) are allowed. Any other characters are replaced following an encoding scheme: + +* spaces are replaced by underscores (``_``); +* dashes ``-`` are replaced by ``_minus_``; +* underscores are replaced by ``_underscore_``; + +The ``encode_class_name`` function provided in ``easybuild.tools.filetools`` returns the expected class name +for a given software name; for example: + +```shell +$ python3 -c "from easybuild.tools.filetools import encode_class_name; print(encode_class_name('netCDF-Fortran'))" +EB_netCDF_minus_Fortran +``` + +**Python class name for *generic* easyblocks** + +For *generic* easyblocks, the class name does *not* include an ``EB_`` prefix (since there is no need for an escaping +mechanism) and hence the name is fully free to choose, taking into account the restriction to alphanumeric characters +and underscores. + +For code style reasons, the class name should start with a capital letter and use CamelCasing. + +Examples include ``Bundle``, ``ConfigureMake``, ``CMakePythonPackage``. + +### Python module name and location + +The *filename* of the Python module is directly related to the name of Python class it provides. + +It should: + +* *not* include the ``EB_`` prefix of the class name for software-specific easyblocks; +* consists only of lower-case alphanumeric characters (``[a-z0-9]``) and underscores (``_``); + * dashes (``-``) are replaced by underscores (``_``); + * any other non-alphanumeric characters (incl. spaces) are simply dropped; + +Examples include: + +* ``gcc.py`` (for *GCC*) +* ``netcdf_fortran.py`` (for *netCDF-Fortran*) +* ``gamess_us.py`` (for *GAMESS (US)*) + +The ``get_module_path`` function provided by the EasyBuild framework in the +``easybuild.framework.easyconfig.easyconfig`` module returns the (full) +module location for a particular software name or easyblock class name. For example: + +```python +>>> from easybuild.framework.easyconfig.easyconfig import get_module_path +>>> get_module_path('netCDF-Fortran') +'easybuild.easyblocks.netcdf_fortran' +>>> get_module_path('EB_netCDF_minus_Fortran') +'easybuild.easyblocks.netcdf_fortran' +``` + +The location of the Python module is determined by whether the easyblock is generic or software-specific. +Generic easyblocks are located in the ``easybuid.easyblocks.generic`` namespace, while software-specific easyblocks +live in the ``easybuild.easyblocks`` namespace directly. + +To keep things organised, the actual Python module files +for software-specific easyblocks are kept in 'letter' subdirectories, +rather than in one large '``easyblocks``' directory +(see +[https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/](https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/)), +but this namespace is collapsed transparently by EasyBuild (you don't need to import from letter subpackages). + +To let EasyBuild pick up one or more new or customized easyblocks, you can use the [``--include-easyblocks``](https://docs.easybuild.io/en/latest/Including_additional_Python_modules.html#including-additional-easyblocks-include-easyblocks) +configuration option. As long as both the filename of the Python module and the name of the Python class +are correct, EasyBuild will use these easyblocks when needed. + +## Structure of an easyblock + +The example below shows the overal structure of an easyblock: + +```python +from easybuild.framework.easyblock import EasyBlock +from easybuild.tools.run import run_cmd + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def configure_step(self): + """Custom implementation of configure step for Example""" + + # run configure.sh to configure the build + run_cmd("./configure.sh --install-prefix=%s" % self.installdir) +``` + + +Each easyblock includes an implementation of a ``class`` that (directly or indirectly) derives from the abstract +``EasyBlock`` class. + +Typically some useful functions provided by the EasyBuild framework are imported at the top of the Python module. + +In the class definition, one or more '``*_step``' methods (and perhaps a couple of others) are redefined, +to implement the corresponding step in the build and installation procedure. + +Each easyblock *must* implement the ``configure``, ``build`` and ``install`` steps, since these are not implemented +in the abstract ``EasyBlock`` class. This could be done explicitly by redefining the corresponding ``*_step`` methods, +or implicitly by deriving from existing (generic) easyblocks. + +The full list of methods that can be redefined in an easyblock can be consulted in +the [API documentation](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyblock.html). + +## Deriving from existing easyblocks + +When implementing an easyblock, it is common to derive from an existing (usually generic) easyblock, +and to leverage the functionality provided by it. This approach is typically used when only a specific part +of the build and installation procedure needs to be customised. + +In the (fictitious) example below, we derive from the generic ``ConfigureMake`` easyblock to redefine the ``configure`` +step. In this case, we are *extending* the ``configure`` step as implemented by ``ConfigureMake`` rather than +redefining it entirely, since we call out to the original ``configure_step`` method at the end. + +```python +from easybuild.easyblocks.generic.configuremake import ConfigureMake +from easybuild.tools.filetools import copy_file + + +class EB_Example(ConfigureMake): + """Custom easyblock for Example""" + + def configure_step(self): + """Custom implementation of configure step for Example""" + + # use example make.cfg for x86-64 + copy_file('make.cfg.x86', 'make.cfg') + + # call out to original configure_step implementation of ConfigureMake easyblock + super(EB_Example, self).configure_step() +``` + +## Easyconfig parameters + +All of the easyconfig parameters that are defined in an easyconfig file +are available via the ``EasyConfig`` instance that can be accessed through ``self.cfg`` in an easyblock. +For instance, if the easyconfig file specifies + +```python +name = 'example' +version = '2.5.3' +versionsuffix = '-Python-3.7.4' +``` + +then these three parameters are accessible within an easyblock via ``self.cfg['name']``, ``self.cfg['version']`` +and ``self.cfg['versionsuffix']``. + +A few of the most commonly used parameters can be referenced directly: + +* **``self.name``** is equivalent with ``self.cfg['name']``; +* **``self.version``** is equivalent with ``self.cfg['version']``; +* **``self.toolchain``** is equivalent with ``self.cfg['toolchain']``; + + +### Updating parameters + +You will often find that you need to *update* some easyconfig parameters in an easyblock, +for example ``configopts`` which specifies options for the configure command. + +Because of implementation details (related to +how template values like ``%(version)s`` are handled), you need to be a bit careful here... + +To completely redefine the value of an easyconfig parameter, you can use simple assignment. For example: + +```python +self.cfg['example'] = "A new value for the example easyconfig parameter." +``` + +If want to *add* to the existing value however, you *must* use the ``self.cfg.update`` method. For example: + +```python +self.cfg.update('some_list', 'example') +``` + +This will ***not*** work (because ``self.cfg['some_list']`` does not return a reference to the original value, +but to a *temporary copy* thereof): + +```python +# anti-pattern, this does NOT work as expected! +self.cfg['some_list'].append('example') +``` + +### Custom parameters + +Additional custom easyconfig parameters can be defined in an easyblock to allowing steering its behaviour. +This is done via the ``extra_options`` *static* method. Custom parameters can be specified to be mandatory. + +The example below shows how this can be implemented: + +```python +from easybuild.easyblocks.generic.configuremake import ConfigureMake +from easybuild.framework.easyconfig import CUSTOM, MANDATORY + + +class EB_Example(ConfigureMake): + """Custom easyblock for Example""" + + @staticmethod + def extra_options(): + """Custom easyconfig parameters for Example""" + extra_vars = { + 'required_example_param': [None, "Example required custom parameter", MANDATORY], + 'optional_example_param': [None, "Example optional custom parameter", CUSTOM], + } + return ConfigureMake.extra_options(extra_vars) +``` + +The first element in the list of a defined custom parameter corresponds to the default value for that parameter +(both ``None`` in the example above). The second element provides some informative help text, and the last element +indicates whether the parameter is mandatory (``MANDATORY``) or just an optional custom parameter (``CUSTOM``). + +## Easyblock constructor + +In the ``class`` constructor of the easyblock, i.e. the ``__init__`` method, one or more class variables +can be initialised. These can be used for sharing information between different ``*_step`` methods in the easyblock. + +For example: + +```python +from easybuild.framework.easyblock import EasyBlock + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def __init__(self, *args, **kwargs): + """Constructor for Example easyblock, initialises class variables.""" + + # call out to original constructor first, so 'self' (i.e. the class instance) is initialised + super(EB_Example, self).__init__(*args, **kwargs) + + # initialise class variables + self.example_value = None + self.example_list = [] +``` + +## File operations + +File operations is a common use case for implementing easyblocks, hence the EasyBuild framework provides a +number of useful functions related to this, including: + +* ``read_file()``: read file at a specified location and returns its contents; + +* ``write_file(, )`` at a specified location with provided contents; + to append to an existing file, use ``append=True`` as an extra argument; + +* ``copy_file(, )`` to copy an existing file; + +* ``apply_regex_substitutions(, )`` to patch an existing file; + +All of these functions are provided by the [``easybuild.tools.filetools``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html) module. + +## Executing shell commands + +For executing shell commands two functions are provided by the +[``easybuild.tools.run``](https://docs.easybuild.io/en/latest/api/easybuild.tools.run.html) module: + +* ``run_cmd()`` to run a non-interactive shell command; + +* ``run_cmd_qa(, )`` to run an interactive shell command; + +Both of these accept a number of optional arguments: + +* ``simple=True`` to just return ``True`` or ``False`` to indicate a successful execution, + rather than the default return value, i.e., a tuple that provides the command output and the exit code (in that + order); + +* ``path=`` to run the command in a specific subdirectory; + +The ``run_cmd_qa`` function takes two additional specific arguments: + +* ``no_qa=`` to specify a list of patterns to recognize non-questions; + +* ``std_qa=`` to specify regular expression patterns for common questions, and the matching answer; + +## Manipulating environment variables + +To (re)define environment variables, the ``setvar`` function provided by the +[``easybuild.tools.environment``](https://docs.easybuild.io/en/latest/api/easybuild.tools.environment.html) +module should be used. + +This makes sure that the changes being made to the specified environment variable are kept track of, +and that they are handled correctly under ``--extended-dry-run``. + +## Logging and errors + +It is good practice to include meaningful log messages in the ``*_step`` methods being customised in the easyblock, +to enrich the EasyBuild log with useful information for later debugging or diagnostics. + +For logging, the provided ``self.log`` logger class should be used. +You can use the ``self.log.info`` method to log an informative message. +Similar methods are available for logging debug messages (``self.log.debug``), which are +only emitted when ``eb`` is run with debugging mode enabled (``--debug`` or ``-d``), +and for logging warning messages (``self.log.warning``). + +If something goes wrong, you can raise an ``EasyBuildError`` instance to report the error. + +For example: + +```python +from easybuild.framework.easyblock import EasyBlock +from easybuild.tools.build_log import EasyBuildError +from easybuild.tools.run import run_cmd + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def configure_step(self): + """Custom implementation of configure step for Example""" + + cmd = "./configure --prefix %s" % self.installdir) + out, ec = run_cmd(cmd) + + success = 'SUCCESS' + if success in out: + self.log.info("Configuration command '%s' completed with success." % cmd) + else: + raise EasyBuildError("Pattern '%s' was not found in output of '%s'." % (success, cmd)) +``` + +## Custom sanity check + +For software-specific easyblocks, a custom sanity check is usually included to verify that the installation was +successful or not. + +This is done by redefining the ``sanity_check_step`` method in the easyblock. For example: + +```python +from easybuild.framework.easyblock import EasyBlock + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def sanity_check_step(self): + """Custom sanity check for Example.""" + + custom_paths = { + 'files': ['bin/example'], + 'dirs': ['lib/examples/'], + } + custom_commands = ['example --version'] + + # call out to parent to do the actual sanity checking, pass through custom paths and commands + super(EB_Example, self).sanity_check_step(custom_paths=custom_paths, custom_commands=custom_commands) +``` + +You can both specify file paths and subdirectories to check for, which are specified relative to the installation directory, +as well as simple commands that should execute successfully after completing the installation and loading the generated module file. + +It is up to you how extensive you make the sanity check, but it is recommended to make the check as complete +as possible to catch any potential build or installation problems that may occur, while ensuring that it can +run relatively quickly (in seconds, or at most a couple of minutes). + +## Version-specific parts + +In some cases version-specific actions or checks need to be included in an easyblock. +For this, it is recommended to use ``LooseVersion`` rather than directly comparing version numbers using string values. + +For example: + +```python +from distutils.version import LooseVersion +from easybuild.framework.easyblock import EasyBlock + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def sanity_check_step(self): + """Custom sanity check for Example.""" + + custom_paths = { + 'files': [], + 'dirs': [], + } + + # in older versions, the binary used to be named 'EXAMPLE' rather than 'example' + if LooseVersion(self.version) < LooseVersion('1.0'): + custom_paths['files'].append('bin/EXAMPLE') + else: + custom_paths['files'].append('bin/example') + + super(EB_Example, self).sanity_check_step(custom_paths=custom_paths) +``` + +## Compatibility with ``--extended-dry-run`` and ``--module-only`` + +Some special care must be taken to ensure that an easyblock is fully compatible with ``--extended-dry-run`` / ``-x`` +(see [Inspecting install procedures](../basic_usage/#inspecting-install-procedures)) and ``--module-only``. + +For compatibility with ``--extended-dry-run``, you need to take into account that specified operations +like manipulating files or running shell commands will not actually be executed. You can check +whether an easyblock is being run in dry run mode via ``self.dry_run``. + +For example: + +```python +from easybuild.framework.easyblock import EasyBlock +from easybuild.tools.build_log import EasyBuildError +from easybuild.tools.run import run_cmd + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def configure_step(self): + """Custom implementation of configure step for Example""" + + cmd = "./configure --prefix %s" % self.installdir) + out, ec = run_cmd(cmd) + + success = 'SUCCESS' + if success in out: + self.log.info("Configuration command '%s' completed with success." % cmd) + + # take into account that in dry run mode we won't get any output at all + elif self.dry_run: + self.log.info("Ignoring missing '%s' pattern since we're running in dry run mode." % success) + + else: + raise EasyBuildError("Pattern '%s' was not found in output of '%s'." % (success, cmd)) +``` + +For ``--module-only``, you should make sure that both the ``make_module_step``, including the ``make_module_*`` submethods, +and the ``sanity_check_step`` methods do not make any assumptions about the presence of certain environment variables, or that class variables have been defined already. + +This is required because under ``--module-only`` the large majority of the ``*_step`` functions are +simply *skipped* entirely. So, if the ``configure_step`` method is responsible for defining class variables that are +picked up in ``sanity_check_step``, the latter may run into unexpected initial values like ``None``. +A possible workaround is to define a separate custom method to define the class variables, and to call out to this +method from ``configure_step`` and ``sanity_check_step`` (for the latter, conditionally, i.e., only if the class +variables still have the initial values). + +For example: + +```python +from easybuild.framework.easyblock import EasyBlock + + +class EB_Example(EasyBlock): + """Custom easyblock for Example""" + + def __init__(self, *args, **kwargs): + """Easyblock constructor.""" + super(EB_Example, self).__init__(*args, **kwargs) + + self.command = None + + def set_command(self): + """Initialize 'command' class variable.""" + # $CC environment variable set by 'prepare' step determines exact command + self.command = self.name + '-' + os.getenv('CC') + + def configure_step(self): + """Custom configure step for Example.""" + + self.set_command() + self.cfg.update('configopts', "COMMAND=%s" % self.command) + + super(EB_Example, self).configure_step() + + def sanity_check_step(self): + """Custom implementation of configure step for Example""" + + if self.command is None: + self.set_command() + + super(EB_Example, self).sanity_check_step(custom_commands=[self.command]) +``` + +## Exercise + +### Exercise I.1 + +Try implementing a new custom easyblock for ``eb-tutorial``, which derives directly +from the base ``EasyBlock`` class. + +Your easyblock should: + +* define a custom mandatory easyconfig parameter named ``message``; +* run `cmake` to configure the installation, which includes at least: + * specifying the correct installation prefix (using the `-DCMAKE_INSTALL_PREFIX=...` option); + * passing down the value of ``message`` easyconfig parameter via `-DEBTUTORIAL_MSG=...` +* run `make` to build `eb-tutorial`; +* run `make install` to install the generated binary; +* perform a custom sanity check to ensure the installation is correct; +* pick up on commonly used easyconfig parameters like `configopts` and `preinstallopts` where appropriate; + +??? success "(click to show solution)" + + Here's a complete custom easyblock for ``eb-tutorial`` that derives from the base ``EasyBlock`` class, + which should be included in a file named ``eb_tutorial.py``. + + We need to implement the ``configure_step``, ``build_step``, and ``install_step`` methods in + order to have a fully functional easyblock. + + The configure, build, and install steps take into account the corresponding easyconfig + parameters that allow customizing these commands from an easyconfig file. + + ```python + from easybuild.framework.easyblock import EasyBlock + from easybuild.framework.easyconfig import MANDATORY + from easybuild.tools.run import run_cmd + + + class EB_eb_minus_tutorial(EasyBlock): + """Custom easyblock for eb-tutorial.""" + + @staticmethod + def extra_options(): + extra = EasyBlock.extra_options() + extra.update({ + 'message': [None, "Message that eb-tutorial command should print", MANDATORY], + }) + return extra + + def configure_step(self): + """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" + + cmd = ' '.join([ + self.cfg['preconfigopts'], + "cmake", + "-DCMAKE_INSTALL_PREFIX='%s'" % self.installdir, + "-DEBTUTORIAL_MSG='%s'" % self.cfg['message'], + self.cfg['configopts'], + ]) + run_cmd(cmd) + + def build_step(self): + """Build step for eb-tutorial""" + + cmd = ' '.join([ + self.cfg['prebuildopts'], + "make", + self.cfg['buildopts'], + ]) + run_cmd(cmd) + + def install_step(self): + """Install step for eb-tutorial""" + + cmd = ' '.join([ + self.cfg['preinstallopts'], + "make install", + self.cfg['installopts'], + ]) + run_cmd(cmd) + + def sanity_check_step(self): + custom_paths = { + 'files': ['bin/eb-tutorial'], + 'dirs': [], + } + custom_commands = ['eb-tutorial'] + return super(EB_eb_minus_tutorial, self).sanity_check_step(custom_paths=custom_paths, + custom_commands=custom_commands) + ``` + +### Exercise I.2 + +Try implementing another new custom easyblock for ``eb-tutorial``, +which derives from the generic ``CMakeMake`` easyblock. + +Your easyblock should only: + +* define a custom mandatory easyconfig parameter named ``message``; +* pass down the value of ``message`` easyconfig parameter via `-DEBTUTORIAL_MSG=...` +* perform a custom sanity check to ensure the installation is correct; + +??? success "(click to show solution)" + + When deriving from the ``CMakeMake`` generic easyblock, there is a lot less to worry about. + + We only need to customize the ``configure_step`` method to ensure that the ``-DEBTUTORIAL_MSG`` configuration + option is specified; the ``CMakeMake`` easyblock already takes care of specifying the location of + the installation directory (and a bunch of other configuration options, like compiler commands and flags, etc.). + + Implementing the ``build_step`` and ``install_step`` methods is no longer needed, + the standard procedure that is run by the ``CMakeMake`` generic easyblock is fine, + and even goes beyond what we did in the previous exercise (like building in parallel with ``make -j``). + + ```python + from easybuild.easyblocks.generic.cmakemake import CMakeMake + from easybuild.framework.easyconfig import MANDATORY + from easybuild.tools.run import run_cmd + + class EB_eb_minus_tutorial(CMakeMake): + """Custom easyblock for eb-tutorial.""" + + @staticmethod + def extra_options(): + extra = CMakeMake.extra_options() + extra.update({ + 'message': [None, "Message that eb-tutorial command should print", MANDATORY], + }) + return extra + + def configure_step(self): + """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" + self.cfg.update('configopts', "-DEBTUTORIAL_MSG='%s'" % self.cfg['message']) + + super(EB_eb_minus_tutorial, self).configure_step() + + def sanity_check_step(self): + custom_paths = { + 'files': ['bin/eb-tutorial'], + 'dirs': [], + } + custom_commands = ['eb-tutorial'] + return super(EB_eb_minus_tutorial, self).sanity_check_step(custom_paths=custom_paths, + custom_commands=custom_commands) + ``` + +*[[next: PArt 3: OVerview]](3_00_part3_advanced.md)* diff --git a/docs/2022-CSC_and_LO/3_00_part3_advanced.md b/docs/2022-CSC_and_LO/3_00_part3_advanced.md new file mode 100644 index 00000000..708c57f8 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_00_part3_advanced.md @@ -0,0 +1,9 @@ +# Part III: Advanced topics + +*[[back to start page]](index.md)* + +* [Using EasyBuild as a library](3_01_easybuild_library.md) *(hands-on)* +* [Using hooks to customise EasyBuild](3_02_hooks.md) *(hands-on)* +* [Submitting installations as Slurm jobs](3_03_slurm_jobs.md) *(hands-on)* +* [Module naming schemes (incl. hierarchical)](3_04_module_naming_schemes.md) *(hands-on)* +* [GitHub integration to facilitate contributing to EasyBuild](3_05_github_integration.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/3_01_easybuild_library.md b/docs/2022-CSC_and_LO/3_01_easybuild_library.md new file mode 100644 index 00000000..c037f49f --- /dev/null +++ b/docs/2022-CSC_and_LO/3_01_easybuild_library.md @@ -0,0 +1,293 @@ +# Using EasyBuild as a library + +*[[back: Configuring EasyBuild]](1_07_configuration.md)* + +--- + +You can use EasyBuild as a Python library and implement a script to automate a particular task. + +All the functionality provided by the EasyBuild framework can be leveraged, +so you only have to glue things together and add the missing pieces. + + +## Setting up the EasyBuild configuration + +Before you call any of the functions provided by the EasyBuild framework, +you should initialize EasyBuild by **setting up the configuration**. +Some of the EasyBuild framework functions assume that EasyBuild has been initialized properly, +and calling them without doing so will result in errors. + +To set up the EasyBuild configuration, you should call the ``set_up_configuration`` function +that is provided by the ``easybuild.tools.options`` module. + +This function takes a couple of optional arguments that are useful in the context of +a Python script that uses EasyBuild as a library: + +* Via ``args`` you can provide a list of command line arguments that should be passed to the option parser. + By default, the standard command line arguments are picked up via ``sys.args``, which may not be what you want. +* By passing ``silent=True`` you can specify that the ``set_up_configuration`` function should not print + anything when it is called. By default, it will print the location to the temporary EasyBuild log file. + +### Cleaning up the temporary directory + +When EasyBuild is configured, a temporary directory specific to that EasyBuild session will be created automatically. +You should clean up that directory, especially if you will be creating temporary files, or if the script will be run +often. + +Note that cleaning up the temporary directory implies removes the temporary log files, +so you probably only want to do this if no errors occurred. + +### Minimal example script + +Here is a minimal (and pretty useless) example Python script, which sets up the EasyBuild configuration +and cleans up the temporary directory: + +```python +#!/usr/bin/env python3 +from easybuild.tools.filetools import remove_dir +from easybuild.tools.options import set_up_configuration + +opts, _ = set_up_configuration(args=[], silent=True) + +remove_dir(opts.tmpdir) +``` + + +## Example use cases + +Once the EasyBuild configuration has been set up, the functions provided by the EasyBuild framework +can be called from a Python script (or directly from the Python interpreter). + +A full overview of all functions is available via the [EasyBuild API documentation](https://docs.easybuild.io/en/latest/api/easybuild.html). + +We highlight a couple commonly used functions in the sections below. + +### File operations + +The [``easybuild.tools.filetools``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html) +module provides a bunch of functions for performing file operations. + +That includes straightforward things like reading, writing, and copying files +(see [``read_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.read_file), [``write_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.write_file), [``copy_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.copy_file)), but also more specific functionality like applying a patch file ([``apply_patch``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.apply_patch)), +computing different types of checksums for a file ([``compute_checksum``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.compute_checksum)), and downloading files +([``download_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.download_file)). + +Here is a trivial example where we download a file from a specified URL to a temporary directory, +compute the SHA256 checksum, and then copy the downloaded file to the current working directory: + +```python +#!/usr/bin/env python3 +import os +import sys +import tempfile + +from easybuild.tools.filetools import download_file, compute_checksum, copy_file, remove_dir +from easybuild.tools.options import set_up_configuration + +url = sys.argv[1] + +opts, _ = set_up_configuration() + +fn = os.path.basename(url) +tmp_path = os.path.join(opts.tmpdir, fn) +download_file(fn, url, tmp_path) + +sha256 = compute_checksum(tmp_path, checksum_type='sha256') +print("SHA256 checksum for %s: %s" % (fn, sha256)) + +target_dir = os.getcwd() +copy_file(tmp_path, target_dir) +print("%s copied to %s" % (fn, target_dir)) + +# cleanup temp dir +remove_dir(opts.tmpdir) +``` + +Example output: +```shell +$ export TEST_URL='https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.0.1.tar.gz' +$ python3 download_and_copy.py $TEST_URL +== Temporary log file in case of crash /tmp/eb-or_xhaw8/easybuild-ewooa00c.log +SHA256 checksum for eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 +eb-tutorial-1.0.1.tar.gz copied to /home/example +``` +We are deliberately *not* specifying the `args` optional argument to the `set_up_configuration` call here, +so we can pass EasyBuild configuration options as command line arguments to this script. +Of course, only those configuration options that are taken into account by the EasyBuild +framework functions being called have any impact. For example: + +```shell +# wait for max. 1h for the download to complete +python3 download_and_copy.py $TEST_URL --download-timeout 3600 +``` + +### Running shell commands + +To run shell commands, we can leverage the ``run_cmd`` functions from the ``easybuild.tools.run`` module. + +Here is a simple example for running the '`make`' command via a Python script: + +```python +#!/usr/bin/env python3 +import sys + +from easybuild.tools.filetools import remove_dir, which +from easybuild.tools.run import run_cmd +from easybuild.tools.options import set_up_configuration + +opts, _ = set_up_configuration(args=[]) + +cmd = 'make' +cmd_path = which(cmd) +if cmd_path: + print(">>> '%s' command found at %s" % (cmd, cmd_path)) +else: + sys.stderr.write("ERROR: '%s' command not found!\n" % cmd) + sys.exit(1) + +cmd = ' '.join(["make"] + sys.argv[1:]) +out, ec = run_cmd(cmd) + +print("\n>>> Output of '%s' (exit code %s):\n\n%s" % (cmd, ec, out)) + +remove_dir(opts.tmpdir) +``` + +Note that in this case it is important to use `args=[]` when calling `set_up_configuration`, +to avoid that the additional script arguments which are options for the '`make`' command +are picked up by the EasyBuild option parser. + +Example usage: + +```shell +$ python3 run_make.py --version +== Temporary log file in case of crash /tmp/eb-zfvbceg6/easybuild-7tynetaj.log + +>> Output of 'make --version' (exit code 0): + +GNU Make 3.81 +``` + +### Interacting with the modules tool + +You can interact with the environment modules tool in a Python script using the +[``easybuild.tools.modules``](https://docs.easybuild.io/en/latest/api/easybuild.tools.modules.html) +Python module that is a part of the EasyBuild framework. +The modules tool that will be used is determined by the active EasyBuild configuration. + +Here is an example script that checks which modules are available and currently loaded, +loads the default module file for ``bzip2``, and inspects the resulting changes to the environment. + +```python +#!/usr/bin/env python3 +import os +import sys + +from easybuild.tools.filetools import remove_dir +from easybuild.tools.modules import get_software_root_env_var_name, modules_tool +from easybuild.tools.options import set_up_configuration + +opts, _ = set_up_configuration() + +# obtain ModulesTool instance for preferred modules tool (determined by active EasyBuild configuration) +mod_tool = modules_tool() +print("Active modules tool: %s version %s" % (mod_tool.NAME, mod_tool.version)) + +avail_modules = mod_tool.available() +print("Found %d available modules in total" % len(avail_modules)) + +avail_eb_modules = mod_tool.available('EasyBuild') +print("Found %d available modules for EasyBuild: %s" % (len(avail_eb_modules), ', '.join(avail_eb_modules))) + +loaded_modules = mod_tool.loaded_modules() +print("%d modules are currently loaded: %s" % (len(loaded_modules), ', '.join(loaded_modules))) + +# load default module for bzip2, check changes to environment +name = 'bzip2' +env_var_name = get_software_root_env_var_name(name) +if any(m.startswith(name + '/') for m in avail_modules): + + print("Current $%s value: %s" % (env_var_name, os.getenv(env_var_name, '(no set)'))) + print("Loading (default) '%s' module..." % name) + + mod_tool.load([name]) + print("Loaded modules: %s" % ', '.join(mod_tool.loaded_modules())) + + # inspect $_LMFILES_ environment variable to determine path to loaded bzip2 module file + for mod_file_path in os.getenv('_LMFILES_').split(':'): + if name in mod_file_path: + print("Path to loaded %s module: %s" % (name, mod_file_path)) + break + + # $EBROOTBZIP2 should be set now (if the bzip2 module was installed with EasyBuild) + print("Current $%s value: %s" % (env_var_name, os.getenv(env_var_name, '(no set)'))) +else: + sys.stderr.write("No modules available for %s\n" % name) + sys.exit(1) + +remove_dir(opts.tmpdir) +``` + +### Parsing easyconfig files + +Here is another small example Python script, which uses the EasyBuild framework functionality +to locate and parse an easyconfig file, and inspect the value of specific easyconfig parameters. + +We define a small helper function named ``parse_easyconfig``, because the EasyBuild framework API +is a bit awkward to use for this simple use case. + +```python +#!/usr/bin/env python3 +import sys + +from easybuild.framework.easyconfig.tools import det_easyconfig_paths, parse_easyconfigs +from easybuild.tools.options import set_up_configuration + + +def parse_easyconfig(ec_fn): + """ + Helper function: find and parse easyconfig with specified filename, + and return parsed easyconfig file (an EasyConfig instance). + """ + # determine path to easyconfig file + ec_path = det_easyconfig_paths([ec_fn])[0] + + # parse easyconfig file; + # the 'parse_easyconfigs' function expects a list of tuples, + # where the second item indicates whether or not the easyconfig file was automatically generated or not + ec_dicts, _ = parse_easyconfigs([(ec_path, False)]) + + # only retain first parsed easyconfig, ignore any others (which are unlikely anyway) + return ec_path, ec_dicts[0]['ec'] + + +# check whether required arguments are provided +if len(sys.argv) < 3: + sys.stderr.write("ERROR: Usage: %s ") + sys.exit(1) + +ec_fn = sys.argv[1] +keys = sys.argv[2:] + +set_up_configuration(args=[], silent=True) + +ec_path, ec = parse_easyconfig(ec_fn) + +print("Inspecting %s ..." % ec_path) +for key in keys: + print("%s: %s" % (key, ec[key])) +``` + +Example usage: + +```shell +$ ./inspect_easyconfig.py Subread-2.0.0-GCC-8.3.0.eb name version sources sanity_check_paths +name: Subread +version: 2.0.0 +sources: ['subread-2.0.0-source.tar.gz'] +sanity_check_paths: {'files': ['bin/exactSNP', 'bin/featureCounts', 'bin/subindel', 'bin/subjunc', 'bin/sublong', 'bin/subread-align', 'bin/subread-buildindex'], 'dirs': ['bin/utilities']} +``` + +--- + +*[[next: Using hooks to customise EasyBuild]](3_02_hooks.md)* diff --git a/docs/2022-CSC_and_LO/3_02_hooks.md b/docs/2022-CSC_and_LO/3_02_hooks.md new file mode 100644 index 00000000..a2663ff0 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_02_hooks.md @@ -0,0 +1,190 @@ +# Using hooks to customise EasyBuild + +*[[back: Using EasyBuild as a library]](3_01_easybuild_library.md)* + +--- + +Sometimes you may have the need to customise the behaviour of EasyBuild, +while you want to avoid copying easyconfig files in order to make small changes +(like adding an additional configuration option), having to maintain a set +of modified easyblocks, or using a patched version of the EasyBuild framework. + +EasyBuild provides support for implementing so-called *hooks*, +which are small functions that are triggered automatically at specific times. + +Through these hook functions you can take additional actions, or even tweak +internal data structures, to change the software installation procedure that EasyBuild +would normally execute. + +This is a very powerful feature, but it is important to aware of some details in order +to make good use of it. + +Support for hooks is also covered in detail in the EasyBuild documentation, see +[here](https://docs.easybuild.io/en/latest/Hooks.html). + + +## Implementing and using hooks + +Using hooks is done by implementing functions with specific names in a Python module, +and configuring EasyBuild to use them by specifying the path to the file that contains them +via the ``hooks`` configuration option. + +For example, if the file named ``my_eb_hooks.py`` in your home directory contains the implementation +of your hooks, you can configure EasyBuild to use them by defining the ``$EASYBUILD_HOOKS`` environment +variable: + +```shell +export EASYBUILD_HOOKS=$HOME/my_eb_hooks.py +``` + +## Available hooks + +EasyBuild supports various hooks, the full list can be consulted via ``eb --avail-hooks``. + +There are 3 different types of hooks: + +- start/end hooks +- parse hook +- pre/post-step hooks + +These are discussed in more details in the sections below. + +### Start/end hooks + +The first type of hooks are the ``start`` and ``end`` hooks, which are respectively triggered +at the very start of the EasyBuild session (right after setting up the EasyBuild configuration), +and at the very end of the session (right before stopping the logger and doing final cleanup). + +These hooks are only called *once* for every EasyBuild session, no matter how many installations +are performed during that session. + +The ``end`` hook is *not* triggered if an error occurred during one of the installations, +only on successful completion of the EasyBuild session. + +These hooks can be implementing by defining a function named ``start_hook`` and ``end_hook`` +in the Python module file that is provided via the ``hooks`` configuration option. +No arguments are provided when calling these hooks. + +### Parse hook + +The ``parse`` hook is triggered right after an easyconfig file is being parsed, +*before* EasyBuild sets up its internal data structures based on the parsed easyconfig. + +If you want to dynamically change one or more easyconfig parameters without changing the corresponding +easyconfig files, using this hook may be appropriate. + +Note that parsing an easyconfig file can happen for a variety of reasons, +not only when the easyconfig file will actually be installed. EasyBuild will also +parse easyconfig files to check whether they resolve required dependencies, +to check whether the corresponding module file is already installed, etc. + +This hook can be implemented via a function named ``parse_hook``, and exactly one +argument is provided when it is called: the [``EasyConfig``](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyconfig.easyconfig.html#easybuild.framework.easyconfig.easyconfig.EasyConfig) +instance that represents the parsed easyconfig file. + +### Pre/post-step hooks + +The third type of hooks are *pre/post-step* hooks, which are triggered right before or +after a particular installation step is executed. + +EasyBuild performs each software installation by stepping through over a dozen different methods, +and for each of these steps there a pre- and post-hook is triggered, which results in over 30 +additional hooks. + +To use any of these hooks, you need to implement a function that follow a strict naming scheme: +``__hook``. For example, the hook that is triggered right before the ``configure`` +step is run is a function named ``pre_configure_hook``. + +Every time these hooks are called, a single argument is provided: an [``EasyBlock``](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyblock.html#easybuild.framework.easyblock.EasyBlock) +instance that represents the easyblock that is being used to perform the installation. +The parsed easyconfig file can be accessed via the ``cfg`` class variable of the ``EasyBlock`` instance. + +These hooks are useful for influencing the installation procedure at a particular stage. + +## Caveats + +There are a couple of important caveats to take into account when implementing hooks. + +### Breaking EasyBuild with hooks + +Since hooks allow you to inject custom code into EasyBuild at runtime, +it is also easy to break EasyBuild by using hooks... + +Make sure to carefully test your hook implementations, and constrain the actions +you take a much as possible, for example by adding conditions to control for which +software names you will actually modify the installation procedure, etc. + +Any errors that are triggered or raised while a hook function is running +will interrupt the EasyBuild session. + +So don't forget: with great power comes great responsibility! + +### Template values + +Depending on the type of hook, you may observe "raw" values of easyconfig parameters where +template values have not been resolved yet, or values in which template values have been resolved already. + +In the ``parse`` hook, you will always see unresolved template values. + +In the pre/post-step hooks you will see resolved template values, +unless you explicitly disable templating. + +To obtain easyconfig parameter values with unresolved template values in step hooks, +you can use the ``disable_templating`` [context manager](https://docs.python.org/3/reference/compound_stmts.html#with). +For example: + +```python +from easybuild.framework.easyconfig.easyconfig import disable_templating +from easybuild.tools.build_log import print_warning + +def pre_source_hook(eb): + """Print warning when software version was found in 'raw' name of source file.""" + with disable_templating(eb.cfg): + for src in eb.cfg['sources']: + if eb.version in src: + msg = "Software version '%s' found in name of source file (%s), " % (eb.version, src) + msg += "please use %(version)s template value instead!" + print_warning(msg) +``` + + +### Manipulating easyconfig parameters + +If you want update a particular easyconfig parameter without overwriting the existing value, +a bit of care has to be taken: you should use the ``update`` method of the ``EasyConfig`` instance +for this, unless you disable template resolution. This is particularly important when +updating easyconfig parameters that have *mutable* value (like a ``list`` or ``dict``). + +Here's a correct example of a pre-install hook: + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # always copy the README directory too when installing pigz + eb.cfg.update('files_to_copy', 'README') +``` + +This seemingly equivalent implementation will ***not*** work (the value of the `files_to_copy` +easyconfig parameter will *not* be updated): + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # incorrect way of adding 'README' to 'files_to_copy' (DON'T USE THIS!) + eb.cfg['files_to_copy'].append('README') +``` + +To use this coding style successfully, you have to disable the templating mechanism +when updating the easyconfig parameter: + +```python +def pre_install_hook(eb): + if eb.name == 'pigz': + # this works, but it is better to use the 'update' method instead... + with disable_templating(eb.cfg): + eb.cfg['files_to_copy'].append('README') +``` + +--- + +*[[next: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* diff --git a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md new file mode 100644 index 00000000..0ffa12a9 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md @@ -0,0 +1,304 @@ +# Submitting installations as Slurm jobs + +*[[back: Using hooks to customise EasyBuild]](3_02_hooks.md)* + +--- + +EasyBuild can submit jobs to different backends including Slurm to install software, +to *distribute* the often time-consuming installation of a set of software applications and +the dependencies they require to a cluster. + +This is done via the ``--job`` command line option. + +It is important to be aware of some details before you start using this, which we'll cover here. + +## Configuration + +The EasyBuild configuration that is active at the time that ``eb --job`` is used +will be *passed down* into the submitted job automatically, via command line options to the ``eb`` +command that is run in the job script. + +This includes not only command line options used next to ``--job``, but also configuration settings +that are specified via an [EasyBuild configuration file](configuration.md#configuration-files) or through +[``$EASYBUILD_*`` environment variables](configuration.md#easybuild_-environment-variables). + +This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables +that are in place in the job environment are most likely *irrelevant*, since configuration settings +they specify they will most likely be overruled by the corresponding command line options. + + +## Using ``eb --job`` + +### Job backend + +The default job backend in EasyBuild v4.x is [``GC3Pie``](https://gc3pie.readthedocs.io). +To let EasyBuild submit jobs to Slurm instead, you should set the ``job-backend`` configuration setting +to ``Slurm``, for example by setting the corresponding environment variable: + +```shell +export EASYBUILD_JOB_BACKEND='Slurm' +``` + +### Job resources + +To submit an installation as a job, simply use ``eb --job``: + +```shell +eb example.eb --job +``` + +By default, EasyBuild will submit single-core jobs requesting for 24 hours of walltime. +You can tweak the requested resources via the ``job-cores`` and ``job-max-walltime`` configuration options. +For example: + +```shell +# submit job to install example, using 5 cores and 2 hours of max. walltime +eb example.eb --job --job-cores 5 --job-max-walltime 2 +``` + +Note that not all ``job-*`` configuration settings apply to all job backends, +see the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Submitting_jobs.html) for more details. + +### Controlling Slurm submission options + +When using Slurm as a job backend, EasyBuild will automatically generate job scripts which +use the ``eb`` command to perform a single installation. These scripts will be submitted +using the ``sbatch`` command. + +EasyBuild currently doesn't provide away to customize the Slurm submission options, +for example to submit to a particular partition, or to use a particular account, +build you can set the corresponding ``$SBATCH_*`` environment variables prior to running ``eb --job``. + +For example, to specify a particular account that should be used for the jobs submitted by EasyBuild +(equivalent with using the ``-A`` or ``--account`` command line option for ``sbatch``): + +```shell +export SBATCH_ACCOUNT='example_project' +``` + +Or to submit to a particular Slurm partition (equivalent with the ``-p`` or ``--partition`` option for ``sbatch``): + +```shell +export SBATCH_PARTITION='example_partition' +``` + +For more information about supported ``$SBATCH_*`` environment variables, +see the [Slurm documentation](https://slurm.schedmd.com/sbatch.html#lbAJ). + +## Combining ``--job`` and ``--robot`` + +If one or more dependencies are still missing for the software you want to install, +you can combine ``--job`` and ``--robot`` to get EasyBuild to submit a *separate* job +for each of the installations. These jobs will *not* ``--robot``, they will each only +perform a single installation. + +Dependencies between jobs will be "registered" at submission time, so Slurm will put jobs +on hold until the jobs that install the required (build) dependencies have completed successfully, +and cancel jobs if the job to install a dependency failed for some reason. + +## Attention points + +There are a couple of important things to keep an eye on when submitting installations as jobs... + +### Differences on cluster workernodes + +Sometimes the resources available on the login nodes and cluster workernodes are slightly different, +and you may need to take this into account in your EasyBuild configuration. + +For example, plenty of disk space may be available in the `/tmp` temporary filesystem on a login node, +while the workernodes require you to use a different location for temporary files and directories. +As a result, you may need to slightly change your EasyBuild configuration when submitting installations +as jobs, to avoid that they fail almost instantly due to a lack of disk space. + +Keep in mind that the active EasyBuild configuration is passed down into the submitted jobs, +so any configuration that is present on the workernodes may not have any effect. + +For example, if you commonly use `/tmp/$USER` for build directories on a login node, +you may need to tweak that when submitting jobs to use a different location: + +```shell +# EasByuild is configured to use /tmp/$USER on the login node +login01 $ eb --show-config | grep buildpath +buildpath (E) = /tmp/example + +# use /localdisk/$USER for build directories when submitting installations as jobs +login01 $ eb --job --buildpath /localdisk/$USER example.eb --robot +``` + +### Temporary log files and build directories + +The temporary log file that EasyBuild creates is most likely going to end up on the local disk +of the workernode on which the job was started (by default in `$TMPDIR` or `/tmp`). +If an installation fails, the job will finish and temporary files will likely be cleaned up instantly, +which may leave you wondering about the actual cause of the failing installation... + +To remedy this, there are a couple of EasyBuild configuration options you can use: + +* You can use ``--tmp-logdir`` to specify a different location where EasyBuild should store temporary log files, + for example: + ```shell + $ eb --job example.eb --tmp-logdir $HOME/eb_tmplogs + ``` + +* If you prefer having the entire log file stored in the Slurm job output files, + you can use ``--logtostdout`` when submitting the jobs. This will result in extensive logging + to your terminal window when submitting the jobs, but it will also make EasyBuild + log to ``stdout`` when the installation is running in the job, and hence the log messages will be + captured in the job output files. + +The same remark applies to build directories: they should be on a local filesystem (to avoid problems +that often occur when building software on a parallel filesystem like GPFS or Lustre), +which will probably be cleaned up automatically when a job fails. Here it is less easy to provide +general advice on how to deal with this, but one thing you can consider is retrying the installation +in an interactive job, so you can inspect the build directory after the installation fails. + +### Lock files + +EasyBuild creates [locks](https://docs.easybuild.io/en/latest/Locks.html) +to prevent that the same installation is started multiple times on +different system to the same installation directory. + +If an installation fails or gets interrupted, EasyBuild cleans up those locks automatically. + +However, if a Slurm job that is using EasyBuild to install software gets cancelled (because it +ran out of walltime, tried to consume too much memory, through an ``scancel`` command, etc.), +EasyBuild will not get the chance to clean up the lock file. + +If this occurs you will need to either clean up the lock file (which is located in the `software/.locks` +subdirectory of ``installpath``) manually, or re-submit the job with ``eb --job --ignore-locks``. + +## Example + +As an example, we will let EasyBuild submit jobs to install ``AUGUSTUS`` with the ``foss/2020b`` toolchain. + +### Configuration + +Before using ``--job``, let's make sure that EasyBuild is properly configured: + +```shell +# use $HOME/easybuild for software, modules, sources, etc. +export EASYBUILD_PREFIX=$HOME/easybuild + +# use ramdisk for build directories +export EASYBUILD_BUILDPATH=/dev/shm/$USER + +# use Slurm as job backend +export EASYBUILD_JOB_BACKEND=Slurm +``` + +In addition, add the path to the centrally installed software to ``$MODULEPATH`` via ``module use``: + +```shell +module use /easybuild/modules/all +``` + +Load the EasyBuild module: + +```shell +module load EasyBuild +``` + +Let's assume that we also need to inform Slurm that jobs should be submitted into a particular account: + +```shell +export SBATCH_ACCOUNT=example_project +``` + +This will be picked up by the ``sbatch`` commands that EasyBuild will run to submit the software installation jobs. + + +### Submitting jobs to install AUGUSTUS + +Now we can let EasyBuild submit jobs for AUGUSTUS. + +Let's first check what is still missing: + +```shell +$ eb AUGUSTUS-3.4.0-foss-2020b.eb --missing +... +11 out of 61 required modules missing: + +* HTSlib/1.11-GCC-10.2.0 (HTSlib-1.11-GCC-10.2.0.eb) +* lpsolve/5.5.2.11-GCC-10.2.0 (lpsolve-5.5.2.11-GCC-10.2.0.eb) +* Boost/1.74.0-GCC-10.2.0 (Boost-1.74.0-GCC-10.2.0.eb) +* GSL/2.6-GCC-10.2.0 (GSL-2.6-GCC-10.2.0.eb) +* SAMtools/1.11-GCC-10.2.0 (SAMtools-1.11-GCC-10.2.0.eb) +* BCFtools/1.11-GCC-10.2.0 (BCFtools-1.11-GCC-10.2.0.eb) +* METIS/5.1.0-GCCcore-10.2.0 (METIS-5.1.0-GCCcore-10.2.0.eb) +* BamTools/2.5.1-GCC-10.2.0 (BamTools-2.5.1-GCC-10.2.0.eb) +* MPFR/4.1.0-GCCcore-10.2.0 (MPFR-4.1.0-GCCcore-10.2.0.eb) +* SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0.eb) +* AUGUSTUS/3.4.0-foss-2020b (AUGUSTUS-3.4.0-foss-2020b.eb) +``` + +Several dependencies are not installed yet, so we will need to use ``--robot`` to ensure that +EasyBuild also submits jobs to install these first. + +To speed up the installations a bit, we will request 10 cores for each submitted job (via ``--job-cores``). +That should be sufficient to let each installation finish in (well) under 1 hour, +so we only request 1 hour of walltime per job (via ``--job-max-walltime``). + +In order to have some meaningful job output files, we also enable trace mode (via ``--trace``). + +``` +$ eb AUGUSTUS-3.4.0-foss-2020b.eb --job --job-cores 10 --job-max-walltime 1 --robot --trace +... +== resolving dependencies ... +... +== List of submitted jobs (11): Boost-1.74.0-GCC-10.2.0 (Boost/1.74.0-GCC-10.2.0): 1000011; GSL-2.6-GCC-10.2.0 (GSL/2.6-GCC-10.2.0): 1000004; SAMtools-1.11-GCC-10.2.0 (SAMtools/1.11-GCC-10.2.0): 1000005; HTSlib-1.11-GCC-10.2.0 (HTSlib/1.11-GCC-10.2.0): 1000006; BCFtools-1.11-GCC-10.2.0 (BCFtools/1.11-GCC-10.2.0): 1000001; lpsolve-5.5.2.11-GCC-10.2.0 (lpsolve/5.5.2.11-GCC-10.2.0): 1000007; BamTools-2.5.1-GCC-10.2.0 (BamTools/2.5.1-GCC-10.2.0): 1000008; METIS-5.1.0-GCCcore-10.2.0 (METIS/5.1.0-GCCcore-10.2.0): 1000009; MPFR-4.1.0-GCCcore-10.2.0 (MPFR/4.1.0-GCCcore-10.2.0): 1000010; SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0): 1000002; AUGUSTUS-3.4.0-foss-2020b (AUGUSTUS/3.4.0-foss-2020b): 1000003 +== Submitted parallel build jobs, exiting now +``` + +### Inspecting the submitted jobs + +Once EasyBuild has submitted the jobs, we can inspect them via Slurm's ``squeue`` command: + +``` +$ squeue -u $USER -la + JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON) +1000001 small BCFtools user123 PENDING 0:00 2:00:00 1 (Dependency) +1000002 small SuiteSpa user123 PENDING 0:00 2:00:00 1 (Dependency) +1000003 small AUGUSTUS user123 PENDING 0:00 2:00:00 1 (Dependency) +1000004 small GSL-2.6- user123 RUNNING 0:21 2:00:00 1 node003 +1000005 small SAMtools user123 RUNNING 0:21 2:00:00 1 node007 +1000006 small HTSlib-1 user123 RUNNING 0:21 2:00:00 1 node007 +1000007 small lpsolve- user123 RUNNING 0:21 2:00:00 1 node011 +1000008 small BamTools user123 RUNNING 0:21 2:00:00 1 node011 +1000009 small METIS-5. user123 RUNNING 0:21 2:00:00 1 node013 +1000010 small MPFR-4.1 user123 RUNNING 0:21 2:00:00 1 node029 +1000011 small Boost-1. user123 RUNNING 0:24 2:00:00 1 node029 +``` + +Note that 3 jobs can not be started yet, because those installations require on one or more +missing dependencies. As soon as the jobs for those dependencies (successfully) complete, +these jobs will be able to start. + +### Final result + +After about 20 minutes, AUGUSTUS and all missing dependencies should be installed: + +``` +$ ls -lrt $HOME/easybuild/modules/all/*/*.lua | tail -11 +-rw-rw----. 1 example example 1634 Mar 29 10:13 /users/example/easybuild/modules/all/HTSlib/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1792 Mar 29 10:13 /users/example/easybuild/modules/all/SAMtools/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1147 Mar 29 10:13 /users/example/easybuild/modules/all/BamTools/2.5.1-GCC-10.2.0.lua +-rw-rw----. 1 example example 957 Mar 29 10:13 /users/example/easybuild/modules/all/lpsolve/5.5.2.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1549 Mar 29 10:13 /users/example/easybuild/modules/all/METIS/5.1.0-GCCcore-10.2.0.lua +-rw-rw----. 1 example example 1525 Mar 29 10:14 /users/example/easybuild/modules/all/GSL/2.6-GCC-10.2.0.lua +-rw-rw----. 1 example example 1221 Mar 29 10:15 /users/example/easybuild/modules/all/MPFR/4.1.0-GCCcore-10.2.0.lua +-rw-rw----. 1 example example 1678 Mar 29 10:15 /users/example/easybuild/modules/all/BCFtools/1.11-GCC-10.2.0.lua +-rw-rw----. 1 example example 1292 Mar 29 10:21 /users/example/easybuild/modules/all/Boost/1.74.0-GCC-10.2.0.lua +-rw-rw----. 1 example example 1365 Mar 29 10:28 /users/example/easybuild/modules/all/SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0.lua +-rw-rw----. 1 example example 2233 Mar 29 10:30 /users/example/easybuild/modules/all/AUGUSTUS/3.4.0-foss-2020b.lua + +$ module use $HOME/easybuild/modules/all + +$ module avail AUGUSTUS + +-------- /users/hkenneth/easybuild/modules/all -------- + AUGUSTUS/3.4.0-foss-2020b +``` + +--- + +*[[next: Module naming schemes]](3_04_module_naming_scheme.md)* diff --git a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md new file mode 100644 index 00000000..14229d97 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md @@ -0,0 +1,617 @@ +# Module naming schemes + +*[[back: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* + +--- + +Up until now we have used EasyBuild's default module naming scheme (`EasyBuildMNS`), +which produces module files with names that closely resemble to the names of the +corresponding easyconfig files. +For example, when installing `h5py-3.1.0-foss-2020b.eb` the generated module was named `h5py/3.1.0-foss-2020b`. + +EasyBuild supports several different module naming schemes: + +```shell +$ eb --avail-module-naming-schemes +List of supported module naming schemes: + CategorizedHMNS + CategorizedModuleNamingScheme + EasyBuildMNS + HierarchicalMNS + MigrateFromEBToHMNS +``` + +In this part of the tutorial we will take a closer look at `HierarchicalMNS`, +which is the standard **hierarchical** module naming scheme included with EasyBuild. + +We will also take a quick look at implementing our own custom module naming scheme. + +## Flat vs hierarchical + +The default module naming scheme `EasyBuildMNS` is an example of regular *"flat"* module naming scheme, which is characterized by: + +* all module files are directly available for loading; +* each module name uniquely identifies a particular installation; + +In contrast, a *hierarchical* module naming scheme +consists of a *hierarchy* of module files. + +The typical module hierarchy has 3 levels: + +* a **core** level, where module files for software that was installed using the + [`system` toolchain](../introduction/#system-toolchain) are kept; +* a **compiler** level, where module files for software that was installed using a +*compiler-only toolchain* are stored; +* and an **MPI** level, which houses module files for software that was installed using a toolchain that includes (at least) a compiler and MPI component; + +Here is a simple example of such a 3-level module hierarchy: + +
+ +In this example the core level only includes a single module `GCC/9.3.0`, +while the compiler level includes two modules: `OpenMPI/4.0.3` and `MPICH/3.3.2`. +In the MPI level, three modules are available: one for `FFTW`, one for `ScaLAPACK`, and one for `HDF5`. + +Initially only the modules on the top level of a module hierarchy are available for loading. +If you run "`module avail`" with the example module hierarchy, you will only see the `GCC/9.3.0` module. + +Some modules in the top level of the hierarchy act as a "gateway" to modules in the +next level below. +To make additional modules available for loading one of these gateway modules has to be loaded. In our exampe, loading the `GCC/9.3.0` module results in two additional modules coming into view from the compiler level, as indicated by the arrows: the modules for `OpenMPI` and `MPICH`. These correspond to installations of `OpenMPI` +and `MPICH` that were built using `GCC/9.3.0` as a (compiler-only) toolchain. + +Similarly, the `OpenMPI/4.0.3` module serves as a gateway to the three modules in the MPI level. Only by loading the `OpenMPI` module will these additional three modules become +available for loading. They correspond to software installations built using the `gompi/2020a` toolchain that +consists of the `GCC/9.3.0` compiler module and the `OpenMPI/4.0.3` MPI module. Software installed using +`foss/2020a` (which is a full toolchain that also includes OpenBLAS, FFTW and ScaLAPACK) would also be stored +in this level of the module hierarchy. + +The characteristics of a module hierarchy are: + +* not all module files are directly available for loading; +* some modules serve as a gateway to more modules; +* to access some software installations you will first need to load one or more gateway modules in order + to use them; + +You can probably think of other ways to organize module files in a hierarchical module tree, but here we will stick to the standard core / compiler / MPI hierarchy. + +### Pros & cons + +So why go through all this trouble of organizing modules hierarchically? + +There are a couple of advantages to this approach: + +* shorter module names; +* less overwhelming list of available modules; +* only compatible modules can be loaded together; + +However, the are some minor disadvantages too: + +* not all existing modules are directly visible; +* gateway modules may have little meaning to end users; + +#### Length of module names + +When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For our `HDF5` installation for example, +we have `HDF5/1.10.6-gompi-2020a` as module name. The `-gompi-2020a` part of the name refers to the toolchain that was +used for this installation, but it may be confusing to some people (what kind of Pokémon is a "gompi"?!). + +In the example module hierarchy shown above, the module for `HDF5` is named `HDF5/1.10.6` which is basically the bare +essentials: software name and version. That's way better, nice and clean! + +#### Amount of available modules + +The output of "`module avail`" can be quite overwhelming if lots of module files +are installed and a flat module naming scheme is used, since *all* modules are +*always* available. +EasyBuild makes it very easy to install lots of software, +so the number of installed modules can easily grow into the hundreds or even thousands... + +This often explosive growth of modules is less of an issue when using a hierarchical module naming scheme, since +initially only a small set of modules are available, and relatively limited +groups of additional modules become available as gateway modules are loaded. + +#### Loading compatible modules + +Since all modules are available at once when using a flat module naming scheme, you can easily load +modules together that are not compatible with each other. + +Imagine loading two modules that were built with a different compiler toolchain (different compiler, +different MPI library). That's likely to end in tears, unless you have the necessary technical expertise +to fully understand what is going on, *and* you are being very careful... + +In a module hierarchy this can be prevented, since modules for software that were installed with a +different compiler and/or a different MPI library are located in a different part of the module hierarchy, +and so these modules can not be loaded together easily. + + +#### Visibility of existing modules + +One downside of a module hierarchy is that not all existing modules are directly available for loading +or are even *visible* to the user, since the output of "`module avail`" only shows a subset of all modules. + +[Lmod](https://lmod.readthedocs.io) has a solution for this though: it provides a separate "`module spider`" +command to search for module files throughout the entire module hierarchy. So as long as the end users are +aware of this additional command, it should not be difficult to discover which software installations exist +and how they can be accessed. The "`module spider`" command will inform the user which of the gateway modules +need to be loaded in order to load a specific module file. + +#### Semantics of gateway modules + +An additional potential problem of a module hierarchy is that the semantics of the gateway modules may not be clear +to end users. They may wonder why they need to pick a specific compiler and MPI library, or which of the +available options is the best one. Maybe there are not even be aware what exactly a "compiler" is, or how it is +relevant to the software they need in their bioinformatics pipeline... + +This can be partially resolved by loading a default compiler and MPI module so a particular set of modules +is available right after login, which could be the ones used in the most recent toolchain, or the +recommended versions. More experienced users could then leverage the "`module spider`" command to navigate +the module hierarchy. + +## Using a custom module naming scheme + +Next to the module naming schemes that are included with EasyBuild, +you can also define your own module naming scheme (MNS), and configure EasyBuild to use it. + +### Implementation + +To use a custom module naming scheme, you need to implement a Python module file, +where you define a Python class that derives from the general [``ModuleNamingScheme``](https://docs.easybuild.io/en/latest/api/easybuild.tools.module_naming_scheme.mns.html#easybuild.tools.module_naming_scheme.mns.ModuleNamingScheme) class. + +For a flat module naming scheme, it is sufficient to implement the ``det_full_module_name`` method, +which should return a string value (the full module name). +You may also need to customize the ``is_short_modname_for`` method, which verifies whether +a given (short) module name is for the software with a particular given name (or not). + +The argument provided to ``det_full_module_name`` can be a small Python dictionary which only specifies +the value of a handful of easyconfig parameters (`name`, `version`, `toolchain`, and `versionsuffix`), +or an ``EasyConfig`` instance which represents a parsed easyconfig file and contains values for *all* +known easyconfig parameters. + +For simple module naming schemes, just have ``name``, ``version``, ``toolchain``, and ``versionsuffix`` +available is sufficient. If it is not, you can list which additional easyconfig parameters are required for +your module naming scheme via a class constant ``REQUIRED_KEYS``. + +A fairly straightforward example of a *flat* module name scheme is the default EasyBuild module naming scheme +[``EasyBuildMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/easybuild_mns.py). + +For a *hierarchical* module naming scheme, various additional methods have to be implemented. + +Some of these, like ``det_module_subdir`` and ``det_short_module_name``, determine which part of +the (full) module name determines the location of the module in the module hierarchy, and which part +is the user-facing (or "short") module name. Others, like ``det_modpath_extensions``, +determine which modules are *gateway* modules that open up an additional level of the hierarchy +(by *extending* the ``$MODULEPATH``). + +A typical example of a hierarchical module naming scheme is [``HierarchicalMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/hierarchical_mns.py) that comes with EasyBuild (which is quite complex because it includes quite a bit of special handling for particular compiler +toolchains), which implements the traditional ``Core``-``Compiler``-``MPI`` module hierarchy we discussed +above. + +### Configuring EasyBuild + +To let EasyBuild use a custom module naming scheme, you need to: + +* specify the path to the Python module file implementing it via the ``include-module-naming-schemes`` + EasyBuild configuration option; + +* indicate that you also want to *use* this custom module naming scheme via the ``module-naming-scheme`` + EasyBuild configuration option. + +For example: + +```shell +export EASYBUILD_INCLUDE_MODULE_NAMING_SCHEMES=$HOME/easybuild/example_mns.py +export EASYBUILD_MODULE_NAMING_SCHEME=ExampleMNS +``` + +#### Example custom module naming scheme + +Here is an example of a custom module naming scheme, where: + +* the ``versionsuffix`` goes directly after the ``version``; +* all dashes are replaced by underscores; +* all module names are lowercase; + +Note that we also need to customise the ``is_short_modname_for`` method, +to make sure it returns ``True`` when the EasyBuild framework checks whether +``scipy_bundle/2020.11_foss_2020b`` is a module name for ``SciPy-bundle``. + +```python +import os + +from easybuild.tools.module_naming_scheme.mns import ModuleNamingScheme + + +class ExampleMNS(ModuleNamingScheme): + + REQUIRED_KEYS = ['name', 'version', 'versionsuffix', 'toolchain'] + + def det_full_module_name(self, ec): + """ + Determine full module name: + - all lowercase + - replace all dashes with underscores + """ + parts = [ec['version']] + + # versionsuffix directly after version (but only if it's not empty) + if ec['versionsuffix']: + parts.append(ec['versionsuffix']) + + # only add toolchain name/version for non-system toolchain + tc = ec['toolchain'] + if tc['name'].lower() != 'system': + parts.extend([tc['name'], tc['version']]) + + modname = ec['name'] + '/' + '_'.join(parts) + + modname = modname.replace('-', '_').replace('__', '_') + + return modname.lower() + + def is_short_modname_for(self, short_modname, name): + """Determine whether short module name is a module for the software with specified name.""" + return short_modname.startswith(name.lower().replace('-', '_') + '/') +``` + +We can see what the module names with this module naming scheme would like like via ``eb -D``: + +``` +$ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D + ... + * [ ] $CFGS/g/GCC/GCC-10.2.0.eb (module: gcc/10.2.0) + ... + * [ ] $CFGS/p/Python/Python-2.7.18-GCCcore-10.2.0.eb (module: python/2.7.18_gcccore_10.2.0) + ... + * [ ] $CFGS/o/OpenMPI/OpenMPI-4.0.5-GCC-10.2.0.eb (module: openmpi/4.0.5_gcc_10.2.0) + ... + * [ ] $CFGS/s/SciPy-bundle/SciPy-bundle-2020.03-foss-2020a-Python-2.7.18.eb (module: scipy_bundle/2020.03_python_2.7.18_foss_2020a) +``` + +## Example module hierarchy: HDF5 + +!!! Warning + **This exercise will only work if you have write access to the software installation prefix**, + because EasyBuild will try to copy the installation log file to each installlation directory. + +Now that we know more about hierarchical module naming schemes, +let us see how EasyBuild can help us with generating a hierarchical module tree. + +In this example we will use EasyBuild to generate modules organised in a hierarchy +for some of the software that is already installed in the prepared environment. + +The good news is that the existing installations can be reused. There is absolutely no need +to reinstall the software, we are just creating a different "view" on these software installations. + +### Preparing the environment + +Before running EasyBuild to generate a hierarchical module tree, we have to be a bit careful +with preparing our environment. + +**We must absolutely avoid mixing modules from a flat and hierarchical module naming scheme!** + +Some module files will have the same name in both module trees (like `GCC/10.2.0` for example), +but their contents will be different. +Mixing modules from a flat and hierarchical module tree *will* trigger problems... + +So we have to make sure that the module files we already have in `/easybuild` are *not* visible. +The easiest way to do this is to unload all modules (using "`module purge`") +and resetting the module search path to be empty, which we can do with "`module unuse $MODULEPATH`". + +```shell +module purge +module unuse $MODULEPATH +``` + +In this part of the tutorial, we are assuming you are *not* using an EasyBuild installation provided through +a module. We have just made all modules unavailable, so we would have to first +install EasyBuild again in our hierarchical module tree before we can continue. + +**We strongly recommend using an EasyBuild installation that was [installed via "`pip install`" +or "`pip3 install`"](../installation/#method-1-using-pip-recommended) in this part of the tutorial.** + +An easy way to do this is in the prepared environment is to run: + +```shell +pip3 install --user easybuild +export PATH=$HOME/.local/bin:$PATH +export EB_PYTHON=python3 +``` + +### Configuring EasyBuild + +First of all, we need to make sure that EasyBuild is properly configured. +We can do this by defining this set of environment variables: + +```shell +export EASYBUILD_PREFIX=$HOME/easybuild +export EASYBUILD_BUILDPATH=/tmp/$USER +export EASYBUILD_INSTALLPATH_SOFTWARE=/easybuild/software +export EASYBUILD_MODULE_NAMING_SCHEME=HierarchicalMNS +export EASYBUILD_INSTALLPATH_MODULES=$HOME/hmns/modules +``` + +To make sure we didn't make any silly mistakes, we double check using `eb --show-config`: + + +```shell +$ eb --show-config +# +# Current EasyBuild configuration +# (C: command line argument, D: default value, E: environment variable, F: configuration file) +# +buildpath (E) = /tmp/example +containerpath (E) = /home/example/easybuild/containers +installpath (E) = /home/example/easybuild +installpath-modules (E) = /home/example/hmns/modules +installpath-software (E) = /easybuild/software +module-naming-scheme (E) = HierarchicalMNS +packagepath (E) = /home/example/easybuild/packages +prefix (E) = /home/example/easybuild +repositorypath (E) = /home/example/easybuild/ebfiles_repo +robot-paths (D) = /home/example/.local/easybuild/easyconfigs +sourcepath (E) = /home/example/easybuild/sources +``` + +There are a couple of things worth pointing out here: + +* We have defined the `module-naming-scheme` configuration setting to `HierarchicalMNS`, + which makes EasyBuild use the included standard hierarchical module naming scheme (the classic + core / compiler / MPI one we discussed above). +* We have specified *different* locations for the software (via `installpath-software`) + and the module files (via `installpath-modules`). This is important because we want to + reuse the software that is already installed in `/easybuild/software` while we want to + generate an entirely new module tree for it (in `$HOME/hmns/modules`). + +The other configuration settings are the same as before, and mostly irrelevant for this example. + +### Generating modules for HDF5 + +Let us now generate a hierarchical module tree for `HDF5` and all of its dependencies, +including the toolchain. That sounds complicated, and it sort of is since there are +a lot of details you have to get right for the module hierarchy to works as intended, +but EasyBuild can do all the hard work for us. + +The steps we will have to go through are: + +* Tell EasyBuild we want to "install" the `HDF5-1.10.7-gompi-2020b.eb` easyconfig file; +* Enable dependency resolution via `--robot`; +* Instruct EasyBuild to only generate the module files, not to install the software (since it is + there already in `/easybuild/software`), via the `--module-only` option. + +These steps translate to this single `eb` command: + +``` +$ eb HDF5-1.10.7-gompi-2020b.eb --robot --module-only +... +== building and installing MPI/GCC/10.2.0/OpenMPI/4.0.5/HDF5/1.10.7... +... +== sanity checking... +== cleaning up [skipped] +== creating module... +... +== COMPLETED: Installation ended successfully (took 9 sec) +... +== Build succeeded for 41 out of 41 +``` + +This should take a couple of minutes in total, for generating 41 modules. +Remember that this also includes generating module files for the toolchain and +all of its components. + +In addition, there is a bit more going on one that just generating module files, +since the sanity check step is still being run for each of the installations +when using `--module-only` to ensure the installation is actually functional. +After all, there is no point in generating a module for an obviously broken +installation... + +### Loading the HDF5 module + +After generating the hierarchical module tree for HDF5, how do we access the HDF5 installation through it? + +Here's what the module tree looks like on disk: + +``` +$ ls $HOME/hmns/modules/all +Compiler Core MPI +``` + +Those are basically the 3 levels in the module hierarchy we showed in our example earlier. + +The starting point is the top level of the module hierarchy named `Core`: + +``` +module use $HOME/hmns/modules/all/Core +``` + +Let us see what that gives us in terms of available modules: + +``` +$ module avail + +--------------------- /home/easybuild/hmns/modules/all/Core --------------------- + Bison/3.5.3 GCCcore/10.2.0 flex/2.6.4 help2man/1.47.4 + Bison/3.7.1 (D) M4/1.4.18 gettext/0.21 ncurses/6.2 + GCC/10.2.0 binutils/2.35 gompi/2020b zlib/1.2.11 +``` + + +Nice and short module names, but only a limited set of them. + +We know a module file exists for `HDF5`, but we can't see it yet (and hence +we can't load it either). + +``` +$ module avail HDF5 +No module(s) or extension(s) found! +Use "module spider" to find all possible modules and extensions. +``` + +Let us see if `module spider` is of any help, as "`module avail`" so kindly suggests: + +``` +$ module spider HDF5 +... + + You will need to load all module(s) on any one of the lines below + before the "HDF5/1.10.7" module is available to load. + + GCC/10.2.0 OpenMPI/4.0.5 +``` + +This tells us we need to load two gateway modules before we can load the module +for HDF5. + +Let us start with loading the `GCC` compiler module: + +``` +module load GCC/10.2.0 +``` + +And then check again which modules are available: + +``` +$ module avail + +-------------- /home/easybuild/hmns/modules/all/Compiler/GCC/10.2.0 -------------- + OpenMPI/4.0.5 + +------------ /home/easybuild/hmns/modules/all/Compiler/GCCcore/10.2.0 ------------ + Autoconf/2.69 UCX/1.9.0 libpciaccess/0.16 + ... + Szip/2.1.1 libfabric/1.11.0 zlib/1.2.11 (L,D) + Autoconf/2.69 XZ/5.2.5 libtool/2.4.6 + +--------------------- /home/easybuild/hmns/modules/all/Core --------------------- + Bison/3.5.3 GCCcore/10.2.0 (L) flex/2.6.4 help2man/1.47.4 + Bison/3.7.1 (D) M4/1.4.18 gettext/0.21 ncurses/6.2 + GCC/10.2.0 (L) binutils/2.35 gompi/2020b zlib/1.2.11 +``` + +Good news, we now have additional modules available! + +The compiler level of our hierarchy actually consists of two directories here: `Compiler/GCCcore/10.2.0` +and `Compiler/GCC/10.2.0`. The modules in the `GCCcore` directory are ones we can use in other compiler +toolchains that use GCC 10.2.0 as a base compiler (the details of that are out of scope here). + +The module we are interested in is `OpenMPI/4.0.5`, which is another gateway module. + +Remember that the "`module spider`" output told us that there does indeed exist a module for `HDF5`, but that +we need to load *both* the `GCC/10.2.0` and `OpenMPI/4.0.5` modules first. + +So, let us do exactly that (remember that `GCC/10.2.0` is already loaded): + +``` +module load OpenMPI/4.0.5 +``` + +If you now check the output of "`module avail`" again, you should see the `HDF5/1.10.7` module: + +``` +$ module avail + +-------- /home/easybuild/hmns/modules/all/MPI/GCC/10.2.0/OpenMPI/4.0.5 ------- + HDF5/1.10.7 + +------------ /home/easybuild/hmns/modules/all/Compiler/GCC/10.2.0 ------------ + OpenMPI/4.0.5 (L) + +... +``` + +To use HDF5, we need to load this `HDF5/1.10.7` module. We can verify that the installation works +using one of the commands provided by HDF5, `h5dump` for example: + +``` +module load HDF5/1.10.7 +``` + +``` +$ h5dump --version +h5dump: Version 1.10.7 +``` + +If you now check which modules are loaded via "`module list`", you will notice that all module names +and nice and short now, which is one of the advantages of using a hierarchical module tree: + +```shell +$ module list + +Currently Loaded Modules: + 1) GCCcore/10.2.0 5) numactl/2.0.13 9) hwloc/2.2.0 13) PMIx/3.1.5 + 2) zlib/1.2.11 6) XZ/5.2.5 10) libevent/2.1.12 14) OpenMPI/4.0.5 + 3) binutils/2.35 7) libxml2/2.9.10 11) UCX/1.9.0 15) Szip/2.1.1 + 4) GCC/10.2.0 8) libpciaccess/0.16 12) libfabric/1.11.0 16) HDF5/1.10.7 +``` + +## Exercise + +Now it is your turn! + +Try to get a feeling for how a hierarchical module tree works by: + +* installing the missing modules for the `SciPy-bundle-2020.11-foss-2020b.eb` in the module hierarchy we + generated for HDF5; +* figure out where the `SciPy-bundle` module is located in the hierarchy, and then also load it; + +You can verify your work by running this command (since `pandas` is one of the Python packages included +in the `SciPy-bundle` installation): + +```shell +python -c 'import pandas; print(pandas.__version__)' +``` + +Start from a clean slate, by first running: + +```shell +module purge +module unuse $MODULEPATH +``` + +??? success "(click to show solution)" + + * Step 0: check which modules are still missing, using `--missing` or `-M`: + ```shell + eb SciPy-bundle-2020.11-foss-2020b.eb -M + ``` + The output should tell you that 15 out of 50 required modules are still missing. + + * Install the missing modules in the module hierarchy we have generated in `$HOME/hmns/modules`: + ```shell + eb SciPy-bundle-2020.11-foss-2020b.eb --robot --module-only + ``` + Don't forget to use both `--robot` (to enable dependency resolution) and `--module-only` + (to only run the sanity check and generate module files, not install the software again). + + * Start at the top of the module hierarchy (the `Core` level), + and run module spider to check which gateway modules to load to make `SciPy-bundle` available: + ```shell + module use $HOME/hmns/modules/all/Core + module spider SciPy-bundle/2020.11 + ``` + * Load the gateway modules: + ```shell + module load GCC/10.2.0 OpenMPI/4.0.5 + ``` + * Check that the `SciPy-bundle` module is available, and load it: + ```shell + $ module avail SciPy-bundle + ----- /home/example/hmns/modules/all/MPI/GCC/10.2.0/OpenMPI/4.0.5 ------ + SciPy-bundle/2020.11 + ``` + ``` + module load SciPy-bundle/2020.11 + ``` + * Run the test command: + ``` + $ python -c 'import pandas; print(pandas.__version__)' + 1.1.4 + ``` + +!!! Warning + **This exercise will only work if you have write access to the software installation prefix**, + because EasyBuild will try to copy the installation log file to each installlation directory. + +--- + +*[[next: GitHub integration]](3_05_github_integration.md)* diff --git a/docs/2022-CSC_and_LO/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_05_github_integration.md new file mode 100644 index 00000000..59dcbb67 --- /dev/null +++ b/docs/2022-CSC_and_LO/3_05_github_integration.md @@ -0,0 +1,442 @@ +# GitHub integration to facilitate contributing to EasyBuild + +*[[back: Module naming schemes]](3_04_module_naming_schemes.md)* + +--- + +To contribute changes to the EasyBuild code (framework or easyblocks) or easyconfigs, +you will need to be a bit familiar with Git and GitHub. Or maybe not? + +## Manual contribution procedure + +0) Create and setup a [GitHub account](https://github.com/join) (and register your SSH public key); + +1) Clone and fork the appropriate GitHub repository, for example when contributing an easyconfig file: + +```shell +git clone git@github.com:easybuilders/easybuild-easyconfigs.git +cd easybuild-easyconfigs +git remote add my_fork git@github.com:your_github_account/easybuild-easyconfigs.git +``` + +**You should change '`your_github_account`' in the last line to your own GitHub user name!** + +2) Create and check out a new branch, starting from the (up-to-date) ``develop`` branch: + +``` +git checkout develop +git pull origin develop +git checkout -b example +``` + +3) Stage the changes you want to contribute, after you make sure that your easyconfig file has the +[correct filename](../basic_usage/#easyconfig-filenames), and that it's located in the appropriate directory. + +```shell +mkdir -p easybuild/easyconfigs/e/example/ +mv example.eb easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb +git add easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb +``` + +4) Commit those changes with a sensible commit message: + +```shell +git commit -m "This is just an example" +``` + +5) Push your branch to your fork of the repository on GitHub: + +```shell +git push my_fork example +``` + +6) Open the pull request through the GitHub web interface, making sure that: + +* the target branch is correct (should be `develop`); +* an appropriate title is used; +* a short description of the changes is provided; +* the changes are indeed the ones you want to propose; +* clicking the (correct) green button; + +
Pfft
+ +That didn't exactly motivate you to contribute, did it... + +## Github integration features + +Over the years we noticed that some people were keen on contributing to EasyBuild, +but they were not very familiar with Git or GitHub. That meant they had to overcome a +relatively steep learning curve before they could contribute... + +
Gandalf vs Git
+ +In addition, the contribution workflow can be a bit daunting and time consuming, +even if you're already familiar with the procedure. You will have dozens of +branches flying around in no time, and if you get stuck in a weird corner +with `git` you may quickly end up demotivated. + +This is frustrating not only for the people who wanted to contribute but +also for the EasyBuild maintainers, and it doesn't agree with the philosophy of +a project that aims to *automate* tedious software installation procedures. + +At the end of 2015 efforts were made to tackle this issue by implementing +GitHub integration features in EasyBuild, which automate the contribution +workflow by running `git` commands and interacting with the [GitHub API](https://developer.github.com/v3/). + +We will briefly go over some of these features here, but they are also covered in detail [in the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html). + +### Requirements & configuration + +First of all, the GitHub integration features impose a couple of additional [requirements](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) +and configuration. + + +**Additional dependencies** + +Both the `GitPython` and `keyring` Python packages as well as the `keyrings.cryptfile` add-on package must be installed. +In the prepared environment, you can do this via: + +```shell +pip3 install --user GitPython keyring keyrings.cryptfile +``` + +!!! Note + You may experiences problems installing the ``cryptography`` Python packages, + which is a dependency of keyring. The underlying cause is that you need to have + the [``Rust``](https://www.rust-lang.org/) compiler installed to install the latest version + of ``cryptography`` (see [here](https://github.com/pyca/cryptography/issues/5771)). + + You can work around this issue using: + + ```shell + pip3 install --user 'cryptography<3.4' + ``` + +**SSH public key in GitHub account** + +You need to have a GitHub account that has your SSH public key registered in it +(via [https://github.com/settings/keys](https://github.com/settings/keys)). + +If you need to generate an SSH key pair, you can run the following command: + +```shell +ssh-keygen -t rsa -b 4096 +``` + +You can copy the SSH public key from the output of this command: + +```shell +cat .ssh/id_rsa.pub +``` + + +**Forked repository in GitHub** + +In addition, you must have *forked* the EasyBuild repository you want to contribute to +(for example [https://github.com/easybuilders/easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs)). + +**EasyBuild configuration, incl. GitHub token** + +You also have to configure EasyBuild a bit more, so it knows about your +GitHub user name *and* has a GitHub token available in order to perform actions +in GitHub with your credentials. + +To do this, you should define the `github-user` configuration option and +run the "`eb --install-github-token`" command: + +```shell +# replace 'ebtutorial' with your own GitHub username! +$ export EASYBUILD_GITHUB_USER=ebtutorial +$ eb --install-github-token +``` + +To create a GitHub token: + +* Visit [https://github.com/settings/tokens](https://github.com/settings/tokens). +* Click *"Personal access tokens"*. +* Click followed by *"Generate new token"*. +* Give the token a name (for example *"Token for EasyBuild"*). +* Select both the '`repo`' and '`gist`' scopes. +* Click the green *"Generate token"* button. +* Copy the generated token. +* Paste the token when asked by `--install-github-token` (and hit *Enter*). +* Enter a password to encrypt your GitHub token. + +The output should look something like this: + +```shell +$ eb --install-github-token +== temporary log file in case of crash /tmp/eb-9z0bdve9/easybuild-hfpti62w.log +Token: +Validating token... +Token seems to be valid, installing it. +Please set a password for your new keyring: +Please confirm the password: +Token 'fed..987' installed! +``` + + +**Checking status of GitHub integration** + +You can check the status of the GitHub integration using "`eb --check-github`": + +```shell +$ eb --check-github +== temporary log file in case of crash /tmp/eb-4ckdlyfy/easybuild-gp69ev2w.log + +Checking status of GitHub integration... + +Making sure we're online...OK + +* GitHub user...ebtutorial => OK +Please enter password for encrypted keyring: +* GitHub token...fed..987 (len: 40) => OK (validated) +* git command...OK ("git version 1.8.3.1; ") +* GitPython module...OK (GitPython version 3.1.3) +* push access to ebtutorial/easybuild-easyconfigs repo @ GitHub...OK +* creating gists...OK +* location to Git working dirs... not found (suboptimal) + +All checks PASSed! + +Status of GitHub integration: +* --from-pr: OK +* --new-pr: OK +* --review-pr: OK +* --update-pr: OK +* --upload-test-report: OK +``` + +If you see '`OK`' for each of the status checks, you're all set +to try out the GitHub integration features! + +!!! Note + If your SSH private key is protected with a password, you may need + to enter your password a couple of times when running "`eb --check-github`". + + You can avoid this by [using an SSH agent](https://help.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent). + +!!! Note + If you see the ``push access`` check fail with ``Failed to fetch branch 'main'``, + you will need to rename the ``master`` branch in your fork of the ``easybuild-easyconfigs`` + repository from ``master`` to ``main`` (this is required since EasyBuild v4.3.3). + + This can be done via the *pencil* icon at + [https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches](https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches) + (replace ``YOUR_GITHUB_ACCOUNT`` with the name of your GitHub account in this URL). + +### Creating pull requests + +The GitHub integration in EasyBuild allows you to **create pull requests +using the `eb` command**, without even leaving your shell environment. +How cool is that‽ + +To create a pull request to the `easybuild-easyconfigs` repository, +you can either do it in a single go by +running "`eb --new-pr`" and passing it one or more easyconfig files to add +into the pull request. + +The more detailed option is to first create a branch in your repository fork +in GitHub via "`eb --new-branch-github`" and then later open the pull request +via "`eb --new-pr-from-branch`". This method can be useful when preparing multiple +interdependent pull requests, or to check whether your changes pass the unit tests +(which are run automatically in the GitHub Actions CI environment for +all branches pushed to your fork). + +The `--new-pr` option can also be used to open pull requests to the easyblocks +and framework repositories, and it will even automatically determine the target +repository based on the contents of the files you provide. Whoa! + +You can control the target repository for your pull request using +`--pr-target-account` (default is `easybuilders`) and `--pr-target-repo`. + +If you want you can carefully double check your work before actually +opening the pull request by doing a dry run via "`eb --dry-run --new-pr`" +or "`eb -D --new-pr`". + +Finally, you can use "`eb --preview-pr`" to see how the easyconfig files +for which you plan to create a pull request differ from existing easyconfig +files. + +### Updating pull requests + +To update an existing pull request with additional changes +you can use "`eb --update-pr`" and pass the pull request ID, +alongside the paths to the updated files. + +If you have only created a branch (for example via `eb --new-branch-github`) +you can update it via `--update-branch-github` in the same way, +passing the branch name instead of a pull request ID. + +### Using a pull request + +Next to creating and updating branches and pull requests +you can also *use* easyconfig files and easyblocks from a pull request, +regardless of its status (open, merged, or closed). This is particularly +useful when testing contributions, or to install software for which +support is not yet included in the latest EasyBuild release. + +Using the `--from-pr` option you can install easyconfig files from the +pull request with specified ID. By default all easyconfig files that are +touched by the pull request will be installed, but you can specify +particular ones to use as well. It is generally advised to also use the +`--robot` option to ensure that the easyconfig files are installed in the +correct order with respect to dependencies. + +Similarly, using a new or updated easyblock from a pull request is as simple +as using the `--include-easyblocks-from-pr` option. And of course you can +combine it with `--from-pr`! + +Via `--upload-test-report` you can let EasyBuild submit a comment into the +easyconfig pull request to show that the installation worked on your system. This is +useful for others to know, in particular EasyBuild maintainers, since the comment +will include information about your system (OS, processor, etc.) and your EasyBuild configuration. + +## Demo + +That is a lot to digest, so let us make this a bit more concrete with an example: +we will open a pull request for the [`eb-tutorial` example software](../adding_support_software/#example) to *a fork* of the [`easybuild-easyconfigs` repository](https://github.com/easybuilders/easybuild-easyconfigs) using the `eb` command, +and submit a test report in it. + +!!! Note + Make sure that you have correctly configured the GitHub integration, + [see above](#requirements-configuration). + +### Creating pull request + +We first configure EasyBuild to target the `ebtutorial` GitHub account rather +than the default `easybuilders` GitHub organisation, +by defining the `pr-target-account` configuration setting: + +```shell +export EASYBUILD_PR_TARGET_ACCOUNT=ebtutorial +``` + +In the output of "`eb --show-config`" you should see a line like this: + +``` +pr-target-account (E) = ebtutorial +``` + +We only do this to avoid that lots of pull requests for the `eb-tutorial` +example software are opened in the [central easyconfigs repository](https://github.com/easybuilders/easybuild-easyconfigs). + +Opening a pull request is as simple as running "`eb --new-pr`" and passing +the easyconfig file: + +```shell +$ eb --new-pr example.eb +== temporary log file in case of crash /tmp/eb-ggr6scbq/easybuild-hnk271xj.log +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... +== fetching branch 'develop' from https://github.com/ebtutorial/easybuild-easyconfigs.git... +== copying files to /tmp/eb-ggr6scbq/git-working-dirxwk1fzaw/easybuild-easyconfigs... +== pushing branch '20200622095415_new_pr_eb-tutorial100' to remote 'github_ebtutorial_qgtfU' (git@github.com:ebtutorial/easybuild-easyconfigs.git) +Enter passphrase for key '/home/example/.ssh/id_rsa': +Please enter password for encrypted keyring: + +Opening pull request +* target: ebtutorial/easybuild-easyconfigs:develop +* from: ebtutorial/easybuild-easyconfigs:20200622095415_new_pr_eb-tutorial100 +* title: "{tools}[GCC/10.2.0] eb-tutorial v1.0.1" +* labels: new +* description: +""" +(created using `eb --new-pr`) + +""" +* overview of changes: + easybuild/easyconfigs/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb | 26 ++++++++++++++++++++++++++ + 1 file changed, 26 insertions(+) + +Opened pull request: https://github.com/ebtutorial/easybuild-easyconfigs/pull/ +== Temporary log file(s) /tmp/eb-ggr6scbq/easybuild-hnk271xj.log* have been removed. +== Temporary directory /tmp/eb-ggr6scbq has been removed. +``` + +Take a moment to grasp what we did here: we ran **a single `eb` command** which +took care of the **[whole contribution procedure](#contribution-procedure)** for us, including: + +* Cloning the `easybuilders/easybuild-easyconfigs` repository and checking out the `develop` branch (in a temporary + directory); +* Picking a sensible name for a branch and creating it; +* Adding the `eb-tutorial` easyconfig file to the branch, in the correct location + (`easybuild/easyconfigs/e/eb-tutorial/`) and with the correct filename (`eb-tutorial-1.0.1-GCC-10.2.0.eb`); +* Pushing the branch to our fork (`example/easybuild-easyconfigs`); +* Actually opening the pull request, using an informative title. + +That is so... easy! + +This feature not only *significantly* lowers the bar for contributing, +it also saves quite a bit of time since you don't need to double check +various details (like targeting the `develop` branch) or spend time on +coming up with a nice looking title or funny branch name (although you +still can if you really want to). + +There are a couple of nice side effects too, like not having any local branches +to tidy up on once the pull request gets merged (since `--new-pr` created the +branch only in a temporary directory). + +If many contributions are made via `--new-pr` it also simplifies the task +of EasyBuild maintainers, since pull requests opened this way have a particular +structure to them and thus are easier to digest because they look familiar. + +### Uploading test report + +After opening the pull request, we should also upload a test report to show that the installation is working. +This is just as easy as creating the pull request. + +First make sure that the pre-installed software in the prepared environment +is available, since the required dependencies for `eb-tutorial` are already +installed there: + +```shell +module use /easybuild/modules/all +``` + +You can verify which dependencies are still missing using `--from-pr` combined with `--missing`: + +```shell +# change '1' to the ID of your own pull request (see output of --new-pr) +$ eb --from-pr 1 --missing +== temporary log file in case of crash /tmp/eb-ioi9ywm1/easybuild-e3v0xa1b.log +Please enter password for encrypted keyring: +== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... + +1 out of 20 required modules missing: + +* eb-tutorial/1.0.1-GCC-10.2.0 (eb-tutorial-1.0.1-GCC-10.2.0.eb) +``` + +Uploading a test report boils down to combining `--from-pr` with `--upload-test-report`: + +```shell +# change '1' to the ID of your own pull request (see output of --new-pr) +$ eb --rebuild --from-pr 1 --upload-test-report +Please enter password for encrypted keyring: +... +== processing EasyBuild easyconfig /tmp/eb-bnb1pv3n/files_pr65/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb +== building and installing eb-tutorial/1.0.1-GCC-10.2.0... +... +== COMPLETED: Installation ended successfully (took 2 sec) +... +Adding comment to easybuild-easyconfigs issue #65: 'Test report by @ebtutorial +**SUCCESS** +Build succeeded for 1 out of 1 (1 easyconfigs in this PR) +example - Linux centos linux 7.8.2003, x86_64, Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz (haswell), Python 3.6.8 +See https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 for a full test report.' +== Test report uploaded to https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 and mentioned in a comment in easyconfigs PR#1 +``` + +Note that we may need to use `--rebuild` here since `eb-tutorial` may already be installed. + +This results in a comment being added to the pull request: + +
Test report comment
+ +The gist linked from this comment provides more detailed information: + +
Test report gist
+ +--- + +*[[next: Overview]](index.md)* From 6dbfb52a8915e2a96af1fdc1202b399bbd03d296 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 30 Mar 2022 18:58:28 +0200 Subject: [PATCH 138/159] Troubleshooting section reworked for LUMI. --- docs/2022-CSC_and_LO/2_01_troubleshooting.md | 159 +++++++++++++------ 1 file changed, 108 insertions(+), 51 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_01_troubleshooting.md b/docs/2022-CSC_and_LO/2_01_troubleshooting.md index 86a2f1d3..30f2f5cc 100644 --- a/docs/2022-CSC_and_LO/2_01_troubleshooting.md +++ b/docs/2022-CSC_and_LO/2_01_troubleshooting.md @@ -4,8 +4,8 @@ --- -When installing scientific software you are bound to run into problems -that make the installation fail sooner or later, even when using EasyBuild. +Whatever tool you use, when installing scientific software you'll +be running into problems rather sooner than later. In this part we take a look at how you can **troubleshoot a failing installation**, and focus on a couple of EasyBuild aspects that can be helpful in that context, @@ -50,8 +50,8 @@ $ eb example.eb == building... == FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): build failed (first 300 chars): cmd "make" exited with exit code 2 and output: -/usr/bin/g++ -O2 -ftree-vectorize -march=native -fno-math-errno -std=c++14 -c -o core.o core.cpp -g++: error: unrecognized command line option '-std=c++14' (took 1 sec) +/usr/bin/g++ -O2 -ftree-vectorize -march=znver2 -fno-math-errno -c -o core.o core.cpp +cc1plus: error: bad value (‘znver2’) for ‘-march=’ switch (took 1 sec) == Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log ERROR: Build of /home/easybuild/subread.eb failed (err: ...) ``` @@ -60,22 +60,24 @@ Let's break this down a bit: during the `build` step of the installation procedure EasyBuild was running `make` as a shell command, which failed (exit code 2, so not zero). The `make` command tripped over the compilation of `core.cpp` that failed because -`-std=c++14` is not a known option to the `g++` command. +`-march=znver2` is not a known option to the `g++` command. OK fine, but now what? Can you spot something suspicious here? Wait a minute... Why is `make` using `/usr/bin/g++` for the compilation?! That's not where our toolchain compiler is installed, -that's somewhere under `/easybuild/software`. +that's somewhere under `/opt/cray/pe/gcc`. Let's see what `/usr/bin/g++` is: -```shell +``` $ /usr/bin/g++ --version -g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) +g++ (SUSE Linux) 7.5.0 ``` -Oh my that's pretty ancient (GCC 4.8.5 was released in June 2015). -That could definitely explain why it doesn't know about the C++14 standard yet... +Oh my that's an ancient compiler (7.5 was released on November 14, 2019, +a few months after the release of the Zen2 architecture, but +the base version, 7.1, is really from May 2, 2017, long before the Zen2 architecture was around) +That could definitely explain why it doesn't know about the Zen2 architecture yet... Your next step in this case should probably be figuring out why `/usr/bin/g++` is being used rather than just `g++`, which would @@ -98,7 +100,7 @@ In some cases there won't be any useful information in there however, since the actual error message(s) could only appear way later, perhaps even after the command was already running for several minutes. -In that case, you will have the dive into the log file that is created by EasyBuild for +In that case, you will have to dive into the log file that is created by EasyBuild for every installation, which is located in the unique temporary directory for the EasyBuild session. See for example this output line from our earlier example error message: @@ -122,14 +124,14 @@ directory for successful installation, into the `easybuild` subdirectory. For example: ``` -/easybuild/software/HDF5/1.10.7-gompi-2020b/easybuild/easybuild-HDF5-1.10.7-20210308.214453.log +/appl/lumi/SW/LUMI-21.12/L/EB/ncurses/6.2-cpeGNU-21.12/easybuild/easybuild-ncurses-6.2-20220302.110244.log ``` ### Last log The `eb` command supports a handy little option that prints the location to the most recently updated build log. You can leverage this to quickly -open the build log of the last failed EasyBuild session in an editor: +open the build log of the last ***failed*** EasyBuild session in an editor: ``` vim $(eb --last-log) @@ -206,7 +208,10 @@ For software using a classic `configure` script, you may have to locate and inspect the `config.log` file in the build directory to determine the underlying cause of an error. For software using CMake as a configuration tool you often have to check in `CMakeOutput.log` or `CMakeError.log` for clues, which are sneakily hidden by CMake in a `CMakeFiles` subdirectory of the build directory. -As a side note here: as EasyBuild does not clean out old and failed builds you will need to eventually manually remove these build directories from the `buildpath` directory. +As a side note here: as EasyBuild does not clean out old and failed builds you will need to eventually manually remove these build directories from the `buildpath` directory. On the login nodes of LUMI this is currently not much of an issue as `$XDG_RUNTIME_DIR` is (ab)used for build and +temporary files and that directory is cleaned automatically. However, when building on compute nodes, where `$XDG_RUNTIME_DIR` is +not available, it is essential to manually clean that directory as the space is not automatically cleaned when your session +ends. The `EasyBuild-user` module does define the bash function `clear-eb` that can be used to clear that space. ## Exercise @@ -214,7 +219,7 @@ Let's work our way through a less smooth software installation, using the easyconfig file that is provided below. Don't worry if most of this is still unclear to you, we'll get -to writing easyconfig files from scratch [later in this tutorial](../creating_easyconfig_files). +to writing easyconfig files from scratch [later in this tutorial](2_02_creating_easyconfig_files). ```python easyblock = 'MakeCp' @@ -225,7 +230,7 @@ version = '2.0.1' homepage = 'http://subread.sourceforge.net' description = "High performance read alignment, quantification and mutation discovery" -toolchain = {'name': 'GCC', 'version': '8.5.0'} +toolchain = {'name': 'PrgEnv-gnu', 'version': '21.10'} # download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz sources = ['subread-%(version)s-source.tar.gz'] @@ -253,25 +258,23 @@ Do you spot any potential problems yet with this easyconfig file? ***Preparation*** -Start by copying the text above in a file named `subread.eb`, +Start by copying the text above in a file named `subread.eb` +(which does not follow the EasyBuild conventions but that is not a problem for this exercise), so you can gradually fix the problem you'll encounter. Also make sure that the pre-installed software stack is available, -and that the EasyBuild module is loaded (unless you installed EasyBuild +and that the EasyBuild-user module is loaded (unless you installed EasyBuild yourself): ``` -module use /easybuild/modules/all -module load EasyBuild +module load LUMI/21.12 +module load EasyBuild-user ``` -For this exercise, make sure EasyBuild is configured to -use `$HOME/easybuild` as `prefix`, and to use `/tmp/$USER` as `buildpath`: - -```shell -export EASYBUILD_PREFIX=$HOME/easybuild -export EASYBUILD_BUILDPATH=/tmp/$USER -``` +This will configure EasyBuild correctly for this exercise, though if you already have +an existing EasyBuild user installation you may want to work in a different one +by pointing `$EBU_USER_PREFIX` to the desired work directory before loading +`LUMI/21.12`. Check your configuration via `eb --show-config`. @@ -297,7 +300,7 @@ the easyconfig file? ``` $ eb subread.eb ... - == FAILED: Installation ended unsuccessfully (build directory: /tmp/example/Subread/2.0.1/GCC-8.5.0): build failed (first 300 chars): + == FAILED: Installation ended unsuccessfully (build directory: /run/user/XXXXXXXX/easybuild/build/Subread/2.0.1/cpeGNU-21.12): build failed (first 300 chars): Couldn't find file subread-2.0.1-source.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order): ... ``` @@ -315,12 +318,10 @@ the easyconfig file? (in the `sourcepath` directory): ``` curl -OL https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz - mv subread-2.0.1-source.tar.gz $HOME/easybuild/sources/s/Subread/ + mkdir -p $EBU_USER_PREFIX/sources/s/Subread + mv subread-2.0.1-source.tar.gz $EBU_USER_PREFIX/sources/s/Subread/ ``` - If downloading is problematic for some reason, the source tarball is also available - in `/easybuild/sources/s/Subread`. - Or, we can change the easyconfig file to specify the location where the easyconfig file can be downloaded from: ```python @@ -335,9 +336,9 @@ the easyconfig file? The source tarball is fairly large (23MB), so don't be alarmed if the download takes a little while. ```shell - $ ls -lh $HOME/easybuild/sources/s/Subread + $ ls -lh $EBU_USER_PREFIX/sources/s/Subread total 23M - -rw-rw-r-- 1 easybuild easybuild 23M Jun 13 17:42 subread-2.0.1-source.tar.gz + -rw-rw-r-- 1 XXXXXXXX XXXXXXXX 23M Mar 30 16:08 subread-2.0.1-source.tar.gz ``` --- @@ -350,26 +351,52 @@ What's wrong now? How can you fix it quickly? Take into account that we just want to get this software package installed, we don't care too much about details like the version of the dependencies or -the *toolchain* here... +the toolchain here... ??? success "(click to show solution)" The installation fails because the easyconfig specifies that GCC 8.5.0 should be used as toolchain: + ``` $ eb subread.eb ... - == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-8.5.0): build failed (first 300 chars): - No module found for toolchain: GCC/8.5.0 (took 1 sec) + ERROR: Failed to process easyconfig /pfs/lustrep3/users/kurtlust/easybuild-tutorial/Troubleshooting/subread.eb: Toolchain PrgEnv-gnu not found, + available toolchains: ... + ... + ``` + + `PrgEnv-gnu` is an HPE Cray PE module that may look like a toolchain - it certainly has + the same function: provide compiler, MPI and basic math libraries - but it is not + recognised as a toolchain by EasyBuild. EasyBuild prefers to manage its own modules so that it knows + well what is in it which is not the case with the `PrgEnv-*` modules from the Cray PE + as the content may differe between systems and as the versions of the compilers etc. that + are loaded differ on other modules that are loaded. Hence we created Cray-specific toolchains. + You'll actually find two series of Cray toolchains in the list of available toolchains. The + `CrayGNU`, `CrayIntel`, `CrayPGI` and `CrayCCE` are included with the EasyBuild distribution + and where developed by CSCS for their systems using Environment Modules. These were not compatible + with the initial releases of the Cray PE with Lmod modules so new ones were developed on which we + also built for the LUMI toolchains. Those are called `cpeCray`, `cpeGNU`, `cpeAOCC` and `cpeAMD` + and are maintained by LUST and available via the LUMI repositories. + + Changing the toolchain name to `cpeGNU` is not enough to solve all problems though: + + ``` + $ eb subread.eb + ... + ERROR: Build of /pfs/lustrep3/users/kurtlust/easybuild-tutorial/Troubleshooting/subread.eb failed (err: 'build failed (first 300 chars): + No module found for toolchain: cpeGNU/21.10') + ... ``` - We don't have this GCC version installed, but we do have GCC 10.2.0: + We don't have this `cpeGNU` version installed, but we do have GCC 21.12: ```shell - $ module avail GCC/ - ----------------- /easybuild/modules/all ------------------ - GCC/10.2.0 + $ module avail cpeGNU/ + ----- Infrastructure modules for the software stack LUMI/21.12 on LUMI-L ----- + cpeGNU/21.12 + ... ``` So let's try using that instead. @@ -377,7 +404,7 @@ the *toolchain* here... Edit the easyconfig file so it contains this: ```python - toolchain = {'name': 'GCC', 'version': '10.2.0'} + toolchain = {'name': 'cpeGNU', 'version': '21.12'} ``` --- @@ -392,13 +419,13 @@ Can you fix the next problem you run into? The compilation fails, but the error message we see is incomplete due to EasyBuild truncating the command output (only the 300 first characters of the output are shown): ``` - == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-10.2.0): build failed (first 300 chars): - cmd " make -j 1 -f Makefile.Linux CFLAGS="-fast -fcommon"" exited with exit code 2 and output: - gcc -mtune=core2 -O3 -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\""2.0.1"\" -D_FILE_OFFSET_BITS=64 -fmessage-length=0 -ggdb -fast -fcommon -c -o core.o core.c - gcc: error: unrecognized command line opti (took 1 sec) + == FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12): build failed + (first 300 chars): cmd " make -j 256 -f Makefile.Linux CFLAGS="-fast -fcommon"" exited with exit code 2 and output: + gcc -mtune=core2 -O3 -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\""2.0.1"\" -D_FILE_OFFSET_BITS=64 -fmessage-length=0 + -ggdb -fast -fcommon -I/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86 (took 4 secs) ``` - If you open the log file and scroll to the end, + If you open the log file (e.g., with `view $(eb --last-log)`) and scroll to the end, the error is pretty obvious: ``` gcc: error: unrecognized command line option -fast; did you mean -Ofast? @@ -416,7 +443,8 @@ Can you fix the next problem you run into? to hard specify compiler flags (certainly not incorrect ones). The comment above the `buildopts` definition makes it clear that the `-fcommon` flag *is* required though, because GCC 10 became a bit stricter by - using `-fno-common` by default. Note that we are using `-fcommon` + using `-fno-common` by default (and we're using GCC 11 in `cpeGNU/21.12`). + Note that we are using `-fcommon` as an escape mechanism here: it would be better to fix the source code and create a patch file instead. @@ -433,6 +461,32 @@ Can you fix the next problem you run into? Note that we need to be careful with quotes here: we use inner double quotes to ensure that `$CFLAGS` will be expanded to its value when the build command is run. + As you can see after the change by doing a dry-run: + + ``` + $ eb subread.eb -x + ... + + Defining build environment... + + export BLAS_INC_DIR='/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/include' + ... + export CC='cc' + export CFLAGS='-O2 -ftree-vectorize -fno-math-errno' + ... + [build_step method] + running command "make -j 256 -f Makefile.Linux CFLAGS="$CFLAGS -fcommon"" + (in /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12/Subread-2.0.1/src) + ... + ``` + + EasyBuild will launch the command + ``` + make -j 256 -f Makefile.Linux CFLAGS="$CFLAGS -fcommon" + ``` + in a shell where `CFLAGS` is defined and set to an appropriate value (determined by + defaults in EasyBuild, settings in the EasyBuild configuration and settings in the + easyconfig file that we shall discuss later). --- @@ -449,10 +503,11 @@ Don't give up now, try one last time and fix the last problem that occurs... ``` $ eb subread.eb ... - == FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/Subread/2.0.1/GCC-10.2.0): build failed (first 300 chars): - Sanity check failed: sanity check command featureCounts --version exited with code 255 + == FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12): + build failed (first 300 chars): Sanity check failed: sanity check command featureCounts --version exited with code 255 (output: featureCounts: unrecognized option '--version' ... + ... ``` If you look at the full output in the log file you can see @@ -473,12 +528,14 @@ Don't give up now, try one last time and fix the last problem that occurs... --- -In the end, you should be able to install Subread 2.0.1 with the GCC 10.2.0 toolchain by fixing the problems with the `subread.eb` easyconfig file. +In the end, you should be able to install Subread 2.0.1 with the cpeGNU 21.12 toolchain by fixing the problems with the `subread.eb` easyconfig file. Check your work by manually loading the module and checking the version via the `featureCounts` command, which should look like this: ```shell +$ module load Subread/2.0.1-cpeGNU-21.12 +... $ featureCounts -v featureCounts v2.0.1 ``` From 7f4a8a7761330f5fa0ce3c772e577fc4743dc1be Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 1 Apr 2022 18:15:00 +0200 Subject: [PATCH 139/159] Section on creating easyconfig files. --- .../2_02_creating_easyconfig_files.md | 527 +++++++++--------- 1 file changed, 278 insertions(+), 249 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md index d46e003a..99ac7d88 100644 --- a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md @@ -4,8 +4,9 @@ --- -We already know how to install easyconfig files that are included with EasyBuild, -but what about installing software for which no easyconfig is available yet? +We already know how to install easyconfig files that are provided by LUMI, +but what about installing software for which no easyconfig is available yet, +or for which we need to adapt an existing easyconfig? To do this we will need to **create additional easyconfig files**, since every software installation performed by EasyBuild is done based on an easyconfig file. @@ -14,10 +15,10 @@ In this part of the tutorial we will look at the guts of easyconfig files and ev ## Easyconfigs vs easyblocks -Before we dive into writing [easyconfig files](../introduction/#easyconfig-files), -let us take a brief look at how they relate to [easyblocks](../introduction/#easyblocks). +Before we dive into writing [easyconfig files](../1_05_terminology/#easyconfig-files), +let us take a brief look at how they relate to [easyblocks](../1_05_terminology/#easyblocks). -As we discussed [earlier](../introduction/#terminology), an easyconfig file (`*.eb`) is required +As we discussed [earlier](../1_05_terminology/#terminology), an easyconfig file (`*.eb`) is required for each installation performed by EasyBuild which specifies the details of the installation (which software version, toolchain, etc.), while the installation procedure is implemented @@ -30,34 +31,47 @@ together with a custom *software-specific* easyblock? This is not an easy question to answer in a general sense, since it depends on several factors: the complexity of the software you want to get installed, how much flexibility you want, how "intelligent" the installation procedure should be with respect to the compiler toolchain and dependencies -that are used for the installation, etc. +that are used for the installation, whether you want an easyconfig that is more user-focused (with easy-to-understand +parameters) or one that is more focused towards people who understand installation commands, etc. -In a nutshell, custom software-specific easyblocks are "do once and forget": they are central solution to peculiarities in the installation procedure of a particular software package. +In a nutshell, custom software-specific easyblocks are "do once and forget": they are central solution to peculiarities in the installation procedure of a particular software package. However, they also hide a lot from direct view, so if things go +wrong, it is often harder to debug the exact problem. And since a single easyblock has to cover +multiple software versions, multiple toolchains and multiple possible configurations of the package, +they are often harder to develop and certainly harder to test. As a result many of the easyblocks +included with EasyBuild work poorly on HPE Cray systems, e.g., because they want to add compiler flags +specific for a certain compiler and don't recognise the Cray compilers. Reasons to consider implementing a software-specific easyblock rather than using a generic easyblock include: -* 'critical' values for easyconfig parameters required to make installation succeed; -* toolchain-specific aspects of the build and installation procedure (e.g., configure options); -* interactive commands that need to be run; -* custom (configure) options for dependencies; -* having to create or adjust specific (configuration) files; -* 'hackish' usage of a generic easyblock; -* complex or very non-standard installation procedure; - -Implementing easyblocks is out of scope for this basic tutorial, for more information please consult -the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). +- 'critical' values for easyconfig parameters required to make installation succeed; +- toolchain-specific aspects of the build and installation procedure (e.g., configure options); +- interactive commands that need to be run; + *For example: The easyblock to [install maple](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/m/maple.py) + responds to a number of interactive questions.* +- custom (configure) options for dependencies; + *For example, the [easyblock for VMD](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/v/vmd.py) + will add configure options depending on the dependency list of the package.* +- having to create or adjust specific (configuration) files; +- 'hackish' usage of a generic easyblock; +- complex or very non-standard installation procedure; + *For example, the [easyblock to install the gcc compilers from source](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/g/gcc.py), + bootstrapping with the system compiler and then re-installing with itself.* + +For implementing easyblocks we refer to the +["Implementing easyblocks" section of this tutorial](2_03_implementing_easyblocks.md) +and the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). ## Writing easyconfig files Writing an easyconfig file boils down to defining a set of easyconfig parameters in a text file, which we give a '`.eb`' extension by convention. The name of an easyconfig file doesn't matter when using it directly to install software, but it does matter when EasyBuild needs to find it -to resolve a dependency for example (as we [discussed earlier](../basic_usage/#easyconfig-filenames)). +to resolve a dependency for example (as we [discussed earlier](../1_08_basic_usage/#easyconfig-filenames)). The syntax for easyconfig files is *Python syntax*: you are basically defining a bunch of Python variables that correspond to easyconfig parameters. -The order in which the easyconfig parameters are defined doesn't matter, but we generally try to strick to a particular +The order in which the easyconfig parameters are defined doesn't matter, but we generally try to stick to a particular order which roughly corresponds to the order in which the easyconfig parameters are used during the installation. That is mostly helpful for humans staring at easyconfig files or comparing them. @@ -65,13 +79,13 @@ That is mostly helpful for humans staring at easyconfig files or comparing them. A limited number of easyconfig parameters are *mandatory*, they must be defined in every easyconfig file: -* `name`: the name of the software to install; -* `version`: the version of the software to install; -* `homepage`: a URL to the website of the software; -* `description`: a short description of the software; -* `toolchain`: the compiler toolchain to use for the installation; +- `name`: the name of the software to install; +- `version`: the version of the software to install; +- `homepage`: a URL to the website of the software; +- `description`: a description of the software; +- `toolchain`: the compiler toolchain to use for the installation; -**`name`, `version`** +#### `name`, `version` It should be no surprise that specifying the name and version of the software you want to install is mandatory. This information may influence the value of several other easyconfig parameters (like the name of the source file), and is also used to the determine the name of the module file to install. @@ -81,7 +95,7 @@ name = 'example' version = '1.0' ``` -**`homepage`, `description`** +#### `homepage`, `description` The homepage and description are included in the generated module file for the installation. That way the "`module show`" command provides some useful high-level information about the installation. @@ -91,10 +105,8 @@ homepage = 'https://example.org' description = "This is just an example." ``` -Usually it does not matter whether you use single or double quotes to specify string values, -but you will often see that single quotes are used for values that don't have spaces (words) -and double quotes for values that do have spaces (sentences). There is no technical reason for -this, it just feels more natural to some people. There are cases where it is important to use +Usually it does not matter whether you use single or double quotes to specify string values. +There are cases where it is important to use the right type of quotes however, we will get back to that later (keep it in mind for the exercises!). For multi-line descriptions, you will need to use "triple quoting" (which is standard Python syntax): @@ -105,23 +117,43 @@ description = """This is an example It is spread across multiple lines.""" ``` -**`toolchain`** +If no homepage is known for the package, the convention in the EasyBuild community is to use +`(none)` as the value. + +The `description` field is used in two different places in the LUA module file generated by +EasyBuild: +- In the help block, for the first section of the help information +- If there is no `whatis` parameter in the easyconfig file, it is also used to generate + a whatis-command with the description in the module file. + +*Note:* We ask people who contribute easyconfigs to LUMI to instead use a decent description +that is useful in the help information, and to specify a short description via the `whatis` +parameter: + +```python +whatis = ['Description: Blosc is an extremely fast, multi-threaded, meta-compressor library'] +``` + +Note that in this case the word "Description:" has to be explicitly specified. -EasyBuild also requires that the [compiler toolchain](../introduction/#toolchains) is specified, via the `toolchain` + +#### `toolchain` + +EasyBuild also requires that the [compiler toolchain](../1_05_terminology/#toolchains) is specified, via the `toolchain` easyconfig parameter. -This can either be the [`system` toolchain](../introduction/#system-toolchain), for which a constant named `SYSTEM` is available: +This can either be the [`system` toolchain](../1_05_terminology/#system-toolchain), for which a constant named `SYSTEM` is available: ```python toolchain = SYSTEM ``` -Usually we specify a 'proper' toolchain like the compiler-only toolchain GCC 10.2.0 which we used before, -or the full toolchain `foss` 2020b. The name and version of the toolchain can be specified using a small Python dictionary, +Usually we specify a 'proper' toolchain like the `cpeGNU/21.12` toolchain we have used bvefore. +The name and version of the toolchain can be specified using a small Python dictionary, for example: ```python -toolchain = {'name': 'GCC', 'version': '10.2.0'} +toolchain = {'name': 'cpeGNU', 'version': '21.12'} ``` ### Commonly used parameters @@ -177,8 +209,11 @@ Some things worth pointing out here: file. This way the software version is only specified in one place and the easyconfig file is easier to update to other software versions. A list of template values can be consulted via the EasyBuild command line via the `--avail-easyconfig-templates` option, or in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_templates.html). -* Source files can also be specified in ways other than just using a filename, see the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-param-sources-alt) for more information. -* Specified checksums are usually SHA256 checksum values, but [other types are also supported](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html?highlight=checksums#checksums). +* Source files can also be specified in ways other than just using a filename, see the + [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-param-sources-alt) for more information. + It is also possible to download a given commit from a GitHub repository. +* Specified checksums are usually SHA256 checksum values, but + [other types are also supported](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html?highlight=checksums#checksums). #### Easyblock @@ -189,7 +224,9 @@ This is not mandatory however, because by default EasyBuild will determine the e name of the software. If '`example`' is specified as software name, EasyBuild will try to locate a software-specific easyblock named `EB_example` (in a Python module named `example.py`). Software-specific easyblocks follow the convention that the class name starts with `'EB_`', followed by the software name -(where some characters are replaced, like '`-`' with '`_minus_`'). +(where some characters are replaced, like '`-`' with '`_minus_`'). It is possible to use different +naming conventions for software-specific easyblocks, but then EasyBuild will not automatically detect +that there is one for the package and it will also need to be specified via the `easyblock` parameter. **Generic easyblocks** @@ -234,7 +271,7 @@ tar_config_opts* Override tar settings as determined by configure. [defau #### Dependencies -You will often need to list one or more [dependencies](../introduction/#dependencies) that are required +You will often need to list one or more [dependencies](../1_05_terminology/#dependencies) that are required to install or run the software. We distinguish between two main different types of dependencies: runtime dependencies and build dependencies. @@ -248,12 +285,17 @@ software once it is installed. The modules for these dependencies will be loaded set up by EasyBuild during the installation, but they will *not* be loaded by the generated module file. You can specify build dependencies via the `builddependencies` easyconfig parameter. One typical example of a build dependency is `CMake`, which is only needed for configuring -the build. +the build. On LUMI we define a `buildtools` module for each version of the `LUMI` software stack +which contains most popular build tools. This ensures that we use a consistent set of build tools +and that they do not change over the life time of a stack, also in case of OS upgrades, to +increase the reproducibility of the build process Here is a simple example of specifying dependencies: ```python -builddependencies = [('CMake', '3.18.4')] +builddependencies = [ + ('buildtools', '%(toolchain_version)s', '', True) +] dependencies = [ ('Python', '3.8.2'), @@ -268,20 +310,38 @@ The name and version of a dependency is specified with a 2-tuple (a tuple with t In some cases additional information may have to be provided, as is shown in the example above for the `SciPy-bundle` dependency where a 3rd value is specified corresponding to the `versionsuffix` value of this dependency. -If this is not specified, it is assumed to be the empty string (`''`). +If this is not specified, it is assumed to be the empty string (`''`). Note how we use the '`%(pyver)s'` template value in the `SciPy-bundle` dependency -specification, to avoid hardcoding the Python version in different places. +specification, to avoid hardcoding the Python version in different places. (Though this +specific parameter is less useful on LUMI as we currently try to build on top of `cray-python`.) + +The `buildtools` build dependency shows that there is a fourth parameter specifying the toolchain +used for that dependency and is needed if that toolchain is different from the one used in the example. +As it is not possible to load several Cray toolchains together (they are not in a hierachical relation) +the only useful value on LUMI is `True` which tells that `buildtools` is build with the `SYSTEM` +toolchain. Here also we use a template, `%(toolchain_version)s` which - as its name suggests - expands +to the version of the toolchain, as we version our `buildtools` modules after the version of the Cray +toolchains for which they are intended. + +See also the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#dependencies) +for additional options on specifying dependencies. That page specifies two more dependency types: -See also the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#dependencies) for additional options on specifying dependencies. +* `hiddendependencies` are currently not used on LUMI, and if we would use them in the future, it will likely + be through a way that does not require this parameter. +* `osdependencies` can be used to let EasyBuild check if certain needed OS packages are installed. + See, e.g., the [easyconfigs for the `buildtools` package](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/easybuild/easyconfigs/b/buildtools) + on LUMI. #### Version suffix -In some cases you may want to build a particular software package in different configurations, or include a label in the module name to highlight a particular aspect +In some cases you may want to build a particular software package in different configurations, +or include a label in the module name to highlight a particular aspect of the installation. -The `versionsuffix` easyconfig parameter can be used for this purpose. The name of this parameter implies that this label will be added after the +The `versionsuffix` easyconfig parameter can be used for this purpose. +The name of this parameter implies that this label will be added after the software version (and toolchain label) in the standard module naming scheme. If you are configuring the software to build with a particular non-default value, @@ -303,7 +363,8 @@ dependencies = [('Python', '2.7.18')] Even though Python 2 is officially dead and buried some scientific software still requires it, and mixing modules where some use Python 2 and other use Python 3 doesn't work well. -The `versionsuffix` label is helpful to inform the user that a particular Python version is required by the installation. +The `versionsuffix` label is helpful to inform the user that a particular Python version is +required by the installation. #### Customizing configure, build, test, and install commands @@ -330,6 +391,7 @@ prebuildopts = 'export HDF5_PREFIX="$EBROOTHDF5" && ' installopts = "PREFIX='%(installdir)s'" ``` +(again an imaginary example as on LUMI we advise to use the Cray-provided HDF5 modules.) Here we are: @@ -406,7 +468,13 @@ Finally, you will usually see the `moduleclass` easyconfig parameter to be defin moduleclass = 'lib' ``` -This is done to categorize software, and it is used to group the generated module files into smaller sets ([remember what we saw when installing software earlier](../basic_usage/#using-installed-software)). +This is done to categorize software, and it is used to group the generated module files into smaller sets +([remember what we saw when installing software earlier](../1_08_basic_usage/#using-installed-software)). + +This is currently not used on LUMI since we feel that (a) it is not easy to explain to users how they can +then only make certain classes that are useful to them visible and more importantly (b) since it is not +always intuitive to decide which moduleclass should be used for a package, or from a user 's perspective, +in which category to look for a package. ## Generating tweaked easyconfigs @@ -428,7 +496,7 @@ eb example-1.2.3.eb --try-software-version 1.2.4 Or, to try using a different compiler toolchain you can use `--try-toolchain`: ```shell -eb example-1.2.3-foss-2020b.eb --try-toolchain intel,2020b +eb example-1.2.3-foss-2020b.eb --try-toolchain cpeCray,21.12 ``` It is important to keep in mind the *"try"* aspect here: while easyconfigs that @@ -477,20 +545,18 @@ You can consult the unpacked sources at [https://github.com/easybuilders/easybui ### Preparation -Make sure EasyBuild is properly configured before you start: +Make sure that a proper version of the `LUMI` software stack is loaded +(we recommend `LUMI/21.12` for the session for which this tutorial was designed) +and also `EasyBuild-user` is loaded to configure EasyBuild to install in +`$HOME/EasyBuild` or `$EBU_USER_PREFIX`. -```shell -export EASYBUILD_PREFIX=$HOME/easybuild -export EASYBUILD_BUILDPATH=/tmp/$USER +``` shell +module load LUMI/21.12 +module load EasyBuild-user` ``` -and that the installed software in `/easybuild` is available: -```shell -module use /easybuild/modules/all -``` - -### Mandatory parameters +### Easyblock Let's start by getting the mandatory easyconfig parameters defined in the easyconfig file: @@ -499,48 +565,26 @@ name = 'eb-tutorial' version = '1.0.1' homepage = 'https://easybuilders.github.io/easybuild-tutorial' -description = "EasyBuild tutorial example" -``` -If we try using this (very) minimal easyconfig file, EasyBuild will inform us that we failed to specify one of the mandatory easyconfig parameters: `toolchain`: +whatis = [ 'Description: EasyBuild tutorial example'] -``` -$ eb example.eb -== Temporary log file in case of crash /tmp/eb-90j723rl/easybuild-q21plqvx.log -== found valid index for /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs, so using it... -ERROR: Failed to process easyconfig /home/example/example.eb: mandatory parameters not provided in pyheader: toolchain +description = """ +This is a short C++ example program that can be buid using CMake. +""" ``` -We will use `GCC/10.2.0` as toolchain, since we know it is already installed in `/easybuild` in the prepared environment, so we also define the `toolchain` easyconfig parameter: - -```python -toolchain = {'name': 'GCC', 'version': '10.2.0'} -``` - -In addition, we'll also specify the `moduleclass`. -This is not required, but it is usually set to a sensible value: - -```python -moduleclass = 'tools' -``` - -The default value is '`base`', at least '`tools`' has *some* meaning. - -### Easyblock - -Let us see what happens if we take our current easyconfig file for a spin: +Let's see what EasyBuild does with this very minimal easyconfig file: ```shell -$ eb example.eb -== temporary log file in case of crash /tmp/eb-8_vxjfn7/easybuild-k3aaoan2.log -ERROR: Failed to process easyconfig /home/example/example.eb: +$ eb eb-tutorial.eb +== Temporary log file in case of crash /run/user/XXXXXXXXX/easybuild/tmp/eb-k_82wotb/easybuild-wg0k_reb.log +ERROR: Failed to process easyconfig /pfs/lustrep4/users/XXXXXXXX/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v1.eb: No software-specific easyblock 'EB_eb_minus_tutorial' found for eb-tutorial ``` -That didn't get us very far... - -The error shows that there is no software-specific easyblock available for installing the software with the name '`eb-tutorial`'. -Does that mean we have to implement an easyblock? +It is not mandatory to specify an easyblock in the easyconfig. However, in the absense of that +specification, EasyBuild goes looking for an application-specific easyblock with the standard name, +in this case `EB_eb_minus_tutorial`, which it does not have. Does that mean we have to implement an easyblock? In this simple case it doesn't, since we can leverage one of the available *generic easyblocks*. But, which one? @@ -559,81 +603,90 @@ easyblock = 'CMakeMake' The "`easyblock =`" line is usually at the top of the easyconfig file, but strictly speaking the order of the parameter definitions doesn't matter (unless one is defined in terms of another one). -### CMake build dependency -Does using the `CMakeMake` generic easyblock help at all? +### Mandatory parameters + +When trying this improved easyconfig file, EasyBuild will inform us that we failed to specify +one of the mandatory easyconfig parameters: `toolchain`: ``` $ eb example.eb -== temporary log file in case of crash /tmp/eb-yutbor1p/easybuild-4jc9v1u9.log -== found valid index for /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs, so using it... -== processing EasyBuild easyconfig /home/example/example.eb -== building and installing eb-tutorial/1.0.1-GCC-10.2.0... -== fetching files... -== creating build dir, resetting environment... -== unpacking... -== patching... -== preparing... -== configuring... -== FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/ebtutorial/1.0.1/GCC-10.2.0): -build failed (first 300 chars): cmd " cmake -DCMAKE_INSTALL_PREFIX=/home/example/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER='gcc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=native -fno-math-errno' -DCMAKE_CXX_COMPILER='g++' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=native (took 0 sec) +== Temporary log file in case of crash /run/user/XXXXXXXXX/easybuild/tmp/eb-22om7hut/easybuild-ipozjipf.log +ERROR: Failed to process easyconfig /pfs/lustrep4/users/XXXXXXXX/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v2.eb: +mandatory parameters not provided in pyheader: toolchain ``` -It did help: EasyBuild made an attempt to configure the build using the `cmake` command, but that failed almost -instantly. We need to dive into the log file to see the actual reason. By starting at the end of the log file and -scrolling up, you should be able to locate the following error message: +We will use `cpeCray/21.12` as toolchain, so we also define the `toolchain` easyconfig parameter: +```python +toolchain = {'name': 'cpeCray', 'version': '21.12'} ``` -/bin/bash: cmake: command not found + +In addition, we'll also specify the `moduleclass`. +This is not required, but it is usually set to a sensible value: + +```python +moduleclass = 'tools' ``` -Ah, that explains it, `cmake` isn't even installed on this system. Or is it? +The default value is '`base`', at least '`tools`' has *some* meaning. -```shell -$ module avail CMake +### CMake build dependency ---------------------------- /easybuild/modules/all ---------------------------- - CMake/3.18.4-GCCcore-10.2.0 -``` +The `CMakeMake` easyblock needs the `cmake` command. On LUMI we are lucky as +`cmake` is already installed in the OS. It may be a somewhat older version, but for this +program is is probably enough. Yet in general it is better to use sufficiently recent +build tools, and `cmake` is one of those tools that is typically entered as a build +dependency. After all, the less you rely on the OS, the more likely it becomes that +your easyconfig is useful for other sites also. -Since a module is available for `CMake` that is compatible with the toolchain we are using (GCC 10.2.0), -we can use it as a dependency for the installation. -It is only needed for building the software, not for running it, so it's only a *build* dependency: +In the [section on Lmod](../1_02_Lmod#module-extensions) we've already seen that on LUMI +the `cmake` command is available through the `buildtools` modules, and as discussed in +other examples on this page, LUMI has one for every `LUMI` software stack with its version +number the same as the stack and corresponding toolchains. It is a good practice to +add this module as a build dependency: ```python -builddependencies = [('CMake', '3.18.4')] +builddependencies = [ + ('buildtools', '%(toolchain_version)s', '', True) +] ``` -There is usually no need to specify toolchain for (build) dependencies, EasyBuild will automatically consider -[subtoolchains](../terminology/#toolchains) compatible with the specified toolchain to locate module for the dependencies. +In a more traditional EasyBuild setup with the common toolchains, there is usually no need to specify +the toolchain for (build) dependencies. EasyBuild will automatically consider +[subtoolchains](../1_05_terminology/#toolchains) compatible with the specified toolchain to locate +modules for the dependencies. However, the Cray PE toolchains on LUMI are currently not part of +such a hierarchy and the `SYSTEM` toolchain we used for `buildtools` is not automatically considered +which is why we need the 4-element version of the dependency specification. -You can verify this via `eb -D` (equivalent with `eb --dry-run`): +You can verify that EasyBuild now locates the dependency via `eb -D` (equivalent with `eb --dry-run`): ``` -$ eb example.eb -D +$ eb eb-tutorial.eb -D ... - * [x] /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs/g/GCC/GCC-10.2.0.eb (module: GCC/10.2.0) - * [x] /easybuild/software/EasyBuild/4.3.3/easybuild/easyconfigs/c/CMake/CMake-3.18.4-GCCcore-10.2.0.eb (module: CMake/3.18.4-GCCcore-10.2.0) - * [ ] /home/example/example.eb (module: eb-tutorial/1.0.1-GCC-10.2.0) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) + * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeCray/cpeCray-21.12.eb (module: cpeCray/21.12) + * [ ] /pfs/lustrep4/users/kulust/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v4.eb (module: eb-tutorial/1.0.-cpeCray-21.12) ``` ### Sources -If you try again after adding `CMake` as a build dependency, you will see the installation fail again in the +If you try again after adding `buildtools` as a build dependency, you will see the installation fail again in the configuration step. Inspecting the log file reveals this: ``` -CMake Error: The source directory "/tmp/example/ebtutorial/1.0.1/GCC-10.2.0" does not appear to contain CMakeLists.txt. +CMake Error: The source directory "/run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12" does not appear to contain CMakeLists.txt. ``` -Wait, but there *is* a `CMakeLists.txt`, we can see it in the [unpacked sources](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1)! +Wait, but there *is* a `CMakeLists.txt`, we can see it in the +[unpacked sources](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1)! Let's inspect the build directory: ``` -$ ls /tmp/$USER/ebtutorial/1.0.1/GCC-10.2.0 +$ ls /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12 easybuild_obj -$ ls /tmp/$USER/ebtutorial/1.0.1/GCC-10.2.0/easybuild_obj +$ ls /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12/easybuild_obj $ ``` @@ -672,7 +725,8 @@ If now we try installing the easyconfig file again, EasyBuild complains that it can't find the specified source file anywhere: ``` -Couldn't find file eb-tutorial-1.0.1.tar.gz anywhere, and downloading it didn't work either... +Couldn't find file eb-tutorial-1.0.1.tar.gz anywhere, and downloading it didn't work either... +Paths attempted (in order):... ``` To let EasyBuild automatically download the source file if it is not available yet, @@ -716,11 +770,13 @@ expanded when the configure command is run (see [Exercise 7.1](#exercises)). Hopefully that brings us closer to getting the installation to work... ``` -$ eb example.eb +$ eb eb-tutorial.eb .... == sanity checking... -== FAILED: Installation ended unsuccessfully (build directory: /tmp/easybuild/ebtutorial/1.0.1/GCC-10.2.0): build failed (first 300 chars): -Sanity check failed: no (non-empty) directory found at 'lib' or 'lib64' in /home/easybuild/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 (took 2 sec) +== ... (took 2 secs) +== FAILED: Installation ended unsuccessfully (build directory: /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12): +build failed (first 300 chars): Sanity check failed: no (non-empty) directory found at 'lib' or 'lib64' in +/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 (took 7 secs) ``` It got all the way to the sanity check step, that's great! @@ -729,9 +785,9 @@ The sanity check failed because no '`lib`' or `'lib64'` directory was found. Indeed: ``` -$ ls $HOME/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0 +$ ls /users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 bin -$ ls $HOME/easybuild/software/eb-tutorial/1.0.1-GCC-10.2.0/bin +$ ls /users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 eb-tutorial ``` @@ -757,10 +813,11 @@ a module, we know it will work as expected. By enabling trace mode via `--trace` we can get some more information too: ```shell -$ eb example.eb --module-only --trace +$ eb eb-tutorial.eb --module-only --trace ... == sanity checking... >> file 'bin/eb-tutorial' found: OK + >> loading modules: eb-tutorial/1.0.1-cpeCray-21.12... >> running command 'eb-tutorial' ... >> result for command 'eb-tutorial': OK ... @@ -773,7 +830,6 @@ To convince yourself that the installation works as intended, try to load the `e run the `eb-tutorial` command yourself: ``` -$ module use $HOME/easybuild/modules/all $ module load eb-tutorial $ eb-tutorial Hello from the EasyBuild tutorial! @@ -820,12 +876,73 @@ You can let EasyBuild determine *and* inject these SHA256 checksums automatically via `eb --inject-checksums`: ``` -$ eb example.eb --inject-checksums +$ eb eb-tutorial.eb --inject-checksums ... == injecting sha256 checksums for sources & patches in example.eb... == * eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 ``` +Finally, we should consider changing the name of the easyconfig that we just developed +to align with the EasyBuild conventions as otherwise it would not be found when used as +a dependency of another package. In this case, the name should be +`eb-tutorial-1.0.0-cpeCray-21.12.eb`. In fact, EasyBuild stored a processed version +of our easyconfig with that name in the repository: + +``` +$ ls $EASYBUILD_REPOSITORYPATH +eb-tutorial +$ ls $EASYBUILD_REPOSITORYPATH/eb-tutorial +eb-tutorial-1.0.1-cpeCray-21.12.eb +$ cat $EASYBUILD_REPOSITORYPATH/eb-tutorial/eb-tutorial-1.0.1-cpeCray-21.12.eb +... +``` + +Note that EasyBuild has added an additional parameter to the easyconfig file, `buildstats`, +with a lot of information about how easybuild was called and properties of the node on which +it was run. This file is still a valid easyconfig file though from which we can build the +program again. + +Let's also inspect the installation directory when the whole build process has finished successfully. +This is very easy after loading the module, as EasyBuild-generated modules define a number of environment +variables for each module: + +``` +$ module load eb-tutorial +$ env | grep TUTORIAL +EBVERSIONEBMINTUTORIAL=1.0.1 +EBDEVELEBMINTUTORIAL=/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12/easybuild/eb-tutorial-1.0.1-cpeCray-21.12-easybuild-devel +EBROOTEBMINTUTORIAL=/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 +``` +The most interesting one of those variables is the `EBROOT` variable which points to the installation directory. +As variable names cannot contain minus signs, the minus in the module name is replaced with `MIN` in the name +of the variable (which is not the most consistent thing however as in the name of an easyblock it is replaced +with `minus`). + +Let's have a look in that directory: + +``` +$ ls $EBROOTEBMINTUTORIAL +bin easybuild +$ ls EBROOTEBMINTUTORIAL/easybuild +easybuild-eb-tutorial-1.0.1-20220401.184518.log +easybuild-eb-tutorial-1.0.1-20220401.184518_test_report.md +eb-tutorial-1.0.1-cpeCray-21.12-easybuild-devel +eb-tutorial-1.0.1-cpeCray-21.12.eb +reprod +$ ls EBROOTEBMINTUTORIAL/easybuild/reprod +easyblocks +eb-tutorial-1.0.1-cpeCray-21.12.eb +eb-tutorial-1.0.1-cpeCray-21.12.env +hooks +ls $EBROOTEBMINTUTORIAL/easybuild/reprod/easyblocks +cmakemake.py configuremake.py +``` + +As you can see, EasyBuild has also created the `easybuild` subdirectory (and it actually told us about that +at the end of the installation) which contains a lot of information about the build, also to make it easier +to reproduce a build process afterwards. + + --- ## Exercises @@ -854,6 +971,9 @@ a bit more personal: include the username of the account that was used to instal ``` To re-install the `example.eb` easyconfig, you will need to use `eb --rebuild` or `eb --force`. + EasyBuild will also complain about modules that are loaded already if you tested the program above. + A good way to get rid of all those modules on LUMI is to simply use `module purge`. You don't need + to reload the software stack, but you will need to load `EasyBuild-user` again. --- @@ -874,125 +994,34 @@ You can leverage the `eb-tutorial` easyconfig file we have composed in the examp ??? success "(click to show solution)" You can use the `--try-software-version` option for this: ```shell - $ eb example.eb --try-software-version 1.1.0 + $ eb eb-tutorial.eb --try-software-version 1.1.0 ... - == building and installing eb-tutorial/1.1.0-GCC-10.2.0... + == building and installing eb-tutorial/1.1.0-cpeCray-21.12... ... == COMPLETED: Installation ended successfully (took 4 sec) ``` To test: ``` - $ module use $HOME/easybuild/modules/all - $ module load eb-tutorial/1.1.0-GCC-10.2.0 + $ module load eb-tutorial/1.1.0-cpeCray-21.12 $ eb-tutorial I have a message for you: - Hello from the EasyBuild tutorial! + Hello from the EasyBuild tutorial! I was installed by XXXXXXXX. ``` (`eb-tutorial` version 1.0.1 doesn't print "`I have a message for you:`") ---- - -***Exercise E.3**** - Installing py-eb-tutorial 1.0.0* - -Try composing an easyconfig file for the `py-eb-tutorial` example software, which is a tiny Python package. -The source tarball can be downloaded from this link: [py-eb-tutorial-1.0.0.tar.gz](https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/py-eb-tutorial-1.0.0.tar.gz). - -A couple of tips: - -* There is a [generic easyblock](#easyblock) available for installing Python packages, which will come in useful here. - -* By default EasyBuild performs an `import` check when install Python packages, using a Python module name that is derived from the software name by default, which will be incorrect in this case. You can specify the correct name to use in the import check by specifying it via the `options` -easyconfig parameter in your easyconfig file: - ```python - options = {'modulename': 'example'} - ``` - (you will need to change '`example`' here, of course) - -* Leverage the software that is already pre-installed in `/easybuild` in the prepared environment. - Remember that some already installed modules may be a *bundle* of a couple of other software packages. - -Please also take this into account: - -* Unfortunately this software doesn't come with documentation. That is done to make it an example that - is representative for software that you may run into in the wild (it's *not* because - we were lazy when preparing the exercises, really!). - You can inspect the sources of this software [here](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/py-eb-tutorial-1.0.0). Definitely take a look at the `setup.py` file, it includes some clues - about the requirements to get this software installed. - -* Make sure the installation actually *works*, by checking that the `py-eb-tutorial` command runs correctly. - Maybe you will need to make sure other required software is available as well, for it to work correctly... - - -??? success "(click to show solution)" - - Here is a complete working easyconfig file for `py-eb-tutorial`: - ```python - - easyblock = 'PythonPackage' - - name = 'py-eb-tutorial' - version = '1.0.0' - versionsuffix = '-Python-%(pyver)s' - - homepage = 'https://easybuilders.github.io/easybuild-tutorial' - description = "EasyBuild tutorial Python example" - - source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] - sources = [SOURCE_TAR_GZ] - checksums = ['fcf73a9efc65527a210b993e8889d41ebf05977eef1f6a65ebac3188152cd496'] - - toolchain = {'name': 'foss', 'version': '2020b'} - - dependencies = [ - ('Python', '3.8.6'), - ('SciPy-bundle', '2020.11'), - ('eb-tutorial', '1.0.1'), - ] - - use_pip = True - - options = {'modulename': 'eb_tutorial'} - - sanity_check_paths = { - 'files': ['bin/py-eb-tutorial'], - 'dirs': ['lib/python%(pyshortver)s/site-packages'], - } - - sanity_check_commands = ["py-eb-tutorial"] - - moduleclass = 'tools' + EasyBuild has also created a new easyconfig for this configuration and stored + in the repository and the `easybuild` subdirectory from the installation + directory. As on LUMI the repository is in the search path we can actually copy + the file back to the current directory: + ``` + eb --copy-ec eb-tutorial-1.1.0-cpeCray-21.12. ``` + Some of the formatting is lost though and the checksum is still missing, so you may want + to do some cleaning up. ``` + eb eb-tutorial-1.1.0-cpeCray-21.12.eb --inject-checksum + ``` + - Some remarks: - - * We used the `PythonPackage` generic easyblock. There is also a `PythonBundle` easyblock for installing - bundles of Python packages, which is used for `SciPy-bundle` for example. But we don't need that here, - since we are only dealing with a single Python package. - - * The `versionsuffix` is not strictly needed, but it's common to tag Python packages with the Python version - for which they were installed. - * The SHA256 checksum for the source tarball was added automatically via `eb py-eb-tutorial.eb --inject-checksums`. - - * `py-eb-tutorial` only wants to be installed with `pip install`, so we had to set `use_pip = True`. - You can consult the custom easyconfig parameters supported by the `PythonPackage` easyblock via - "`eb -a -e PythonPackage`", see the `EASYBLOCK-SPECIFIC` part of the output. - Even when the default installation mechanism used by `PythonPackage` - (which consists of running `python setup.py install`) works fine, - it is recommended to instruct EasyBuild to use `pip install` instead. - - * By default EasyBuild will try to import `py_eb_tutorial`, while the actual name of the Python package - provided by `py-eb-tutorial` is just `eb_tutorial`. We fixed this by specifying the correct Python module name to - use via `options`. - - * Strictly speaking we don't need to specify a custom `sanity_check_paths`, since the default used - by Python package is already pretty decent (it will check for a non-empty `lib/python3.8/site-packages` - directory in the installation). We also want to make sure the `py-eb-tutorial` command is available in - the `bin` subdirectory however. Hardcoding to `python3.8` can be avoided using the `%(pyshortver)s` - template value. - - * A good way to check whether the `py-eb-tutorial` command works correctly is by running it as a sanity check - command. If the `eb-tutorial` command is not available the `py-eb-tutorial` command will fail, - since it basically just runs the `eb-tutorial` command. So we need to include `eb-tutorial` as a (runtime) - dependency in the `py-eb-tutorial` easyconfig file. +--- *[[next: Implementing easyblocks]](2_03_implementing_easyblocks.md)* From 1d2744197e9aea4eee18339a4d3868282523fbcf Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 1 Apr 2022 19:12:05 +0200 Subject: [PATCH 140/159] Adding in an additional section about external modules. --- docs/2022-CSC_and_LO/2_00_part2_using.md | 3 ++- docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md | 2 +- ...lementing_easyblocks.md => 2_04_implementing_easyblocks.md} | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-) rename docs/2022-CSC_and_LO/{2_03_implementing_easyblocks.md => 2_04_implementing_easyblocks.md} (99%) diff --git a/docs/2022-CSC_and_LO/2_00_part2_using.md b/docs/2022-CSC_and_LO/2_00_part2_using.md index e033499c..c50036be 100644 --- a/docs/2022-CSC_and_LO/2_00_part2_using.md +++ b/docs/2022-CSC_and_LO/2_00_part2_using.md @@ -4,4 +4,5 @@ * [Troubleshooting](2_01_troubleshooting.md) *(hands-on)* * [Creating easyconfig files](2_02_creating_easyconfig_files.md) *(hands-on)* -* [Implementing easyblocks](2_03_implementing_easyblocks.md) *(hands-on)* +* [Using external modules from the Cray PE](2_03_external_modules.md) +* [Implementing easyblocks](2_04_implementing_easyblocks.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md index 99ac7d88..f7d6a646 100644 --- a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md @@ -1024,4 +1024,4 @@ You can leverage the `eb-tutorial` easyconfig file we have composed in the examp --- -*[[next: Implementing easyblocks]](2_03_implementing_easyblocks.md)* +*[[next: Using external modules from the Cray PE]](2_03_external_modules.md)* diff --git a/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md similarity index 99% rename from docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md rename to docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md index 5355548e..e0c07510 100644 --- a/docs/2022-CSC_and_LO/2_03_implementing_easyblocks.md +++ b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md @@ -1,6 +1,6 @@ # Implementing easyblocks -*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* +*[[back: Using external modules from the Cray PE]](2_03_external_modules.md)* --- From 4fd663817bd20f9bafaa35bf8f86ca01fa45e26f Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 1 Apr 2022 19:12:42 +0200 Subject: [PATCH 141/159] Adding in a section taken from a previous tutorial and integrating some of the material in an earlier section on the Cray PE. --- docs/2022-CSC_and_LO/1_03_CPE.md | 222 ++++++++++++++---- docs/2022-CSC_and_LO/2_03_external_modules.md | 81 +++++++ 2 files changed, 259 insertions(+), 44 deletions(-) create mode 100644 docs/2022-CSC_and_LO/2_03_external_modules.md diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md index 077fbc36..d3a2ff84 100644 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_03_CPE.md @@ -15,76 +15,207 @@ tools and to configure the environment in a flexible way. ## Cray PE components -- **Cray Compiling Environment (CCE)**: +### Cray compiler environments - CCE consists of Cray compilers performing code analysis during compilation to generate highly optimized code. - Supported languages include Fortran, C and C++, and UPC (Unified Parallel C). - The Cray C/C++ compiler is basically the Clang/LLVM compiler with a back-end configured by HPE Cray that also - adds some Cray-specific options while the Cray Fortran compiler uses an HPE-Cray specific front-end - with a back-end based on LLVM. The Fortran compiler supports most of Fortran 2018 and tends to be - considerably stricter than the GNU or Intel Fortran compilers. +Cray PE supports multiple compilers, Cray and third party compilers as well: AOCC, Cray, Intel, GNU. - The CCE compilers will also support OpenMP offload to NVIDIA and AMD GPUs but that is still work in progress - at the time of the development of this tutorial. The Fortran compiler (but not the C/C++ compiler) also supports - OpenACC for offloading to GPU. +Users can access the compilers loading a programming environment module (see below). - The classic pre-LLVM Cray C/C++ compiler is not available on Cray EX systems. +Compilers in the Cray Programming Environment are almost always used through the Cray compiler wrappers. In fact, +until recently, some compilers could not be used without the wrappers as they failed to even detect their own +include files and some components. The behaviour of the wrapper will depend on the other modules that are +loaded in the environment. E.g., there is no separate wrapper for MPI as the MPI header files and libraries are +automatically activated through the regular wrappers as soon as a Cray MPI module is loaded. -- **Cray Scientific and Mathematical Libraries (CSML)**: +The commands to invoke compiler wrappers are `ftn` (Fortran), `cc` (C), `CC` (C++). They wrap automatically +to the right compilers based on the programming environment module that is loaded and the compiler module. - A set of high performance libraries providing portability for scientific applications, sparse and dense linear - algebra (BLAS, LAPACK, ScaLAPACK) and fast Fourier transforms (FFTW). +The online help can be accessed with the `-help` option. E.g.: `cc -help`, `CC -help`. +One of the most frequently used options of the compiler wrappers is `-craype-verbose`: +``` + ftn -help | grep verbose -A 1 + -craype-verbose Print the command which is forwarded + to compiler invocation +``` +More information is available with the `info` or `man` commands. +E.g.: both `info cc` or `man cc` will display the man page of the C compiler wrapper. - There is a special (more limited) version of the library with GPU-optimized routines. +The compiler wrappers call the correct compiler in the currently loaded programming environment, +with appropriate options to build and link applications with relevant libraries, as required by the modules loaded. +Besides a number of generic options, they also will pass all other options to the underlying compile, +so you can still use the regular compiler options also. - CSML also includes the HDF5 and netCDF libraries, in versions with and without MPI. +The compiler wrappers should replace direct calls to compiler drivers in Makefiles and build scripts +to ensure that the proper libraries are used. -- **Cray Message Passing Toolkit**: +!!! Note + For system libraries, only dynamic linking is supported by compiler wrappers on the Cray EX system - A collection of software libraries used to perform data transfers between nodes running in parallel applications. - It includes the Message Passing Interface (MPI) and OpenSHMEM parallel programming models. +--- - The HPE Cray MPI implementation is based on the MPICH3 sources from ANL. However, some algorithms are tweaked - specifically for HPE Cray hardware and the Cray interconnects. +#### Cray Compiling Environment (CCE) -- **Cray Environment Setup and Compiling Support**: - - Libraries supporting code compilation and setting up the development environment, including compiler drivers. +The Cray Compiling Environment is set by the module `PrgEnv-cray`, which is loaded by default at login. +The compiler itself is contained in the `cce` module. -- **Cray Performance and Analysis Tools**: +CCE consists of Cray compilers performing code analysis during compilation to generate highly optimized code. +Supported languages include Fortran, C and C++, and UPC (Unified Parallel C). +The Cray C/C++ compiler is basically the Clang/LLVM compiler with a back-end configured by HPE Cray that also +adds some Cray-specific options while the Cray Fortran compiler uses an HPE-Cray specific front-end +with a back-end based on LLVM. The Fortran compiler supports most of Fortran 2018 and tends to be +considerably stricter than the GNU or Intel Fortran compilers. - Tools to analyze the performance and behavior of programs that are run on Cray systems, and a Performance API (PAPI). +The CCE compilers will also support OpenMP offload to NVIDIA and AMD GPUs but that is still very immature and +very much work in progress at the time of the development of this tutorial. +The Fortran compiler (but not the C/C++ compiler) also supports +OpenACC for offloading to GPU and is being updated to the newest versions of this standard. -- **Cray Debugging Support Tools**: +The classic pre-LLVM Cray C/C++ compiler is not available on Cray EX systems. - Debugging tools, including `gdb4hpc` and `Valgrind4hpc`. +Compiler-specific manpages can be accessed on the system with `man crayftn`, `man craycc` or `man crayCC`. + +More details are given in the +[Cray Fortran Reference Manual](https://support.hpe.com/connect/s/search?language=en_US#q=Cray%20Fortran%20Reference%20manual&t=All&sort=relevancy&numberOfResults=25) + and the + [Cray Compiling Environment Release](https://support.hpe.com/connect/s/search?language=en_US#q=Cray%20Compiling%20Environment%20Release&t=All&sort=relevancy&numberOfResults=25) +that unfortunately are hidden deep in the new [support pages of HPE](https://support.hpe.com/). +The [Clang Compiler User’s Manual](https://clang.llvm.org/docs/UsersManual.html) +is another source of information for the Cray C and C++ Clang compilers. + +For more information about compiler pragmas and directives, see `man intro_directives` on the system. + +--- + +### Third-Party Compilers + +#### GNU + +The GNU C/C++ and Fortran compilers are probably the best supported third party +compiler in the Cray PE. + +Compiler-specific manpages can be accessed on the system with `man gfortran`, `man gcc` or `man g++`. + +More details are provided by the [GCC online documentation](https://gcc.gnu.org/onlinedocs). + + +#### AOCC + +There AOCC compilers, the AMD Optimizing C/C++ Compiler and matching fortran compilers, +AMD's compiler offering for CPU-only systems, have a matching programming environment module +and are a full citizen of the Cray PE. + +Compiler-specific documentation is available in the [AOCC User Guide](https://developer.amd.com/amd-aocc/#userguide). + +Cray provides a bundled package of support libraries to install into the PE environment to enable AOCC, +and Cray PE utilities such as debuggers and performance tools work with AOCC. + + +#### AMD ROCm compilers + +The AMD ROCm compilers (in the AMD world sometimes known as AOMP) are supported on systems +with AMD CPUs. However, at the time of writing of this tutorial they are not yet available in the +LUMI environment and the integration with the Cray environment still seems to be work in progress. + + +#### Intel (not on LUMI) + +The Cray PE also provides a programming environment module to enable +the Intel® oneAPI compiler and tools. + +The documentation is available in the [Intel® oneAPI Programming Guide](https://software.intel.com/content/www/us/en/develop/documentation/oneapi-programming-guide/top/oneapi-development-environment-setup.html) + +Cray provides a bundled package of support libraries to install into the Cray PE to enable the Intel compiler, allowing +utilities such as debuggers and performance tools to work with it. + + +#### NVIDIA HPC toolkit (not on LUMI) -- **Third-party compilers**: +The NVIDIA HPC Toolkit compilers (formerly PGI) are supported on systems with NVIDIA GPUs. - The Cray PE also supports several third-party compilers. Not all compilers are supported on all systems: - - The GNU C/C++ and Fortran compilers will be present on most systems with the Cray PE. +### Cray Scientific and Math Library - - The AMD AOCC C/C++ and Fortran compilers are supported on systems with AMD CPUs. +* Manpages: `intro_libsci`, `intro_fftw3` - - The AMD ROCm compilers (sometimes known as AOMP) are supported on systems with AMD GPU technology. +The Cray Scientific and Math Libraries (CSML, also known as LibSci) are a collection of numerical routines +optimized for best performance on Cray systems. - - The Intel C/C++ and Fortran compilers (also the oneAPI versions) are supported on systems with Intel - processors. HPE Cray does not include them in licenses for AMD CPUs. +These libraries satisfy dependencies for many commonly used applications on Cray systems for a wide variety of domains. - - The NVIDIA HPC Toolkit compilers are supported on systems with NVIDIA GPUs. +When the module for a CSML package (such as `cray-libsci` or `cray-fftw`) is loaded, +all relevant headers and libraries for these packages are added to the compile +and link lines of the `cc`, `ftn`, and `CC` compiler wrappers, so linking with them is +completely transparant (to the extent that users wonder where the libraries are). + +The CSML collection contains the following Scientific Libraries: + +* BLAS (Basic Linear Algebra Subroutines) +* BLACS (Basic Linear Algebra Communication Subprograms) +* CBLAS (Collection of wrappers providing a C interface to the Fortran BLAS library) +* IRT (Iterative Refinement Toolkit) +* LAPACK (Linear Algebra Routines) +* LAPACKE (C interfaces to LAPACK Routines) +* ScaLAPACK (Scalable LAPACK) +* `libsci_acc` (library of Cray-optimized BLAS, LAPACK, and ScaLAPACK routines) +* HDF5 +* NetCDF (Network Common Data Format) +* FFTW3 (the Fastest Fourier Transforms in the West, release 3) + + +### Cray Message Passing Toolkit + +* Manpages: `intro_mpi` + +* Website: [http://www.mpi-forum.org](http://www.mpi-forum.org) + +MPI is a widely used parallel programming model that establishes a practical, portable, efficient, +and flexible standard for passing messages between ranks in parallel processes. + +Cray MPI is derived from Argonne National Laboratory MPICH and implements the MPI-3.1 standard +as documented by the MPI Forum in MPI: A Message Passing Interface Standard, Version 3.1. + +Support for MPI varies depending on system hardware. To see which functions and environment variables the +system supports, please have a look at the corresponding man pages with `man intro_mpi` on the system. + +Note that though on LUMI at the time of the tutorial both the OpenFabric Interface (OFI) based and UCX-based +versions of the library are supported, the finalised LUMI system will only support OFI. + + +### DSMML + +* Modules: `cray-dsmml` +* +* Manpages: `intro_dsmml` +* +* Website: [https://pe-cray.github.io/cray-dsmml](https://pe-cray.github.io/cray-dsmml) + +Distributed Symmetric Memory Management Library (DSMML) is a HPE Cray proprietary memory management library. + +DSMML is a standalone memory management library for maintaining distributed shared symmetric memory +heaps for top-level PGAS languages and libraries like Coarray Fortran, UPC, and OpenSHMEM. + +DSMML allows user libraries to create multiple symmetric heaps and share information with other libraries. + +Through DSMML, interoperability can be extracted between PGAS programming models. + +Further details are available in the man page on the system with `man intro_dsmml`. + + +### Other components + +- **Cray Performance and Analysis Tools**: + + Tools to analyze the performance and behavior of programs that are run on Cray systems, and a Performance API (PAPI). + +- **Cray Debugging Support Tools**: + + Debugging tools, including `gdb4hpc` and `Valgrind4hpc`. --- ## Configuring the Cray PE through modules -Compilers in the Cray Programming Environment are almost always used through the Cray compiler wrappers. In fact, -until recently, some compilers could not be used without the wrappers as they failed to even detect their own -include files and some components. The behaviour of the wrapper will depend on the other modules that are -loaded in the environment. E.g., there is no separate wrapper for MPI as the MPI header files and libraries are -automatically activated through the regular wrappers as soon as a Cray MPI module is loaded. - Multiple releases of the HPE Cray PE can be installed simultaneously on the system and users can mix-and-match components. However, it is also possible to load only components for a specific release of the PE. Cray PE versions have version numbers of the form ``yy.mm``, e.g., ``22.02`` for the version released in @@ -95,7 +226,7 @@ Below we only discuss those modules that are important when building software wi profilers, etc., are not included in the list. -### The PrgEnv-* modules +### The programming environment modules The usual way to initialise the Cray PE is by loading one of the ``PrgEnv-*`` modules. This module will then load the appropriate compiler, compiler wrappers and other libraries. Some of the components that will be @@ -117,7 +248,10 @@ All ``PrgEnv-*`` modules belong to the same module family. Hence Lmod will autom already loaded ``PrgEnv-*`` module when you load a different one. -### Selecting the version through the cpe module +### Selecting the version through the cpe meta-module + +The Cray PE on the EX system provides the meta-module `cpe`: the purpose of the meta-module is +similar to the scope of the `cdt` and `cdt-cuda` meta-modules available on the XC systems. Loading one of the ``cpe/yy.mm`` modules (e.g., ``cpe/22.02``) has the following effects: diff --git a/docs/2022-CSC_and_LO/2_03_external_modules.md b/docs/2022-CSC_and_LO/2_03_external_modules.md new file mode 100644 index 00000000..5d6632c0 --- /dev/null +++ b/docs/2022-CSC_and_LO/2_03_external_modules.md @@ -0,0 +1,81 @@ +# Using external modules from the Cray PE + +*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* + +--- + +## What are external modules? + +EasyBuild supports the use of modules that were not installed via EasyBuild. +We refer to such modules as [external modules](https://docs.easybuild.io/en/latest/Using_external_modules.html). +These modules do not define the `EBROOT*` and `EBVERSION*` environment variables that EasyBuild would define +and uses internally in several easyblocks and some easyconfig files. + +This feature is used extensively on Cray systems to interface with the Cray PE (which comes with its own +modules and cannot be installed via EasyBuild): +[external modules can be used as dependencies](https://docs.easybuild.io/en/latest/Using_external_modules.html#using-external-modules-as-dependencies), +by including the module name in the dependencies list, +along with the `EXTERNAL_MODULE` constant marker. + +For example, to specify the module `cray-fftw` as a dependency, you should write the following in your easyconfig file: +``` python +dependencies = [('cray-fftw', EXTERNAL_MODULE)] +``` + +For such dependencies, EasyBuild will: + +* load the module before initiating the software build and install procedure + +* include a `module load` statement in the generated module file (for runtime dependencies) + +!!! Note + The default version of the external module will be loaded unless a specific version is given as dependency, + and here that version needs to be given as part of the name of the module and not as the second element in the + tuple. + + ```python + dependencies = [('cray-fftw/3.3.8.12', EXTERNAL_MODULE)] + ``` + +If the specified module is not available, EasyBuild will exit with an error message stating that the dependency +can not be resolved because the module could not be found, without searching for a matching easyconfig file +from which it could generate the module. + + +--- + +## EasyBuild Metadata for external modules + +[Metadata](https://docs.easybuild.io/en/latest/Using_external_modules.html#metadata-for-external-modules) + can be supplied to EasyBuild for external modules: using the `--external-modules-metadata` +configuration option, the location of one or more metadata files can be specified. + +The files are expected to be in INI format, with a section per module name +and key-value assignments specific to that module. + +The external modules metadata file can be also defined with the corresponding environment variable: +``` +echo $EASYBUILD_EXTERNAL_MODULES_METADATA +/apps/common/UES/jenkins/production/easybuild/cpe_external_modules_metadata-21.04.cfg +``` + +The following keys are +[supported by EasyBuild](https://docs.easybuild.io/en/latest/Using_external_modules.html#supported-metadata-values): + +* name: software name(s) provided by the module +* version: software version(s) provided by the module +* prefix: installation prefix of the software provided by the module + +For instance, the external module version loaded by the dependency `cray-fftw` can be specified as follows: +```ini +[cray-fftw] +name = FFTW +prefix = FFTW_DIR/.. +version = 3.3.8.10 +``` + +The environment variable `$EBROOTFFTW` will also be defined according to the `prefix` specified in the metadata file. + +--- + +*[[next: Implementing easyblocks]](2_04_implementing_easyblocks.md)* From 0b02bd1bb62233bfce99a046e2fff85dda31537b Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 5 Apr 2022 15:54:42 +0200 Subject: [PATCH 142/159] Implementing EasyBlocks adapted for LUMI. --- .../2_04_implementing_easyblocks.md | 113 ++++++++++++++++-- 1 file changed, 101 insertions(+), 12 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md index e0c07510..a91b0dc8 100644 --- a/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md +++ b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md @@ -27,7 +27,10 @@ by most GNU software packages. *Software-specific* easyblocks implement the build and installation procedure for a particular software package. Typically this involves highly customised steps, for example specifying dedicated configuration options, creating or adjusting specific files, executing non-standard shell commands, etc. Usually a custom implementation of the -sanity check is also included. +sanity check is also included. Much of the work done in software-specific easyblocks can often also be done +in generic easyblocks using parameters such as ``confdigopts`` etc., but a software-specific easyblock can +hide some of that complexity from the user. Other software-specific easyblocks implement very specific +installation procedures that do not fit in one of the generic ones. Using a generic easyblock requires specifying the ``easyblock`` parameter in the easyconfig file. If it is not specified, EasyBuild will try and find the software-specific easyblock derived from the software name. @@ -38,7 +41,8 @@ for an easyblock (see below). ## Naming -Easyblocks need to follow a strict naming scheme, to ensure that EasyBuild can pick them up as needed. This involves two aspects: +Easyblocks need to follow a strict naming scheme, to ensure that EasyBuild can pick them up automatically as needed. +This involves two aspects: * the name of the Python class; * the name and location of the Python module file. @@ -52,7 +56,7 @@ Because of limitations in Python on characters allowed in names of Python classe only alphanumeric characters and underscores (``_``) are allowed. Any other characters are replaced following an encoding scheme: * spaces are replaced by underscores (``_``); -* dashes ``-`` are replaced by ``_minus_``; +* dashes ``-`` are replaced by ``_minus_`` (note the inconsistency with the naming of ``EBROOT`` and ``EBVERSION`` variables); * underscores are replaced by ``_underscore_``; The ``encode_class_name`` function provided in ``easybuild.tools.filetools`` returns the expected class name @@ -117,6 +121,11 @@ To let EasyBuild pick up one or more new or customized easyblocks, you can use t configuration option. As long as both the filename of the Python module and the name of the Python class are correct, EasyBuild will use these easyblocks when needed. +On LUMI, the EasyBuild configuration modules take care of setting this parameter (using the corresponding environment +variable), pointing to custom easyblocks in the LUMI software stack itself and a repo (with a fixed name) that users +can create themselves. At this moment it does not yet include possible other easyblock repositories in other repositories. + + ## Structure of an easyblock The example below shows the overal structure of an easyblock: @@ -232,7 +241,7 @@ self.cfg['some_list'].append('example') ### Custom parameters -Additional custom easyconfig parameters can be defined in an easyblock to allowing steering its behaviour. +Additional custom easyconfig parameters can be defined in an easyblock to steer its behaviour. This is done via the ``extra_options`` *static* method. Custom parameters can be specified to be mandatory. The example below shows how this can be implemented: @@ -256,7 +265,9 @@ class EB_Example(ConfigureMake): ``` The first element in the list of a defined custom parameter corresponds to the default value for that parameter -(both ``None`` in the example above). The second element provides some informative help text, and the last element +(both ``None`` in the example above). The second element provides some informative help text +(which can then be displayed with ``eb -a -e ``, eg, ``eb -a -e EB_GCC``), +and the last element indicates whether the parameter is mandatory (``MANDATORY``) or just an optional custom parameter (``CUSTOM``). ## Easyblock constructor @@ -519,6 +530,29 @@ class EB_Example(EasyBlock): super(EB_Example, self).sanity_check_step(custom_commands=[self.command]) ``` + +## Easyblocks in the Cray ecosystem + +The generic easyblocks are usually rather independent of compilers etc. and tend to work well with all toolchains. +However, software-specific easyblocks may contain code that is specific for certain toolchains and are often only +tested with the common toolchains (foss and intel and their subtoolchains). Many of those easyblocks will fail +on Cray systems (or any system that uses other toolchains) as they don't recognise the compiler and rather than +implementing some generic behaviour that may or may not work, produce an error message instead that the compiler +toolchain is not supported. + +Several packages on LUMI therefore use generic easyblocks rather than the software-specific easyblocks that may +exist for those applications. Adapting those software-specific easyblocks for LUMI poses an interesting maintenance +problem. Either one could decide to not contribute back to the community, but this implies then that all modifications +made to the corresponding easyblocks in the EasyBuild distribution should be monitored and implemented in the custom +easyblocks for Cray also. On the other hand, contributing back to the community also poses two problems. First it +would also require to implement the Cray toolchains as used on LUMI in the core of EasyBuild (which already contains +a different set of toolchains targeted more at how the Cray PE works with the regular environment modules), and that +only makes sense if these toolchains are first extended to not only cover the programming environments supported on +LUMI but also the Intel and NVIDIA programming environments. Second, the EasyBuild community has no easy way of testing +any modification made to such an easyblock on a Cray PE system. Hence every update made in the community may break +the Cray PE support again. + + ## Exercise ### Exercise I.1 @@ -570,9 +604,9 @@ Your easyblock should: cmd = ' '.join([ self.cfg['preconfigopts'], - "cmake", - "-DCMAKE_INSTALL_PREFIX='%s'" % self.installdir, - "-DEBTUTORIAL_MSG='%s'" % self.cfg['message'], + 'cmake', + '-DCMAKE_INSTALL_PREFIX=\'%s\'' % self.installdir, + '-DEBTUTORIAL_MSG="%s"' % self.cfg['message'], self.cfg['configopts'], ]) run_cmd(cmd) @@ -582,7 +616,7 @@ Your easyblock should: cmd = ' '.join([ self.cfg['prebuildopts'], - "make", + 'make', self.cfg['buildopts'], ]) run_cmd(cmd) @@ -592,7 +626,7 @@ Your easyblock should: cmd = ' '.join([ self.cfg['preinstallopts'], - "make install", + 'make install', self.cfg['installopts'], ]) run_cmd(cmd) @@ -607,6 +641,58 @@ Your easyblock should: custom_commands=custom_commands) ``` + We also need to adapt our easyconfig file for ``eb-tutorial``: + + - The ``easyblock`` line is no longer needed as we will rely on the automatic selection of the + software-specific easyblock. + - We don't need to define the message through ``configopts`` but via the easyblock-specific + configuration parameter ``message``. In fact, we were so careful when implementing the ``configure_step`` + that even variable expansion will still work so we can still include ``$USER`` in the message. + - The sanity check is also no longer needed as it is done by the software-specific easyblock. + + So the easyconfig file simplifies to: + + ```python + name = 'eb-tutorial' + version = "1.1.0" + + homepage = 'https://easybuilders.github.io/easybuild-tutorial' + + whatis = [ 'Description: EasyBuild tutorial example'] + + description = """ + This is a short C++ example program that can be buid using CMake. + """ + + toolchain = {'name': 'cpeCray', 'version': '21.12'} + + builddependencies = [ + ('buildtools', '%(toolchain_version)s', '', True) + ] + + source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] + sources = [SOURCE_TAR_GZ] + checksums = ['def18b69b11a3ec34ef2a81752603b2118cf1a57e350aee41de9ea13c2e6a7ef'] + + message = 'Hello from the EasyBuild tutorial! I was installed by $USER.' + + moduleclass = 'tools' + + ``` + + Running this example on LUMI is a little tricky as using ``--include-easyblocks`` to point EasyBuild to + our new easyblock interfers with settings already made by the EasyBuild configuration modules (``EasyBuild-user``) + and causes error messages about the toolchains. So either the easyblock needs to be copied to the user location + that can be found by looking at the output of ``eb --show-config`` or we simply need to extend the list of + easyblocks that EasyBuild searches with the easyblocks in the current directory: + + ``` shell + EASYBUILD_INCLUDE_EASYBLOCKS="$EASYBUILD_INCLUDE_EASYBLOCKS,./*.py" + ``` + + + + ### Exercise I.2 Try implementing another new custom easyblock for ``eb-tutorial``, @@ -648,7 +734,7 @@ Your easyblock should only: def configure_step(self): """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" - self.cfg.update('configopts', "-DEBTUTORIAL_MSG='%s'" % self.cfg['message']) + self.cfg.update('configopts', '-DEBTUTORIAL_MSG="%s"'% self.cfg['message']) super(EB_eb_minus_tutorial, self).configure_step() @@ -662,4 +748,7 @@ Your easyblock should only: custom_commands=custom_commands) ``` -*[[next: PArt 3: OVerview]](3_00_part3_advanced.md)* + This is a much simpler easyblock as we already use all the logic that has been written for us to build + with CMake. + +*[[next: Part 3: Overview]](3_00_part3_advanced.md)* From cca08127ef090058cd6707b74238aa65e8f171ee Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 5 Apr 2022 15:54:55 +0200 Subject: [PATCH 143/159] Correction of typos. --- docs/2022-CSC_and_LO/2_03_external_modules.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_03_external_modules.md b/docs/2022-CSC_and_LO/2_03_external_modules.md index 5d6632c0..b19313ff 100644 --- a/docs/2022-CSC_and_LO/2_03_external_modules.md +++ b/docs/2022-CSC_and_LO/2_03_external_modules.md @@ -8,10 +8,10 @@ EasyBuild supports the use of modules that were not installed via EasyBuild. We refer to such modules as [external modules](https://docs.easybuild.io/en/latest/Using_external_modules.html). -These modules do not define the `EBROOT*` and `EBVERSION*` environment variables that EasyBuild would define +External modules do not define the `EBROOT*` and `EBVERSION*` environment variables that EasyBuild would define and uses internally in several easyblocks and some easyconfig files. -This feature is used extensively on Cray systems to interface with the Cray PE (which comes with its own +External modules are used extensively on Cray systems to interface with the Cray PE (which comes with its own modules and cannot be installed via EasyBuild): [external modules can be used as dependencies](https://docs.easybuild.io/en/latest/Using_external_modules.html#using-external-modules-as-dependencies), by including the module name in the dependencies list, @@ -47,7 +47,7 @@ from which it could generate the module. ## EasyBuild Metadata for external modules [Metadata](https://docs.easybuild.io/en/latest/Using_external_modules.html#metadata-for-external-modules) - can be supplied to EasyBuild for external modules: using the `--external-modules-metadata` +can be supplied to EasyBuild for external modules: using the `--external-modules-metadata` configuration option, the location of one or more metadata files can be specified. The files are expected to be in INI format, with a section per module name @@ -76,6 +76,10 @@ version = 3.3.8.10 The environment variable `$EBROOTFFTW` will also be defined according to the `prefix` specified in the metadata file. +On LUMI, users in generally don't need to be too concerned about the metadata file as the EasyBuild-user (and other hidden +EasyBuild configuration modules) take care of pointing to the right metadata file, which is specific for each version of the +Cray PE and hence each version of the LUMI software stack. + --- *[[next: Implementing easyblocks]](2_04_implementing_easyblocks.md)* From ed39db44465aa8d9a112b0db75b98c0f8528c2a4 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 5 Apr 2022 15:55:13 +0200 Subject: [PATCH 144/159] Using EasyBuild as a library corrected for LUMI. --- .../2022-CSC_and_LO/3_01_easybuild_library.md | 26 ++++++++++++------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_01_easybuild_library.md b/docs/2022-CSC_and_LO/3_01_easybuild_library.md index c037f49f..f0c255eb 100644 --- a/docs/2022-CSC_and_LO/3_01_easybuild_library.md +++ b/docs/2022-CSC_and_LO/3_01_easybuild_library.md @@ -254,10 +254,12 @@ def parse_easyconfig(ec_fn): # parse easyconfig file; # the 'parse_easyconfigs' function expects a list of tuples, - # where the second item indicates whether or not the easyconfig file was automatically generated or not + # where the second item indicates whether or not the easyconfig file was + # automatically generated or not ec_dicts, _ = parse_easyconfigs([(ec_path, False)]) - # only retain first parsed easyconfig, ignore any others (which are unlikely anyway) + # only retain first parsed easyconfig, ignore any others (which are unlikely + # anyway) return ec_path, ec_dicts[0]['ec'] @@ -278,15 +280,21 @@ for key in keys: print("%s: %s" % (key, ec[key])) ``` -Example usage: +Example usage (ensure that ``EasyBuild-user`` is loaded as EasyBuild should be +configured properly!): -```shell -$ ./inspect_easyconfig.py Subread-2.0.0-GCC-8.3.0.eb name version sources sanity_check_paths -name: Subread -version: 2.0.0 -sources: ['subread-2.0.0-source.tar.gz'] -sanity_check_paths: {'files': ['bin/exactSNP', 'bin/featureCounts', 'bin/subindel', 'bin/subjunc', 'bin/sublong', 'bin/subread-align', 'bin/subread-buildindex'], 'dirs': ['bin/utilities']} ``` +$ ./inspect_easyconfig.py GMP-6.2.1-cpeCray-21.12.eb name version sources sanity_check_paths +Inspecting /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/GMP/GMP-6.2.1-cpeCray-21.12.eb ... +name: GMP +version: 6.2.1 +sources: ['gmp-6.2.1.tar.bz2'] +sanity_check_paths: {'files': ['include/gmp.h', 'include/gmpxx.h', 'lib/libgmp.a', 'lib/libgmp.la', 'lib/libgmp.so', 'lib/libgmpxx.a', 'lib/libgmpxx.la', 'lib/libgmpxx.so', 'lib/pkgconfig/gmp.pc', 'lib/pkgconfig/gmpxx.pc'], 'dirs': ['share/info']} +``` + +This feature can be used, e.g., to extract information from easyconfig files to generate documentation +for the library of easyconfig files. + --- From 1a1c143070af0cebc9c8e07417729c33d4db97d9 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 5 Apr 2022 15:55:33 +0200 Subject: [PATCH 145/159] Section about hooks extended with references to additional examples. --- docs/2022-CSC_and_LO/3_02_hooks.md | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_02_hooks.md b/docs/2022-CSC_and_LO/3_02_hooks.md index a2663ff0..f91f7d7c 100644 --- a/docs/2022-CSC_and_LO/3_02_hooks.md +++ b/docs/2022-CSC_and_LO/3_02_hooks.md @@ -19,9 +19,32 @@ would normally execute. This is a very powerful feature, but it is important to aware of some details in order to make good use of it. -Support for hooks is also covered in detail in the EasyBuild documentation, see -[here](https://docs.easybuild.io/en/latest/Hooks.html). - +Support for hooks is also +[covered in detail in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Hooks.html). + +## Why hooks? + +Hooks can be implemented for various reasons. Some examples used by some sites in the EasyBuild +community: + +- Enforce some site policies on easyconfig files. A practical example for LUMI could be imposing + the use of the `whatis` parameter and ensuring that it contains a `Description` line to have a + more consistent format for module files. +- Ensuring that some information is always in the module file. E.g., on LUMI the `site_contacts` + parameter is added automatically when installing in the central stack if it is not already in + the easyconfig file to refer to the LUST support pages. +- Modify the behaviour of standard easyconfig files that come with EasyBuild to adapt them to the system + while allowing users to simply use the standard easyconfig files. E.g., we could consider this option + to offer the common EasyBuild FOSS toolchain and its subtoolchains to users on LUMI-C, while using hooks to + adapt, e.g., the easyconfig files for Open MPI to work on LUMI. + +Some site have really large hook files to implement policies and modify standard EasyBuild build recipes, e.g., +- [The hooks file from Jülich Supercomputing Centre](https://github.com/easybuilders/JSC/blob/2022/Custom_Hooks/eb_hooks.py) +- ComputeCanada [cc_hooks_common.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_common.py) + and [cc_hooks_gento.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_gentoo.py) +- [The hooks file from the EESSI software stack](https://github.com/EESSI/software-layer/blob/main/eb_hooks.py). + [EESSI](https://www.eessi-hpc.org/) is an effort to build a software stack distributed via + [CernVM-FS](https://cernvm.cern.ch/fs/) using EasyBuild to build all software. ## Implementing and using hooks From 5da4db5528f2099f0c3520b440d53e80a945a1b2 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 13:57:15 +0200 Subject: [PATCH 146/159] Slurm job submission from EasyBuild a bit reworked. --- docs/2022-CSC_and_LO/3_03_slurm_jobs.md | 109 +++++++++++++++--------- 1 file changed, 69 insertions(+), 40 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md index 0ffa12a9..567f8abd 100644 --- a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md +++ b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md @@ -6,12 +6,30 @@ EasyBuild can submit jobs to different backends including Slurm to install software, to *distribute* the often time-consuming installation of a set of software applications and -the dependencies they require to a cluster. +the dependencies they require to a cluster. Each individual package is installed in a separate +job and job dependencies are used to manage the dependencies between package so that no build +is started before the dependencies are in place. This is done via the ``--job`` command line option. It is important to be aware of some details before you start using this, which we'll cover here. +!!! Warning "This section is not supported on LUMI, use at your own risk" + + EasyBuild on LUMI is currently not fully configured to support job submission via Slurm. Several + changes would be needed to the configuration of EasyBuild, including the location of the + temporary files and build directory. Those have to be made by hand. + + Due to the setup of the central software stack, this feature is currently useless to install + the central stack. For user installations, there are also limitations as the enviornment + on the compute nodes is different from the login nodes so, e.g., different locations for + temporary files are being used. These would only be refreshed if the EasyBuild configuration + modules are reloaded on the compute nodes which cannot be done currently in the way Slurm + job submission is set up in EasyBuild. + + Use material in this section with care; it has not been completely tested. + + ## Configuration The EasyBuild configuration that is active at the time that ``eb --job`` is used @@ -25,6 +43,8 @@ that are specified via an [EasyBuild configuration file](configuration.md#config This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables that are in place in the job environment are most likely *irrelevant*, since configuration settings they specify they will most likely be overruled by the corresponding command line options. +It does also imply however that the EasyBuild configuration that is in place when ``eb --job`` is used +does also work on the compute nodes to which the job is submitted. ## Using ``eb --job`` @@ -39,6 +59,9 @@ to ``Slurm``, for example by setting the corresponding environment variable: export EASYBUILD_JOB_BACKEND='Slurm' ``` +On LUMI this is taken care of in the EasyBuild configuration modules such as ``EasyBuild-user``. + + ### Job resources To submit an installation as a job, simply use ``eb --job``: @@ -73,13 +96,13 @@ For example, to specify a particular account that should be used for the jobs su (equivalent with using the ``-A`` or ``--account`` command line option for ``sbatch``): ```shell -export SBATCH_ACCOUNT='example_project' +export SBATCH_ACCOUNT='project_XXXXXXXXX' ``` Or to submit to a particular Slurm partition (equivalent with the ``-p`` or ``--partition`` option for ``sbatch``): ```shell -export SBATCH_PARTITION='example_partition' +export SBATCH_PARTITION='small' ``` For more information about supported ``$SBATCH_*`` environment variables, @@ -113,24 +136,29 @@ as jobs, to avoid that they fail almost instantly due to a lack of disk space. Keep in mind that the active EasyBuild configuration is passed down into the submitted jobs, so any configuration that is present on the workernodes may not have any effect. -For example, if you commonly use `/tmp/$USER` for build directories on a login node, -you may need to tweak that when submitting jobs to use a different location: +For example, on LUMI it is possible to use ``$XDG_RUNTIME_DIR`` on the login nodes which has +the advantage that any leftovers of failed builds will be cleaned up when the user ends their last +login session on that node, but it is not possible to do so on the compute nodes. ```shell # EasByuild is configured to use /tmp/$USER on the login node -login01 $ eb --show-config | grep buildpath -buildpath (E) = /tmp/example +uan01 $ eb --show-config | grep buildpath +buildpath (E) = /run/user/XXXXXXXX/easybuild/build -# use /localdisk/$USER for build directories when submitting installations as jobs -login01 $ eb --job --buildpath /localdisk/$USER example.eb --robot +# use /dev/shm/$USER for build directories when submitting installations as jobs +login01 $ eb --job --buildpath /dev/shm/$USER/easybuild example.eb --robot ``` + ### Temporary log files and build directories -The temporary log file that EasyBuild creates is most likely going to end up on the local disk -of the workernode on which the job was started (by default in `$TMPDIR` or `/tmp`). -If an installation fails, the job will finish and temporary files will likely be cleaned up instantly, -which may leave you wondering about the actual cause of the failing installation... +The problems for the temporary log files are twofold. First, they may end up in a place +that is not available on the compute nodes. E.g., for the same reasons as for the build +path, the LUMI EasyBuild configuration will place the temporary files in a subdirectory of +``$XDG_RUNTIME_DIR`` on the loginnodes but a subdirectory of ``/dev/shm/$USER`` on the +compute nodes. The second problem however is that if an installation fails, those log files are +not even accessible anymore which may leave you wondering about the actual cause of the failing +installation... To remedy this, there are a couple of EasyBuild configuration options you can use: @@ -139,6 +167,7 @@ To remedy this, there are a couple of EasyBuild configuration options you can us ```shell $ eb --job example.eb --tmp-logdir $HOME/eb_tmplogs ``` + This will move at least the log file to a suitable place. * If you prefer having the entire log file stored in the Slurm job output files, you can use ``--logtostdout`` when submitting the jobs. This will result in extensive logging @@ -146,11 +175,13 @@ To remedy this, there are a couple of EasyBuild configuration options you can us log to ``stdout`` when the installation is running in the job, and hence the log messages will be captured in the job output files. -The same remark applies to build directories: they should be on a local filesystem (to avoid problems -that often occur when building software on a parallel filesystem like GPFS or Lustre), -which will probably be cleaned up automatically when a job fails. Here it is less easy to provide -general advice on how to deal with this, but one thing you can consider is retrying the installation -in an interactive job, so you can inspect the build directory after the installation fails. +The build directory of course also suffers from the problem of being no longer accessible if the +installation fails, but there it is not so easy to find a solution. Building on a shared file system +is not only much slower, but in particular on parallel file systems like GPFS/SpectrumScale, Lustre +or BeeGFS buiding sometimes fails in strange ways. One thing you can consider if you cannot do the +build on a login node (e.g., because the code is not suitable for cross-compiling or the configure +system does tests that would fail on the login node), is to rety the installation in an +interactive job, so you can inspect the build directory after the installation fails. ### Lock files @@ -171,37 +202,37 @@ subdirectory of ``installpath``) manually, or re-submit the job with ``eb --job As an example, we will let EasyBuild submit jobs to install ``AUGUSTUS`` with the ``foss/2020b`` toolchain. +!!! Warning "This example does not work on LUMI" + + Note that this is an example using the FOSS common toolchain. For this reason it does not work on + LUMI. + ### Configuration Before using ``--job``, let's make sure that EasyBuild is properly configured: ```shell -# use $HOME/easybuild for software, modules, sources, etc. -export EASYBUILD_PREFIX=$HOME/easybuild +# Load the EasyBuild-user module (central installations will not work at all +# using job submission) +module load LUMI/21.12 +module load partition/C +module load EasyBuild-user # use ramdisk for build directories -export EASYBUILD_BUILDPATH=/dev/shm/$USER +export EASYBUILD_BUILDPATH=/dev/shm/$USER/build +export EASYBUILD_TMPDIR=/dev/shm/$USER/tmp # use Slurm as job backend export EASYBUILD_JOB_BACKEND=Slurm ``` -In addition, add the path to the centrally installed software to ``$MODULEPATH`` via ``module use``: -```shell -module use /easybuild/modules/all -``` - -Load the EasyBuild module: - -```shell -module load EasyBuild -``` - -Let's assume that we also need to inform Slurm that jobs should be submitted into a particular account: +We will also need to inform Slurm that jobs should be submitted into a particular account, and +in a particular partition: ```shell -export SBATCH_ACCOUNT=example_project +export SBATCH_ACCOUNT=project_XXXXXXXXX +export SBATCH_PARTITION='small' ``` This will be picked up by the ``sbatch`` commands that EasyBuild will run to submit the software installation jobs. @@ -234,14 +265,14 @@ $ eb AUGUSTUS-3.4.0-foss-2020b.eb --missing Several dependencies are not installed yet, so we will need to use ``--robot`` to ensure that EasyBuild also submits jobs to install these first. -To speed up the installations a bit, we will request 10 cores for each submitted job (via ``--job-cores``). +To speed up the installations a bit, we will request 8 cores for each submitted job (via ``--job-cores``). That should be sufficient to let each installation finish in (well) under 1 hour, so we only request 1 hour of walltime per job (via ``--job-max-walltime``). In order to have some meaningful job output files, we also enable trace mode (via ``--trace``). ``` -$ eb AUGUSTUS-3.4.0-foss-2020b.eb --job --job-cores 10 --job-max-walltime 1 --robot --trace +$ eb AUGUSTUS-3.4.0-foss-2020b.eb --job --job-cores 8 --job-max-walltime 1 --robot --trace ... == resolving dependencies ... ... @@ -278,7 +309,7 @@ these jobs will be able to start. After about 20 minutes, AUGUSTUS and all missing dependencies should be installed: ``` -$ ls -lrt $HOME/easybuild/modules/all/*/*.lua | tail -11 +$ ls -lrt $HOME/EasyBuild/modules/.../*.lua | tail -11 -rw-rw----. 1 example example 1634 Mar 29 10:13 /users/example/easybuild/modules/all/HTSlib/1.11-GCC-10.2.0.lua -rw-rw----. 1 example example 1792 Mar 29 10:13 /users/example/easybuild/modules/all/SAMtools/1.11-GCC-10.2.0.lua -rw-rw----. 1 example example 1147 Mar 29 10:13 /users/example/easybuild/modules/all/BamTools/2.5.1-GCC-10.2.0.lua @@ -291,11 +322,9 @@ $ ls -lrt $HOME/easybuild/modules/all/*/*.lua | tail -11 -rw-rw----. 1 example example 1365 Mar 29 10:28 /users/example/easybuild/modules/all/SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0.lua -rw-rw----. 1 example example 2233 Mar 29 10:30 /users/example/easybuild/modules/all/AUGUSTUS/3.4.0-foss-2020b.lua -$ module use $HOME/easybuild/modules/all - $ module avail AUGUSTUS --------- /users/hkenneth/easybuild/modules/all -------- +-- EasyBuild managed user software for software stack ... -- AUGUSTUS/3.4.0-foss-2020b ``` From ad1142fb8285b59c7b4725b3f971099f966095db Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 15:42:35 +0200 Subject: [PATCH 147/159] Some explanation in the overview of part 3 --- docs/2022-CSC_and_LO/1_02_Lmod.md | 82 +++++++- docs/2022-CSC_and_LO/3_00_part3_advanced.md | 6 + .../3_04_module_naming_schemes.md | 192 +++++++----------- 3 files changed, 152 insertions(+), 128 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md index 0caa8a47..d6d73a6c 100644 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ b/docs/2022-CSC_and_LO/1_02_Lmod.md @@ -73,12 +73,54 @@ to a hierarchy with 3 levels: 3. The ``MPI`` level, containing libraries and applications that depend on the compiler used and the MPI implementation. +??? Example "A simple Lmod hierarchy with a single compiler" + + Here is a simple example of such a 3-level module hierarchy + (that almost could have been generated by EasyBuild): + +
+ + In this example the ``Core`` level only includes a single module `GCC/9.3.0`, + while the ``Compiler`` level includes two modules: `OpenMPI/4.0.3` and `MPICH/3.3.2`. + In the ``MPI `` level, three modules are available: one for `FFTW`, one for `ScaLAPACK`, + and one for `HDF5`. + + Initially only the modules on the top level of a module hierarchy are available for loading. + If you run "`module avail`", the command that is used to view all modules that are available + for loading, with this example module hierarchy, you will only see the `GCC/9.3.0` module. + + Some modules in the top level of the hierarchy act as a "gateway" to modules in the + next level below. + To make additional modules available for loading one of these gateway modules has to be loaded. + In our example, loading the `GCC/9.3.0` module results in two additional modules coming into + view from the ``Compiler`` level, as indicated by the arrows: the modules for `OpenMPI` and `MPICH`. + These correspond to installations of `OpenMPI` + and `MPICH` that were built using `GCC/9.3.0`. + + Similarly, the `OpenMPI/4.0.3` module serves as a gateway to the three modules in the ``MPI`` level. + Only by loading the `OpenMPI` module will these additional three modules become + available for loading. They correspond to software installations built using the ``GCC/9.3.0`` + compiler with ``OpenMPI/4.0.3``. + Now assume that we have two compilers in the hierarchy, Compiler_A and Compiler_B. Their modules would reside -at the ``Core`` level. Both copilers provide the same MPI implementation, MPI_C. So there would be two modules +at the ``Core`` level. Both compilers provide the same MPI implementation, MPI_C. So there would be two modules for ``MPI_C`` in two different subdirectories at the ``Compiler`` level. And further assume that we have an application, Appl_E, compiled with both Compiler_A and Compiler_B and using MPI_C. For that application there would also be two module files at the ``MPI`` level, one in a subdirectory corresponding ao Compiler_A and MPI_C and one -in a subdirectory corresponding to Compiler_B and MPI_C. To be able to load the module for Appl_E, a user should +in a subdirectory corresponding to Compiler_B and MPI_C. + +```mermaid +graph TD; +A[Compiler_A] --> AC[MPI_C]; +A --> AD[MPI_D] +B[Compiler_B] --> BC[MPI_C]; +AC --> ACE[Appl_E]; +AD --> ADE[Appl_E] +BC --> BCE[Appl_E]; +``` + + +To be able to load the module for Appl_E, a user should first load Compiler_A, then load MPI_C and only then is it possible to load the module for Appl_E: ```bash @@ -102,6 +144,23 @@ and depending on those adapt the path to the binaries, several very simple modul logic, and one could add an Appl_E module for a different compiler or MPI implementation without touching any of the already existing module files for that application. +Similarly, if after + +```bash +module load Compiler_A MPI_C Appl_E +``` + +one does + +```bash +module load MPI_D +``` + +thent MPI_C gets unloaded, Lmod notices that it also has to unload/deactivate Appl_E, then will load MPI_D for Compiler_A and +finally will notice that there is an equivalent Appl_E module available again, and Lmod will load that one also. However, +now loading Compiler_B will cause a warning that MPI_D and Appl_E have been deactivated as there is no module name MPI_D in +any version for Compiler_B. + ### Building blocks @@ -126,8 +185,9 @@ Some mechanisms in Lmod make implementing a hierarchy fairly easy (though there ### Implementation details -The above example could be implemented using 6 module files: One for each compiler, two for the MPI module and -two for the application module. +The above example could be implemented using 8 module files: One for each compiler, three for the MPI modules +(two for MPI_C and one for MPI_D) and +three for the application modules. ``` moduleroot @@ -139,8 +199,10 @@ moduleroot ├── Compiler │ ├── Compiler_A │ │ └── version_A -│ │ └── MPI_C -│ │ └── version_C.lua +│ │ ├── MPI_C +│ │ │ └── version_C.lua +│ │ └── MPI_D +│ │ └── version_D.lua │ └── Compiler_B │ └── version_B │ └── MPI_C @@ -148,8 +210,12 @@ moduleroot └── MPI ├── Compiler_A │ └── version_A - │ └── MPI_C - │ └── version_C + │ ├── MPI_C + │ │ └── version_C + │ │ └── Appl_E + │ │ └── version_E.lua + │ └── MPI_D + │ └── version_D │ └── Appl_E │ └── version_E.lua └── Compiler_B diff --git a/docs/2022-CSC_and_LO/3_00_part3_advanced.md b/docs/2022-CSC_and_LO/3_00_part3_advanced.md index 708c57f8..28a24990 100644 --- a/docs/2022-CSC_and_LO/3_00_part3_advanced.md +++ b/docs/2022-CSC_and_LO/3_00_part3_advanced.md @@ -2,6 +2,12 @@ *[[back to start page]](index.md)* +In this section we mostly cover "good to know that they exist" features as they are not used +on LUMI, or not really accessible to regular user installations that are performed with the +LUMI ``EasyBuild-user`` module. Hooks are used on LUMI, but it is not really advised to +overwrite the centrally defined hooks with a local file. And the whole structure of the +EasyBuild integration is also set up to make use of the GitHub integration in the future. + * [Using EasyBuild as a library](3_01_easybuild_library.md) *(hands-on)* * [Using hooks to customise EasyBuild](3_02_hooks.md) *(hands-on)* * [Submitting installations as Slurm jobs](3_03_slurm_jobs.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md index 14229d97..d0b4ebe8 100644 --- a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md +++ b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md @@ -4,21 +4,24 @@ --- -Up until now we have used EasyBuild's default module naming scheme (`EasyBuildMNS`), -which produces module files with names that closely resemble to the names of the +Up until now we have used the naming scheme in use on LUMI which is for many practical purposes +is indistinguishable from the default EasyBuild naming scheme (`EasyBuildMNS`). +It produces module files with names that closely resemble to the names of the corresponding easyconfig files. -For example, when installing `h5py-3.1.0-foss-2020b.eb` the generated module was named `h5py/3.1.0-foss-2020b`. +For example, when installing `zlib-1.2.11-cpeGNU-21.12.eb` the generated module was named +`zlib/1.2.11-cpeGNU-21.12`. EasyBuild supports several different module naming schemes: ```shell $ eb --avail-module-naming-schemes List of supported module naming schemes: - CategorizedHMNS - CategorizedModuleNamingScheme - EasyBuildMNS - HierarchicalMNS - MigrateFromEBToHMNS + EasyBuildMNS + LUMI_FlatMNS + MigrateFromEBToHMNS + HierarchicalMNS + CategorizedHMNS + CategorizedModuleNamingScheme ``` In this part of the tutorial we will take a closer look at `HierarchicalMNS`, @@ -28,52 +31,54 @@ We will also take a quick look at implementing our own custom module naming sche ## Flat vs hierarchical -The default module naming scheme `EasyBuildMNS` is an example of regular *"flat"* module naming scheme, which is characterized by: +!!! Note -* all module files are directly available for loading; -* each module name uniquely identifies a particular installation; + This text is very much based on the generic EasyBuild tutorials as we cannot demonstrate + all aspects on LUMI as it is configured today. -In contrast, a *hierarchical* module naming scheme -consists of a *hierarchy* of module files. - -The typical module hierarchy has 3 levels: + On LUMI, the complete module scheme is partly hierarchical, but EasyBuild uses a flat naming + scheme. The two levels in the hierarchy that are present to deal with different versions of + the software stack and the various types of compute nodes, are not managed by EasyBuild. -* a **core** level, where module files for software that was installed using the - [`system` toolchain](../introduction/#system-toolchain) are kept; -* a **compiler** level, where module files for software that was installed using a -*compiler-only toolchain* are stored; -* and an **MPI** level, which houses module files for software that was installed using a toolchain that includes (at least) a compiler and MPI component; +The default module naming scheme `EasyBuildMNS` +and the custom `LUMI_FlatMNS` naming scheme are both examples of regular *"flat"* +module naming schemes, which are characterized by: -Here is a simple example of such a 3-level module hierarchy: +- all module files are directly available for loading; +- each module name uniquely identifies a particular installation; -
+In contrast, a *hierarchical* module naming scheme +consists of a *hierarchy* of module files. +A fairly typical 3-level schme (``Core``, ``Compiler`` and ``MPI``) has been +discussed in the [section on Lmod](../1_02_Lmod#lmod-hierarchy). +This typical Lmod hierarcny would map very well on the EasyBuild common toolchains. -In this example the core level only includes a single module `GCC/9.3.0`, -while the compiler level includes two modules: `OpenMPI/4.0.3` and `MPICH/3.3.2`. -In the MPI level, three modules are available: one for `FFTW`, one for `ScaLAPACK`, and one for `HDF5`. +In fact, for the example -Initially only the modules on the top level of a module hierarchy are available for loading. -If you run "`module avail`" with the example module hierarchy, you will only see the `GCC/9.3.0` module. +
-Some modules in the top level of the hierarchy act as a "gateway" to modules in the -next level below. -To make additional modules available for loading one of these gateway modules has to be loaded. In our exampe, loading the `GCC/9.3.0` module results in two additional modules coming into view from the compiler level, as indicated by the arrows: the modules for `OpenMPI` and `MPICH`. These correspond to installations of `OpenMPI` -and `MPICH` that were built using `GCC/9.3.0` as a (compiler-only) toolchain. +software at the ``Core`` level would be installed with the ``GCCcore`` and ``SYSTEM`` +toolchains. Software at the ``Compiler`` level would be installed with the ``GCC`` +toolchain, while software at the ``MPI`` level after loading the ``OpenMPI`` module +would be installed using the regular ``foss`` toolchain or the ``gompi`` toolchain +(see the diagram in the +["Common toolchains" section of the page on terminology](../1_05_terminology/#common-toolchains)). -Similarly, the `OpenMPI/4.0.3` module serves as a gateway to the three modules in the MPI level. Only by loading the `OpenMPI` module will these additional three modules become -available for loading. They correspond to software installations built using the `gompi/2020a` toolchain that -consists of the `GCC/9.3.0` compiler module and the `OpenMPI/4.0.3` MPI module. Software installed using -`foss/2020a` (which is a full toolchain that also includes OpenBLAS, FFTW and ScaLAPACK) would also be stored -in this level of the module hierarchy. +On LUMI, where software is installed through the Cray Programming Environment with no real choice of +MPI implementation, a two-level arrangement would still make a lot of sense, with at the ``Core`` level +all software compiled with the SYSTEM toolchain while there could be a ``PrgEnv`` level for software +compiled with a particula programming enviornment aka cepGNU/cpeCray/cpeAOCC toolchain. Such a scheme +is used on the Cray systems at CSCS. -The characteristics of a module hierarchy are: +To recap, the characteristics of a module hierarchy are: * not all module files are directly available for loading; * some modules serve as a gateway to more modules; * to access some software installations you will first need to load one or more gateway modules in order to use them; -You can probably think of other ways to organize module files in a hierarchical module tree, but here we will stick to the standard core / compiler / MPI hierarchy. +You can probably think of other ways to organize module files in a hierarchical module tree, but here +we will stick to the standard core / compiler / MPI hierarchy. ### Pros & cons @@ -81,22 +86,24 @@ So why go through all this trouble of organizing modules hierarchically? There are a couple of advantages to this approach: -* shorter module names; +* shorter module names (or at least for the version part of the name); * less overwhelming list of available modules; * only compatible modules can be loaded together; However, the are some minor disadvantages too: -* not all existing modules are directly visible; +* not all existing modules are directly visible, so users have to learn how to find + modules using ``module spider`` etc.; * gateway modules may have little meaning to end users; #### Length of module names -When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For our `HDF5` installation for example, -we have `HDF5/1.10.6-gompi-2020a` as module name. The `-gompi-2020a` part of the name refers to the toolchain that was +When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For a `HDF5` installation +with the EasyBuild common toolchains for example, +one might have `HDF5/1.10.6-gompi-2020a` as module name. The `-gompi-2020a` part of the name refers to the toolchain that was used for this installation, but it may be confusing to some people (what kind of Pokémon is a "gompi"?!). -In the example module hierarchy shown above, the module for `HDF5` is named `HDF5/1.10.6` which is basically the bare +In the example module hierarchy shown above, the module for `HDF5` could simply be named `HDF5/1.10.6` which is basically the bare essentials: software name and version. That's way better, nice and clean! #### Amount of available modules @@ -130,8 +137,10 @@ and so these modules can not be loaded together easily. One downside of a module hierarchy is that not all existing modules are directly available for loading or are even *visible* to the user, since the output of "`module avail`" only shows a subset of all modules. -[Lmod](https://lmod.readthedocs.io) has a solution for this though: it provides a separate "`module spider`" -command to search for module files throughout the entire module hierarchy. So as long as the end users are +As we have discussed in [the Lmod section](../LMOD). +[Lmod](https://lmod.readthedocs.io) provides a powerful solution to search for modules through the +``module spider`` and to some extent also the ``module keyword`` commands. +So as long as the end users are aware of this additional command, it should not be difficult to discover which software installations exist and how they can be accessed. The "`module spider`" command will inform the user which of the gateway modules need to be loaded in order to load a specific module file. @@ -140,7 +149,7 @@ need to be loaded in order to load a specific module file. An additional potential problem of a module hierarchy is that the semantics of the gateway modules may not be clear to end users. They may wonder why they need to pick a specific compiler and MPI library, or which of the -available options is the best one. Maybe there are not even be aware what exactly a "compiler" is, or how it is +available options is the best one. Maybe they are not even aware what exactly a "compiler" is, or how it is relevant to the software they need in their bioinformatics pipeline... This can be partially resolved by loading a default compiler and MPI module so a particular set of modules @@ -148,10 +157,15 @@ is available right after login, which could be the ones used in the most recent recommended versions. More experienced users could then leverage the "`module spider`" command to navigate the module hierarchy. +On LUMI this would mean loading a default software stack, but due to the the default modules are currently +loaded on LUMI this was not possible to accomplish without losing other functionality of the module tree. + ## Using a custom module naming scheme Next to the module naming schemes that are included with EasyBuild, -you can also define your own module naming scheme (MNS), and configure EasyBuild to use it. +you can also define your own module naming scheme (MNS), and configure EasyBuild to use it +(which is eactly what has been done on LUMI to remove a feature of the default ``EasyBuildMNS`` scheme +that we do not use). ### Implementation @@ -254,7 +268,9 @@ class ExampleMNS(ModuleNamingScheme): return short_modname.startswith(name.lower().replace('-', '_') + '/') ``` -We can see what the module names with this module naming scheme would like like via ``eb -D``: +We can see what the module names with this module naming scheme would like like via ``eb -D``. +E.g., for the common toolchains (the example will not work on LUMI as the default easyconfig +files are not in the robot- and the search path): ``` $ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D @@ -270,9 +286,10 @@ $ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D ## Example module hierarchy: HDF5 -!!! Warning - **This exercise will only work if you have write access to the software installation prefix**, - because EasyBuild will try to copy the installation log file to each installlation directory. +!!! Warning "Example not suitable for LUMI" + **This exercise is meant for a system where the common toolchains can be used and requires an + indpendent EasyBuild installation in your personal file space**, + because EasyBuild will try to copy the installation log file to each installation directory. Now that we know more about hierarchical module naming schemes, let us see how EasyBuild can help us with generating a hierarchical module tree. @@ -308,7 +325,7 @@ a module. We have just made all modules unavailable, so we would have to first install EasyBuild again in our hierarchical module tree before we can continue. **We strongly recommend using an EasyBuild installation that was [installed via "`pip install`" -or "`pip3 install`"](../installation/#method-1-using-pip-recommended) in this part of the tutorial.** +or "`pip3 install`"](../1_06_installation#method-1-using-pip) in this part of the tutorial.** An easy way to do this is in the prepared environment is to run: @@ -376,7 +393,9 @@ The steps we will have to go through are: * Tell EasyBuild we want to "install" the `HDF5-1.10.7-gompi-2020b.eb` easyconfig file; * Enable dependency resolution via `--robot`; -* Instruct EasyBuild to only generate the module files, not to install the software (since it is +* Assuming the software would have been installed already with the default naming scheme + in a different module directory, instruct EasyBuild to only generate the module files, + not to install the software (since it is there already in `/easybuild/software`), via the `--module-only` option. These steps translate to this single `eb` command: @@ -544,73 +563,6 @@ Currently Loaded Modules: 4) GCC/10.2.0 8) libpciaccess/0.16 12) libfabric/1.11.0 16) HDF5/1.10.7 ``` -## Exercise - -Now it is your turn! - -Try to get a feeling for how a hierarchical module tree works by: - -* installing the missing modules for the `SciPy-bundle-2020.11-foss-2020b.eb` in the module hierarchy we - generated for HDF5; -* figure out where the `SciPy-bundle` module is located in the hierarchy, and then also load it; - -You can verify your work by running this command (since `pandas` is one of the Python packages included -in the `SciPy-bundle` installation): - -```shell -python -c 'import pandas; print(pandas.__version__)' -``` - -Start from a clean slate, by first running: - -```shell -module purge -module unuse $MODULEPATH -``` - -??? success "(click to show solution)" - - * Step 0: check which modules are still missing, using `--missing` or `-M`: - ```shell - eb SciPy-bundle-2020.11-foss-2020b.eb -M - ``` - The output should tell you that 15 out of 50 required modules are still missing. - - * Install the missing modules in the module hierarchy we have generated in `$HOME/hmns/modules`: - ```shell - eb SciPy-bundle-2020.11-foss-2020b.eb --robot --module-only - ``` - Don't forget to use both `--robot` (to enable dependency resolution) and `--module-only` - (to only run the sanity check and generate module files, not install the software again). - - * Start at the top of the module hierarchy (the `Core` level), - and run module spider to check which gateway modules to load to make `SciPy-bundle` available: - ```shell - module use $HOME/hmns/modules/all/Core - module spider SciPy-bundle/2020.11 - ``` - * Load the gateway modules: - ```shell - module load GCC/10.2.0 OpenMPI/4.0.5 - ``` - * Check that the `SciPy-bundle` module is available, and load it: - ```shell - $ module avail SciPy-bundle - ----- /home/example/hmns/modules/all/MPI/GCC/10.2.0/OpenMPI/4.0.5 ------ - SciPy-bundle/2020.11 - ``` - ``` - module load SciPy-bundle/2020.11 - ``` - * Run the test command: - ``` - $ python -c 'import pandas; print(pandas.__version__)' - 1.1.4 - ``` - -!!! Warning - **This exercise will only work if you have write access to the software installation prefix**, - because EasyBuild will try to copy the installation log file to each installlation directory. --- From 7f742b912782e6bbfc6da7704ed11e70b6e1b475 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 16:22:58 +0200 Subject: [PATCH 148/159] GitHub integration section, mostly old text and not all suited for LUMI. --- .../3_05_github_integration.md | 28 +++++++++++++++---- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/docs/2022-CSC_and_LO/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_05_github_integration.md index 59dcbb67..6d04ce7f 100644 --- a/docs/2022-CSC_and_LO/3_05_github_integration.md +++ b/docs/2022-CSC_and_LO/3_05_github_integration.md @@ -4,6 +4,20 @@ --- +!!! Warning "Preliminary not for LUMI and many other Cray systems" + + Not everything in this section is as useful for EasyBuild installations that build + on the Cray Programming Environment. E.g., the default easyconfig repository does + not contain easyconfigs for the Cray PE. + + Moreover, due to the restricted operating system on the compute nodes of big + Cray systems such as the EX series (COS instead of the full SUSE linux), + the GitHub integration does not work on the compute nodes of a typical setup as + one of the required Python packages fails to work as it needs the DBus daemon. + + Since GitHub integration is not yet really used on LUMI, this section is not fully + tested but left in the tutorial for completeness. + To contribute changes to the EasyBuild code (framework or easyblocks) or easyconfigs, you will need to be a bit familiar with Git and GitHub. Or maybe not? @@ -94,11 +108,6 @@ and configuration. **Additional dependencies** Both the `GitPython` and `keyring` Python packages as well as the `keyrings.cryptfile` add-on package must be installed. -In the prepared environment, you can do this via: - -```shell -pip3 install --user GitPython keyring keyrings.cryptfile -``` !!! Note You may experiences problems installing the ``cryptography`` Python packages, @@ -294,6 +303,15 @@ will include information about your system (OS, processor, etc.) and your EasyBu ## Demo +!!! Warning "Not currently suited for LUMI" + + This is text from a previous (non-Cray) version of the tutorial. The demo does require + a number of files not installed on LUMI after going through this tutorial, + and someone who can master the repository used to clean up again. + However, having a look at the output (which was generated in the spring of 2021 on + the CSC system puhti) still offers some information, so the demo was left in this + version of the tutorial. + That is a lot to digest, so let us make this a bit more concrete with an example: we will open a pull request for the [`eb-tutorial` example software](../adding_support_software/#example) to *a fork* of the [`easybuild-easyconfigs` repository](https://github.com/easybuilders/easybuild-easyconfigs) using the `eb` command, and submit a test report in it. From 10280f8733edd83188ded658269c294251ba7f95 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 16:34:33 +0200 Subject: [PATCH 149/159] Adapt the structure and include an additional reading section. --- docs/2022-CSC_and_LO/3_05_github_integration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/2022-CSC_and_LO/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_05_github_integration.md index 6d04ce7f..6e19cb9a 100644 --- a/docs/2022-CSC_and_LO/3_05_github_integration.md +++ b/docs/2022-CSC_and_LO/3_05_github_integration.md @@ -457,4 +457,4 @@ The gist linked from this comment provides more detailed information: --- -*[[next: Overview]](index.md)* +*[[next: Additional reading]](4_00_additional_reading.md)* From bf1dd5e5d6513c188aadd261193180ac5a0cd637 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 17:08:35 +0200 Subject: [PATCH 150/159] Additional reading section. --- docs/2022-CSC_and_LO/4_00_additional_reading.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/2022-CSC_and_LO/4_00_additional_reading.md b/docs/2022-CSC_and_LO/4_00_additional_reading.md index 14f8454c..6e6066fe 100644 --- a/docs/2022-CSC_and_LO/4_00_additional_reading.md +++ b/docs/2022-CSC_and_LO/4_00_additional_reading.md @@ -11,7 +11,11 @@ EasyBuild. - [EasyBuild YouTube channel](https://www.youtube.com/c/EasyBuilders) - This tutorial is an evolution of the +<<<<<<< HEAD [EasyBuild tutorial prepared for the LUMI User Support Team, spring '21](https://easybuilders.github.io/easybuild-tutorial/2021-lust/) +======= + [EasyBuild tutorial prepared for the LUMI User Suppport Team, spring '21](https://easybuilders.github.io/easybuild-tutorial/2021-lust/) +>>>>>>> eafaa943 (Additional reading section.) given by Kenneth Hoste (UGent, EasyBuild lead developer) and Luca Marsella (CSCS) - [Recordings are available on YouTube](https://www.youtube.com/watch?v=JTRw8hqi6x0&list=PLhnGtSmEGEQh573bk3BeOj_KCRBBiA5OT) - The EasyBuild setup on LUMI is partly insprired on the setup used at CSCS on their Cray systems From 50a3aa1daf11f0e9ffde20663d84e8c25d360853 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 6 Apr 2022 17:43:28 +0200 Subject: [PATCH 151/159] Correcting a number of links. --- docs/2022-CSC_and_LO/1_06_installation.md | 4 ++-- docs/2022-CSC_and_LO/1_07_configuration.md | 6 +++--- docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md | 6 +++--- docs/2022-CSC_and_LO/2_03_external_modules.md | 2 +- docs/2022-CSC_and_LO/3_03_slurm_jobs.md | 6 +++--- 5 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_06_installation.md b/docs/2022-CSC_and_LO/1_06_installation.md index 6e68b500..24c71d5e 100644 --- a/docs/2022-CSC_and_LO/1_06_installation.md +++ b/docs/2022-CSC_and_LO/1_06_installation.md @@ -424,7 +424,7 @@ We do so only for the framework and easyblocks files as the easconfig files are install EasyBuild. Instead we create our own EasyConfig file for EasyBuild which contains some additional packages that enable extra features in EasyBuild and also provide more information to Lmod. Next the configuration module for EasyBuild (see the next section, -["Configuring EasyBuild"](1_06_configuring.md), for more information) and use the temporary +["Configuring EasyBuild"](1_07_configuration), for more information) and use the temporary installation of EasyBuild with our own EasyConfig file to do a proper installation of EasyBuild with module in the final location. @@ -468,7 +468,7 @@ eb --show-config ``` This should tell you that EasyBuild (ab)uses `$HOME/.local/easybuild` as a default location. -More on configuring EasyBuild in the [next part of the tutorial](configuration.md). +More on configuring EasyBuild in the [next part of the tutorial](configuration). #### System information diff --git a/docs/2022-CSC_and_LO/1_07_configuration.md b/docs/2022-CSC_and_LO/1_07_configuration.md index 17e60afe..a1c2f3de 100644 --- a/docs/2022-CSC_and_LO/1_07_configuration.md +++ b/docs/2022-CSC_and_LO/1_07_configuration.md @@ -280,7 +280,7 @@ The default `EasyBuildMNS` module naming scheme roughly corresponds to the filen and consists of the software name followed by a combination of the software version, toolchain and an optional label (which corresponds to the value of the `versionsuffix` easyconfig parameter): `/<-toolchain>`. Just like with [names of easyconfig -files](terminology.md#easyconfig-files), the `<-toolchain>` part is omitted when the `system` toolchain is used, +files](1_05_terminology#easyconfig-files), the `<-toolchain>` part is omitted when the `system` toolchain is used, and the `` value is empty by default. !!! Note "Module naming scheme on LUMI" @@ -366,9 +366,9 @@ across different software installations. They define *configuration settings*, such as the location where software should be installed, or the syntax that should be used when generating module files. -An [easyconfig file](terminology.md#easyconfig-files) (`*.eb`) on the other hand *specifies the details for one particular software installation*. +An [easyconfig file](1_05_terminology#easyconfig-files) (`*.eb`) on the other hand *specifies the details for one particular software installation*. It does this by defining a set of *easyconfig parameters*, which tell EasyBuild the name and version -of the software to install, which [toolchain](terminology.md#toolchains) and [easyblock](terminology.md#easyblocks) to use, etc. +of the software to install, which [toolchain](1_05_terminology#toolchains) and [easyblock](1_05_terminology#easyblocks) to use, etc. For each software installation performed by EasyBuild, there is a corresponding easyconfig file. There typically are only a handful of configuration files used however, for example a system-level diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md index f7d6a646..f9bec25a 100644 --- a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md +++ b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md @@ -1,6 +1,6 @@ # Creating easyconfig files -*[[back: Troubleshooting]](2_02_troubleshooting.md)* +*[[back: Troubleshooting]](2_02_troubleshooting)* --- @@ -58,7 +58,7 @@ Reasons to consider implementing a software-specific easyblock rather than using bootstrapping with the system compiler and then re-installing with itself.* For implementing easyblocks we refer to the -["Implementing easyblocks" section of this tutorial](2_03_implementing_easyblocks.md) +["Implementing easyblocks" section of this tutorial](2_03_implementing_easyblocks) and the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). ## Writing easyconfig files @@ -1024,4 +1024,4 @@ You can leverage the `eb-tutorial` easyconfig file we have composed in the examp --- -*[[next: Using external modules from the Cray PE]](2_03_external_modules.md)* +*[[next: Using external modules from the Cray PE]](2_03_external_modules)* diff --git a/docs/2022-CSC_and_LO/2_03_external_modules.md b/docs/2022-CSC_and_LO/2_03_external_modules.md index b19313ff..80a29712 100644 --- a/docs/2022-CSC_and_LO/2_03_external_modules.md +++ b/docs/2022-CSC_and_LO/2_03_external_modules.md @@ -1,6 +1,6 @@ # Using external modules from the Cray PE -*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* +*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files)* --- diff --git a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md index 567f8abd..283a53a0 100644 --- a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md +++ b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md @@ -37,8 +37,8 @@ will be *passed down* into the submitted job automatically, via command line opt command that is run in the job script. This includes not only command line options used next to ``--job``, but also configuration settings -that are specified via an [EasyBuild configuration file](configuration.md#configuration-files) or through -[``$EASYBUILD_*`` environment variables](configuration.md#easybuild_-environment-variables). +that are specified via an [EasyBuild configuration file](1_07_configuration#configuration-files) or through +[``$EASYBUILD_*`` environment variables](1_07_configuration#easybuild_-environment-variables). This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables that are in place in the job environment are most likely *irrelevant*, since configuration settings @@ -330,4 +330,4 @@ $ module avail AUGUSTUS --- -*[[next: Module naming schemes]](3_04_module_naming_scheme.md)* +*[[next: Module naming schemes]](3_04_module_naming_scheme)* From 5119b5e6e84a929b2c585a3f5ad9798f27a4e867 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 7 Apr 2022 17:26:56 +0200 Subject: [PATCH 152/159] Restructuring for nicer navigation bar on the left. --- docs/2022-CSC_and_LO/1_00_part1_intro.md | 12 - .../2022-CSC_and_LO/1_01_what_is_easybuild.md | 321 ----- docs/2022-CSC_and_LO/1_02_Lmod.md | 820 ----------- docs/2022-CSC_and_LO/1_03_CPE.md | 418 ------ .../1_04_LUMI_software_stack.md | 101 -- docs/2022-CSC_and_LO/1_05_terminology.md | 294 ---- docs/2022-CSC_and_LO/1_06_installation.md | 545 -------- docs/2022-CSC_and_LO/1_07_configuration.md | 610 --------- docs/2022-CSC_and_LO/1_08_basic_usage.md | 1216 ----------------- .../1_Intro/1_01_what_is_easybuild.md | 75 + docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md | 32 + .../1_Intro/1_06_installation.md | 4 + docs/2022-CSC_and_LO/2_00_part2_using.md | 8 - docs/2022-CSC_and_LO/2_01_troubleshooting.md | 545 -------- .../2_02_creating_easyconfig_files.md | 1027 -------------- docs/2022-CSC_and_LO/2_03_external_modules.md | 85 -- .../2_04_implementing_easyblocks.md | 754 ---------- docs/2022-CSC_and_LO/3_00_part3_advanced.md | 15 - .../2022-CSC_and_LO/3_01_easybuild_library.md | 301 ---- docs/2022-CSC_and_LO/3_02_hooks.md | 213 --- docs/2022-CSC_and_LO/3_03_slurm_jobs.md | 333 ----- .../3_04_module_naming_schemes.md | 569 -------- .../3_05_github_integration.md | 460 ------- docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md | 1 - .../3_Advanced/3_03_slurm_jobs.md | 4 + 25 files changed, 115 insertions(+), 8648 deletions(-) delete mode 100644 docs/2022-CSC_and_LO/1_00_part1_intro.md delete mode 100644 docs/2022-CSC_and_LO/1_01_what_is_easybuild.md delete mode 100644 docs/2022-CSC_and_LO/1_02_Lmod.md delete mode 100644 docs/2022-CSC_and_LO/1_03_CPE.md delete mode 100644 docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md delete mode 100644 docs/2022-CSC_and_LO/1_05_terminology.md delete mode 100644 docs/2022-CSC_and_LO/1_06_installation.md delete mode 100644 docs/2022-CSC_and_LO/1_07_configuration.md delete mode 100644 docs/2022-CSC_and_LO/1_08_basic_usage.md delete mode 100644 docs/2022-CSC_and_LO/2_00_part2_using.md delete mode 100644 docs/2022-CSC_and_LO/2_01_troubleshooting.md delete mode 100644 docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md delete mode 100644 docs/2022-CSC_and_LO/2_03_external_modules.md delete mode 100644 docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md delete mode 100644 docs/2022-CSC_and_LO/3_00_part3_advanced.md delete mode 100644 docs/2022-CSC_and_LO/3_01_easybuild_library.md delete mode 100644 docs/2022-CSC_and_LO/3_02_hooks.md delete mode 100644 docs/2022-CSC_and_LO/3_03_slurm_jobs.md delete mode 100644 docs/2022-CSC_and_LO/3_04_module_naming_schemes.md delete mode 100644 docs/2022-CSC_and_LO/3_05_github_integration.md diff --git a/docs/2022-CSC_and_LO/1_00_part1_intro.md b/docs/2022-CSC_and_LO/1_00_part1_intro.md deleted file mode 100644 index bdfddfc5..00000000 --- a/docs/2022-CSC_and_LO/1_00_part1_intro.md +++ /dev/null @@ -1,12 +0,0 @@ -# Part I: Introduction to EasyBuild on Cray systems - -*[[back to start page]](index.md)* - -- [What is EasyBuild?](1_01_what_is_easybuild.md) -- [The Lmod module system](1_02_Lmod.md) -- [The HPE Cray Programming Environment and Lmod](1_03_CPE.md) -- [Terminology](1_04_terminology.md) - -- [Installation](installation.md) *(hands-on)* -- [Configuration](configuration.md) *(hands-on)* -- [Basic usage](basic_usage.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md deleted file mode 100644 index 979c6e90..00000000 --- a/docs/2022-CSC_and_LO/1_01_what_is_easybuild.md +++ /dev/null @@ -1,321 +0,0 @@ -# What is EasyBuild? - -*[[back: Overview]](index.md)* - -

EasyBuild logo

- -**[EasyBuild](http://easybuild.io)** is a **software build and installation framework** -that manages (scientific) software in an efficient way, without compromising on performance. -Easybuild is especially tailored towards [**High Performance Computing (HPC)**](https://en.wikipedia.org/wiki/Supercomputer) systems, -and provides a **command-line interface** to facilitate the installation and management of a scientific software stack. - ---- - - - - - -## Elevator pitch - -EasyBuild intends to relieve HPC support teams from manually managing -software installations while at the same time **providing a consistent and well performing -scientific software stack** to end users of HPC systems. - -It serves as a **uniform interface for installing scientific software** -and saves valuable time (and frustration) by the automation of tedious, boring and repetitive tasks. - -In addition, EasyBuild can **empower scientific researchers to self-manage their software stack**, -and it can serve as a tool that can be leveraged for **building optimized container images**. - -The project has grown to become a **platform for collaboration** among HPC sites worldwide. - ---- - -On LUMI we pay a lot of attention to empowering users to create their custom software stack which -is best suited for their project without being confronted with hundreds of modules that are not -useful to them. It also enables them to easily customize the build instructions to create a -project-specific build or even to install software with custom patches to adapt to their needs. -For this we offer both [EasyBuild](https://easybuild.io/) and [Spack](https://spack.io/) -as tools but obviously this tutorial concentrates on EasyBuild alone. - -EasyBuild can also be used to [generate container recipes](https://docs.easybuild.io/en/latest/Containers.html) -that will then use EasyBuild when building -the container to install platform-optimised software in the container. However, as that functionality -is still under development on LUMI it is not yet covered in this LUMI-specific version of the tutorial. - - ---- - -## Key features - -EasyBuild is capable of **fully autonomously installing (scientific) software**, -including making sure that all necessary dependencies are installed, -and automatically generating environment module files. - -***No*** **admin privileges are required**: it is sufficient to have write permissions -to the preferred software installation prefix. - -EasyBuild is [**highly configurable**](https://docs.easybuild.io/en/latest/Configuration.html) via configuration files, -environment variables, and command line options. Various elements of the functionality can be -[**dynamically extended**](https://docs.easybuild.io/en/latest/Including_additional_Python_modules.html) via plugins, -and [**hooks**](https://docs.easybuild.io/en/latest/Hooks.html) are available for further site-specific customizations. - -The installation procedure executed by EasyBuild is thoroughly -[**logged**](https://docs.easybuild.io/en/latest/Logfiles.html), and is fully transparent via support for -[**dry runs**](https://docs.easybuild.io/en/latest/Extended_dry_run.html) and -[**tracing**](https://docs.easybuild.io/en/latest/Tracing_progress.html) the software installation procedure -as it is performed. - -EasyBuild supports using a **custom module naming scheme**, allows for -*hierarchical* module naming schemes, and **integrates with various other tools** ranging from resource managers ([Slurm](https://slurm.schedmd.com) and [GC3Pie](https://github.com/gc3pie/gc3pie)), -container tools ([Singularity](https://github.com/hpcng/singularity) and [Docker](https://www.docker.com)), -packaging tools ([FPM](https://fpm.readthedocs.io)), and so on. - ---- - -## Focus points - -EasyBuild was created specifically for **installing scientific software on HPC systems**, -which is reflected in some of the design choices that were made. - - -### Performance - -EasyBuild strongly prefers to **build software from source code**, whenever possible. - -This is important to ensure that the binaries that are installed can maximally exploit -the capabilities of the system architecture on which the software will be run. - -For that same reason, EasyBuild out-of-the-box **optimizes software for the processor architecture of the build host** -by default, via compiler options like ``-march=native`` (GCC), ``-xHost`` (Intel compilers), etc. -This behaviour [can be changed via the ``--optarch`` configuration setting](https://docs.easybuild.io/en/latest/Controlling_compiler_optimization_flags.html). -When using the HPE Cray Programming Environment, the target modules are used instead. -In principle it is possible to cross-compile with EasyBuild, but this comes with some pitfalls. -By default EasyBuild tries to run tests if the build process of a package provides that -functionality, or can run simple tests of its own, and of course these tests will fail if -the host CPU does not support the full instruction set of the target CPU. Secondly, more -and more software has broken build processes where developers thought that it was a good -idea to hard-code options such as ``-march=native`` or ``--xHost`` in a way that EasyBuild -cannot easily overwrite those. - - -### Reproducibility - -In addition to performance, **reproducibility of installations** is a core aspect of EasyBuild. - -TODO correct link in the next lines. - -Most software installations performed with EasyBuild use a **particular compiler toolchain**, -with which we aim to be in control over the build environment and avoid relying on tools and libraries -provided by the operating system. For similar reasons, we try to **provide all required dependencies through EasyBuild** as well, -with a few notable exceptions, like ``OpenSSL`` for security reasons, and Infiniband and GPU drivers which -are too closely intertwined with the operating system. - -For both toolchains and dependencies, **fixed software versions** are specified in the -easyconfig files. That way, easyconfig files can easily be shared with others: -if they worked for you it is very likely that they will work for others too, because the vast majority of the -software stack is controlled by EasyBuild. - -As we shall see when discussing the HPE Cray Programming Environment, the reproducibility is a bit more limited -in that environment due to the way the programming environment selects shared libraries at runtime. - - -### Community effort - -In a number of different ways, we try to encourage EasyBuild users to **collaborate** and help each other out. - -We actively recommend people to report problems and bugs, to submit ideas for additional features and improvements, -and to [**contribute back**](https://docs.easybuild.io/en/latest/Contributing.html) when possible, be it -by opening pull requests to the [GitHub repositories](https://github.com/easybuilders) or the [documentation](https://docs.easybuild.io). - -The EasyBuild community also provides packages for two standard toolchains, called the -[**common toolchains**](https://easybuild.readthedocs.io/en/latest/Common-toolchains.html). -The so-called ``foss`` is based on free and open software (GNU compilers, FlexiBlas with OpenBLAS backend, LaPACK, FFTW and Open MPI) -while the ``intel`` toolchain uses the full stack of Intel compilers, libraries and MPI. Currently these toolchains -cannot be supported on LUMI though as getting Open MPI to run on LUMI is challenging and as the Intel libraries and -some MPI versions also produce slow or incorrect results on AMD CPUs (and it is not clear if Intel MPI is compatible -with SlingShot 11). -Focusing the efforts on those two popular common toolchains makes it possible to support a lot of build recipes that are -collected in the [central repository](https://github.com/easybuilders/easybuild-easyconfigs). -They can still serve as a basis to develop build recipes for the HPE Cray Programming Environment. - -Last but not least, EasyBuild provides various [**GitHub integration features**](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) -that greatly facilitate the contribution process: opening, updating, and testing pull requests, -reviewing incoming contributions, and much more can all be done directly from the EasyBuild -command line. This not only saves time, effort, brain cycles, and mouse clicks for contributors, -but it also makes the review process for *maintainers* significantly easier. -All together this leads to improved stability and consistency. - ---- - -## What EasyBuild is ***not*** - -EasyBuild is ***not*** **YABT (Yet Another Build Tool)**: it does *not* replace established build -tools like CMake or ``make``, it wraps around them. -If the installation procedure of a software package involves running some unholy trinity of tools while whispering -the correct magic incantations, EasyBuild automates this process for you. - -It is ***not*** **a replacement for traditional Linux package managers** like ``yum``, ``dnf`` or ``apt``. -EasyBuild relies on certain tools and libraries provided by the operating system. This includes glibc, OpenSSL, -drivers for Infiniband and GPUs, and so on. It is required that these tools are installed and managed by other means. -This is typically done via the package management tool that comes with your Linux distribution. - -Finally, EasyBuild is ***not*** **a magic solution to all your (software installation) problems**. -You may, and probably still will occasionally, run into compiler errors unless somebody has already taken care of the problem for you. -This is even more likely to happen in the Cray environment as builds are not often tested with the Cray compilers -or even with the AMD compilers. - ---- - -## Requirements - -Since EasyBuild is primarily intended for installing (scientific) software on HPC systems, the main target is -systems running some flavor of **GNU/Linux**. EasyBuild also works on *macOS*, -but significant effort would be required to use it extensively there. - - - -EasyBuild is implemented in [**Python**](https://www.python.org), and is compatible with Python 2.7 and 3.5+. -Since [Python 2 is no longer maintained](https://blog.python.org/2020/04/python-2718-last-release-of-python-2.html), -it is recommended to use a recent version of Python 3 for running EasyBuild. - -An [**environment modules**](https://en.wikipedia.org/wiki/Environment_Modules_(software)) tool is required for running EasyBuild. -The modern Lua-based implementation **[Lmod](https://lmod.readthedocs.io)** is strongly recommended, -but other (Tcl-based) implementations are also supported. -Both Tcl and Lua are supported as syntax for environment module files. -On LUMI Lmod is used as the module tool. - ---- - -## Development - - -EasyBuild is **actively developed** by a worldwide community under the -[**GPLv2 open source license**](https://www.gnu.org/licenses/old-licenses/gpl-2.0.html) -via the [**``easybuilders``** organisation on GitHub](https://github.com/easybuilders), in the following repositories: - -- [easybuild-framework](https://github.com/easybuilders/easybuild-framework) -- [easybuild-easyblocks](https://github.com/easybuilders/easybuild-easyblocks) -- [easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs) -- [easybuild](https://github.com/easybuilders/easybuild) (website, documentation) -- [easybuild-tutorial](https://github.com/easybuilders/easybuild-tutorial) (the generic EasyBuild tutorials) - -On LUMI the generic EasyBuild installation is used but extended and adapted using LUMI-specifics -plugins and hooks maintained in the [LUMI software stack repository](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack). - -Stable versions of EasyBuild have been released roughly every 6-8 weeks since 2012, -and are published via the *Python Package Index (PyPI)*, -see [https://pypi.org/project/easybuild](https://pypi.org/project/easybuild/). - -**Comprehensive testing** practices are applied throughout the -development cycle, with extensive suites of unit and integration tests being run in a CI environment (GitHub Actions), -consistent testing of incoming contributions (pull requests), and thorough regression testing before every release. - ---- - -## A worldwide community - -EasyBuild has grown into a **community project** since it was first released publicy in 2012, -and is used and developed by dozens (hundreds?) of HPC centres and consortia worldwide. - -Since 2016 the EasyBuild community has gathered for a yearly [**EasyBuild User Meeting**](https://easybuild.io/eum), -which is an open and highly interactive event that provides a great opportunity to meet fellow EasyBuild users, -developers, and maintainers, and to discuss related topics, learn about new aspects of the tool, and how other HPC -sites leverage it, and integrate it with other tools like [Jenkins](https://www.youtube.com/watch?v=bzW45cfRUMU), -[Ansible](https://www.youtube.com/watch?v=Qr2udsAjkG0), [CernVM-FS](https://www.youtube.com/watch?v=_0j5Shuf2uE), -[ReFrame](https://www.youtube.com/watch?v=KN9QwQIrso4) etc. - -HPC sites, consortia, and companies that use EasyBuild include (but are not limited to): - -VSC logo -JSC logo -HPCNow! logo -Fred Hutch logo -Vienna BioCenter logo -CÉCI logo -Compute Canada logo -UiO logo -BEAR logo -SNIC logo -National University of Singapore logo -IT4Innovations logo -Texas A&M logo -TU Dresden logo -University of Melbourne logo -University of Luxembourg logo -EPCC logo -Cyprus Institute logo -University of Basel logo -Microsoft logo - ---- - -## Getting help - -### *Documentation* - -If you have any questions about EasyBuild, make sure to check the EasyBuild documentation first, -which is available at [**https://docs.easybuild.io**](https://docs.easybuild.io). - -It is not unlikely that an answer to your question is available there already. -The [search box](https://docs.easybuild.io/en/latest/search.html) on the top left may be helpful. - -### GitHub - - - -You can open an issue in one of the EasyBuild repositories in [the `easybuilders` GitHub organisation](https://github.com/easybuilders/) to -ask questions, report problems or share suggestions and ideas for additional -improvements. - -Please try to post the issue in the most suitable repository. If you are -not sure which repository is most suited just go ahead and open the issue anyway, -the EasyBuild maintainers can move your issue if deemed appropriate. - -Also here you can use the search box on the top left, to see if somebody else -has already asked a similar question or reported a similar problem. - -### Mailing list - -Feel free to ask any questions or provide feedback via the EasyBuild mailing list -`easybuild@lists.ugent.be`. - -Keep in mind that **you must be subscribed to the mailing list in order to -post messages there!** You can subscribe via [this link](https://lists.ugent.be/wws/subscribe/easybuild). - -In 2016-2017 over 1,000 messages were posted to the EasyBuild mailing list. -More recently this has decreased to about 700 messages per year, mostly due -to the EasyBuild Slack channel. - -### Slack - -Slack logo - -The [EasyBuild Slack](https://easybuild.slack.com/) is a quick and easy way -to get in touch with the EasyBuild community. -Over 400 people have joined it so far, and there is usually somebody awake in there -who may be able to help you out. - -**To join the EasyBuild Slack request an invitation via -[https://easybuild-slack.herokuapp.com](https://easybuild-slack.herokuapp.com/).** - -If you are not a big fan of Slack, you can also consider joining the `#easybuild` -IRC channel (available on the [Freenode IRC network](https://freenode.net/)). -A bot relays messages between this IRC channel and the `#general` channel in the EasyBuild Slack. - - -### Conference calls - -Every other week on Wednesday an EasyBuild conference call is set up to discuss recent -developments, provide an outlook to the next EasyBuild release, and -provide answers to questions that can be raised by the people attending the call. - -These conference calls are open to anyone interested in the EasyBuild project, -and have been organised since November 2013. - -More information, including the dates and times of the next calls, is available -[on the EasyBuild Wiki](https://github.com/easybuilders/easybuild/wiki/Conference-calls). - ---- - -*[[next: The Lmod module system]](1_02_Lmod.md)* diff --git a/docs/2022-CSC_and_LO/1_02_Lmod.md b/docs/2022-CSC_and_LO/1_02_Lmod.md deleted file mode 100644 index d6d73a6c..00000000 --- a/docs/2022-CSC_and_LO/1_02_Lmod.md +++ /dev/null @@ -1,820 +0,0 @@ -# The Lmod module system - -*[[back: What is EasyBuild?]](1_01_what_is_easybuild.md)* - ---- - -## Modules - -*Module* is a massively overloaded term in (scientific) software and IT in general -(kernel modules, Python modules, and so on). -In the context of EasyBuild, the term 'module' usually refers to an **environment module (file)**. - -[Environment modules](https://en.wikipedia.org/wiki/Environment_Modules_(software)) is a well established concept -on HPC systems: it is a way to specify changes that should be made to one or more -[environment variables](https://en.wikipedia.org/wiki/Environment_variable) in a -[shell](https://en.wikipedia.org/wiki/Shell_(computing))-agnostic way. A module file -is usually written in either [Tcl](https://en.wikipedia.org/wiki/Tcl) or -[Lua](https://en.wikipedia.org/wiki/Lua_(programming_language)) syntax, -and specifies which environment variables should be updated, and how (append, -prepend, (re)define, undefine, etc.) upon loading the environment module. -Unloading the environment module will restore the shell environment to its previous state. - -Environment module files are processed via a **modules tool**, of which there -are several conceptually similar yet slightly different implementations. -The oldest module tool still in use today is Environment Modules 3.2, implemented in C and -supporting module files written in Tcl. After a gap in developement, Xavier Delaruelle of CEA -developed [Environment Modules 4 and 5](https://sourceforge.net/projects/modules/) which is -fully implemented on Tcl. An alternative module tool is [Lmod](https://lmod.readthedocs.io), -developed by Robert McLay at TACC and implemented in LUA. This tool supports natively LUA -module files but also offers a high degree of compatibility with Tcl-based module files -developed for Environment Modules fia a translation layer and some API translation. - -The Cray PE offers a choice between the old-style Environment Modules 3.2 and Lmod, but no -packages or official support for Environment Modules 4 or 5. At the user level, -Environment Modules 3.2 and Lmod have many commands in common, but with different options. -Lmod also has some powerful features that are lacking in Environment Modules 3.2. - -!!! Note "The Cray PE on LUMI" - On LUMI, Lmod was selected as the module tool. One area where there are significant - differences between Environment Modules 3.2 (and also the newer versions) and Lmod is - in the commands for discovering modules on the system. If you are not familiar with Lmod - and its commands for users, it is worthwile to read the - [LUMI documentation page on Lmod](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/). - Some of those commands are also discussed on this page. - ---- - -## Lmod hierarchy - -### User view - -Lmod supports a module hierarchy. In a hierarchy, there is a distinction between the *installed -modules* and the *available modules*. Available modules are those that can be loaded directly -without first loading any other module, while the installed modules is the complete set of -modules that one could load one way or another. A typical use case -is a hierarchy to deal with different compilers on a system and different MPI implementations. -After all, it is a common practice to only link libraries and application code compiled with the -same compiler to avoid compatibility problems between compilers (and to be able to use advanced -features such as link time optimization). This is even more important for MPI, as Open MPI and -MPCIH-derived MPI implementations have incompatible Application Binary Interfaces. This would lead -to a hierarchy with 3 levels: - -1. The ``Core`` level containing the modules for the compilers themselves, e.g., one or more versions - of the GNU compiler suite and one or more versions of LLVM-based compilers. - - Loading a compiler module would then make the next level available: - -2. The ``Compiler`` level, containing modules for libraries and packages that only rely on the compilers - but do not use MPI, as well as the MPI modules, e.g., a version of Open MPI and a version of MPICH. - - Loading one of the MPI modules would then make the next level available: - -3. The ``MPI`` level, containing libraries and applications that depend on the compiler used and the MPI - implementation. - -??? Example "A simple Lmod hierarchy with a single compiler" - - Here is a simple example of such a 3-level module hierarchy - (that almost could have been generated by EasyBuild): - -
- - In this example the ``Core`` level only includes a single module `GCC/9.3.0`, - while the ``Compiler`` level includes two modules: `OpenMPI/4.0.3` and `MPICH/3.3.2`. - In the ``MPI `` level, three modules are available: one for `FFTW`, one for `ScaLAPACK`, - and one for `HDF5`. - - Initially only the modules on the top level of a module hierarchy are available for loading. - If you run "`module avail`", the command that is used to view all modules that are available - for loading, with this example module hierarchy, you will only see the `GCC/9.3.0` module. - - Some modules in the top level of the hierarchy act as a "gateway" to modules in the - next level below. - To make additional modules available for loading one of these gateway modules has to be loaded. - In our example, loading the `GCC/9.3.0` module results in two additional modules coming into - view from the ``Compiler`` level, as indicated by the arrows: the modules for `OpenMPI` and `MPICH`. - These correspond to installations of `OpenMPI` - and `MPICH` that were built using `GCC/9.3.0`. - - Similarly, the `OpenMPI/4.0.3` module serves as a gateway to the three modules in the ``MPI`` level. - Only by loading the `OpenMPI` module will these additional three modules become - available for loading. They correspond to software installations built using the ``GCC/9.3.0`` - compiler with ``OpenMPI/4.0.3``. - -Now assume that we have two compilers in the hierarchy, Compiler_A and Compiler_B. Their modules would reside -at the ``Core`` level. Both compilers provide the same MPI implementation, MPI_C. So there would be two modules -for ``MPI_C`` in two different subdirectories at the ``Compiler`` level. And further assume that we have an -application, Appl_E, compiled with both Compiler_A and Compiler_B and using MPI_C. For that application there would -also be two module files at the ``MPI`` level, one in a subdirectory corresponding ao Compiler_A and MPI_C and one -in a subdirectory corresponding to Compiler_B and MPI_C. - -```mermaid -graph TD; -A[Compiler_A] --> AC[MPI_C]; -A --> AD[MPI_D] -B[Compiler_B] --> BC[MPI_C]; -AC --> ACE[Appl_E]; -AD --> ADE[Appl_E] -BC --> BCE[Appl_E]; -``` - - -To be able to load the module for Appl_E, a user should -first load Compiler_A, then load MPI_C and only then is it possible to load the module for Appl_E: - -```bash -module load Compiler_A MPI_C Appl_E -``` - -What is interesting is what happens if the user now loads Compiler_B: - -```bash -module load Compiler_B -``` - -In a properly designed and implemented hierarchy, Lmod will unload Compiler_A which will also trigger the unloading/deactivation -of MPI_C and Appl_E. It will then load the module for Compiler_B and proceed with looking if it can find another module for -MPI_C. That will then be loaded which now makes a different module for Appl_E available, which Lmod will proceed to load. If it -cannot find an exact match for the version, Lmod will even try to locate a different version. Hence the situation after loading -Compiler_B is that now modules are loaded for Compiler_B, MPI_C for Compiler_B and Appl_E for Compiler_A with MPI_C. -All this requires very little effort from the module file programmer and very little logic in the module files. E.g., rather -then implementing a single module file for Appl_E that would require logic to see which compiler and MPI implementation is loaded -and depending on those adapt the path to the binaries, several very simple modules need to be written with very little -logic, and one could add an Appl_E module for a different compiler or MPI implementation without touching any of the already -existing module files for that application. - -Similarly, if after - -```bash -module load Compiler_A MPI_C Appl_E -``` - -one does - -```bash -module load MPI_D -``` - -thent MPI_C gets unloaded, Lmod notices that it also has to unload/deactivate Appl_E, then will load MPI_D for Compiler_A and -finally will notice that there is an equivalent Appl_E module available again, and Lmod will load that one also. However, -now loading Compiler_B will cause a warning that MPI_D and Appl_E have been deactivated as there is no module name MPI_D in -any version for Compiler_B. - - -### Building blocks - -Some mechanisms in Lmod make implementing a hierarchy fairly easy (though there are a lot of hidden pitfalls) - -- The *MODULEPATH* environment variable determines which modules are available. MODULEPATH is different from any other - path-style variable in Lmod in that any change will immediately trigger a re-evaluation of which modules are available - and trigger deactivating modules that are no longer available when a directory is removed from the MODULEPATH or - looking for alternatives for deactivated modules when a directory is added to the MODULEPATH. - -- The *"one name rule"*: Lmod cannot have two modules loaded with the same name (but a different version). By default, when loading - a module with the name of an already loaded module, Lmod will automatically swap the old one with the new one, i.e., unload the - already loaded module and load the new one. - -- The *family* concept: It is possible to declare a module to be part of a family using a command in the module file. No two modules - of the same family can be loaded at the same time, and Lmod will again by default auto-swap the already loaded one with the one - being loaded. The procedure is different though as Lmod now first has to read the new module file to discover the family, and this - may lead to more side effects. But that discussion is outside the scope of this tutorial. - - The family concept was for a long time a unique feature of Lmod, but it has been added now also to Environment Modules version 5.1. - - -### Implementation details - -The above example could be implemented using 8 module files: One for each compiler, three for the MPI modules -(two for MPI_C and one for MPI_D) and -three for the application modules. - -``` -moduleroot -├── Core -│ ├── Compiler_A -│ │ └── version_A.lua -│ └── Compiler_B -│ └── version_B.lua -├── Compiler -│ ├── Compiler_A -│ │ └── version_A -│ │ ├── MPI_C -│ │ │ └── version_C.lua -│ │ └── MPI_D -│ │ └── version_D.lua -│ └── Compiler_B -│ └── version_B -│ └── MPI_C -│ └── version_C.lua -└── MPI - ├── Compiler_A - │ └── version_A - │ ├── MPI_C - │ │ └── version_C - │ │ └── Appl_E - │ │ └── version_E.lua - │ └── MPI_D - │ └── version_D - │ └── Appl_E - │ └── version_E.lua - └── Compiler_B - └── version_B - └── MPI_C - └── version_C - └── Appl_E - └── version_E.lua -``` - -Besides the module functions needed to create the environment needed to run the compiler, the module file for -Compiler_A would need only two lines to implement the hierarchy: - -```Lua -family('Compiler') -prepend_path('MODULEPATH', 'moduleroot/Compiler/Compiler_A/version_A') -``` - -There are now two different ``version_C.lua`` files. One contains the necessary calls to module functions to -initialise the environment to use the version compiled with Compiler_A/version_A while the other contains the -necessary functions to do that for Compiler_B/version_B. Again, two more lines are needed to implement the hierarchy. -E.g., for ``moduleroot/Compiler/Compiler_A/version_A/MPI_C/version_C.lua``: - -```Lua -family('MPI') -prepend_path('MODULEPATH', 'moduleroot/MPI/Compiler_A/version_A/MPI_C/version_C') -``` - -Finally two vesions of the ``version_E.lua`` file are needed, one to prepare the environment for using the -package with Compiler_A anmd MPI_C and one for using the package with Compiler_B and MPI_C. However, these -are just regular modules and no additions are needed to work for the hierarchy. - -Both EasyBuild and Spack support Lmod hierarchies and with these tools it is also fairly automatic to create -different versions of the module files for each compiler and MPI library used to build the application. When -hand-writing modules it may be more interesting to have a generic module which would work for all those cases -and that is also possible with Lmod. Lmod does have a range of *introspection functions* that a module can use -to figure out its name, version and place in the module tree. All that would be needed is that the various -instances of the module file are at the correct location in the module tree and link to the generic file which -can be outside the module tree. In fact, this feature is used on LUMI to implement the modules that load a -particular version of the hardware for a particular section of LUMI. - ---- - -## Finding modules - -In a hierarchical setup, not all modules are available at login. This implies that a user cannot use -``module avail`` to discover which software is available on the system. To this end Lmod has powerful -search commands. It is important to understand how these commands work to ensure that the proper information -is included in the module files to improve discoverability of software. - -!!! Note "Documentation in the LUMI documentation" - Extensive information on search commands with examples of how to use them on LUMI can be found - in the [LUMI documentation]([https://)](https://docs.lumi-supercomputer.eu/), in - [the computing section, "Module environment page", "Finding modules" section](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/#finding-modules). - - -### module spider command - -The available modules at any point in time are often only a subset of all installed modules on a -system. However, Lmod provides the ``module spider`` command to search for a module with a given name -among all installed modules and to tell you how this module can be loaded (i.e., which other modules -need to be loaded to make the module available). - -The ``module spider`` command has three levels, producing different outputs: - - 1. ``module spider`` without further arguments will produce a list of all - installed software and show some basic information about those packages. - Some packages may have an ``(E)`` behind their name and will appear in blue - (in the default colour scheme) which means that they are part of a different - package. These are called *extensions* of packages or modules. - This is explained a little further in this page. - - Note that ``module spider`` will also search in packages that are hidden from - being displayed. These packages can be loaded and used. However administrators - may have decided to hide them - either because they are not useful to regular users or because they think that - they will rarely or never be directly loaded by a user and want to avoid - overloading the module display. - - 2. ``module spider `` will search for the specific package. This - can be the name of a module, but it will also search some other information - that can be included in the modules. The search is also case-insensitive. - E.g., on LUMI - ```bash - module spider GNUplot - ``` - will show something along the lines of - ``` - ------------------------------------------------------------------ - gnuplot: - ------------------------------------------------------------------ - Description: - Gnuplot is a portable command-line driven graphing utility - - Versions: - gnuplot/5.4.2-cpeCray-21.08 - gnuplot/5.4.2-cpeGNU-21.08 - ``` - so even though the capitalisation of the name was wrong, it can tell us that - there are two versions of gnuplot. The ``cpeGNU-21.08`` and ``cpeCray-21.08`` - tell that the difference is the compiler that was used to install gnuplot, - being the GNU compiler (PrgEnv-gnu) and the Cray compiler (PrgEnv-cray) - respectively. - - In some cases, if there is no ambiguity, `module spider` will actually - already produce help about the package, which is the next level. - - 3. `module spider /` will show more help information - about the package, including information on which other modules need to be - loaded to be able to load the package. E.g., - ```bash - module spider git/2.35.1 - ``` - will return something along the lines of - ``` - ------------------------------------------------------------------- - git: git/2.35.1 - ------------------------------------------------------------------- - Description: - Git is a free and open source distributed version control - system - - You will need to load all module(s) on any one of the lines below - before the "git/2.35.1" module is available to load. - - CrayEnv - LUMI/21.12 partition/C - LUMI/21.12 partition/D - LUMI/21.12 partition/G - LUMI/21.12 partition/L - - Help: - ``` - (abbreviated output). Note that it also tells you which other modules need - to be loaded. You need to choose the line which is appropriate for you and - load all modules on that line, not the whole list of in this case 9 - modules. - -!!! failure "Known issue" - The Cray PE uses Lmod in an unconventional manner with the hierarchy not - build fully in the way Lmod expects. As a consequence Lmod is not always - able to generate the correct list of modules that need to be loaded to make - a package available, and the list of ways to make a module available may - also be incomplete. - - The problem is somewhat aggrevated on LUMI because the Cray PE hierarchy sits - next to the hierarchy of the software stack as the Cray PE is installed - separately and hence cannot be integrated in the way the Lmod developer had - in mind. - - -#### Module extensions - -Certain packages, e.g., Python, Perl or R, get a lot of their functionality through -other packages that are installed together with them and extend the functionity, -e.g., NumPy and SciPy for Python. Installing all those packages as separate modules -to make it easy to see if they are installed or not on a system would lead to an -overload of modules on the system. - -Similary, admins of a software stack may chose to bundle several libraries or tools -that are often used together in a single module (and single installation directory), -e.g., to reduce module clutter but also to reduce the length of the search paths for -binaries, libraries or manual pages to speed up loading of applications. - -Lmod offers a way to make those individual packages installed in a module discoverable -by declaring them as *extensions* of the module. The ``module spider`` command will -search for those too. - - 1. ``module spider`` without further arguments: The output may contain lines similar - to - ``` - ----------------------------------------------------------------------- - The following is a list of the modules and extensions currently available: - ----------------------------------------------------------------------- - Autoconf: Autoconf/2.71 (E) - - CMake: CMake/3.21.2 (E), CMake/3.22.2 (E) - ``` - which tells that ``Autoconf`` and ``CMake`` are not available as modules themselves - but as extensions of another module, and it also tells the versions that are available, - though that list may not be complete (and is not always complete for modules either - as it is limited to one line of output). - -2. ``module spider `` will search for extensions also. E.g., - ``` - module spider CMake - ``` - on LUMI will return something along the lines of - ``` - ----------------------------------------------------------------------- - CMake: - ----------------------------------------------------------------------- - Versions: - CMake/3.21.2 (E) - CMake/3.22.2 (E) - ``` - (output abbreviated). - This tells that there is no ``CMake`` module on the system but that two versions - of ``CMake`` are provided in another module. - - 3. `module spider /` will show more information on the - extension, including which module provides the extension and which other modules - have to be loaded to make that module available. E.g., on LUMI, - ``` - module spider CMake/3.22.2 - ``` - will output something along the lines of - ``` - ----------------------------------------------------------------------- - CMake: CMake/3.22.2 (E) - ----------------------------------------------------------------------- - This extension is provided by the following modules. To access the - extension you must load one of the following modules. Note that any - module names in parentheses show the module location in the software - hierarchy. - - buildtools/21.12 (LUMI/21.12 partition/L) - buildtools/21.12 (LUMI/21.12 partition/G) - buildtools/21.12 (LUMI/21.12 partition/D) - buildtools/21.12 (LUMI/21.12 partition/C) - buildtools/21.12 (CrayEnv) - ``` - (output abbreviated and slightly reformatted for readability). This tells that - ``CMake/3.22.2`` is provided by the ``bvuildtools/21.12`` module and that there - are 5 different ways to make that package available. - -??? Bug "Restrictions with older Lmod versions" - At the time of development of this tutorial, Cray is still using the pretty old - 8.3.1 version of Lmod. Even though extensions were supported since Lmod version 8.2.5, - Lmod 8.3.1 has several problems: - - - It is not possible to hide extensions in the output of ``module avail``, a feature - that only became available in version 8.5. This may be annoying to many users as - the extension list of packages such as Python, R and Perl can be very long (the - default EasyBuild installation of R contains on the order of 600 packages). - - For that reason on LUMI extensions are only used for some modules. - - - ``module avail`` also shows extensions for modules that are not available which - makes no sense. This bug was only corrected in Lmod 8.6.13 and 8.6.14. - - -### module keyword - -Another search command that is sometimes useful is `module keyword`. It really -just searches for the given word in the short descriptions that are included in -most module files and in the name of the module. The output is not always -complete since not all modules may have a complete enough short description. - -Consider we are looking for a library or package that supports MP3 audio -encoding. -```bash -module keyword mp3 -``` -will return something along the lines of -``` ----------------------------------------------------------------- - -The following modules match your search criteria: "mp3" ----------------------------------------------------------------- - - LAME: LAME/3.100-cpeCray-21.08, LAME/3.100-cpeGNU-21.08 - LAME is a high quality MPEG Audio Layer III (mp3) encoder -``` -though the output will depend on the version of Lmod. This may not be the most -useful example on a supercomputer, but the library is in fact needed to be able -to install some other packages even though the sound function is not immediately -useful. - -??? bug "Know issue: Irrelevant output" - At the moment of the development of this tutorial, this command actually - returns a lot more output, referring to completely irrelevant extensions. - This is a bug in the HPE-Cray-provided version of Lmod (8.3.1 at the time - of development of this tutorial) that was only solved in more recent versions. - - -### module avail - -The `module avail` command is used to show only available modules, i.e., modules -that can be loaded directly without first loading other modules. It can be used -in two ways: - - 1. Without a further argument it will show an often lengthy list of all - available modules. Some modules will be marked with `(D)` which means that - they are the default module that would be loaded should you load the module - using only its name. - - 2. With the name of a module (or a part of the name) it will show all modules - that match that (part of) a name. E.g., - ```bash - module avail gnuplot - ``` - will show something along the lines of - ``` - ------ EasyBuild managed software for software stack LUMI/21.08 on LUMI-L ------ - gnuplot/5.4.2-cpeCray-21.08 gnuplot/5.4.2-cpeGNU-21.08 (D) - - Where: - D: Default Module - (output abbreviated). - ``` - but - ```bash - module avail gnu - ``` - will show you an often lengthy list that contains all packages with gnu - (case insensitive) in their name or version. - - -### Getting help - -One way to get help on a particular module has already been discussed on this -page: ``module spider /`` will produce help about the package as -soon as it can unambiguously determine the package. It is the only command that -can produce help for all installed packages. The next two commands can only -produce help about available packages. - -A second command is ``module whatis`` with the name or name and version of a -module. It will show the brief description of the module that is included in -most modules on the system. If the full version of the module is not given, it -will display the information for the default version of that module. - -The third command is ``module help``. Without any further argument it will display -some brief help about the module command. However, when used as -``module help `` or ``module help /`` it will produce help for either the -default version of the package (if the version is not specified) or the -indicated version. - - -### Implementation details - -Lmod works by executing the module file. However, the actions of all Lmod-defined -functions will depend upon the mode in which Lmod is executing the module function, -and the module file can also detect in which mode it is executing. -Modes include "load", "unload"but also "spider". E.g., when te mode is "load", the -``setenv`` function will set an environment variable to the indicated value while in -"unload" mode that environment variable will be unset, and in "spider" mode the -environment variable is left untouched. The working of ``prepend_path``, a function -that modifies PATH-style variables, depends a bit on how Lmod is configured (as it is -possible to work with reference counts), but in its most basic mode, ``prepend_path`` -will add a given directory to a given PATH-style environment variable (or move it to -the front of the PATH-style variable if the directory is already in there), while in -"unload" mode that specific directory will be removed from the PATH (but no error will -be generated should the directory that is used as the argument not be part of the path -in that PATH-style variable). When the mode is "spider", the function has special behaviour -if it is used to change the ``MODULEPATH``. It will then note the change and add that -directory to the list of directories that has to be searched for module files. -This makes ``module spider`` a very expensive command as it may have to traverse a lot -of directories and has to execute all module files in there. Therefor Lmod will build -a so-called spider cache which can be pre-built in the system for certain directories -and otherwise will be build in the user's home directory (in the ``.lmod.d/.cache`` -subdirectory). Our experience is that this cache tends to be rather fragile, -in particular on Cray systems (and that has been confirmed in discussions with -people with access to some other Cray systems) so from time to time Lmod fails to -note changes to the modules, at least when using commands such as ``module spider``. -The actual loading and unloading of the module is not based on cached information. - -Lmod has several functions that can be used in module files to provide the information -that Lmod needs for the search-related and help commands. - -The ``help`` function defines the long help text used by ``module help`` and by -``module spider`` as soon as there is no ambiguity anymore about which module is -being searched for. - -The ``whatis`` function is used to provide short information about a module. That -information is then used by ``module whatis`` and ``module keyword`` , but also -for brief information shown by ``module spider`` when multiple modules or versions -of modules are found by the command. A module file can contain multiple ``whatis`` -commands and the Lmod manuel suggests to use those lines as a kind of database -record. See, e.g., -[the Lmod manual page with module file examples](https://lmod.readthedocs.io/en/latest/100_modulefile_examples.html?highlight=whatis). -One such example is -```Lua -whatis("Name: valgrind") -whatis("Version: 3.7.0") -whatis("Category: tools") -whatis("URL: http://www.valgrind.org") -whatis("Description: memory usage tester") -``` -It is not all that important to include all those lines in a module file, but some of -those lines get a special treatment from Lmod. The line starting with ``Description`` -is used by ``module spider`` to provide some brief information about the module if it -is not totally resolved. This comes with a limitation though: It is not show for each -version of the module, so ideally all "GROMACS" modules should contain the same -description line and use other lines to provide further information about what -distinguished a particular version. -Likewise the ``Category:`` line is used by the ``spider_decoration`` hook that can be -used to add decoration to the spider level 1 output. -All in all the ``whatis`` function if often overlooked in Lmod-based module functionx -but it is a very useful function to include in the proper way in module files. - -A third function that provides information to the search commands is ``extensions``. -It can be used to list up the extensions supported by the module. The argument list -may seem strange as it takes only a single argument, a string of comma-separated ``extension/version`` -elements, but that is because the number of arguments to a function is limited in -Lua and that limit can actually be met easily by modules for Python, Perl or R packages. - - ---- - -## Some warnings about writing modulefiles - -***This section is very technical and only useful if you want to manually implement -modules that depend on each other one way or another.*** - -Lmod cannot guarantee that the order of unloading the modules will be the inverse of -the order in which they were loaded. Moreover, unloading a module is not done by reverting -stored actions done when loading the module, but by executing the modulefile again -in a mode that reverts certain actions. This can lead to subtle problems when modulefiles -communicate with each other through environment variables or by detecting which other -modules are loaded. These problems are usually solved by using a proper hierarchy -and basing actions of modulefiles on their position in the hierarchy. - -One case where passing information between modules through environment variables will -go wrong is when that environment variable is subsequently used to compute a directory -name that should be added to a PATH-like variable. Assume we have two versions of -a ``MyPython`` module, e.g., ``MyPython/2.7.18`` and ``MyPython/3.6.10``. That module then -sets an environment variable ``PYTHON_API_VERSION`` to either ``2.7`` or ``3.6``. -Next we have a module ``MyPythonPackage`` that makes a number of Python packages available -for both Python modules. However, as some Python packages have to be installed separately -for each Python version, it does so by adding a directory to the environment variable -``PYTHON_PATH`` that contains the version which it gets by using the Lua function -``os.getenv`` to request the value of ``PYTHON_API_VERSION``. - -One problem becomes clear in the following scenario: -``` bash -module load MyPython/2.7.18 -module load MyPythonPackage/1.0 -module load MyPython/3.6.10 -``` -The ``module load MyPythonPackage`` will find the environment variable ``PYTHON_PACKAGE_API`` -with the value ``2.7`` as set by ``module load MyPython/2.7.18`` and hence add the directory -for the packages for version 2.7 to ``PYTHON_PATH``. The ``module load MyPython/3.6.10`` -command will trigger two operations because of the *"one name rule"*: First it will -automatically unload ``MyPython/2.7.18`` (which will unset ``PYTHON_API_VERSIUON``) and -next it will load ``MyPython/3.6.10`` which will set ``PYTHON_API_VERSION`` to ``3.6``. -However, ``MyPythonPackage`` is not reloaded so the ``PYTHON_PATH`` variable will now point -to the wrong directory. One would be tempted to think that the easy fix for the user would -be to reload ``MyPythonPackage/1.0``: -``` bash -module load MyPythonPackage/1.0 -``` -Because of the *"one name rule"* this will again trigger an unload followed by a load -of the module. The problem is in the unload. One would expect that first unloading -``MyPythonPackage`` would remove the 2.7 directory from the ``PYTHON_PATH`` but it -will not. Lmod does not remeber that last time it loaded ``MyPythonPackage`` it added -the 2.7 directory to ``PythonPath``. Instead it will execute the commands in the -modulefile and reverse certain commands. Since ``PYTHON_API_VERSION`` has now the value -``3.6``, it will try to remove the directory for version ``3.6`` which is not in the -``PYTHON_PATH``. The subsequent load will then add the 3.6 directory to ``PYTHON_PATH`` -so the environment variable now contains both directories. - -In this simple case, a ``module purge`` after the first two ``module load`` commands would -still work as Lmod is able to figure out the right order to unload modules, but in more -complicated examples this may also go wrong. However, a ``module purge`` command after -the load of ``MyPython/3.6.10`` would also fail to clean up the environment as it would -still fail to remove the 2.7 directory from ``PYTHONPATH``. - -??? Note "Running the example" - To test this example for yourself, create a directory and add that directory to - the ``MODULEPATH`` using ``module use``. In that directory, create the following - subdirectories and files: - 1. ``MyPython/2.7.18.lua`` with content: - ``` lua - LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) - setenv( 'PYTHON_API_VERSION', '2.7' ) - ``` - 2. ``MyPython/3.6.10.lua`` with content: - ``` lua - LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) - setenv( 'PYTHON_API_VERSION', '3.6' ) - ``` - 3. ``MyPythonPackage/1.0.lua`` with content: - ``` lua - LmodMessage( 'In ' .. myModuleFullName() .. ' in mode ' .. mode() ) - LmodMessage( 'PYTHON_API_VERSION = ' .. ( os.getenv( 'PYTHON_API_VERSION' ) or '') ) - prepend_path( 'PYTHON_PATH', 'someroot/python' .. - ( os.getenv( 'PYTHON_API_VERSION' ) or 'TTT' ) .. '/packages' ) - LmodMessage( 'PYTHON_PATH = ' .. ( os.getenv( 'PYTHON_PATH' ) or '') ) - ``` -??? Note "Solution with a hierarchy" - The better way in Lmod to implement the above scenario would be in a module hierarchy. - - Just to show the power of Lmod introspection functions combined with a proper hierarchy - we present a solution using only one version of the code for ``MyPython`` and one version - of the code for ``MyPythonPackages``. - - It is best to start from a clean directory. In that directory, create: - - 1. The files ``level1/MyPython/2.7.18.lua`` and ``level1/MyPython/3.6.10.lua``, - both with the same contents: - ``` lua - LmodMessage( '\nIn ' .. myModuleFullName() .. ' in mode ' .. mode() ) - - local api_version = myModuleVersion():match( '(%d+%.%d+)%..*' ) - - -- Set the variable PYTHON_API_VERSION but not for internal use in the modules. - setenv( 'PYTHON_API_VERSION', api_version ) - - local module_root = myFileName():match( '(.*)/level1/' .. myModuleFullName() ) - prepend_path( 'MODULEPATH', pathJoin( module_root, 'level2/PythonAPI', api_version ) ) - LmodMessage( 'MODULEPATH is now\n ' .. - os.getenv( 'MODULEPATH' ):gsub( ':', '\n ' ) ) - ``` - - 2. The files ``level2/PythonAPI/2.7/MyPythonPackage/1.0.lua`` and - ``level2/PythonAPI/3.6/MyPythonPackage/1.0.lua``, both with the contents: - ``` lua - LmodMessage( '\nIn ' .. myFileName() .. ' in mode ' .. mode() ) - - local python_api_version = myFileName():match( '.*/level2/PythonAPI/([^/]+)/.*' ) - LmodMessage( 'Detected Python API version from hierarchy: ' .. python_api_version ) - LmodMessage( 'Detected Python API version from environment: ' .. - ( os.getenv( 'PYTHON_API_VERSION' ) or '' ) ) - - prepend_path( 'PYTHON_PATH', 'someroot/python' .. python_api_version .. '/packages' ) - - LmodMessage( 'PYTHON_PATH = ' .. (os.getenv( 'PYTHON_PATH' ) or '') ) - ``` - - Now add the ``level1`` subdirectory to ``MODULEPATH``, e.g., if you're in the directory - containing the ``level1`` and ``level2`` subdirectories: - ``` bash - module use $PWD/level1 - ``` - and then try the following commands: - ``` bash - module avail - module load MyPython/2.7.18 - module avail - module load MyPythonPackage/1.0 - module load MyPython/3.6.10 - ``` - and pay attention to the output. - - Initially ``module avail`` will show none of the ``MyPythonPackage`` modules. These are - installed modules but not available modules. ``module load MyPython/2.7.18`` will set the - environment variable ``PYTHON_API_VERSION`` to ``2.7`` and also add a directory to the front - of the ``MODULEPATH`` with the directory name ending on ``level2/PythonAPI/2.7``. Now - ``module avail`` will show the ``MyPythonPackage/1.0`` module. - - The ``MyPythonPackage`` shows two ways to get the version of the Python API to use for - determining the right directory to add to ``PYTHON_PATH``. The fragile way is to enquire - the value of the environment variable ``PYTHON_API_VERSION`` set by loading ``MyPython/2.7.18``. - The more robust way is to use the Lmod introspection function ``myFileName()`` which returns - the full path and file name of the module file that is executing, and extracting the version - from the path with a pattern matching function. In this particular situation both computed - values are the same so both would have worked to correctly add - ``somedir/python2.7/packages`` to the front of ``PYTHON_PATH``. - - The next command, ``module load MyPython/3.6.10`` triggers a chain of events. - - First, Lmod notices that there is already a module loaded with the same name, so it will - unload ``MyPython/2.7.18``. This will unset the environment variable ``PYTHON_API_VERSION`` - (the inverse operation of ``setenv``) and will remove the ``.../level2/PythonAPI/2.7`` - subdirectory from the ``MODULEPATH`` (the inverse action of ``prepend_path``). - - Now due to - the change of the ``MODULEPATH`` the ``MyPythonPackage/1.0`` module which was loaded from - ``.../level2/PythonAPI/2.7`` is no longer available so Lmod will continue with unloading - that module. The interesting bit now is that ``PYTHON_API_VERSION`` is unset. So had we - computed the name of the directory to add to ``PYTHON_PATH`` using the value of that - environment variable, the module would have failed to compute the correct directory name - to remove so ``prepend_path`` would have left the ``PYTHON_PATH`` environment variable - untouched. However, by computing that value from the directory of the modulefile, we get - the right value and can correctly remove ``somedir/python2.7/packages`` from ``PYTHON_PATH``. - Lmod will also remember that the module was only unloaded due to a change in the - ``MODULEPATH`` and not because a user explicitly unloaded the module. I.e., it considers - the module as deactivated but not as unloaded. - - Lmod proceeds with loading the ``MyPython/3.6.10`` module. This will now set - ``PTHON_API_VERSION`` to ``3.6`` and add a directory with name ending on - ``level2/PythonAPI/3.6`` to ``MODULEPATH``. - - Things are not done yet though. As the ``MODULEPATH`` has changed, Lmod looks at its list - of deactivated modules and notices that a different version of ``MyPythonPackage/1.0`` is - now available. Hence it will now automatically load that module from the - ``.../level2/PythonAPI/3.6`` subdirectory so that that module now correctly detects - that ``somedir/python3.6/package`` should be added to ``PYTHON_PATH``. - - Hence at the end of the cycle we have again a correctly configured environment with no - trace of the ``2.7`` version that was loaded initially and with no action required from - the user to ensure that ``MyPythonPackage`` is unloaded and reloaded to ensure the - correct configuration. - - This idea is used on LUMI to implement the various versions of the software stack with - for each software stack also optimised binaries for each of the node types. - - ---- - -## Further reading - -- [Lmod documentation](https://lmod.readthedocs.io/en/latest/index.html) -- [Lmod on LUMI in the LUMI documentation](https://docs.lumi-supercomputer.eu/computing/Lmod_modules/) -- [Documentation of Environment Modules 5](https://modules.readthedocs.io/en/latest/), - an alternative to Lmod (though not currently supported by HPE Cray) - - ---- - -*[[next: The Cray Programming Environment]](1_03_CPE.md)* diff --git a/docs/2022-CSC_and_LO/1_03_CPE.md b/docs/2022-CSC_and_LO/1_03_CPE.md deleted file mode 100644 index d3a2ff84..00000000 --- a/docs/2022-CSC_and_LO/1_03_CPE.md +++ /dev/null @@ -1,418 +0,0 @@ -# The HPE Cray Programming Environment - -*[[back: The Lmod module system]](1_02_Lmod.md)* - ---- - -On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated -as Cray PE). The environment provides several tools, including compilers, communication libraries, -optmized math libraries and various other libraries, analyzers and debuggers. - -The Cray PE is made available through *environment modules* tha allow to select particular versions of -tools and to configure the environment in a flexible way. - ---- - -## Cray PE components - -### Cray compiler environments - -Cray PE supports multiple compilers, Cray and third party compilers as well: AOCC, Cray, Intel, GNU. - -Users can access the compilers loading a programming environment module (see below). - -Compilers in the Cray Programming Environment are almost always used through the Cray compiler wrappers. In fact, -until recently, some compilers could not be used without the wrappers as they failed to even detect their own -include files and some components. The behaviour of the wrapper will depend on the other modules that are -loaded in the environment. E.g., there is no separate wrapper for MPI as the MPI header files and libraries are -automatically activated through the regular wrappers as soon as a Cray MPI module is loaded. - -The commands to invoke compiler wrappers are `ftn` (Fortran), `cc` (C), `CC` (C++). They wrap automatically -to the right compilers based on the programming environment module that is loaded and the compiler module. - -The online help can be accessed with the `-help` option. E.g.: `cc -help`, `CC -help`. -One of the most frequently used options of the compiler wrappers is `-craype-verbose`: -``` - ftn -help | grep verbose -A 1 - -craype-verbose Print the command which is forwarded - to compiler invocation -``` -More information is available with the `info` or `man` commands. -E.g.: both `info cc` or `man cc` will display the man page of the C compiler wrapper. - -The compiler wrappers call the correct compiler in the currently loaded programming environment, -with appropriate options to build and link applications with relevant libraries, as required by the modules loaded. -Besides a number of generic options, they also will pass all other options to the underlying compile, -so you can still use the regular compiler options also. - -The compiler wrappers should replace direct calls to compiler drivers in Makefiles and build scripts -to ensure that the proper libraries are used. - -!!! Note - For system libraries, only dynamic linking is supported by compiler wrappers on the Cray EX system - ---- - -#### Cray Compiling Environment (CCE) - -The Cray Compiling Environment is set by the module `PrgEnv-cray`, which is loaded by default at login. -The compiler itself is contained in the `cce` module. - -CCE consists of Cray compilers performing code analysis during compilation to generate highly optimized code. -Supported languages include Fortran, C and C++, and UPC (Unified Parallel C). -The Cray C/C++ compiler is basically the Clang/LLVM compiler with a back-end configured by HPE Cray that also -adds some Cray-specific options while the Cray Fortran compiler uses an HPE-Cray specific front-end -with a back-end based on LLVM. The Fortran compiler supports most of Fortran 2018 and tends to be -considerably stricter than the GNU or Intel Fortran compilers. - -The CCE compilers will also support OpenMP offload to NVIDIA and AMD GPUs but that is still very immature and -very much work in progress at the time of the development of this tutorial. -The Fortran compiler (but not the C/C++ compiler) also supports -OpenACC for offloading to GPU and is being updated to the newest versions of this standard. - -The classic pre-LLVM Cray C/C++ compiler is not available on Cray EX systems. - -Compiler-specific manpages can be accessed on the system with `man crayftn`, `man craycc` or `man crayCC`. - -More details are given in the -[Cray Fortran Reference Manual](https://support.hpe.com/connect/s/search?language=en_US#q=Cray%20Fortran%20Reference%20manual&t=All&sort=relevancy&numberOfResults=25) - and the - [Cray Compiling Environment Release](https://support.hpe.com/connect/s/search?language=en_US#q=Cray%20Compiling%20Environment%20Release&t=All&sort=relevancy&numberOfResults=25) -that unfortunately are hidden deep in the new [support pages of HPE](https://support.hpe.com/). -The [Clang Compiler User’s Manual](https://clang.llvm.org/docs/UsersManual.html) -is another source of information for the Cray C and C++ Clang compilers. - -For more information about compiler pragmas and directives, see `man intro_directives` on the system. - ---- - -### Third-Party Compilers - -#### GNU - -The GNU C/C++ and Fortran compilers are probably the best supported third party -compiler in the Cray PE. - -Compiler-specific manpages can be accessed on the system with `man gfortran`, `man gcc` or `man g++`. - -More details are provided by the [GCC online documentation](https://gcc.gnu.org/onlinedocs). - - -#### AOCC - -There AOCC compilers, the AMD Optimizing C/C++ Compiler and matching fortran compilers, -AMD's compiler offering for CPU-only systems, have a matching programming environment module -and are a full citizen of the Cray PE. - -Compiler-specific documentation is available in the [AOCC User Guide](https://developer.amd.com/amd-aocc/#userguide). - -Cray provides a bundled package of support libraries to install into the PE environment to enable AOCC, -and Cray PE utilities such as debuggers and performance tools work with AOCC. - - -#### AMD ROCm compilers - -The AMD ROCm compilers (in the AMD world sometimes known as AOMP) are supported on systems -with AMD CPUs. However, at the time of writing of this tutorial they are not yet available in the -LUMI environment and the integration with the Cray environment still seems to be work in progress. - - -#### Intel (not on LUMI) - -The Cray PE also provides a programming environment module to enable -the Intel® oneAPI compiler and tools. - -The documentation is available in the [Intel® oneAPI Programming Guide](https://software.intel.com/content/www/us/en/develop/documentation/oneapi-programming-guide/top/oneapi-development-environment-setup.html) - -Cray provides a bundled package of support libraries to install into the Cray PE to enable the Intel compiler, allowing -utilities such as debuggers and performance tools to work with it. - - -#### NVIDIA HPC toolkit (not on LUMI) - -The NVIDIA HPC Toolkit compilers (formerly PGI) are supported on systems with NVIDIA GPUs. - - -### Cray Scientific and Math Library - -* Manpages: `intro_libsci`, `intro_fftw3` - -The Cray Scientific and Math Libraries (CSML, also known as LibSci) are a collection of numerical routines -optimized for best performance on Cray systems. - -These libraries satisfy dependencies for many commonly used applications on Cray systems for a wide variety of domains. - -When the module for a CSML package (such as `cray-libsci` or `cray-fftw`) is loaded, -all relevant headers and libraries for these packages are added to the compile -and link lines of the `cc`, `ftn`, and `CC` compiler wrappers, so linking with them is -completely transparant (to the extent that users wonder where the libraries are). - -The CSML collection contains the following Scientific Libraries: - -* BLAS (Basic Linear Algebra Subroutines) -* BLACS (Basic Linear Algebra Communication Subprograms) -* CBLAS (Collection of wrappers providing a C interface to the Fortran BLAS library) -* IRT (Iterative Refinement Toolkit) -* LAPACK (Linear Algebra Routines) -* LAPACKE (C interfaces to LAPACK Routines) -* ScaLAPACK (Scalable LAPACK) -* `libsci_acc` (library of Cray-optimized BLAS, LAPACK, and ScaLAPACK routines) -* HDF5 -* NetCDF (Network Common Data Format) -* FFTW3 (the Fastest Fourier Transforms in the West, release 3) - - -### Cray Message Passing Toolkit - -* Manpages: `intro_mpi` - -* Website: [http://www.mpi-forum.org](http://www.mpi-forum.org) - -MPI is a widely used parallel programming model that establishes a practical, portable, efficient, -and flexible standard for passing messages between ranks in parallel processes. - -Cray MPI is derived from Argonne National Laboratory MPICH and implements the MPI-3.1 standard -as documented by the MPI Forum in MPI: A Message Passing Interface Standard, Version 3.1. - -Support for MPI varies depending on system hardware. To see which functions and environment variables the -system supports, please have a look at the corresponding man pages with `man intro_mpi` on the system. - -Note that though on LUMI at the time of the tutorial both the OpenFabric Interface (OFI) based and UCX-based -versions of the library are supported, the finalised LUMI system will only support OFI. - - -### DSMML - -* Modules: `cray-dsmml` -* -* Manpages: `intro_dsmml` -* -* Website: [https://pe-cray.github.io/cray-dsmml](https://pe-cray.github.io/cray-dsmml) - -Distributed Symmetric Memory Management Library (DSMML) is a HPE Cray proprietary memory management library. - -DSMML is a standalone memory management library for maintaining distributed shared symmetric memory -heaps for top-level PGAS languages and libraries like Coarray Fortran, UPC, and OpenSHMEM. - -DSMML allows user libraries to create multiple symmetric heaps and share information with other libraries. - -Through DSMML, interoperability can be extracted between PGAS programming models. - -Further details are available in the man page on the system with `man intro_dsmml`. - - -### Other components - -- **Cray Performance and Analysis Tools**: - - Tools to analyze the performance and behavior of programs that are run on Cray systems, and a Performance API (PAPI). - -- **Cray Debugging Support Tools**: - - Debugging tools, including `gdb4hpc` and `Valgrind4hpc`. - - ---- - -## Configuring the Cray PE through modules - -Multiple releases of the HPE Cray PE can be installed simultaneously on the system and users can mix-and-match -components. However, it is also possible to load only components for a specific release of the PE. -Cray PE versions have version numbers of the form ``yy.mm``, e.g., ``22.02`` for the version released in -February of 2022. However, each of the components have their own version number and it is not easy to see -which version of a component came with which version(s) of the Cray PE. - -Below we only discuss those modules that are important when building software with EasyBuild. Debuggers, -profilers, etc., are not included in the list. - - -### The programming environment modules - -The usual way to initialise the Cray PE is by loading one of the ``PrgEnv-*`` modules. This module will then -load the appropriate compiler, compiler wrappers and other libraries. Some of the components that will be -loaded are configured through the ``/etc/cray-pe.d/cray-pe-configuration.sh`` file, so the list of modules -may depend on the actual system that you are using. - -The Cray PE supports the following ``PrgEnv-*`` modules. On LUMI, only the first three are currently available: - -| Module | Compiler module | What? | -|:---------------|-----------------|:--------------------------------------------| -| PrgEnv-cray | cce | The Cray Compiling Environment compilers | -| PrgEnv-gnu | gcc | The GNU compilers | -| PrgEnv-aocc | aocc | AMD compilers for CPU-only systems | -| PrgEnv-amd | rocm | AMD ROCm compilers for GPU systems | -| PrgEnv-intel | intel | The Intel compilers | -| PrgEnv-nvidia | nvidia | NVIDIA HPC toolkit compilers (formerly PGI) | - -All ``PrgEnv-*`` modules belong to the same module family. Hence Lmod will automatically unload any -already loaded ``PrgEnv-*`` module when you load a different one. - - -### Selecting the version through the cpe meta-module - -The Cray PE on the EX system provides the meta-module `cpe`: the purpose of the meta-module is -similar to the scope of the `cdt` and `cdt-cuda` meta-modules available on the XC systems. - -Loading one of the ``cpe/yy.mm`` modules (e.g., ``cpe/22.02``) has the following effects: - -- It sets the default versions of each of the Cray PE modules to the version that comes with that - particular release of the Cray PE. E.g., - - ``` bash - module load cpe/22.02 - module load cce - ``` - - would load that version of the ``cce`` compiler that comes with the 22.02 release of the Cray PE. - -- It will reload all already loaded Cray PE modules and switch them over to the version corresponding - to that particular release of the Cray PE. - -???+ Bug "Limitations and bugs" - Due to the way Lmod works and implementation bugs in the ``cpe`` modules, loading the ``cpe`` module - does not always have the desired effect. - - - The Cray PE sets the default version of each module by adding a file to the list of files in the - ``LMOD_MODULERCFILE`` environment variable. This is because Lmod does not re-evaluate the visibility - of modules and the internal list of default version immediately when the value of ``LMOD_MODULERCFILE`` - is changed but only the next time the ``module`` command is executed. Hence - ``` bash - module load cpe/22.02 ; module load cce - ``` - and - ``` - module load cpe/22.02 cce - ``` - do not have the same effect. In the first version, the version of ``cce`` loaded is the version that - corresponds to the 22.02 release of the Cray PE. In the second case however the default version of the - ``cce`` module is determined by whatever list of default modules was used when calling the ``module`` - command so may or may not be the one of the 22.02 release. - - - Loading the ``cpe`` module after loading the other Cray PE modules also does not always have the - desired effect in many versions of the Cray PE. This is because of a bug in the ``cpe `` module that - reloads the modules in the wrong order which may trigger the reload of a module with whatever version - was the default when the ``module`` command was called rather than the version the the ``cpe`` module - intends to (re-)load. - - -### The compiler wrapper module craype - -The ``craype`` module loads the compiler wrappers. There is only one set of compiler wrappers for all compilers. -Which compiler will be called, which libraries will be included, but also processor and GPU target options will -be used, is all determined by other modules. Hence it is in principle possible to use a single Makefile for -a project and still reconfigure the build by loading certain modules. - - -### Target modules - -The targets for CPU and GPU optimization, the network library for MPI bt also some other compiler options, -can be set through target modules: - -- ``craype-x86-*`` (and similar options can be expected on ARM-based systems) set the target for CPU - optimisations. For LUMI, the ``crape-x86-rome``, ``craype-x86-milan`` and ``craype-x86-trento`` modules - are relevant. - - This can also be used to cross-compile to a different target architecture unless the compiler target - gets overwritten by a compiler flag added to the command line through the Makefile or other means, - something that unfortunately happens more and more often in faulty software installation procedures. - -- ``creype-accel-*`` sets the target for OpenMP offload (and likely other technologies in the future). E.g., - loading ``craype-accel-amd-gfx90a`` tells the compilers to target AMD MI200 family GPUs, while loading - ``craype-accel-host`` tells the compiler to use the CPU instead (according to the documentation, the latter - is for ``PrgEnv-cray`` only). - -- ``craype-network-*`` selects the communication library to be used by Cray MPICH. On Slingshot 11 EX systems, - only ``craype-network-ofi`` is supported, but Slingshot 10 EX systems also offer support for UCX through the - ``craype-network-ucx`` module. - -- The ``craype-hugepages*`` modules enable Cray Huge Pages support. To fully enable this support they have to - be used at link-time and at run-time. At link time, support is compiled into the binary wile at run-time they - are used to set the actual size of the huge pages. - - The ``craype-hugepages*`` modules are not supported by all compilers. E.g., the AOCC compiler does not support - huge pages at the moment, and loading the module at link time will cause an error message from the linker. - - -### The compiler modules - -The compiler modules have already been discussed with the ``PrgEnv-*`` modules above. The different regular -compiler modules also all belong to the same family so no two different compilers can be loaded simultaneously -and Lmod will automatically unload the other compiler when a new one is loaded. - - -### The MPI modules - -To load the Cray MPI libraries, both one of the ``craype-network-*`` modules and a compiler module has to be -loaded as the MPI libraries are both network- and compiler specific. - -For some unknown reason, the MPI module for the libfabric (``craype-network-ofi``) transport is called ``cray-mpich`` -while the library for the UCX transport (``craype-network-ucx``) is called ``craype-network-ucx``. As a result, -the MPI module fails to reload automatically when switching between both transports, but it does reload automatically -when switching compilers. - -Loading an MPI module will also automatically configure the regular compiler wrappers to compile with support for -that MPI module. However, the libfabric and UCX versions of the MPI library have compatible interfaces, so it is -always possible to swap between those versions at runtime. - - -### The Cray Scientific libraries - -The Cray Scientific Libraries are loaded through the ``cray-libsci`` module (or ``cray-libsci_acc`` for the GPU -versions). Loading this module makes the BLAS, LAPACK, and ScaLAPACK libraries available, and also the Cray IRT -(Iterative Refinement Toolkit), a Cray-specific library. It will also configure the compiler wrappers to link -with these libraries, so no additional include or link options are needed. - - -### The ``fftw`` module - -As FFTW is a third party library, it gets its own module (``cray-fftw``) and is not included in the ``cray-libsci`` module. - -The ``cray-fftw`` module can only be loaded if one of the processor target modules (the ``craype-x86-*`` modules) -is loaded first. - - ---- - -## Some unexpected behaviour of the Cray PE - -On Cray EX systems, dynamic linking is the preferred way of linking applications. Whith that comes some -unexpected behaviour of the Cray modules. EasyBuild users expect that at run time the versions of the -libraries that are used, are the ones from the modules that are loaded. This is not always the case -for the runtime libraries of the Cray PE. By default the Cray PE will use a default set of libraries -that is determined by the system default version of the Cray PE (which is set by the sysadmins, not -determined by the ``cpe`` module). This can only be avoided by either using\ -rpath-linking (which is also special in the Cray PE as the wrappers activate rpath linking if the -environment variable ``CRAY_ADD_RPATH`` is defined and set to ``yes``) or by manually adjusting -the library path after loading the modules: -``` bash -export LD_LIBRARY_PATH=${CRAY_LD_LIBRARY_PATH}:$LD_LIBRARY_PATH -``` -The latter cannot be easily automated in modulefiles. Any technique that can be used (without actually -reworking the Cray PE modules) has nasty side effects in some scenarios. - -The net result of this feature is that some reproducibility of the results is lost. Programs will -react differently if the system default version of the Cray PE is changed as that will change the -set of default libraries that will be used at runtime unless rpath-linking is used or users -redfine ``LD_LIBRARY_PATH``. - - ---- - -## Further reading - -- [LUMI documentation: "Developing" section](https://docs.lumi-supercomputer.eu/development/) -- The Cray PE is mostly documented through man pages. There used to be some documentation on the - Cray web site also but the documentation system got reworked after the merger with HPE. - The documentation is now in the [HPE Support Centre](https://support.hpe.com/) where it - is very difficult to find the right version of the documents. -- The [PE-Cray](https://github.com/PE-Cray) GitHub project also provides some additional - documentation, including - - [some whitepapers](https://github.com/PE-Cray/whitepapers), - - [documentation for Cray OpenSMHEMX](https://github.com/PE-Cray/cray-openshmemx) and - - [information about Cray DSMML](https://github.com/PE-Cray/cray-dsmml) - though much of that information ia also not updated regularly anymore. - - ---- - -*[[next: LUMI software stacks]](1_04_LUMI_software_stack.md)* diff --git a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md b/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md deleted file mode 100644 index 3bfae70c..00000000 --- a/docs/2022-CSC_and_LO/1_04_LUMI_software_stack.md +++ /dev/null @@ -1,101 +0,0 @@ -# LUMI software stacks (technical) - -*[[back: The Cray Programming Environment]](1_03_CPE.md)* - ---- - -The user-facing documentation on how to use the LUMI software stacks is -avialable in [the LUMI documentation](https://docs.lumi-supercomputer.eu/computing/softwarestacks/). -On this page we focus more on the technical implementation behind it. - ---- - -# An overview of LUMI - -LUMI has different node types providing compute resources: - -- LUMI has 16 login nodes, though many of those are reserved for special purposes and not - available to all users. TODO - - ---- - -## CrayEnv and LUMI modules - -On LUMI, two types of software stacks are currently offered: - - - ``CrayEnv`` (module name) offers the Cray PE and enables one to use - it completely in the way intended by HPE-Cray. The environment also offers a - limited selection of additional tools, often in updated versions compared to - what SUSE Linux, the basis of the Cray Linux environment, offers. Those tools - are installed and managed via EasyBuild. However, EasyBuild is not available - in that partition. - - It also rectifies a problem caused by the fact that there is only one - configuration file for the Cray PE on LUMI, so that starting a login shell - will not produce an optimal set of target modules for all node types. - The ``CrayEnv`` module recognizes on which node type it is running and - (re-)loading it will trigger a reload of the recommended set of target - modules for that node. - - - ``LUMI`` is an extensible software stack that is mostly managed through - [EasyBuild][easybuild]. Each version of the LUMI software stack is based on - the version of the Cray Programming Environment with the same version - number. - - A deliberate choice was made to only offer a limited number of software - packages in the globally installed stack as the setup of redundancy on LUMI - makes it difficult to update the stack in a way that is guaranteed to not - affect running jobs and as a large central stack is also hard to manage. - However, the EasyBuild setup is such that users can easily install - additional software in their home or project directory using EasyBuild build - recipes that we provide or they develop, and that software will fully - integrate in the central stack (even the corresponding modules will be made - available automatically). - - Each ``LUMI`` module will also automatically activate a set of application - modules tuned to the architecture on which the module load is executed. To - that purpose, the ``LUMI`` module will automatically load the ``partition`` - module that is the best fit for the node. After loading a version of the - ``LUMI`` module, users can always load a different version of the ``partition`` - module. - -Note that the ``partition`` modules are only used by the ``LUMI`` module. In the -``CrayEnv`` environment, users should overwrite the configuration by loading their -set of target modules after loading the ``CrayEnv`` module. - - ---- - -## The ``partition`` module - -The ``LUMI`` module currently supports four partition modules, but that number may -be reduced in the future: - -| Partition | CPU target | Accelerator | -|:----------------|-----------------------|:----------------------------| -| ``partition/L`` | ``craype-x86-rome`` | ``craype-accel-host`` | -| ``partition/C`` | ``craype-x86-milan`` | ``craype-accel-host`` | -| ``partition/G`` | ``craype-x86-trento`` | ``craype-accel-amd-gfx90a`` | -| ``partition/D`` | ``craype-x86-rome`` | ``craype-accel-nvidia80`` | - -All ``partition`` modules also load `craype-network-ofi``. - -``pattition/D`` may be dropped in the future as it seems we have no working CUDA setup -and can only use the GPU nodes in the LUMI-D partition for visualisation and not with CUDA. - -Furthermore if it would turn out that there is no advantage in optimizing for Milan -specifically, or that there are no problems at all in running Milan binaries on Rome -generation CPUs, ``partition/L`` and ``partition/C`` might also be united in a single -partition. - - - - - - - ---- - -*[[next: Terminology]](1_05_terminology.md)* - diff --git a/docs/2022-CSC_and_LO/1_05_terminology.md b/docs/2022-CSC_and_LO/1_05_terminology.md deleted file mode 100644 index 8e05012f..00000000 --- a/docs/2022-CSC_and_LO/1_05_terminology.md +++ /dev/null @@ -1,294 +0,0 @@ -# Terminology - -*[[back: LUMI software stacks]](1_04_LUMI_software_stack.md)* - ---- - -Over the years, we have come up with some terminology specific to EasyBuild -to refer to particular components, which we use alongside established terminology relevant to the context -of building and installing software. - -It is important to be familiar with these terms, so we'll briefly cover them one by one. - ---- - -## Toolchains - -A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**, -which are used to build software from source, together with a set of **additional libraries** that provide further core functionality. - -We refer to the different parts of a toolchain as **toolchain components**. - -The *compiler component* typically consists of [C](https://en.wikipedia.org/wiki/C_(programming_language)), -[C++](https://en.wikipedia.org/wiki/C%2B%2B), and [Fortran](https://en.wikipedia.org/wiki/Fortran) -compilers in the context of HPC, but additional compilers (for example, -a [CUDA](https://developer.nvidia.com/cuda-zone) compiler for -[GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units) software) -can also be included. - -Additional toolchain components are usually special-purpose libraries: - -- an MPI library to support distributed computations (for example, [Open MPI](https://www.open-mpi.org/) - or vendor-specific MPI implementations like Intel MPI in a toolchain based on the Intel compilers); - -- libraries providing efficient linear algebra routines ([BLAS](http://performance.netlib.org/blas/), - [LAPACK](http://performance.netlib.org/lapack/), [ScaLAPACK](http://www.netlib.org/scalapack/)); - - These libraries are included because they tend to provide a common API, so mixing them in a single - executable would cause name conflicts when linking. - -- a library supporting computing Fast Fourier Transformations (for example, [FFTW](http://fftw.org/)); - -These libraries are included because they are some of the most popular libraries in scientific software, -but also because they provide an API that is available through different implementations. Mixing -implementations in a single build is a recipe for problems as the various implementations might -define the same symbols. - -A toolchain that includes all of these libraries is referred to as a **full toolchain**, while -a **subtoolchain** is a toolchain that is missing one or more of these libraries. -A **compiler-only toolchain** only consists of compilers (no additional libraries). - -The toolchain concept also maps nicely on the Programming Environment concept in the Cray -Programming Environment. Each ``PrgEnv-*`` module in fact provides a full toolchain in a -typical Cray PE installation, except for the FFTW library. Note that systems could configure -the contents of a ``PrgEnv-*`` module differently as the MPI library and scientific library are -optional. In the Cray PE, the MPI library is provided by the ``cray-mpic`` module, -the BLAS, LAPACK and ScaLAPACK libraries by the ``cray-libsci`` module (for CPU-only nodes) -and the FFTW library through the ``cray-fftw`` module. - -Many EasBuild toolchains are organised in a hierarchy. Each toolchain can use components compiled -with itself or with one of the subtoolchains. The Cray toolchains however are not currently -organised in a hierarchy. - - -### System toolchain - -The **`system` toolchain** is a special case which corresponds to using the compilers and libraries -*provided by the operating system*, rather than using toolchain components that were installed using EasyBuild. - -It used sparingly, mostly to install software where no actual compilation is done or -to build a set of toolchain compilers and its dependencies, since the versions of the system compilers -and libraries are beyond the control of EasyBuild, which could affect the reproducibility of the installation. - -On LUMI however it is used a bit more and it takes some of the functions of the ``GCCcore`` toolchain in -other EasyBuild toolchain hierarchies. - - -### Common toolchains - -The `foss` and `intel` toolchains are also known as the *common toolchains*, -because they are widely adopted by the EasyBuild community. - -The `foss` toolchain consists of all open source components (hence the name: -"FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/), -[ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/). -In recent versions (since 2021a), [FlexiBLAS](https://www.mpi-magdeburg.mpg.de/projects/flexiblas) -is used as the BLAS library, with OpenBLAS and LAPACK as the backend. - -The `intel` toolchain consists of the [Intel C, C++ and Fortran compilers](https://software.intel.com/content/www/us/en/develop/tools/compilers.html) (on top of a `GCC` version -controlled through EasyBuild) alongside the [Intel MPI](https://software.intel.com/content/www/us/en/develop/tools/mpi-library.html) and [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) libraries. - -Roughly every 6 months, a new version of these common toolchains is agreed upon -in the EasyBuild community, after extensive testing. - -The common toolchains are also organised in a toolchain hierarchy: - -```mermaid -graph TD; -A[SYSTEM] --> B[GCCcore]; -B --> C[GCC]; -B --> D[iccifort]; -C --> E[gompi: Adds OpenMPI]; -D --> F[iimpi: Adds Intel MPI]; -D --> G[imkl: Adds Intel MKL]; -E --> H[foss: Adds OpenBLAS, LAPACK, ScaLAPACK, FFTW]; -F --> I[intel]; -G --> I; -``` - -More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html). - - ---- - -## EasyBuild framework - -EasyBuild is written in Python and organised in three layers, represented by the three blocks in the logo: -the EasyBuild framework, easyblocks and Easyconfig files. - -The EasyBuild *framework* consists of a set of Python modules organised in packages (``easybuild.framework``, -``easybuild.toolchains``, ``easybuild.tools``, etc.) that collectively form **the core of EasyBuild**, -and is developed in the [``easybuild-framework`` repository on GitHub](https://github.com/easybuilders/easybuild-framework). - -It implements the **common functionality that you need when building software from source**, -providing functions for unpacking source files, applying patch files, collecting the output produced -by shell commands that are being run and checking their exit code, generating environment module files, etc. - -The EasyBuild framework does *not* implement any specific installation procedure, it only provides -the necessary functionality to facilitate this. - - ---- - -## Easyblocks - -An *easyblock* is **a Python module that implements a specific software installation procedure**, -and can be viewed as a plugin to the EasyBuild framework. Easyblocks can be either *generic* or *software-specific*. - -A **generic easyblock** implements an installation procedure that can be used for -multiple different software packages. Commonly used examples include the ``ConfigureMake`` easyblock -which implements the ubiquitous ``configure``-``make``-``make install`` procedure, and the -``PythonPackage`` easyblock that can be used to install a Python package. - -*In Spack this functionality is provided in the core of the package.* - -A **software-specific easyblock** implements an installation procedure that is specific to a particular software packages. -Infamous examples include the easyblocks we have for ``GCC``, ``OpenFOAM``, ``TensorFlow``, ``WRF``, ... - -The installation procedure performed by an easyblock can be controlled by defining -**easyconfig parameters** in an easyconfig file. - -*In Spack, this functionality is provided by the ``package.py`` files for the various packages. However, EasyBuild -does not need a software-specific easyblock for every package as for many packages the combination of a -generic easyblock with the much simpler description in the easyconfig file is sufficient. -The software-specific easyblocks can be a bit troublesome on Cray PE systems. Some easyblocks contain code that -is specific for certain compilers or toolchains and do not support all compilers or all toolchains.* - -A collection of (generic and software-specific) easyblocks is developed by the EasyBuild community -in the [``easybuild-easyblocks`` repository on GitHub](https://github.com/easybuilders/easybuild-easyblocks). - ---- - -## Easyconfig parameters - -An **easyconfig parameter** specifies a particular aspect of a software installation that should be performed by -EasyBuild. - -Some easyconfig parameters are **mandatory**. The following parameters *must* be defined in *every* easyconfig file: - -* ``name`` and ``version``, which specify the name and version of the software to install; -* ``homepage`` and ``description``, which provide key metadata for the software; -* ``toolchain``, which specifies the compiler toolchain to use to install the software (see - ``toolchains`` tab); - -Other easyconfig parameters are **optional**: they can be used to provide required information, -or to control specific aspects of the installation procedure performed by the easyblock. - -Some commonly used optional easyconfig parameters include: - -* ``easyblock``, which specifies which (generic) easyblock should be used for the installation; -* ``sources`` and ``source_urls``, which specify the list of source files and where to download them; -* ``dependencies`` and ``builddependencies``, which specify the list of (build) dependencies; -* ``configopts``, ``buildopts``, and ``installopts``, which specify options for the configuration/build/install commands, respectively; - -If no value is specified for an optional easyconfig parameter, the corresponding default value will be used. - -There are two groups of easyconfig parameters. *General* easyconfig parameters can be defined for any software -package, and (usually) control a specific aspect of the installation. *Custom* easyconfig parameters are -only supported by certain easyblocks, and only make sense for particular (types of) software. - ---- - -## Easyconfig files - -*Easyconfig files* (or *easyconfigs* for short), are **simple text files written in Python syntax -that specify what EasyBuild should install**. -Each easyconfig file defines the set of **easyconfig parameters** that collectively form a complete specification -for a particular software installation. - -The **filename** of an easyconfig file usually ends with the ``.eb`` extension. -In some contexts the filename is expected to be determined by the value of a handful of key -easyconfig parameters: ``name``, ``version``, ``toolchain`` and ``versionsuffix``. The general format for -the filename of an easyconfig file is: ``--.eb``, -where the toolchain part is omitted when a ``system`` toolchain is used, and the `` can be empty. - -The filename of easyconfig files is particularly relevant when EasyBuild is searching for easyconfig files to resolve dependencies, -since it does this purely based on filenames: interpreting the contents of every (potential) easyconfig file -it encounters would be too expensive. - -In the [``easybuild-easyconfigs`` repository on GitHub](https://github.com/easybuilders/easybuild-easyconfigs), -the EasyBuild community maintains a large (and growing) collection of easyconfig files, for a wide range of -(scientific) software. - -*EasyBuild does not have an equivalent of the concretiser of Spack. EasyBuild always starts from a complete, -precise configuration of a package to install, determined by the easyconfig parameters and possibly a -custom easyblock. Both approaches have their pros and cons. The concretiser in Spack combined with the -configuration of a software package via command line parameters makes it easy to quickly generate a -number of different configurations of a package to test performance or to check which combinations work -well. But one also has to be careful to not cause an explosion of modules and software installations, and -take care of using the appropriate features to ensure that one can generate the same configurations again. -The precise configurations of EayBuild can sometimes slow down things as they may need a lot of file editing, -but when it comes to building a reproducible software stack and sharing precise building instructions, this -approach may be preferred.* - ---- - -## Easystack files - -[**Easystack files**](https://docs.easybuild.io/en/latest/Easystack-files.html) -are a new concept in EasyBuild, providing a way to define a *software stack* -that should be installed by EasyBuild. - -They are written in [YAML syntax](https://yaml.org/), and include a list of *software specifications* -which correspond to a list of easyconfig files, with support for providing specific EasyBuild -configuration options for particular software packages, and including or excluding specific software -packages based on labels. - -The support for using easystack files is currently marked as *experimental*, -which means it is subject to change in future EasyBuild releases, and may be prone to errors. - ---- - -## Extensions - -*Extensions* is the collective term we use for **additional software packages that can be installed -on top of another software package**. Common examples are *Python packages*, *R libraries*, and *Perl modules*. - -As you can tell the common terminology here is a bit messy, so we came up with a unifying term... - -Extensions can be installed in different ways: - -* *stand-alone*, as a separate installation on top of one or more other installations; -* as a part of a *bundle* of extensions that collectively form a separate installation; -* or as an actual *extension* to a specific installation to yield a "batteries included" - type of installation (for examples by adding a bunch of Python packages from PyPI into - a Python installation); - ---- - -## Dependencies - -A *dependency* is a common term in the context of software. It refers to **a software -package that is either strictly required by other software, or that can be leveraged to -enhance other software** (for example to support specific features). - -There are three main types of dependencies for computer software: - -* a **build dependency** is only required when building/installing a software package; - once the software package is installed, it is no longer needed to *use* that software - (examples: `CMake`, `pkg-config`); -* a **link-time dependency** is somewhere in between a build and runtime dependency: - it is only needed when *linking* a software package; it can become either a build or runtime - dependency, depending on exactly how the software is installed (example: `OpenBLAS`); -* a **run-time dependency** (often referred to simply as *dependency*) is a software package that is - required to *use* (or *run*) another software package (example: `Python`); - -EasyBuild currently doesn't treat link-time dependencies differently from run-time dependencies -and hence has no specific easyconfig parameter for them. - ---- - -## Bringing it all together - -The EasyBuild **framework** leverages **easyblocks** to automatically build and install -(scientific) software, potentially including additional **extensions**, using a particular compiler **toolchain**, -as specified in **easyconfig files** which each define a set of **easyconfig parameters**. - -EasyBuild ensures that the specified **(build) dependencies** are in place, -and automatically generates a set of **(environment) modules** that facilitate access to the installed software. - -An **easystack file** can be used to specify a collection of software to install with EasyBuild. - ---- - -*[[next: Installation]](1_06_installation.md)* diff --git a/docs/2022-CSC_and_LO/1_06_installation.md b/docs/2022-CSC_and_LO/1_06_installation.md deleted file mode 100644 index 24c71d5e..00000000 --- a/docs/2022-CSC_and_LO/1_06_installation.md +++ /dev/null @@ -1,545 +0,0 @@ -# Installing EasyBuild - -*[[back: Terminology]](1_05_terminology.md)* - ---- - -Before deep diving into the wonderful world of EasyBuild and getting your hands dirty with -the hands on exercises coming up in this tutorial, you will need to install EasyBuild. - -In this section we outline a couple of different ways of doing this, and also the things you should pay attention to. -By the end, you will have a functional EasyBuild installation that you can use -for the remainder of this tutorial. - -!!! Remark "EasyBuild on LUMI" - - EasyBuild is already installed for each of the LUMI software stacks. The LUMI software - stacks are based on the Cray PE (using the Cray-specific toolchains). - Moreover, in case you want to install a full shadow stack rather than build on top - of one of the LUMI software stacks, scripts are provided to initialise a new version - of the LUMI software stack based on a specific version of the Cray PE, and those - scripts also take care of the installation of EasyBuild. - - Hence this section of the tutorial is only useful for people who want more background - knowledge or who want to experiment with other EasyBuild toolchains. - - -!!! summary - - * *Requirements*: Linux, Python 2.7 or 3.5+, environment modules tool (Lmod is recommended). - * *Installation methods*: - * ``pip install easybuild`` - * installing EasyBuild with EasyBuild - * development setup - * Verify EasyBuild installation using `eb --version` and `eb --help`. - - ---- - -## Requirements - -### Linux - -The main target platform for EasyBuild is *Linux*, since it is heavily focused on installing software -on HPC system where Linux is dominant operating system (to the point where 100% of the current Top500 list of supercomputers are running Linux). - -EasyBuild is also compatible with *macOS*, but the included easyconfig files are heavily focused -on Linux so most software installations supported by EasyBuild won't work out-of-the-box on macOS. -You can still use the EasyBuild command line interface on macOS for other tasks though, -like development or contributing back to the project. - -### Python - -EasyBuild is implemented in Python, and is **compatible with both Python 2.7 and Python 3.5+** -(that is, Python 3.5 or a newer version of Python 3). - -To check which Python version you have, use: - -```shell -python -V -``` - -*No additional Python packages are required by EasyBuild*, the ones that come with the standard -Python distribution are sufficient. Some additional Python packages *can* be leveraged for specific features. -More on that later. - -### Environment modules tool - -**An *environment modules tool* is required for using EasyBuild.** - -We strongly recommend using [Lmod](https://lmod.readthedocs.io), a Lua-based modern environment -modules implementation and the most commonly used modules tool in the EasyBuild community. -Other implementations, like the original Tcl-based one, are also supported. - -To check if you have a modules tool installed, use: - -``` -module --version -``` - -If this produces output that starts with something like "`Modules based on Lua: Version 8.3.1`" you have Lmod installed, -which is the default modules tool used by EasyBuild, and you are all set for installing and using EasyBuild. -Any sufficiently recent Lmod version (8.x or even 7.x) should be fine. - -If you see output that starts with a line like "`VERSION=3.2.10`" or "`Modules Release 4.5.0`", -you have the original Tcl-based environment modules tool installed, either in its original -C-based imnplementation (``VERSION=3.2.10``) or the new Tcl-based implementation developed at CEA -(``Modules Release 4.5.0``) -and *EasyBuild will need to be configured to use it after installation*. - -If the `module` function is not defined either you do not have a modules tool installed -or your environment is not properly set up to use it. In this case, -please refer to the EasyBuild documentation here for more information. - -HPE Cray supports both the old-style Environments Modules (based on version 3.2.10) and Lmod. On LUMI, -we use Lmod as the default module system for users, and EasyBuild is configured to use that one. - ---- - -## EasyBuild as a Python package - -EasyBuild consists of a number of interdependent Python packages, -and is available via both GitHub at https://github.com/easybuilders, -as well as via the standard Python Package Index (PyPI) at -https://pypi.org/project/easybuild. - -As you may be aware the Python packaging ecosystem is bit convoluted, -which is reflected in the many different ways in which you can install a Python package. -In addition, EasyBuild is packaged in 3 components (framework, easyblocks, easyconfigs) -which slightly complicates the installation. - -
- -Nevertheless, you don't need to be a rocket scientist to install EasyBuild (and even if you are, -that's OK too), so don't worry. - -You can install EasyBuild just like you can install any other Python software that is released -via the standard *Python Package Index* (PyPI), through one of the standard Python installation tools -(like `pip`, `virtualenv`, `pipenv`, ...). -And since EasyBuild is a software installation tool in its own right, we actually have a couple -of additional tricks up our sleeve! - -### Python 2 or Python 3? - -For EasyBuild it does not matter much whether you install it on top of Python 2 or Python 3. -Since version 4.5, Python 3 does offer some optional features (requiring additional packages) -that are not available with Python 2. -Since Python 2 is end-of-life, -we strongly recommend using Python 3 if you have the choice. - -By default EasyBuild will use the `python` command to run, -but you can control this if needed via ``$EB_PYTHON``. - - -## Installing EasyBuild - -We present three methods for installing EasyBuild. -It is up to you which one you prefer, both result a fully functional EasyBuild installation. - -Time to get your hands dirty! - -* Method 1: Using ``pip`` -* Method 2: installing EasyBuild with EasyBuild -* Method 3: Development setup - ---- - -### Method 1: Using `pip` - -Since EasyBuild is released as a [Python package on PyPI](https://pypi.org/project/easybuild) -you can install it using `pip`, the most commonly used tool for installing Python packages. - -You may need to take additional steps after the installation, depending on the exact installation command. - -!!! note - There are various other ways of installing Python packages, which we won't cover here. - If you are familiar with other tools like `virtualenv` or `pipenv`, feel free to use those - instead to install EasyBuild. - -#### Running `pip install` - -Installing EasyBuild with `pip` is as simple as running the following command: - -```shell -pip install easybuild -``` - -However, you may need to slightly change this command depending on the context and your personal preferences: - -* To install EasyBuild *system-wide*, you can use `sudo` (if you have admin privileges): - ```shell - sudo pip install easybuild - ``` - -* To install EasyBuild *in your personal home directory*, you can use the `--user` option: - ```shell - pip install --user easybuild - ``` - This will result in an EasyBuild installation in `$HOME/.local/`. - -* To install EasyBuild in a *specific directory* you can use the `--prefix` option: - ```shell - pip install --prefix _PREFIX_ easybuild - ``` - In this command, you should replace '`_PREFIX_`' with the location where you want to have EasyBuild installed - (for example, `$HOME/tools` or `/tmp/$USER`). - -#### `pip` vs `pip3` - -On systems where both Python 2 and Python 3 are installed you may also have different `pip` commands -available. Or maybe `pip` is not available at all, and only "versioned" `pip` commands like `pip3` are -available. - -If you (only) have `pip3` available, you can replace `pip` with `pip3` in any of the `pip install` commands -above. - -If you want to ensure that you are using the ``pip`` installation that corresponds to the Python 3 installation -that you intend to use, you can use ``python3 -m pip`` rather than ``pip3``. - -#### Updating your environment - -If you used the `--user` or `--prefix` option in the `pip install` command, -or if you installed EasyBuild with a `pip` version that does not correspond -to your default Python installation, -you will need to update your environment to make EasyBuild ready for use. -This is not required if you did a system-wide installation in a standard location with the default Python version. - -!!! note - Keep in mind that you will have to make these environment changes again if you start a new shell session. - To avoid this, you can update one of the shell startup scripts in your home directory (`.bashrc` for example). - -#### Updating ``$PATH`` - -Update the `$PATH` environment variable to make sure the `eb` command is available: -```shell - -export PATH=_PREFIX_/bin:$PATH -``` -**Replace '`_PREFIX_`' in this command** with the directory path where EasyBuild was installed into -(use `$HOME/.local` if you used `pip install --user`). - -This is not required if you installing EasyBuild in a standard system location. - -You can check with the `which eb` command to determine whether or not you need to update the ``$PATH`` environment variable. - -#### Updating ``$PYTHONPATH`` - -If you installed EasyBuild to a non-standard location using `pip install --prefix`, -you also need to update the Python search path environment variable -[`$PYTHONPATH`](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH) to instruct Python where -it can find the EasyBuild Python packages. - -This is not required if you used the `--user` option, since Python will automatically consider -`$HOME/.local` when searching for installed Python packages, or if you installed EasyBuild in a standard -system-wide location. - -Update `$PYTHONPATH` by running a command like: - -```shell - -export PYTHONPATH=_PREFIX_/lib/pythonX.Y/site-packages:$PYTHONPATH -``` - -Here, you need to replace the `X` and `Y` with the major and minor version of your Python installation, -which you can determine by running `python -V`. -For example, if you are using Python 3.6, make sure you are using `/python3.6/` in the command to update `$PYTHONPATH`. - -And of course, you again need to **replace '`_PREFIX_`'** with the installation prefix where EasyBuild was installed -into. - -For example: - -```shell -# update $PYTHONPATH if EasyBuild was installed in $HOME/tools with Python 3.6 -export PYTHONPATH=$HOME/tools/lib/python3.6/site-packages:$PYTHONPATH -``` - -#### Setting ``$EB_PYTHON`` - -If you want to control which Python version is used to run EasyBuild, -you can specify the name or the full path to the `python` command that should be used by the `eb` command -via the `$EB_PYTHON` environment variable. - -This may be required when you installing EasyBuild with a version of `pip` that does not correspond -with the default Python version. - -For example, to ensure that `eb` uses `python3.6`: - -```shell -export EB_PYTHON=python3.6 -``` - -#### Setting ``$EB_VERBOSE`` - -To determine which `python` commands are being considered by the `eb` command, -you can define the `$EB_VERBOSE` environment variable. For example: - -```shell -$ EB_VERBOSE=1 eb --version ->> Considering 'python3.6'... ->> 'python3' version: 3.6.8, which matches Python 3 version requirement (>= 3.5) ->> Selected Python command: python3 (/usr/bin/python3.6) ->> python3.6 -m easybuild.main --version -This is EasyBuild 4.3.3 (framework: 4.3.3, easyblocks: 4.3.3) on host example -``` - -!!! Tip "Debugging startup problems" - - Using `ÈB_VERBOSE` is useful if EasyBuild fails to start up and complains it - cannot find a suitable Python executable. Rather often the error message is - caused by a failure elsewhere in EasyBuild. - - ---- - -### Method 2: Installing EasyBuild with EasyBuild - -!!! note - This section covers an alternative method for installing EasyBuild. - - If you already have EasyBuild installed through ``pip``, - you can skip ahead to the next section. - -If you prefer having EasyBuild available through an environment module file, -you can consider installing EasyBuild with EasyBuild. This can be done in 3 steps: - -* Step 1: Installing EasyBuild with ``pip`` into a temporary location (only needed if EasyBuild is not installed yet) -* Step 2: Using EasyBuild to install EasyBuild as a module -* Step 3: Loading the EasyBuild module - -!!! note - A [bootstrap script](https://docs.easybuild.io/en/latest/Installation.html#bootstrapping-easybuild) - is available that automates this procedure, - but is known to be problematic in some contexts, and is not being actively - maintained anymore. - - As a result, we do not recommend using the bootstrap script anymore. - - -#### Step 1: Installing EasyBuild into a temporary location - -If you don't have EasyBuild installed yet, you need to install it in a temporary location first. -The recommended way of doing this is using ``pip``. - -For example, to install EasyBuild into a subdirectory `/tmp/$USER` using the default Python 3 version: - -```shell -# pick installation prefix, and install EasyBuild into it -export EB_TMPDIR=/tmp/$USER/eb_tmp -python3 -m pip install --ignore-installed --prefix $EB_TMPDIR easybuild -``` - -```shell -# update environment to use this temporary EasyBuild installation -export PATH=$EB_TMPDIR/bin:$PATH -export PYTHONPATH=$(/bin/ls -rtd -1 $EB_TMPDIR/lib*/python*/site-packages | tail -1):$PYTHONPATH -export EB_PYTHON=python3 -``` - -#### Step 2: Using EasyBuild to install EasyBuild - -Once you have a working (recent) temporary EasyBuild installation, you can use it to -install EasyBuild as a module. Usually this is done in the location where you would -like to install other software too. - -You can use the ``eb --install-latest-eb-release`` command for this, -combined with the ``--prefix`` option to control which directories are used by EasyBuild for the installation. - -For example, to install the latest version of EasyBuild as a module into ``$HOME/easybuild``: - -```shell -eb --install-latest-eb-release --prefix $HOME/easybuild -``` - -!!! note - You may see a harmless deprecation warning popping up when performing this installation, just ignore it. - -#### Step 3: Loading the EasyBuild module - -Once step 2 is completed, you should be able to load the module that was generated alongside -the EasyBuild installation. You will need to do this every time you start a new shell session. - -First, make the module available by running the following command (which will update the module search path -environment variable `$MODULEPATH`): - -```shell - -module use _PREFIX_/modules/all -``` - -**Replace '`_PREFIX_`'** with the path to the directory that you used when running step 2 -(for example, ``$HOME/easybuild``). - -Then, load the `EasyBuild` module to update your environment and make EasyBuild available for use: - -```shell - -module load EasyBuild -``` - -Note that in this case, we don't need to make any changes to our environment for EasyBuild to work correctly. -The environment module file that was generated by EasyBuild specifies all changes that need to be made. - -### Method 3: Development setup - -If you are planning to make changes to EasyBuild, or if you prefer using the latest *bleeding edge* -version of EasyBuild that is being developed, you can consider *cloning* the 3 main EasyBuild repositories -from GitHub, and updating your environment to run EasyBuild from there. - -This can be done as follows (into ``$HOME/easybuild``): - -```shell -mkdir -p $HOME/easybuild -cd $HOME/easybuild -``` - -```shell -# clone EasyBuild repositories from GitHub -git clone https://github.com/easybuilders/easybuild-framework.git -git clone https://github.com/easybuilders/easybuild-easyblocks.git -git clone https://github.com/easybuilders/easybuild-easyconfigs.git -``` -```shell -# update environment for running EasyBuild from there -export PATH=$HOME/easybuild/easybuild-framework:$PATH -export PYTHONPATH=$HOME/easybuild/easybuild-framework:$HOME/easybuild/easybuild-easyblocks:$HOME/easybuild/easybuild-easyconfigs:$PYTHONPATH - -# control which Python command is used to run EasyBuild -export EB_PYTHON=python3 -``` - - -### Approach on LUMI - -*Documentation on the inner workings of the LUMI software stack can be found in the -[LUMI-SoftwareStack GitHub, docs subdirectory](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/docs).* - -To keep the different versions of the LUMI software stack as independent from one another as possible, -EasyBuild is bootstrapped for each software stack by the ``prepare_LUMI_stack.sh`` script. We use the -system Python 3 for running EasyBuild. That Python version currently doesn't have ``pip`` installed. -EasyBuild is first installed in a temporary work directory from files downloaded from PyPi. These -are each untarred and then installed into their location by running -``python3 setup.py install --prefix=...`` in their unpacked directory. -We do so only for the framework and easyblocks files as the easconfig files are not used to -install EasyBuild. Instead we create our own EasyConfig file for EasyBuild which contains -some additional packages that enable extra features in EasyBuild and also provide more information -to Lmod. Next the configuration module for EasyBuild (see the next section, -["Configuring EasyBuild"](1_07_configuration), for more information) and use the temporary -installation of EasyBuild with our own EasyConfig file to do a proper installation of EasyBuild -with module in the final location. - - -TODO: follow development setup but with downloading rather than cloning, then use that one with the -EasyBuild configuration module to install EasyBuild with itself. - - - -## Verifying the installation - -Regardless of how EasyBuild was installed, you can now run a couple of basic commands to verify the installation: - -#### Checking the version - -To check which EasyBuild version you have installed, run: - -```shell - -eb --version -``` - -The output should match with the latest EasyBuild version. - - -#### Consulting the help output - -You can consult the help output of the `eb` command, which produces a long list of available options -along with a short informative message. - -```shell -eb --help -``` - -#### Showing the default EasyBuild configuration - -To inspect the current EasyBuild configuration, you can use this command: - -```shell -eb --show-config -``` - -This should tell you that EasyBuild (ab)uses `$HOME/.local/easybuild` as a default location. -More on configuring EasyBuild in the [next part of the tutorial](configuration). - -#### System information - -You ask EasyBuild to collect and print some information about the -system you are using it on (OS, CPU, Python, etc.) using this command: - -```shell -eb --show-system-info -``` - -## Updating EasyBuild - -Before we wrap up here, a brief word about updating EasyBuild. - -Once you have EasyBuild installed, the easiest way to update to a newer version is by instructing EasyBuild -to install the latest available version as a module: - -``` - -eb --install-latest-eb-release -``` - -This will result in a *new* EasyBuild installation, which is entirely separate from the EasyBuild installation -you are currently using (so it is *not* an in-place update). -The location where this new EasyBuild version will be installed is determined by the active -EasyBuild configuration. - -If you have installed EasyBuild through ``pip``, and you prefer updating that installation, -you can use ``pip install --upgrade easybuild`` (perhaps with additional options like ``--user`` or ``--prefix``). - -!!! Note "EasyBuild upgrade policy on LUMI" - - Even minor version or patch level updates of EasyBuild may break things (and this has happned - in the past on other systems managed previously by LUMI User Support Team members). Hence a very - conservative upgrade policy is used on LUMI. - - In general we fix the version of EasyBuild within a particular version of the LUMI software stack - and only transition to the latest version when starting a new software stack. The reason is that we - want to be able to rebuild an existing software stack with as little unexpected problems as possible. - A move to a newer version of EasyBuild is acceptable should we run into major problems with an existing - version that cannot be solved in an easyconfig file or LUMI-specific custom easyblock, but in those - cases we will first rebuild the whole software stack on a test setup to ensure that no problems are - introduced by upgrading EasyBuild. - - ---- - -## Exercise (optional) - -*This exercise is not needed to proceed the course when working on LUMI as we will -use the installation that is already present.* - -Install EasyBuild in your home directory. - -Make sure that the EasyBuild installation uses the `python3` command to run, -rather than the standard `python` command. - -Choose your own adventure, or try all these installation methods! - -* install EasyBuild with `pip` (or another very similar command...) using either the `--user` or `--prefix` option; -* perform a (manual) "bootstrap" installation into `$HOME/easybuild`, as outlined in installation method 2; -* set up a development installation; - -Check that the installation works by running the verification commands outlined -[above](#verifying-the-installation). - ---- - -**You'll need a working custom EasyBuild installation for some of the -optional generic EasyBuild exercises, but in general you can use the -installation provided by LUMI.** - - -*[[next: Configuring EasyBuild]](1_07_configuration.md)* diff --git a/docs/2022-CSC_and_LO/1_07_configuration.md b/docs/2022-CSC_and_LO/1_07_configuration.md deleted file mode 100644 index a1c2f3de..00000000 --- a/docs/2022-CSC_and_LO/1_07_configuration.md +++ /dev/null @@ -1,610 +0,0 @@ -# Configuring EasyBuild - -*[[back: Installation]](1_06_installation.md)* - ---- - -After installing EasyBuild, you should **configure** it. - -EasyBuild should work fine out-of-the-box if you use Lmod as your modules tool. -If you are not using Lmod, please see here for more information. - -Nevertheless, we strongly recommend you to inspect the default configuration, -and to configure EasyBuild according to your preferences and the system on which you will use it. - -!!! Note "EasyBuild configuration on LUMI" - - On LUMI serveral configurations of EasyBuild are already available. - - The ``EasyBuild-user`` module is the most important one. It will configure EasyBuild - to install software in either a default location in the user's home directory - (``$HOME/EasyBuild``) or the location pointed to by the environment variable - ``EBU_USER_PREFIX``. The latter can be used to install software in the project directory - so that it is available to all users of the project. It enables the creation of a custom - software environment build on top of the LUMI software stack. Software in this environment is - build in exactly the same way as it would if it were installed in the central repository, but - one can now also easily create custom configurations without confusing other projects with - different or non-standard configurations of software. - - There are two more admin-only configuration modules for EasyBuild. The main one is - ``EasyBuild-production`` which is used for software installations in the central software stack. - The other one is ``EasyBuild-infrastructure`` which is used to install infrastructure modules - that need to be installed in a pure Lmod hierarchy, e.g., the modules that load the toolchains. - - ---- - -## Available configuration settings - -One of the central policies in the EasyBuild project is to avoid hardcoded settings in the codebase. -While this significantly increases the ability to configure EasyBuild to your liking, it also results -in a large amount of available configuration settings. - -The full list of configuration settings can be consulted via `eb --help`, -which shows the corresponding command line option accompanied by a short description. -At the time of writing, *over 240 different configuration settings* are supported by EasyBuild. - -For the sake of this tutorial we will focus on a specific subset of configuration settings, -and cover only the most prominent and important ones. - -We will refer to EasyBuild configuration settings using the names as they appears in the output of `eb --help`, -and omit the leading dashes (`--`) for the sake of clarity. - -Keep in mind that *every* configuration setting can be defined in 3 different ways, -see below for more details. - ---- - -### Overall prefix - -*(default: `HOME/.local/easybuild`)* - -The `prefix` configuration setting specifies the overall prefix that EasyBuild should use, -which **determines the default value for various other configuration settings**: - -* `installpath`: `` -* `buildpath`: `/build` -* `sourcepath`: `/sources` -* `repositorypath` (easyconfigs archive): `/ebfiles_repo` -* `containerpath`: `/containers` - -Here, `` represents the value of the `prefix` configuration setting. - -If one of the configuration settings affected by `prefix` is defined specifically, -the `prefix` value becomes irrelevant for that specific configuration setting. - ---- - -### Install path - -*(default: ``)* - -The **location for both the software installation directories and generated module files** -can be controlled via the `installpath` configuration setting. -Software installation directories will be placed in `/software`, while -`/modules/all` will be used for generated module files. - -The `installpath` location is usually set to a directory on a *shared filesystem* when installing -software for an HPC cluster. Of course, software can also be installed on a local filesystem, -which is particularly useful to test and evaluate software installations. - -Separate configuration settings are available for both software and modules locations, -as well as for controlling the name of the `software` and `modules/all` subdirectories. - -*The EasyBuild community recommends to only change the `installpath` configuration setting to control the location -of software installations and accompanying module files, -such that the software and modules directories are located in the same parent directory, -and the default `software` and `modules/all` names for the subdirectories are used.* - -!!! Note "Path for software and for modules on LUMI" - - On LUMI we do not follow that recommendation. There is a three for the software installations - themselves with subdirectories based on the version of the software stack and LUMI hardware - partition, and a separate tree for the modules organised in a similar way. - - This makes it slightly easier to organise the module tree with user-friendly labeling, but above - all also makes the synchronisation process of the 4 instances of the software directory more robust - as it is now easy to synchonise all modules in the last step, which is a much quicker process than - syncrhonising the software installations. - - We also use short paths for software installations (to avoid overrunning the maximum length of a - shebang line in scripts) while we use longer, more descriptive names for subdirectories in the - module tree. - - ---- - -### Build path - -*(default: `/build`)* - -For each installation it performs, EasyBuild creates a **separate build directory** where software will be compiled -before installing it. This directory is cleaned up automatically when the installation is successfully completed. -To control the location where these build directories are created, you can use the `buildpath` configuration setting. - -Keep in mind that build directories may grow out to several GBs in size during an installation, -and that the commands that run in there can be fairly I/O-intensive since they may involve -manipulating lots of small files. In addition, a build directory that corresponds to a failing installation -is *not* cleaned up automatically, but it will be cleaned up and recycled when the same installation is re-attempted. -Running out of disk space in the location where build directories are created will result in failing -installations. - -It is strongly recommend to use the path to a directory on a *local filesystem* for the value of the -`buildpath` configuration setting, since using a shared filesystem like Lustre or GPFS is known to cause -problems when building certain software packages. Using an in-memory location (like `/dev/shm/$USER`) can -significantly speed up the build process, but may also lead to problems (due to space limitations, -or specific mount options like `noexec`). - -!!! Note "buildpath on LUMI" - - The configuration modules on LUMI will use a RAM disk for the build path. On the login nodes, - ``$XDG_RUNTIME_DIR`` is used as that space is automatically cleared when the last session of a user - ends. However, on the compute nodes a job- or user-specific subdirectory of ``/dev/shm`` is currently used - as ``$XDG_RUNTIME_DIR`` does not exist. - - ---- - -### Source path - -*(default: `/sources`)* - -For most supported software, EasyBuild can **automatically download the source files** required for the installation. -Before trying to download a source file, EasyBuild will first check if it is already present in the source path. - -The locations considered by EasyBuild when checking for available source files, as well as the location to -store downloaded source files, can be controlled via the ``sourcepath`` configuration setting. - -The `sourcepath` value is a colon (`:`) separated list of directory paths. -Each of these paths will be considered in turn when checking for available source files, -until one of them provides the desired source file. Searching for source files is done -based on filename, and a couple of subdirectories are considered. -For example, for a software package named '`Example'`, EasyBuild will consider locations -like `/e/Example/`, `/Example/`, and so on. - -The first path listed in `sourcepath` is the location where EasyBuild will store downloaded source files, -organised by software name through subdirectories, so EasyBuild expects to have write permissions to this path. -For the other paths listed in `sourcepath` only read permissions are required. - -*Make sure you have write permissions to the first path listed in `sourcepath`, so EasyBuild is able -to store downloaded files there. Feel free to list additional paths if you already have a cache of downloaded -files available somewhere.* - -*Storing the downloaded files not only reduces the amount of downloads while developing new easyconfig files -but also greatly helps when recompiling a software stack, as it is not that uncommon that download sites change -of files become unavailable.* - - ---- - -### Easyconfigs archive - -*(default: `/ebfiles_repo`)* - -EasyBuild keeps track of the easyconfig files that were used for installations in the easyconfigs -archive, the location of which is specified by the `repositorypath` configuration setting. - -By default the specified path is assumed to be a regular directory, but using a Git repository -as easyconfigs archive is also supported (for more details, see -[the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Configuration.html#easyconfigs-repository-repository-repositorypath)). - - ---- - -### Modules tool & module syntax - -*(default: `Lmod` as modules tool, and `Lua` as module syntax)* - -By default, EasyBuild assumes you are using **[Lmod](https://lmod.readthedocs.io) as modules tool**. -In addition, it will generate module files in [Lua](https://www.lua.org/) syntax, as supported by Lmod -(next to `Tcl` syntax). - -To diverge from this, you can define the `modules-tool` configuration setting to indicate you -are using a different modules tool; see the output of `eb --avail-modules-tools` for a list of supported -modules tools. -Note that for anything other than Lmod, you *must* make -sure that the actual modules tool binary command is available through `$PATH` (more information -on this [in the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Configuration.html#modules-tool-modules-tool)). - -If you prefer that EasyBuild generates module files in [Tcl](https://tcl.tk/) syntax, you can instruct -it to do so via the `module-syntax` configuration setting. Note that this is *required* if you are using -a modules tool other than Lmod, since only Lmod supports module files in `Lua` syntax. - -*We recommend using EasyBuild with the defaults settings: Lmod as modules tool, and Lua as module syntax.* - ---- - -### Robot search path - -*(default: `robot-paths` specifies the location to the easyconfig files included with EasyBuild installation; -dependency resolution is* ***not*** *enabled)* - -When EasyBuild needs to locate one or more easyconfig files, it will do so via the **robot search path**. -This applies to both easyconfig files that were specified using only their filename as an argument to the -`eb` command, as well as to easyconfigs required to resolve dependencies (more on that later). - -To control the robot search path, two configuration settings are available: `robot` and `robot-paths`. -Both accept a colon-separated list of locations to consider when looking for easyconfig files, -with `robot` having a higher priority than `robot-paths` (i.e., the directories specified with `robot` -appear before those specified by `robot-paths` in the robot search path). - -The key difference between these two configuration settings is that defining `robot` also enables dependency resolution, -and hence makes EasyBuild install missing dependencies, alongside -specifying a list of paths to consider when searching for easyconfig files. -On the other hand, defining `robot-paths` does not have the side effect of enabling dependency resolution. - -In addition, you can use the `--robot` command line option without specifying any paths to it, -to only enable dependency resolution. ***This option is needed on LUMI if you want automatic -dependency resolution as it is disabled on LUMI to prevent accidental mistakes when installing -software.*** - -!!! note "Easyconfig files included with EasyBuild and the robot search path" - Keep in mind that when either of the `robot` or `robot-paths` configuration settings are - defined, the default value corresponding to the location of easyconfigs included with the - EasyBuild is *no longer considered*. - - There are ways around this however, which are outside of the scope of this tutorial. - - For more information, - [see the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Using_the_EasyBuild_command_line.html#controlling-robot-search-path). - - *On LUMI the included easyconfig files cannot be used without changes as they are for the - common toolchains which are currently not supported in the LUMI software stacks.* - - -!!! Note "Easyconfigs archive on LUMI and the robot search path" - - On LUMI we also include the easyconfigs archive at the front of the search path for easyconfig - files. This has the advantage that EasyBuild when checking dependencies will always find the - configuration that is actually installed on the system, even if the easyconfig file in one of - the regular repositories has changed. It also has the disadvantage that one may accidentally - be re-installing with the already installed configuration while one really wants to re-install - the module in a different configuration. - - - ---- - -### Module naming scheme - -*(default: `EasyBuildMNS`)* - -EasyBuild will use a specific naming scheme for the module files it generates for each of the software installations. -This ensures consistency, regardless of who uses EasyBuild to perform the installation. - -Different types of module naming schemes are supported (flat, hierarchical, ...) and you can provide an implementation of -your own custom module naming scheme if desired. A number of different naming schemes are included -with EasyBuild, which you can consult via `eb --avail-module-naming-schemes`. - -The default `EasyBuildMNS` module naming scheme roughly corresponds to the filename of easyconfig files, -and consists of the software name followed by a combination of the software version, toolchain and -an optional label (which corresponds to the value of the `versionsuffix` easyconfig parameter): -`/<-toolchain>`. Just like with [names of easyconfig -files](1_05_terminology#easyconfig-files), the `<-toolchain>` part is omitted when the `system` toolchain is used, -and the `` value is empty by default. - -!!! Note "Module naming scheme on LUMI" - - On LUMI we use a custom variant of the standard EasyBuild flat naming scheme. The EasyBuild default - naming scheme creates an ``all`` subdirectory in the module directory containing all modules, but also - a directory per category, with the categories determined from the ``moduleclass`` easyconfig parameter. - However, that choice is often rather arbitrary as modules can belong to different categories depending - on who you ask. E.g., GROMACS is used in physics, chemistry and life sciences while EasyBuild can only - assign a single category. - - Hence we do not generate the categories and drop the ``all`` level in the module subdirectory. - - ---- - -## Configuration levels - -Configuring EasyBuild can be done in different ways: - -* through one or more configuration files; -* via `$EASYBUILD_*` environment variables; -* using `eb` command line options; - -Each of the methods corresponds to a *configuration level*. - -*Every* configuration setting can be defined via one of these -mechanisms, without exception! - ---- - -### Configuration level hierarchy - -There is a strict **hierarchy between the different configuration levels** supported by EasyBuild. - -Settings defined via a *configuration file* only override default values. - -A configuration setting that is defined via the corresponding *`$EASYBUILD_*` environment variable* -takes precedence over the value specified in a configuration file (if any). - -Finally, values specified through `eb` command line options **always** win, -regardless of whether the corresponding -configuration setting was already defined some other way, be it via a configuration file or -an environment variable. - -For example, let us consider a fictional configuration setting named `magic`: - -* If a value for `magic` is specified in an EasyBuild configuration file, - then this value will only be used if the corresponding environment variable - (`$EASYBUILD_MAGIC`) is *not* defined **and** if the `--magic` command line option is *not* used; -* If the `$EASYBUILD_MAGIC` environment *is* defined however, then its value - will be used for the `this-is-magic` configuration setting; -* *Unless* the `--magic` command line option is used, - since in that case the value provided as an argument there will be used instead. - ---- - -### Configuration files - -*Configuration files* are the most basic way of configuring EasyBuild. -Two types of are supported by EasyBuild: *user-level* and *system-level* -configuration files. The output of `eb --show-default-configfiles` tells you which locations -are considered, and whether any configuration files were found. - -EasyBuild configuration files are written in the standard [INI format](https://en.wikipedia.org/wiki/INI_file), -and the configuration settings are grouped into different *sections*. - -To create an EasyBuild configuration file, the output of `eb --confighelp` is very useful: -it produces the list of all supported configuration settings which are readily grouped in sections -and with every entry commented out, along with accompanying descriptive comments mentioning the default values, -and in the correct syntax. - -*Configuration files are the recommended mechanism to define configuration settings -that should always be in place, regardless of the software you are installing.* - -#### EasyBuild configuration files vs easyconfig files - -EasyBuild configuration files are sometimes confused with easyconfig files, -due to the similar name. However, these are two entirely different concepts! - -EasyBuild configuration files (usually `*.cfg`) are a way of *configuring the general behaviour of EasyBuild* -across different software installations. They define *configuration settings*, -such as the location where software should be installed, or the syntax that should -be used when generating module files. - -An [easyconfig file](1_05_terminology#easyconfig-files) (`*.eb`) on the other hand *specifies the details for one particular software installation*. -It does this by defining a set of *easyconfig parameters*, which tell EasyBuild the name and version -of the software to install, which [toolchain](1_05_terminology#toolchains) and [easyblock](1_05_terminology#easyblocks) to use, etc. - -For each software installation performed by EasyBuild, there is a corresponding easyconfig file. -There typically are only a handful of configuration files used however, for example a system-level -configuration file, perhaps combined with a user-level one. Or there may be no configuration files involved -at all, since EasyBuild can also be configured through other mechanisms: environment variables and command line -options. - ---- - -### `$EASYBUILD_*` environment variables - -A particularly easy way to configure EasyBuild is through *environment variables*. - -At startup, EasyBuild will pick up any environment variable of which the name starts with '`EASYBUILD_`'. -For each of these, it will determine the corresponding configuration setting (or exit with an error if -none was found). - -Mapping the name of a configuration setting to the name of the corresponding environment variable is straightforward: -use capital letters, replace dashes (`-`) with underscores (`_`), and prefix with `EASYBUILD_`. - -For example: the `module-syntax` configuration setting can be specified by defining -the `$EASYBUILD_MODULE_SYNTAX` environment variable: - -```shell -export EASYBUILD_MODULE_SYNTAX=Tcl -``` - -*Configuring via environment variables is especially practical for controlling the EasyBuild configuration -in a more dynamic way. For example, you can implement a simple shell script that defines `$EASYBUILD_*` -environment variables based on the current context (user, hostname, other environment variables), and -configure EasyBuild through [sourcing](https://bash.cyberciti.biz/guide/Source_command) it.* - -!!! note - Keep in mind that environment variables are only defined for the shell session you are currently working in. - If you want to configure EasyBuild through environment variables in a more persistent way, - you can leverage one of the [shell startup scripts](https://bash.cyberciti.biz/guide/Startup_scripts) (for example `$HOME/.bash_profile` or `$HOME/.bashrc`). - ---- - -### `eb` command line options - -Finally, you can also configure EasyBuild by specifying one or options to the `eb` command. - -As mentioned earlier, the values for configuration settings defined this way override the value that -is specified through any other means. So if you want to be sure that a particular configuration setting -is defined the way you want it to be, you can use the corresponding command line option. - -There are various configuration settings for which it only makes sense to use the command line option. -An example of this is letting the `eb` command print the EasyBuild version (via `eb --version`). -Although you could configure EasyBuild to always print its version and then exit whenever the `eb` command is -run, that would not be very useful... - -*Command line options are typically used to define configuration settings that are only relevant to -that particular EasyBuild session. One example is doing a test installation into a temporary directory:* - -```shell -eb --installpath /tmp/$USER example.eb -``` - -## Inspecting the current configuration (`--show-config`) - -Given the large amount of available configuration settings in EasyBuild and the different configuration levels, -you can easily lose track of exactly how EasyBuild is configured. - -Through the `--show-config` command line option you can -easily inspect the currently active EasyBuild configuration. - -The output of `--show-config` includes a sorted list of all configuration settings that are defined to a -*non-default* value, -along with a couple of important ones that are always shown (like `buildpath`, `installpath`, `sourcepath`, and so on). -In addition, it also indicates at which configuration level each setting was defined, -so you can trace down *where* it was defined if needed. - -This is the output produced by `eb --show-config` for the default EasyBuild configuration, -where EasyBuild was installed via `pip install --user` (which results in the value shown for the -`robot-paths` configuration setting): - -```shell -# -# Current EasyBuild configuration -# (C: command line argument, D: default value, E: environment variable, F: configuration file) -# -buildpath (D) = /home/example/.local/easybuild/build -containerpath (D) = /home/example/.local/easybuild/containers -installpath (D) = /home/example/.local/easybuild -repositorypath (D) = /home/example/.local/easybuild/ebfiles_repo -robot-paths (D) = /home/example/.local/easybuild/easyconfigs -sourcepath (D) = /home/example/.local/easybuild/sources -``` - -As shown here, all configuration settings shown follow the default `prefix` value (`$HOME/.local/easybuild`), -and none of the values diverge from the default value, since all entries are marked with `(D)` for "default value"). - -**Example** - -Now let us do some basic configuring and inspect the resulting output of `--show-config`. - -First, create a user-level EasyBuild configuration file to define the `prefix` configuration setting: - -```shell -mkdir -p $HOME/.config/easybuild -echo '[config]' > $HOME/.config/easybuild/config.cfg -echo 'prefix=/apps' >> $HOME/.config/easybuild/config.cfg -``` - -In addition, define the `buildpath` configuration setting using the corresponding -environment variable: - -```shell -export EASYBUILD_BUILDPATH=/tmp/$USER -``` - -Then run `--show-config` while you specify that the `installpath` configuration -setting should be defined as `/tmp/$USER`: - -```shell -$ eb --installpath=/tmp/$USER --show-config -# -# Current EasyBuild configuration -# (C: command line argument, D: default value, E: environment variable, F: configuration file) -# -buildpath (E) = /tmp/easybuild -containerpath (F) = /apps/containers -installpath (C) = /tmp/easybuild -packagepath (F) = /apps/packages -prefix (F) = /apps -repositorypath (F) = /apps/ebfiles_repo -robot-paths (D) = /home/example/.local/easybuild/easyconfigs -sourcepath (F) = /apps/sources -``` - -The output indicates that the `installpath` setting was specified through a command line option (indicated -with `(C)`), that the `buildpath` setting was defined via an environment -variable (indicated with `(E)`), that the `robot-paths` setting still has the default value (indicated with `(D)`), and that all other configuration -settings were specified via a configuration file, some of which indirectly through the `prefix` value (indicated with -`(F)`). - ---- - -## Exercises (optional) - -*These exercises are not very relevant for LUMI as LUMI already offers a complete configuration -also for user installations of software. However, if you are a very advanced user, you may still -want to make changes to that configuration, and all three options (configuration files, -environment variables and command line parameters) are available to users in the LUMI setup.* - -***Exercise C.1* - Configure EasyBuild** - -Configure EasyBuild to use the `easybuild` subdirectory in your home directory for everything, except for: - -* The location of the build directories: use `/tmp/$USER` for this; -* The locations that should be considered when searching for source files: - include both `$HOME/easybuild/sources` and `/easybuild/sources`, but make - sure that source files that are downloaded by EasyBuild are stored in - `$HOME/easybuild/sources`. - -Leave other configuration settings set to their default value. - -??? success "(click to show solution)" - - This is pretty straightforward. - - Here we just define the corresponding environment variables: - - ```shell - export EASYBUILD_PREFIX=$HOME/easybuild - export EASYBUILD_BUILDPATH=/tmp/$USER - export EASYBUILD_SOURCEPATH=$HOME/easybuild/sources:/easybuild/sources - ``` - - The location where EasyBuild should download source files to - must be listed first in the `sourcepath` configuration setting. - - The output of `--show-config` should look like this: - - ```shell - buildpath (E) = /tmp/example - containerpath (E) = /home/example/easybuild/containers - installpath (E) = /home/example/easybuild - packagepath (E) = /home/example/easybuild/packages - prefix (E) = /home/example/easybuild - repositorypath (E) = /home/example/easybuild/ebfiles_repo - robot-paths (D) = /home/example/easybuild/easyconfigs - sourcepath (E) = /home/example/easybuild/sources:/easybuild/sources - ``` - ---- - -***Exercise C.2* - Install a trivial software package with EasyBuild** - -Try running the following command: - -```shell -eb bzip2-1.0.6.eb -``` - -Where do you expect to find the installation? - -??? success "(click to show solution)" - - The software was installed in `$HOME/easybuild`, - since that's how we configured EasyBuild in *Exercise 3.1*: - - ```shell - $ ls $HOME/easybuild - ebfiles_repo modules software sources - ``` - - The actual installation is in `$HOME/easybuild/software`, - while the module file was generated in `$HOME/easybuild/modules/all`: - - ```shell - $ ls $HOME/easybuild/software - bzip2 - $ ls $HOME/easybuild/software/bzip2 - 1.0.6 - $ ls $HOME/easybuild/software/bzip2/1.0.6 - bin easybuild include lib man - ``` - - ```shell - $ ls $HOME/easybuild/modules/all - bzip2 - $ ls $HOME/easybuild/modules/all/bzip2 - 1.0.6.lua - ``` - - The source file for bzip2 1.0.6 was downloaded to `$HOME/easybuild/sources`: - - ```shell - $ ls $HOME/easybuild/sources/b/bzip2 - bzip2-1.0.6.tar.gz - ``` - - We will discuss this in more detail in the next part of the tutorial. - ---- - -*[[next: Basic usage]](1_08_basic_usage.md)* diff --git a/docs/2022-CSC_and_LO/1_08_basic_usage.md b/docs/2022-CSC_and_LO/1_08_basic_usage.md deleted file mode 100644 index f5031e48..00000000 --- a/docs/2022-CSC_and_LO/1_08_basic_usage.md +++ /dev/null @@ -1,1216 +0,0 @@ -# Basic usage - -*[[back: Configuring EasyBuild]](1_07_configuration.md)* - ---- - -Now that we have installed and configured EasyBuild, we can start using it for what it is intended for: -getting scientific software installed without breaking a sweat, or having to resist the urge to -shout out four-letter words. - -We will look at the high-level workflow first, and then cover each aspect in more detail. - -A couple of exercises are available at the end to help to make you more familiar with the -EasyBuild command line interface, so pay attention! - ---- - -## Overall workflow - -Installing software with EasyBuild is as easy as specifying to the **`eb` command** what we -want to install, and then sitting back to enjoy a coffee or tea (or whatever beverage you prefer). - -This is typically done by **specifying the name of one or more easyconfig files**, often combined -with the `--robot` (or `-r`) option to let EasyBuild also install missing dependencies. - -It is recommended to first assess the current situation before letting EasyBuild install the software, -and to check which **dependencies** are already installed and which are still missing. In addition, -you may want to inspect the specifics of the **installation procedure** that will be performed by EasyBuild, -and ensure that the active EasyBuild configuration is what it should be. - ---- - -## Specifying easyconfigs - -Letting EasyBuild know what should be installed can be done by specifying one or more easyconfig files, -which is also the most common way. Alternative methods like using the `--software-name` option won't be -covered in this tutorial, since they are not commonly used. - -Arguments passed to the `eb` command, being anything that is *not* an option (which starts with `-` or `--`) or -is a value for a preceding configuration option, are assumed to refer to easyconfig files (with some exceptions). -These could be: - -* the *(absolute or relative) path* to an easyconfig file; -* the *name* of an easyconfig file; -* the path to a *directory* containing easyconfig files; - -Specified paths to files must of course point to existing files; if not, EasyBuild will print an appropriate error message: - -```shell -$ eb /tmp/does_not_exist.eb -ERROR: Can't find path /tmp/does_not_exist.eb -``` - -When only the *name* of an easyconfig file is specified, EasyBuild will automatically try and locate it. -First, it will consider the *current directory*. If no file with the specified name is found there, -EasyBuild will search for the easyconfig file in the [robot search path](../1_07_configuration/#robot-search-path). - -If the path to an existing *directory* is provided, EasyBuild will walk through the entire directory -(including all subdirectories), retain all files of which the name ends with `.eb`, and (try to) use these -as easyconfig files. - - -#### Example command - -Suppose we have the current situation in our home directory: - -* two (easyconfig) files named `example1.eb` and `example2.eb`; -* a subdirectory named `some_deps`, which has two easyconfig files `dep1.eb` and `dep2.eb` - alongside a text file named `list.txt`; -* a subdirectory named `more_deps` located *in* the `some_deps` subdirectory, - which contains another easyconfig file `dep3.eb`; - -Or, visually represented: - -```shell -example1.eb -example2.eb -some_deps -├── deb2.eb -├── dep1.eb -├── list.txt -└── more_deps - └── dep3.eb -``` - -In this context, we run the following EasyBuild command from our home directory: - -```shell -eb example1.eb bzip2-1.0.6.eb $HOME/example2.eb some_deps -``` - -EasyBuild will interpret each of these arguments as follows: - -* `example1.eb` is the name of a file in the current directory, so it can be used directly; -* `bzip2-1.0.6.eb` is the name of an easyconfig file to locate via the robot search path - (since it does not exist in the current directory); -* `$HOME/example2.eb` specifies the full path to an existing file, which can be used directly; -* `some_deps` is the relative path to an existing directory, so EasyBuild will scan it and find three - easyconfig files: `some_deps/dep1.eb`, `some_deps/dep2.eb` and `some_deps/more_deps/dep3.eb`, - The `list.txt` file will be ignored since its name does not end with `.eb`. - ---- - -### Easyconfig filenames - -Note that the `eb` command does not care how easyconfig files are named, at least to some extent: the `.eb` -file extension *does* matter when easyconfig files are being picked up in subdirectories. - -File names for easyconfigs being mostly irrelevant is only correct with respect to the arguments passed to -the `eb` command however. As we will learn soon, the name of easyconfig files *does* matter (a lot) -when EasyBuild needs to locate easyconfigs that can be used to resolve a specified dependency -(see [here](#enabling-dependency-resolution)). - -This explains why easyconfig files usually adher to a very specific naming scheme, -corresponding to `--.eb`, where: - -* `` represents the software name; -* `` represents the software version; -* `` represents the toolchain used in the easyconfig file, which consists of the toolchain name - and version separated with a dash (`-`), and which is omitted (including the preceding `-`) when the - [`system` toolchain](../1_05_terminology#system-toolchain) is used; -* `` represents the value of the `versionsuffix` easyconfig parameter, - which is sometimes used to distinguish multiple variants of particular software installations - (and is empty by default); - ---- - -### Searching for easyconfigs - -You will frequently need to determine the exact name of an easyconfig file you want to install, -or just check which easyconfigs are available for a given software package. -This can be done by searching for easyconfigs using **`eb --search`** or **`eb -S`**. - -By default all directories listed in the [robot search path](../1_07_configuration#robot-search-path) will be -searched. If you want to search in additional directories without changing the robot search path, -you can use the `search-paths` configuration setting, or you can change the robot search path via either -the `robot` or `robot-paths` configuration options. - -Both the `--search` and `-S` options trigger the same search operation, but yield different output: -`eb --search` will print the full path to each easyconfig file that matches the specified search pattern, -while `eb -S` produces a more concise output. - -For example, let's check which easyconfig files are available for OpenFOAM 8 with a `foss` toolchain: - -```shell -$ eb --search openfoam-8-foss - * /home/example/.local/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-8-foss-2020a.eb - * /home/example/.local/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-8-foss-2020b.eb -``` - -The output is a bit more condensed when using `eb -S`: - -```shell -$ eb -S openfoam-8-foss -CFGS1=/home/example/.local/easybuild/easyconfigs/o/OpenFOAM - * $CFGS1/OpenFOAM-8-foss-2020a.eb - * $CFGS1/OpenFOAM-8-foss-2020b.eb -``` - -Note that the search is performed *case-insensitive*. - -The search pattern can include wildcards like `.*` and/or character groups like `[0-9]`, -or other special characters like `^` or `$` to mark the start/end of the filename, -but you need to be careful that `bash` does not expand these before the `eb` command is started, -so it is recommended to wrap the search pattern in single quotes (`'...'`) when using wildcards. - -For example, to check which easyconfigs are available to install TensorFlow 2.4.1 with the `2020b` version of a toolchain: - -```shell -$ eb -S '^tensorflow-2.4.1.*2020b' -CFGS1=/home/example/.local/easybuild/easyconfigs/t/TensorFlow - * $CFGS1/TensorFlow-2.4.1-foss-2020b.eb - * $CFGS1/TensorFlow-2.4.1-fosscuda-2020b.eb -``` - ---- - -#### Search index - -When searching for easyconfig files, you may see a message like this pop up: - -``` -== found valid index for , so using it... -``` - -This indicates that a **search index** was used for this particular directory which significantly speeds -up the search procedure, especially when the easyconfig files are located on a shared filesystem where metadata -operations involving lots of (small) files can be slow. - -For the easyconfig files included with an EasyBuild release, a search index is readily provided. -For other directories, you can create a search index using `eb --create-index `. -[See the EasyBuild documentation for more information](https://easybuild.readthedocs.io/en/latest/Easyconfigs_index.html). - ---- - -## Inspecting easyconfigs - -Once you have determined the name of the easyconfig file that corresponds to the software you want to install, -you may want to take a closer look at its contents before employing it. - -Since easyconfig files are simple text files (in Python syntax), you could use the ubiquitous `cat` command -or your favorite text editor (`vim`, what else). To avoid that you need to locate the easyconfig file first -and copy-paste the full path to it, you can use **`eb --show-ec`**. - -!!! Hint - To follow the examples below on LUMI, load ``LUMI/21.12`` and ``EasyBuild-user`` - (though results may differ or the examples not work anymore as the software installation - on LUMI evolves). - -For example, let's inspect the contents of the `bzip2-1.0.6.eb` easyconfig file: - -```shell -$ eb --show-ec bzip2-1.0.8-cpeCray-21.12.eb -== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-53o823qb/easybuild-xn6nmt61.log -== Contents of /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeCray-21.12.eb: -# Contributed by Kurt Lust, LUMI project & UAntwerpen - -local_bzip2_version = '1.0.8' # http://www.bzip.org/downloads.html - -name = 'bzip2' -version = local_bzip2_version - -homepage = 'https://www.sourceware.org/bzip2/' - -whatis = [ - 'Description: bzip2 is a freely available, patent free, high-quality data compressor.', - 'The module contains both executables and libraries.' - 'Keywords: BZ2', -] - -description = """ -bzip2 is a freely available, patent free, high-quality data compressor. It -typically compresses files to within 10% to 15% of the best available techniques -(the PPM family of statistical compressors), whilst being around twice as fast -at compression and six times faster at decompression. It is based on the -Burrows-Wheeler block-sorting text compression algorithm and Huffman coding. -""" - -usage = """ -Check the man pages for the available commands or the web-based documentation for the -library functions. -""" - -docurls = [ - 'Web-based documentation: http://www.bzip.org/docs.html', - 'Man pages available for bzcmp, bzdiff, bzegrep, bzfgrep, bzgrep, bzip2, bunzip2, bzless and bzmore', -] - -toolchain = {'name': 'cpeCray', 'version': '21.12'} -toolchainopts = {'pic': True} - -source_urls = ['https://sourceware.org/pub/%(name)s/'] -sources = [SOURCE_TAR_GZ] -patches = ['bzip2-%(version)s-pkgconfig-manpath.patch'] -checksums = [ - 'ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269', # bzip2-1.0.8.tar.gz - 'de11269dc6e4917023af4cee9ff83b204953ad9cde561dbc9d6fc70d9b9578e3', # bzip2-1.0.8-pkgconfig-manpath.patch -] - -builddependencies = [ # Create a reproducible build environment. - ('buildtools', '%(toolchain_version)s', '', True), -] - -local_bzip2_major_minor = '.'.join(local_bzip2_version.split('.')[:2]) - -sanity_check_paths = { - 'files': [ 'lib/pkgconfig/bzip2.pc', 'lib/libbz2.a', 'lib/libbz2.%s' % SHLIB_EXT, - 'lib/libbz2.%s.%s' % (SHLIB_EXT, local_bzip2_major_minor), - 'lib/libbz2.%s.%s' % (SHLIB_EXT, local_bzip2_version), - 'include/bzlib.h' ] + - [ 'bin/b%s' % x for x in ['unzip2', 'zcat', 'zdiff', 'zgrep', 'zip2', 'zip2recover', 'zmore'] ] + - [ 'share/man/man1/bz%s.1' % x for x in ['cmp', 'diff', 'egrep', 'fgrep', 'grep', 'ip2', 'less', 'more'] ], - 'dirs': [] -} - -sanity_check_commands = [ - 'bzip2 --help', - 'pkg-config --libs bzip2', -] - -moduleclass = 'tools' - -== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-53o823qb/easybuild-xn6nmt61.log* have been removed. -== Temporary directory /run/user/10012026/easybuild/tmp/eb-53o823qb has been removed. -``` -The output may actually be longer for an easyconfig file that is already installed on the system -as a new easyconfig file is generated in the repository with some information about the installation -added to it and as on LUMI these are at the front of the robot search path to ensure that the system -finds the right easyconfig file matching with a module on the system. - -We'll get back to what all of this means later... - ---- - -## Checking dependencies - -!!! note - In some of the examples below, we assume that some software is already installed with EasyBuild. - - If you are following hands-on in a prepared environment, make sure these installations are visible - in the output of "`module avail`". - - -Before kicking off an installation, it is good practice to check which of the required dependencies -are already installed, and which ones are still missing. - -This can be helpful to ensure that your EasyBuild configuration is set up correctly, -and to prevent from accidentally installing an entirely new software stack from scratch -in an unintended location (like `$HOME/.local/easybuild`, the -[default installation path](../configuration/#install-path)). - ---- - -### Dry run - -To get a complete overview of all required dependencies, and see which ones are already installed -and which ones aren't yet, you can use **`eb --dry-run`**. - -Since `--dry-run` produces rather verbose output including the full path to each easyconfig file, -there is a more concise equivalent available as well: `eb --dry-run-short`, which is equivalent with **`eb -D`**. - -For example, to check which of the required dependencies for `SAMtools-1.11-GCC-10.2.0.eb` are already installed: - -```shell -$ eb SAMtools-1.14-cpeGNU-21.12.eb -D -== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-oo0lj9lq/easybuild-2cyomy8v.log -Dry run: printing build status of easyconfigs and dependencies -CFGS=/appl/lumi - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeGNU/cpeGNU-21.12.eb (module: cpeGNU/21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ncurses/ncurses-6.2-cpeGNU-21.12.eb (module: ncurses/6.2-cpeGNU-21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/zlib/zlib-1.2.11-cpeGNU-21.12.eb (module: zlib/1.2.11-cpeGNU-21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeGNU-21.12.eb (module: bzip2/1.0.8-cpeGNU-21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12-minimal.eb (module: gettext/0.21-cpeGNU-21.12-minimal) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/XZ/XZ-5.2.5-cpeGNU-21.12.eb (module: XZ/5.2.5-cpeGNU-21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/Brotli/Brotli-1.0.9-cpeGNU-21.12.eb (module: Brotli/1.0.9-cpeGNU-21.12) - * [x] $CFGS/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cURL/cURL-7.78.0-cpeGNU-21.12.eb (module: cURL/7.78.0-cpeGNU-21.12) - * [ ] $CFGS/LUMI-EasyBuild-contrib/easybuild/easyconfigs/h/HTSlib/HTSlib-1.14-cpeGNU-21.12.eb (module: HTSlib/1.14-cpeGNU-21.12) - * [ ] $CFGS/LUMI-EasyBuild-contrib/easybuild/easyconfigs/s/SAMtools/SAMtools-1.14-cpeGNU-21.12.eb (module: SAMtools/1.14-cpeGNU-21.12) -== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-oo0lj9lq/easybuild-2cyomy8v.log* have been removed. -== Temporary directory /run/user/10012026/easybuild/tmp/eb-oo0lj9lq has been removed. -``` - -This output tells you that most of the dependencies required by ``SAMtools-1.14-cpeGNU-21.12.eb`` are -already installed, since they are marked with ``[x]``. However, the easyconfig files for -``HTSLib-1.14-cpeGNU-21.12.eb`` and SAMtools itself are not installed yet, denoted by the -lack of an ``x`` in ``[ ]``. - - ---- - -### Missing dependencies - -If you are only interested in which dependencies are still *missing*, -you can consult the output of **`eb --missing`**, or the equivalent **`eb -M`**. - -For example, for the SAMtools easyconfig file used in the previous example we get (with -some lines removed from the output): - -```shell -$ eb SAMtools-1.14-cpeGNU-21.12.eb -M -2 out of 11 required modules missing: - -* HTSlib/1.14-cpeGNU-21.12 (HTSlib-1.14-cpeGNU-21.12.eb) -* SAMtools/1.14-cpeGNU-21.12 (SAMtools-1.14-cpeGNU-21.12.eb) -``` - -That should be pretty self-explanatory: out of the 113 required dependencies (which includes the `cpeGNU` toolchain -and everything needed to install it), only 2 dependencies (including SAMtools itself) are missing. Great! - ---- - -## Inspecting install procedures - -Finally, before actually installing something you may want to assess *how* exactly EasyBuild is going -to install the software. - -Perhaps you don't trust EasyBuild yet (you will eventually though, hopefully), -or maybe you just want to double check that you have made the right choice before going through with -the actual installation. - -Using **`eb --extended-dry-run`**, or just **`eb -x`** for short, -you can get a **detailed overview of the installation procedure that would be performed by EasyBuild**, -**in a matter of seconds**. - -By means of example, let's inspect some parts of the installation procedure for ``HTSLib-1.14-cpeGNU-21.12.eb``: - -```shell -$ eb HTSlib-1.14-cpeGNU-21.12.eb -x -... - -*** DRY RUN using 'ConfigureMake' easyblock (easybuild.easyblocks.generic.configuremake @ /appl/lumi/SW/LUMI-21.12/common/EB/EasyBuild/4.5.3/lib/python3.6/site-packages/easybuild/easyblocks/generic/configuremake.py) *** - -== building and installing HTSlib/1.14-cpeGNU-21.12... -fetching files... [DRY RUN] - -... - -[prepare_step method] -Defining build environment, based on toolchain (options) and specified dependencies... - -Loading toolchain module... - -module load cpeGNU/21.12 - -Loading modules for dependencies... - -module load buildtools/21.12 -module load zlib/1.2.11-cpeGNU-21.12 -module load bzip2/1.0.8-cpeGNU-21.12 -module load XZ/5.2.5-cpeGNU-21.12 -module load cURL/7.78.0-cpeGNU-21.12 - -... - -Defining build environment... - -... - - export CC='cc' - export CFLAGS='-O2 -ftree-vectorize -fno-math-errno' - -... - -configuring... [DRY RUN] - -[configure_step method] - running command "./configure --prefix=/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/HTSlib/1.14-cpeGNU-21.12" - (in /run/user/10012026/easybuild/build/HTSlib/1.14/cpeGNU-21.12/HTSlib-1.14) - -building... [DRY RUN] - -[build_step method] - running command "make -j 256" - (in /run/user/10012026/easybuild/build/HTSlib/1.14/cpeGNU-21.12/HTSlib-1.14) - -testing... [DRY RUN] - -[test_step method] - -installing... [DRY RUN] - -... - -sanity checking... [DRY RUN] - -[sanity_check_step method] -Sanity check paths - file ['files'] - * bin/bgzip - * bin/tabix - * lib/libhts.so -Sanity check paths - (non-empty) directory ['dirs'] - * include -Sanity check commands - * bgzip --version - * htsfile --version - * tabix --version - -... -``` - -We've obviously trimmed the generated output a bit, but it should be sufficient. - -An overview of the installation procedure is shown, following the installation steps as they would be -performed by EasyBuild. The output above shows: - -* how the build environment will be set up during the `prepare` step, by loading the module for both the - toolchains and the dependencies, and defining a set of environment variables like `$CC`, `$CFLAGS`, etc. -* which command will be executed during the configuration step, and in which directory; -* the list of files and directories that will be checked during the sanity check step; - -If you were concerned about EasyBuild being too much of a black box, that is hopefully resolved now. - -!!! note - It is important to highlight here that the reported installation procedure *may* not be 100% correct, - since the [easyblock](1_05_terminology.md#easyblocks) can change its mind based on the output of shell commands - that were executed, or based on the contents of a file that was generated during the installation. - Since all "actions" that would be performed during the installation are actually skipped when using `eb -x`, - the reported installation procedure could be partially incorrect. - - In addition, the easyblock may trip over the fact that the installation procedure is not actually being - executed, which sometimes leads to an unexpected error. These situations are handled gracefully by `eb -x` - though, which will report any errors that occurred and then continue anyway with inspecting the remainder of the - installation procedure. Although this obviously limits the value of the generated output, - it doesn't make it completely useless. - ---- - -## Installing software - -You should now be able to make an informed decision on which easyconfig file you should use to -make EasyBuild install the software you require. - -As mentioned before, installing an easyconfig is as simple as passing it to the `eb` command. - -So, let's try to install libdap version 3.20.9 with the cpeGNU/21.12 toolchain. - -Let's first check if it has any dependencies that still need to be installed: - -```shell -$ eb libdap-3.20.9-cpeGNU-21.12.eb -D -== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-wm_bk3j6/easybuild-puyu_559.log -Dry run: printing build status of easyconfigs and dependencies - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeGNU/cpeGNU-21.12.eb (module: cpeGNU/21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/zlib/zlib-1.2.11-cpeGNU-21.12.eb (module: zlib/1.2.11-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libtirpc/libtirpc-1.3.2-cpeGNU-21.12.eb (module: libtirpc/1.3.2-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/Brotli/Brotli-1.0.9-cpeGNU-21.12.eb (module: Brotli/1.0.9-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ICU/ICU-69.1-cpeGNU-21.12.eb (module: ICU/69.1-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cURL/cURL-7.78.0-cpeGNU-21.12.eb (module: cURL/7.78.0-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/bzip2/bzip2-1.0.8-cpeGNU-21.12.eb (module: bzip2/1.0.8-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/ncurses/ncurses-6.2-cpeGNU-21.12.eb (module: ncurses/6.2-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libreadline/libreadline-8.1-cpeGNU-21.12.eb (module: libreadline/8.1-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12-minimal.eb (module: gettext/0.21-cpeGNU-21.12-minimal) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/PCRE2/PCRE2-10.37-cpeGNU-21.12.eb (module: PCRE2/10.37-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/XZ/XZ-5.2.5-cpeGNU-21.12.eb (module: XZ/5.2.5-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libxml2/libxml2-2.9.12-cpeGNU-21.12.eb (module: libxml2/2.9.12-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/libxslt/libxslt-1.1.34-cpeGNU-21.12.eb (module: libxslt/1.1.34-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/gettext/gettext-0.21-cpeGNU-21.12.eb (module: gettext/0.21-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/file/file-5.41-cpeGNU-21.12.eb (module: file/5.41-cpeGNU-21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/util-linux/util-linux-2.37.1-cpeGNU-21.12.eb (module: util-linux/2.37.1-cpeGNU-21.12) - * [ ] /pfs/lustrep3/users/kurtlust/LUMI/LUMI-EasyBuild-contrib/easybuild/easyconfigs/l/libdap/libdap-3.20.9-cpeGNU-21.12.eb (module: libdap/3.20.9-cpeGNU-21.12) -== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-wm_bk3j6/easybuild-puyu_559.log* have been removed. -== Temporary directory /run/user/10012026/easybuild/tmp/eb-wm_bk3j6 has been removed. -``` - -and now instal the library: - -```shell -$ eb libdap-3.20.9-cpeGNU-21.12.eb -== Temporary log file in case of crash /run/user/10012026/easybuild/tmp/eb-kfphjoi8/easybuild-kcs00ai5.log -== processing EasyBuild easyconfig -/pfs/lustrep3/users/kurtlust/LUMI/LUMI-EasyBuild-contrib/easybuild/easyconfigs/l/libdap/libdap-3.20.9-cpeGNU-21.12.eb -== building and installing libdap/3.20.9-cpeGNU-21.12... -== fetching files... -== ... (took 2 secs) -== creating build dir, resetting environment... -== unpacking... -== patching... -== preparing... -== ... (took 6 secs) -== configuring... -== ... (took 1 min 6 secs) -== building... -== ... (took 53 secs) -== testing... -== installing... -== ... (took 5 secs) -== taking care of extensions... -== restore after iterating... -== postprocessing... -== sanity checking... -== ... (took 3 secs) -== cleaning up... -== creating module... -== ... (took 2 secs) -== permissions... -== packaging... -== COMPLETED: Installation ended successfully (took 2 mins 20 secs) -== Results of the build can be found in the log file(s) -/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/libdap/3.20.9-cpeGNU-21.12/easybuild/easybuild-libdap-3.20.9-20220329.154535.log -== Build succeeded for 1 out of 1 -== Temporary log file(s) /run/user/10012026/easybuild/tmp/eb-kfphjoi8/easybuild-kcs00ai5.log* have been removed. -== Temporary directory /run/user/10012026/easybuild/tmp/eb-kfphjoi8 has been removed. -``` - -That was... easy. Is that really all there is to it? Well, almost... - -### Enabling dependency resolution - -The libdap installation worked like a charm, but remember that all required dependencies were already -available (see [above](#dry-run)). - -If we try this with the `SAMtools-1.14-cpeGNU-21.12.eb`, for which the required `HTSlib` dependencies is not available yet, it's less successful: - -```shell -$ eb SAMtools-1.14-cpeGNU-21.12.eb -M - -2 out of 11 required modules missing: - -* HTSlib/1.14-cpeGNU-21.12 (HTSlib-1.14-cpeGNU-21.12.eb) -* SAMtools/1.14-cpeGNU-21.12 (SAMtools-1.14-cpeGNU-21.12.eb) -``` - -```shell -$ eb SAMtools-1.14-cpeGNU-21.12.eb -... -== preparing... -== FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12): build failed -(first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12 (took 3 secs) -== Results of the build can be found in the log file(s) -/run/user/10012026/easybuild/tmp/eb-rgj1v43y/easybuild-SAMtools-1.14-20220329.155911.ZtDcX.log - -ERROR: Build of /appl/lumi/LUMI-EasyBuild-contrib/easybuild/easyconfigs/s/SAMtools/SAMtools-1.14-cpeGNU-21.12.eb failed (err: 'build failed (first 300 chars): Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12') -``` - -Oh my, what's this all about? - -If we filter the output a bit and focus on the actual error, the problem is clear: - -``` -Missing modules for dependencies (use --robot?): HTSlib/1.14-cpeGNU-21.12') -``` - -The required dependency `HTSlib/1.14-cpeGNU-21.12` is not installed yet, -and EasyBuild does not automatically install missing dependencies unless it is told to do so -(which we didn't do in the configuration for LUMI). - -It helpfully suggests to use the `--robot` (or '-r') command line option, so let's try that: - -```shell -$ eb SAMtools-1.14-cpeGNU-21.12.eb --robot -... -== resolving dependencies ... -... -== building and installing HTSlib/1.14-cpeGNU-21.12... -... -== COMPLETED: Installation ended successfully (took 13 sec) -... -== building and installing SAMtools/1.14-cpeGNU-21.12... -... -== COMPLETED: Installation ended successfully (took 8 sec) -... -== Build succeeded for 2 out of 2 -``` - -With dependency resolution enabled the `HTSlib` module gets installed first, -before EasyBuild proceeds with installing `SAMtools`. Great! - ---- - -### Trace output - -As you may have noticed if you tried the previous example hands-on, -some installations take a while. -An installation can be spending quite a bit of time during the build step, but what is actually going on there? - -To provide some more feedback as the installation progresses, you can enable the "`trace`" configuration setting. -Let's do this by defining the `$EASYBUILD_TRACE` environment variable, just to avoid having to type `--trace` -over and over again. - -We will redo the installation of `SAMtools-1.14-cpeGNU-21.12.eb` by passing the `--rebuild` -option to the `eb` command (try yourself what happens if you don't use the `--rebuild` option!): - -```shell -$ export EASYBUILD_TRACE=1 -$ eb SAMtools-1.14-cpeGNU-21.12.eb --rebuild -... -== configuring... - >> running command: - [started at: 2022-03-29 18:46:31] - [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] - [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-g7vd83qv.log] - /users/kurtlust/LUMI-user-appl/sources/generic/eb_v4.5.3/ConfigureMake/config.guess - >> command completed: exit 0, ran in < 1s - >> running command: - [started at: 2022-03-29 18:46:31] - [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] - [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-k0etfv8i.log] - ./configure --prefix=/users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L/SAMtools/1.14-cpeGNU-21.12 --build=x86_64-pc-linux-gnu ---host=x86_64-pc-linux-gnu --with-htslib=$EBROOTHTSLIB - >> command completed: exit 0, ran in 00h00m03s -== ... (took 3 secs) -== building... - >> running command: - [started at: 2022-03-29 18:46:34] - [working dir: /run/user/10012026/easybuild/build/SAMtools/1.14/cpeGNU-21.12/samtools-1.14] - [output logged in /run/user/10012026/easybuild/tmp/eb-8p617dr7/easybuild-run_cmd-svcps0yj.log] - make -j 256 CC="cc" CXX="CC" CFLAGS="-O2 -ftree-vectorize -fno-math-errno -fPIC" CXXFLAGS="-O2 -ftree-vectorize -fno-math-errno --fPIC" - >> command completed: exit 0, ran in 00h00m06s -== ... (took 6 secs) -``` - -That's a bit more comforting to stare at... - -SAMtools uses a custom easyblock that is derived from the generic `ConfigureMake` easyblock. -During the *configure* step, the `./configure` command is run with `--build` and `--host` -options added by the generic ConfigureMake easyblock (and the other command, `config.guess` plays -a role in determining the value of those flags). The `--with-htslib=$EBROOTHTSLIB` flag is added -via our easyconfig file to tell SAMtools to use an already available version of HTSlib rather -than the built-in one. - -During the *build* step, the software is actually being compiled -by running the `make` command. EasyBuild automatically uses the available cores on the system (in this case 256). - -We even get a pointer to a log file that contains the output of the command being run, -so we can use `tail -f` to see in detail how it progresses. - -Once the `make` command completes, we get a message that the command completed with a exit code 0 -(implying success), and that it took 3 seconds to run. That's good to know. - -Later during the installation, we now also see this output during the sanity check step: - -``` -== sanity checking... - >> file 'bin/blast2sam.pl' found: OK - >> file 'bin/bowtie2sam.pl' found: OK - >> file 'bin/export2sam.pl' found: OK - >> file 'bin/interpolate_sam.pl' found: OK -... - >> loading modules: SAMtools/1.14-cpeGNU-21.12... - >> running command 'samtools version' ... - >> result for command 'samtools version': OK -``` - -Thanks to enabling trace mode, EasyBuild tells us which files (& directories, but there are non in this case) -it is checking for -in the installation, and which command it is trying to run before declaring it a success. Nice! - -The extra output you get when trace mode is enabled is concise and hence not overwhelming, -while it gives a better insight into what is going on during the installation. -It may also help to spot unexpected actions being taken during the installation early on, -so you can interrupt the installation before it completes, if deemed necessary. - ---- - -## Using installed software - -So far, we have already installed 4 different software packages (SAMtools, HTSlib, libdap); -we even installed SAMtools twice! - -A lot was going on underneath the covers: locating and unpacking -the source tarballs, setting up the build environment, configuring the build, compiling, -creating and populating the installation directory, performing a quick sanity check on the installation, -cleaning things up, and finally generating the environment module file corresponding to the installation. - -That's great, but how do we now actually *use* these installations? - -This is where the generated module files come into play: they form the access portal to the software -installations, and we'll use the ubiquitous `module` command to digest them. - -First, we need to make sure that the modules tool is aware of where the module files for -our installations are located. On LUMI, when using the EasyBuild-user module to configure EasyBuild, -everything is taken care of for you and the LUMI modules will also automatically add the -suitable module directories for user-installed software to the search path for modules. -By default, EasyBuild-config will install in `$HOME/EasyBuild`, but it is possible to -build the installation in a different directory by pointing to it with the environment -variable `EBU_USER_PREFIX`. Of course this variable needs to be set before loading the `LUMI` -module. (Note that one reason why we don't load a software stack by default is that in -the current setup of LUMI this module would be loaded before the user gets the chance to set -that environment variable in `.bash_profile` or `.bashrc`.) - -When loading the `EasyBuild-user` module, the module command will show you were EasyBuild -will install the software and put the modules, and also put its repository of -processed easyconfig file. - -```shell -ml EasyBuild-user - -EasyBuild configured to install software from the LUMI/21.12 software stack for the LUMI/L -partition in the user tree at /users/kurtlust/LUMI-user-appl. - * Software installation directory: /users/kurtlust/LUMI-user-appl/SW/LUMI-21.12/L - * Modules installation directory: /users/kurtlust/LUMI-user-appl/modules/LUMI/21.12/partition/L - * Repository: /users/kurtlust/LUMI-user-appl/ebrepo_files/LUMI-21.12/LUMI-L - * Work directory for builds and logs: /run/user/10012026/easybuild - Clear work directory with clear-eb -``` - -EasyBuild will copy each easyconfig file it installs to -the repository and add some lines to it with information about the installation. It -also has some options that may edit the source easyconfig, e.g., when asking EasyBuild -to try to build with another toolchain. - -You can always check where EasyBuild is installing stuff by checking the output of -`eb --show-config`. - -If you're unsure where EasyBuild is installing stuff at this point, -check the output of `eb --show-config`; the value of the `installpath` configuration setting is what we are interested in now: - -```shell -$ eb --show-config -... -buildpath (E) = /run/user/XXXXXXXX/easybuild/build -... -installpath-modules (E) = /users/XXXXXXXX/LUMI-user/modules/LUMI/21.12/partition/L -installpath-software (E) = /users/XXXXXXXX/LUMI-user/SW/LUMI-21.12/L... -... -repositorypath (E) = /users/XXXXXXXX/LUMI-user/ebrepo_files/LUMI-21.12/LUMI-L -... -sourcepath (E) = /users/XXXXXXXX/LUMI-user/sources:/appl/lumi/sources/easybuild -... -``` - -This is sligthly different from the default EasyBuild setup, where the modules, software, -repository and sources would be installed in respectively the subdirectories `modules`, -`software`, `ebfiles_repo` and `sources` of the directory pointed to by the `installpath` -line. - -The modules directory is also a simplified one from the standard EasyBuild one as that also -provides a module categorisation besides a directory containing all modules. As this categorisation -is largely arbitrary and hard to use in the module system, we decided simply not to use it in -our installation and use a custom naming scheme. - -However, if you would be using EasyBuild on another system with its default configuration, the -above setup would be used. For more information, we refer to the generic EasyBuild tutorials on the -[EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). - -Now the modules tool should be aware of our brand new installations: - -```shell -$ module avail -... --- EasyBuild managed user software for software stack LUMI/21.12 on LUMI-L --- - HTSlib/1.14-cpeGNU-21.12 libdap/3.20.9-cpeGNU-21.12 - SAMtools/1.14-cpeGNU-21.12 - ------ EasyBuild managed software for software stack LUMI/21.12 on LUMI-L ----- -... -``` - -This output shows both the modules for our own installations as well as the "central" installations -(which we omitted for brevity). - -Now we can load these modules and start using these software installations. - -Let's test this for SAMtools. In our current environment, the `samtools` command is not available yet: - -```shell -$ module list - -Currently Loaded Modules: - 1) perftools-base/21.12.0 - 2) cce/13.0.0 - 3) craype/2.7.13 - 4) cray-dsmml/0.2.2 - 5) cray-mpich/8.1.12 - 6) cray-libsci/21.08.1.2 - 7) PrgEnv-cray/8.2.0 - 8) ModuleLabel/label (S) - 9) init-lumi/0.1 (S) - 10) craype-x86-rome - 11) craype-accel-host - 12) libfabric/1.11.0.4.106 - 13) craype-network-ofi - 14) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta - 15) partition/L (S) - 16) LUMI/21.12 (S) - - Where: - S: Module is Sticky, requires --force to unload or purge -$ samtools --bash: samtools: command not found -``` - -Loading the module for SAMtools changes that: - -```shell -$ module load SAMtools/1.14-cpeGNU-21.12 - -Lmod is automatically replacing "cce/13.0.0" with "gcc/11.2.0". -Lmod is automatically replacing "PrgEnv-cray/8.2.0" with "cpeGNU/21.12". - -Due to MODULEPATH changes, the following have been reloaded: - 1) cray-mpich/8.1.12 - -$ module list - -Currently Loaded Modules: - 1) perftools-base/21.12.0 - 2) ModuleLabel/label (S) - 3) init-lumi/0.1 (S) - 4) craype-x86-rome - 5) craype-accel-host - 6) libfabric/1.11.0.4.106 - 7) craype-network-ofi - 8) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta - 9) partition/L (S) - 10) LUMI/21.12 (S) - 11) gcc/11.2.0 - 12) craype/2.7.13 - 13) cray-mpich/8.1.12 - 14) cray-libsci/21.08.1.2 - 15) cray-dsmml/0.2.2 - 16) cpeGNU/21.12 - 17) ncurses/6.2-cpeGNU-21.12 - 18) zlib/1.2.11-cpeGNU-21.12 - 19) bzip2/1.0.8-cpeGNU-21.12 - 20) gettext/0.21-cpeGNU-21.12-minimal - 21) XZ/5.2.5-cpeGNU-21.12 - 22) Brotli/1.0.9-cpeGNU-21.12 - 23) cURL/7.78.0-cpeGNU-21.12 - 24) HTSlib/1.14-cpeGNU-21.12 - 25) SAMtools/1.14-cpeGNU-21.12 - - Where: - S: Module is Sticky, requires --force to unload or purge - -$ samtools --version -samtools 1.14 -Using htslib 1.14 -Copyright (C) 2021 Genome Research Ltd.... -... -``` - -Note that the modules for the required dependencies, including the compiler toolchain (which provides runtime libraries -like `libstdc++.so`), are loaded automatically. The "`module load`" command changes the active environment, -by updating environment variables like `$PATH` for example, to make the software available for use. - -##### Resetting your environment - -To restore your environment to a pristine state in which no modules are loaded, you can either -unload the loaded modules one by one using "`module unload`", or you can unload all of them at once using -"`module purge`". On LUMI, `module purge` will unload all application modules but will not undo the selection -of the software stack. It will reset the software stack though to use those modules that fit best -with the hardware of the current node (i.e., you may find a different `partition` module). - -```shell -$ module purge -The following modules were not unloaded: - (Use "module --force purge" to unload all): - - 1) LUMI/21.12 6) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta - 2) craype-x86-rome 7) partition/L - 3) craype-accel-host 8) init-lumi/0.1 - 4) libfabric/1.11.0.4.106 9) ModuleLabel/label - 5) craype-network-ofi -$ module list - -Currently Loaded Modules: - 1) LUMI/21.12 (S) - 2) craype-x86-rome - 3) craype-accel-host - 4) libfabric/1.11.0.4.106 - 5) craype-network-ofi - 6) xpmem/2.2.40-2.1_3.9__g3cf3325.shasta - 7) partition/L (S) - 8) init-lumi/0.1 (S) - 9) ModuleLabel/label (S) - - Where: - S: Module is Sticky, requires --force to unload or purge -``` - -Runnin `module --force purge` instead will remove all modules, including the `init-lumi` -module which does part of the initialisation. You will not be able to use the software -stacks completely as before without first loading `init-lumi` in its most recent (or default) -version again! - -```shell -$ module --force purge -$ module list -No modules loaded -``` - ---- - -## Stacking software - -Maybe you have overlooked how the software we are playing around with was not only installed across multiple -different installation directories per software, we are also "stacking" our own installations (in `$HOME/EasyBuild` -or `$EBU_USER_PREFIX`) on top of installations that are provided in a totally different location (`/appl/lumi`). - -**EasyBuild doesn't care *where* software is installed: as long as the module file that provides access to it -is available, it is happy to pick it up and use it when required.** - -This implies that end users of LUMI can easily install their -own small software stack on top of what is provided centrally by the LUMI User Support, -for example. They can even -"replace" a central software installation for their purposes if they need to, since the modules tool will -load the first module file that matches the request being made (there are some caveats with this, but we -won't go into those here). - ---- - -## Exercises - -TODO: Rework for LUMI and provide the necessary EasyConfig files. - -***Guidelines*** - -Do yourself a favor: don't peek at the solution until you have made an attempt to solve the exercise yourself! - -Please do not spoil solutions to others before they have been discussed by the tutorial organisers. - -The exercises are based on the easyconfig files included with EasyBuild 4.3.3. - ---- - -***Exercise U.0**** - Making installed software available* - -Before working on the exercises for this part of the tutorial, -make sure that the software that is already installed in the prepared environment is available. - -We will assume that you have a small software stack installed using the `2020b` version of the `foss` toolchain. - -**Tip:** execute a "`module use`" command, and verify with "`module avail`" that a bunch of software modules -are available for loading. - -??? success "(click to show solution)" - - Use the following command to make the modules for the software available that is pre-installed - in the prepared environment: - ```shell - module use /easybuild/modules/all - ``` - - If software is installed in a different location than `/easybuild/` in your environment, - you should adjust the command accordingly. - - ---- - -***Exercise U.1**** - Searching easyconfigs* - -See if EasyBuild provides any easyconfig files for installing GROMACS version 2020/5. - -??? success "(click to show solution)" - To check for available easyconfig files, we can use `eb --search` or `eb -S`: - ```shell - $ eb -S gromacs-2020.5 - == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... - CFGS1=/home/example/.local/easybuild/easyconfigs/g/GROMACS - * $CFGS1/GROMACS-2020.5-fosscuda-2020a-Python-3.8.2.eb - * $CFGS1/GROMACS-2020.5_fix_threads_gpu_Gmxapitests.patch - ``` - This actually shows one easyconfig file but also a patch file. We can also search specifically - for GROMACS 2020.5 in the `foss` and `fosscuda` toolchains using - ```shell - $ eb -S gromacs-2020.5-foss - == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... - CFGS1=/home/example/.local/easybuild/easyconfigs/g/GROMACS - * $CFGS1/GROMACS-2020.5-fosscuda-2020a-Python-3.8.2.eb - ``` - and now we find a single easyconfig file. - ---- - -***Exercise U.2**** - Checking dependencies* - -Check which dependencies are missing to install QuantumESPRESSO version 6.6 with the `2020b` version of the `foss` toolchain. - -??? success "(click to show solution)" - First, we need to determine the name of the easyconfig file for QuantumESPRESSO version 6.6: - ```shell - $ eb -S 'QuantumESPRESSO-6.6.*foss-2020b' - == found valid index for /home/example/.local/easybuild/easyconfigs, so using it... - CFGS1=/home/example/.local/easybuild/easyconfigs/q/QuantumESPRESSO - * $CFGS1/QuantumESPRESSO-6.6-foss-2020b.eb - ``` - To determine which dependencies are missing to install this QuantumESPRESSO easyconfig file, we can use `--missing`: - ```shell - $ eb QuantumESPRESSO-6.6-foss-2020b.eb --missing - - 3 out of 58 required modules missing: - - * libxc/4.3.4-GCC-10.2.0 (libxc-4.3.4-GCC-10.2.0.eb) - * ELPA/2020.11.001-foss-2020b (ELPA-2020.11.001-foss-2020b.eb) - * QuantumESPRESSO/6.6-foss-2020b (QuantumESPRESSO-6.6-foss-2020b.eb) - ``` - (some nonessential output removed). - ---- - -***Exercise U.3**** - Performing a dry run* - -Figure out which command EasyBuild would use to compile -the software provided by the `Bowtie2-2.4.2-GCC-9.3.0.eb` easyconfig file, -without actually installing `Bowtie2`. - -Also, which binaries will EasyBuild check for to sanity check the installation? - -??? success "(click to show solution)" - To inspect the installation procedure, we can use `eb -x Bowtie2-2.4.2-GCC-9.3.0.eb`. - - The output for the build step shows the actual compilation command that would be performed (`make ...`): - - ```shell - [build_step method] - >> running command: - [started at: 2021-03-08 20:15:08] - [working dir: /local_scratch/hkenneth/eb-1wodfohg/__ROOT__/local_scratch/hkenneth/Bowtie2/2.4.2/GCC-9.3.0/Bowtie2-2.4.2] - [output logged in /local_scratch/hkenneth/eb-1wodfohg/easybuild-run_cmd-haojzisn.log] - make -j 48 CC="gcc" CPP="g++" CXX="g++" RELEASE_FLAGS="-O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -std=gnu++98" - (in /local_scratch/hkenneth/Bowtie2/2.4.2/GCC-9.3.0/Bowtie2-2.4.2) - ``` - - If the output you get is less detailed, you may not have set `export EASYBUILD_TRACE=1`. - - The output for the sanity check step shows which binaries are expected to be installed: - ``` - [sanity_check_step method] - Sanity check paths - file ['files'] - * bin/bowtie2 - * bin/bowtie2-align-l - * bin/bowtie2-align-s - * bin/bowtie2-build - * bin/bowtie2-build-l - * bin/bowtie2-build-s - * bin/bowtie2-inspect - * bin/bowtie2-inspect-l - * bin/bowtie2-inspect-s - ``` - ---- - -***Exercise U.4**** - Installing software* - -Install version 3.1.0 of the `h5py` Python package and all missing dependencies, -using the `foss/2020b` toolchain, into `/tmp/$USER/easybuild`, -while leveraging the already installed software available from `/easybuild`. - -Enable trace output so you can see which parts of the installation take a while. - -??? success "(click to show solution)" - First, determine the easyconfig file we can use for this: - ```shell - $ eb -S 'h5py-3.1.0.*foss-2020b' - CFGS1=/home/example/.local/easybuild/easyconfigs/h/h5py - * $CFGS1/h5py-3.1.0-foss-2020b.eb - ``` - - Make sure the pre-install software in `/easybuild/` is available: - ```shell - module use /easybuild/modules/all - ``` - - Check which dependencies are missing to install this `h5py` easyconfig: - ```shell - $ eb h5py-3.1.0-foss-2020b.eb --missing - - 2 out of 63 required modules missing: - - * pkgconfig/1.5.1-GCCcore-10.2.0-python (pkgconfig-1.5.1-GCCcore-10.2.0-python.eb) - * h5py/3.1.0-foss-2020b (h5py-3.1.0-foss-2020b.eb) - ``` - - Install `h5py` by specifying the easyconfig file and enabling dependency resolution via `--robot`, - while indicating that we want to install the software into `/tmp/$USER/easybuild` using the `--installpath` - option. Also make sure that trace mode is enabled by defining the `$EASYBUILD_TRACE` environment variable. - ```shell - $ export EASYBUILD_TRACE=1 - $ eb h5py-3.1.0-foss-2020b.eb --robot --installpath /tmp/$USER/easybuild - ... - == building and installing pkgconfig/1.5.1-GCCcore-10.2.0-python... - ... - == building and installing h5py/3.1.0-foss-2020b... - ... - == building... - >> running command: - [started at: 2020-06-10 21:47:32] - [working dir: /tmp/example/h5py/3.1.0/foss-2020b/h5py-3.1.0] - [output logged in /tmp/eb-rjjkbqe1/easybuild-run_cmd-d_dkc4iz.log] - python setup.py configure --mpi --hdf5=$EBROOTHDF5 && /easybuild/software/Python/3.8.6-GCCcore-10.2.0/bin/python setup.py build - >> command completed: exit 0, ran in 00h01m27s - ... - == COMPLETED: Installation ended successfully (took 2 min 46 sec) - ... - == Build succeeded for 2 out of 2 - ``` - - The trace output shows that most time is spent in the build command, - which runs both `python setup.py configure` and `python setup.py build`. - ---- - -***Exercise U.5**** - Using installed software* - -Using the `h5py` installation from the previous exercise to create an empty HDF5 file, -using the following Python statements: - -```python -import h5py -f = h5py.File("empty.hdf5", "w") -f.close() -``` - -Check the resulting file using the `h5stat` command. - -??? success "(click to show solution)" - First, we need to make the modules tool aware of the module files that were installed into `/tmp/$USER/easybuild`: - ```shell - module use /tmp/$USER/easybuild/modules/all - ``` - - Then we can check the `h5py` module is available, and load it: - ```shell - $ module avail h5py - ------------ /tmp/example/easybuild/modules/all ------------ - h5py/3.1.0-foss-2020b - ``` - - ```shell - module load h5py/3.1.0-foss-2020b - ``` - - The Python code snippet can be run directly on the command line using "`python -c '...'`", since it's tiny: - ```shell - python -c 'import h5py; f = h5py.File("empty.hdf5", "w"); f.close()' - ``` - Of course you can also copy the Python code snippet in a file named `test_h5py.py`, - and then run it with `python test_h5py.py`. - - Checking with the `h5stat` command shows that the resulting `empty.hdf5` is indeed a valid HDF5 file: - ```shell - $ ls -l empty.hdf5 - -rw-rw-r-- 1 example example 800 Jun 10 21:54 empty.hdf5 - - $ h5stat empty.hdf5 - Filename: empty.hdf5 - File information - # of unique groups: 1 - # of unique datasets: 0 - ... - ``` - ---- - -If you've made it through the hands-on exercises, congratulations! - -If not, don't worry too much about it. We covered a lot of ground here, -and it's a lot to take in at once, take your time... - ---- - -*[[next: Part 2: Using EasyBuild]](2_00_part2_using.md)* diff --git a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md index d6ed3f61..421c94cd 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md @@ -1,18 +1,31 @@ # What is EasyBuild? +<<<<<<< HEAD *[[back: Course overview]](../index.md)*

EasyBuild logo

**[EasyBuild](https://easybuild.io)** is a **software build and installation framework** +======= +*[[back: Overview]](index.md)* + +

EasyBuild logo

+ +**[EasyBuild](http://easybuild.io)** is a **software build and installation framework** +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) that manages (scientific) software in an efficient way, without compromising on performance. Easybuild is especially tailored towards [**High Performance Computing (HPC)**](https://en.wikipedia.org/wiki/Supercomputer) systems, and provides a **command-line interface** to facilitate the installation and management of a scientific software stack. --- +<<<<<<< HEAD +======= + + +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) ## Elevator pitch @@ -68,7 +81,11 @@ as it is performed. EasyBuild supports using a **custom module naming scheme**, allows for *hierarchical* module naming schemes, and **integrates with various other tools** ranging from resource managers ([Slurm](https://slurm.schedmd.com) and [GC3Pie](https://github.com/gc3pie/gc3pie)), +<<<<<<< HEAD container tools ([Singularity](https://github.com/apptainer/singularity) and [Docker](https://www.docker.com)), +======= +container tools ([Singularity](https://github.com/hpcng/singularity) and [Docker](https://www.docker.com)), +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) packaging tools ([FPM](https://fpm.readthedocs.io)), and so on. --- @@ -95,7 +112,11 @@ By default EasyBuild tries to run tests if the build process of a package provid functionality, or can run simple tests of its own, and of course these tests will fail if the host CPU does not support the full instruction set of the target CPU. Secondly, more and more software has broken build processes where developers thought that it was a good +<<<<<<< HEAD idea to hard-code options such as ``-march=native`` or ``-xHost`` in a way that EasyBuild +======= +idea to hard-code options such as ``-march=native`` or ``--xHost`` in a way that EasyBuild +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) cannot easily overwrite those. @@ -103,14 +124,24 @@ cannot easily overwrite those. In addition to performance, **reproducibility of installations** is a core aspect of EasyBuild. +<<<<<<< HEAD Most software installations performed with EasyBuild use a **particular compiler toolchain**, +======= +TODO correct link in the next lines. + +Most software installations performed with EasyBuild use a **particular compiler toolchain**, +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) with which we aim to be in control over the build environment and avoid relying on tools and libraries provided by the operating system. For similar reasons, we try to **provide all required dependencies through EasyBuild** as well, with a few notable exceptions, like ``OpenSSL`` for security reasons, and Infiniband and GPU drivers which are too closely intertwined with the operating system. For both toolchains and dependencies, **fixed software versions** are specified in the +<<<<<<< HEAD easyconfig files. That way, easyconfig files can easily be shared with others: +======= +easyconfig files. That way, easyconfig files can easily be shared with others: +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) if they worked for you it is very likely that they will work for others too, because the vast majority of the software stack is controlled by EasyBuild. @@ -127,17 +158,30 @@ and to [**contribute back**](https://docs.easybuild.io/en/latest/Contributing.ht by opening pull requests to the [GitHub repositories](https://github.com/easybuilders) or the [documentation](https://docs.easybuild.io). The EasyBuild community also provides packages for two standard toolchains, called the +<<<<<<< HEAD [**common toolchains**](https://docs.easybuild.io/en/latest/Common-toolchains.html). The so-called ``foss`` is based on free and open software (GNU compilers, FlexiBlas with OpenBLAS backend, LaPACK, FFTW and Open MPI) while the ``intel`` toolchain uses the full stack of Intel compilers, libraries and MPI. Currently these toolchains cannot be supported on LUMI though as getting Open MPI to run on LUMI is challenging and as the Intel libraries and some MPI versions also produce slow or incorrect results on AMD CPUs (and it is not entirely clear if Intel MPI is compatible with SlingShot 11, though in principle it should). +======= +[**common toolchains**](https://easybuild.readthedocs.io/en/latest/Common-toolchains.html). +The so-called ``foss`` is based on free and open software (GNU compilers, FlexiBlas with OpenBLAS backend, LaPACK, FFTW and Open MPI) +while the ``intel`` toolchain uses the full stack of Intel compilers, libraries and MPI. Currently these toolchains +cannot be supported on LUMI though as getting Open MPI to run on LUMI is challenging and as the Intel libraries and +some MPI versions also produce slow or incorrect results on AMD CPUs (and it is not clear if Intel MPI is compatible +with SlingShot 11). +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) Focusing the efforts on those two popular common toolchains makes it possible to support a lot of build recipes that are collected in the [central repository](https://github.com/easybuilders/easybuild-easyconfigs). They can still serve as a basis to develop build recipes for the HPE Cray Programming Environment. +<<<<<<< HEAD Last but not least, EasyBuild provides various [**GitHub integration features**](https://docs.easybuild.io/en/latest/Integration_with_GitHub.html) +======= +Last but not least, EasyBuild provides various [**GitHub integration features**](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) that greatly facilitate the contribution process: opening, updating, and testing pull requests, reviewing incoming contributions, and much more can all be done directly from the EasyBuild command line. This not only saves time, effort, brain cycles, and mouse clicks for contributors, @@ -171,7 +215,11 @@ Since EasyBuild is primarily intended for installing (scientific) software on HP systems running some flavor of **GNU/Linux**. EasyBuild also works on *macOS*, but significant effort would be required to use it extensively there. +<<<<<<< HEAD +======= + +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) EasyBuild is implemented in [**Python**](https://www.python.org), and is compatible with Python 2.7 and 3.5+. Since [Python 2 is no longer maintained](https://blog.python.org/2020/04/python-2718-last-release-of-python-2.html), @@ -187,7 +235,11 @@ On LUMI Lmod is used as the module tool. ## Development +<<<<<<< HEAD +======= + +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) EasyBuild is **actively developed** by a worldwide community under the [**GPLv2 open source license**](https://www.gnu.org/licenses/old-licenses/gpl-2.0.html) via the [**``easybuilders``** organisation on GitHub](https://github.com/easybuilders), in the following repositories: @@ -225,6 +277,7 @@ sites leverage it, and integrate it with other tools like [Jenkins](https://www. HPC sites, consortia, and companies that use EasyBuild include (but are not limited to): +<<<<<<< HEAD a href="https://www.vscentrum.be">VSC logo JSC logo HPCNow! logo @@ -246,6 +299,28 @@ a href="https://www.vscentrum.be">VSC <a href=Cyprus Institute logo University of Basel logo Microsoft logo +======= +VSC logo +JSC logo +HPCNow! logo +Fred Hutch logo +Vienna BioCenter logo +CÉCI logo +Compute Canada logo +UiO logo +BEAR logo +SNIC logo +National University of Singapore logo +IT4Innovations logo +Texas A&M logo +TU Dresden logo +University of Melbourne logo +University of Luxembourg logo +EPCC logo +Cyprus Institute logo +University of Basel logo +Microsoft logo +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) --- diff --git a/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md b/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md index 8eb10e2c..fd894324 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md @@ -6,14 +6,24 @@ On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated as Cray PE). The environment provides several tools, including compilers, communication libraries, +<<<<<<< HEAD optimised math libraries and various other libraries, analyzers and debuggers. The Cray PE is made available through *environment modules* that allow to select particular versions of +======= +optmized math libraries and various other libraries, analyzers and debuggers. + +The Cray PE is made available through *environment modules* tha allow to select particular versions of +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) tools and to configure the environment in a flexible way. --- +<<<<<<< HEAD ## HPE Cray PE components +======= +## Cray PE components +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) ### Cray compiler environments @@ -122,8 +132,12 @@ LUMI environment and the integration with the Cray environment still seems to be The Cray PE also provides a programming environment module to enable the Intel® oneAPI compiler and tools. +<<<<<<< HEAD The documentation is available in the [Intel® oneAPI Programming Guide](https://www.intel.com/content/www/us/en/develop/documentation/oneapi-programming-guide/top/oneapi-development-environment-setup.html) +======= +The documentation is available in the [Intel® oneAPI Programming Guide](https://software.intel.com/content/www/us/en/develop/documentation/oneapi-programming-guide/top/oneapi-development-environment-setup.html) +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) Cray provides a bundled package of support libraries to install into the Cray PE to enable the Intel compiler, allowing utilities such as debuggers and performance tools to work with it. @@ -146,7 +160,11 @@ These libraries satisfy dependencies for many commonly used applications on Cray When the module for a CSML package (such as `cray-libsci` or `cray-fftw`) is loaded, all relevant headers and libraries for these packages are added to the compile and link lines of the `cc`, `ftn`, and `CC` compiler wrappers, so linking with them is +<<<<<<< HEAD completely transparent (to the extent that users wonder where the libraries are). +======= +completely transparant (to the extent that users wonder where the libraries are). +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) The CSML collection contains the following Scientific Libraries: @@ -167,7 +185,11 @@ The CSML collection contains the following Scientific Libraries: * Manpages: `intro_mpi` +<<<<<<< HEAD * Website: [http://www.mpi-forum.org](https://www.mpi-forum.org) +======= +* Website: [http://www.mpi-forum.org](http://www.mpi-forum.org) +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) MPI is a widely used parallel programming model that establishes a practical, portable, efficient, and flexible standard for passing messages between ranks in parallel processes. @@ -185,7 +207,13 @@ versions of the library are supported, the finalised LUMI system will only suppo ### DSMML * Modules: `cray-dsmml` +<<<<<<< HEAD +* Manpages: `intro_dsmml` +======= +* * Manpages: `intro_dsmml` +* +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) * Website: [https://pe-cray.github.io/cray-dsmml](https://pe-cray.github.io/cray-dsmml) Distributed Symmetric Memory Management Library (DSMML) is a HPE Cray proprietary memory management library. @@ -325,7 +353,11 @@ can be set through target modules: ``craype-network-ucx`` module. - The ``craype-hugepages*`` modules enable Cray Huge Pages support. To fully enable this support they have to +<<<<<<< HEAD be used at link-time and at run-time. At link time, support is compiled into the binary while at run-time they +======= + be used at link-time and at run-time. At link time, support is compiled into the binary wile at run-time they +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) are used to set the actual size of the huge pages. The ``craype-hugepages*`` modules are not supported by all compilers. E.g., the AOCC compiler does not support diff --git a/docs/2022-CSC_and_LO/1_Intro/1_06_installation.md b/docs/2022-CSC_and_LO/1_Intro/1_06_installation.md index 4d1ac6c5..47cb5448 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_06_installation.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_06_installation.md @@ -426,7 +426,11 @@ We do so only for the framework and easyblocks files as the easconfig files are install EasyBuild. Instead we create our own EasyConfig file for EasyBuild which contains some additional packages that enable extra features in EasyBuild and also provide more information to Lmod. Next the configuration module for EasyBuild (see the next section, +<<<<<<< HEAD ["Configuring EasyBuild"](../1_07_configuration), for more information) and use the temporary +======= +["Configuring EasyBuild"](1_07_configuration), for more information) and use the temporary +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) installation of EasyBuild with our own EasyConfig file to do a proper installation of EasyBuild with module in the final location. diff --git a/docs/2022-CSC_and_LO/2_00_part2_using.md b/docs/2022-CSC_and_LO/2_00_part2_using.md deleted file mode 100644 index c50036be..00000000 --- a/docs/2022-CSC_and_LO/2_00_part2_using.md +++ /dev/null @@ -1,8 +0,0 @@ -# Part II: Using EasyBuild - -*[[back to start page]](index.md)* - -* [Troubleshooting](2_01_troubleshooting.md) *(hands-on)* -* [Creating easyconfig files](2_02_creating_easyconfig_files.md) *(hands-on)* -* [Using external modules from the Cray PE](2_03_external_modules.md) -* [Implementing easyblocks](2_04_implementing_easyblocks.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/2_01_troubleshooting.md b/docs/2022-CSC_and_LO/2_01_troubleshooting.md deleted file mode 100644 index 30f2f5cc..00000000 --- a/docs/2022-CSC_and_LO/2_01_troubleshooting.md +++ /dev/null @@ -1,545 +0,0 @@ -# Troubleshooting - -*[[back: Overview]](index.md)* - ---- - -Whatever tool you use, when installing scientific software you'll -be running into problems rather sooner than later. - -In this part we take a look at how you can **troubleshoot a failing installation**, -and focus on a couple of EasyBuild aspects that can be helpful in that context, -like the **error messages produced by EasyBuild**, the detailed **log file** that is -available for the installation and the **build directory** where most software is -being compiled before it actually gets installed. - -At the end of this part an exercise is available in which you get -to fix a failing installation, so pay attention! - -## EasyBuild error messages - -When EasyBuild detects that something went wrong, it will usually produce a -short (and hopefully helpful) error message. - -Things that could go wrong during an installation include: - -* missing source or patch files; -* a checksum error on a downloaded source or patch file; -* required dependencies that are not specified in the easyconfig file; -* failing shell commands; -* running out of available memory or disk space; -* a segmentation fault caused by a flipped bit triggered by a cosmic ray ([really, it happens!](https://blogs.oracle.com/linux/attack-of-the-cosmic-rays-v2)); - -Unfortunately this is not an exhaustive list, there are plenty of other -potential problems that could result in a failing installation... - -For each of the shell commands that EasyBuild executes during an -installation, it will check the exit status. -If the exit status is zero, EasyBuild will usually assume that the shell command -ran correctly, and it will continue with the rest of the installation procedure. -If the exit status is anything but zero, EasyBuild will assume that a problem -has occurred, and the installation will be interrupted. - -### Example - -Here is an example of an EasyBuild error message (slightly reformatted for clarity): - -``` -$ eb example.eb -... -== building... -== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): -build failed (first 300 chars): cmd "make" exited with exit code 2 and output: -/usr/bin/g++ -O2 -ftree-vectorize -march=znver2 -fno-math-errno -c -o core.o core.cpp -cc1plus: error: bad value (‘znver2’) for ‘-march=’ switch (took 1 sec) -== Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log -ERROR: Build of /home/easybuild/subread.eb failed (err: ...) -``` - -Let's break this down a bit: during the `build` step of the installation -procedure EasyBuild was running `make` as a shell command, which -failed (exit code 2, so not zero). -The `make` command tripped over the compilation of `core.cpp` that failed because -`-march=znver2` is not a known option to the `g++` command. - -OK fine, but now what? Can you spot something suspicious here? -Wait a minute... Why is `make` using `/usr/bin/g++` for the compilation?! -That's not where our toolchain compiler is installed, -that's somewhere under `/opt/cray/pe/gcc`. - -Let's see what `/usr/bin/g++` is: - -``` -$ /usr/bin/g++ --version -g++ (SUSE Linux) 7.5.0 -``` - -Oh my that's an ancient compiler (7.5 was released on November 14, 2019, -a few months after the release of the Zen2 architecture, but -the base version, 7.1, is really from May 2, 2017, long before the Zen2 architecture was around) -That could definitely explain why it doesn't know about the Zen2 architecture yet... - -Your next step in this case should probably be figuring -out why `/usr/bin/g++` is being used rather than just `g++`, which would -result in using the right compiler version because EasyBuild sets up the build -environment carefully. - -This is a fictitious example of course, but hopefully it gives you a feeling -of how errors that occur during the installation are handled. - -## EasyBuild log files - -Finding the cause of a problem that made the installation fail is, unfortunately, not always that straightforward... - -EasyBuild includes the first 300 characters of the output produced by a failing -shell command in the error message, which is a simple way to try include -actionable information in the error message while not flooding your terminal -with the full output of the command that failed. - -In some cases there won't be any useful information in there however, -since the actual error message(s) could only appear way later, perhaps even after -the command was already running for several minutes. - -In that case, you will have to dive into the log file that is created by EasyBuild for -every installation, which is located in the unique temporary directory for the EasyBuild session. - -See for example this output line from our earlier example error message: - -``` -== Results of the build can be found in the log file(s) /tmp/eb-dbobppfh/easybuild-example-1.0-20200613.145414.aUEJA.log -``` - -You can open this file with your favorite text editor or a tool like `less` -to take a look at the information collected in the log file, which includes -things like: - -* informative messages produced by both the EasyBuild framework and the easyblock - describing how the installation is progressing; -* how the build environment was set up: which modules were loaded, which environment variables were set; -* the exact shell commands that were executed, and in which directory they were run; -* the full output produced by these commands, and their exit code; - -Note that the installation log is also copied into each software installation -directory for successful installation, into the `easybuild` subdirectory. -For example: - -``` -/appl/lumi/SW/LUMI-21.12/L/EB/ncurses/6.2-cpeGNU-21.12/easybuild/easybuild-ncurses-6.2-20220302.110244.log -``` - -### Last log - -The `eb` command supports a handy little option that prints the location -to the most recently updated build log. You can leverage this to quickly -open the build log of the last ***failed*** EasyBuild session in an editor: - -``` -vim $(eb --last-log) -``` - -### Navigating log files - -Usually you want to go to the end of the log file and then work your way up, -either by scrolling or by searching for specific patterns. Here are a couple -of suggestions of patterns you can use to locate errors: - -* `ERROR` -* `Error 1` -* `error: ` -* `failure` -* `not found` -* `No such file or directory` -* `bazel` -* `Segmentation fault` - -Using "`error`" as a search pattern is not very useful: you will hit a lot of log lines -that are not actually errors at all (like the compilation of an `error.c` file), -and you'll miss others that do include errors but mention `ERROR` or `Error` -rather than `error`. - -When using `less` to view a log file, you can navigate it by: - -* hitting '`$`' followed by '`G`' to go to the end of the log file; -* using your arrow keys to scroll up/down; -* typing '`?`' followed by some text and Enter to search backwards for a particular -pattern ('`/`' to search forwards, '`n`' for next match); -* hitting '`q`' to exit; - -It can also be helpful to zoom in on a specific step of the installation procedure, -which you can do by looking for step markers like these: - -``` -== 2020-06-13 01:34:48,816 example INFO configuring... -== 2020-06-13 01:34:48,817 example INFO Starting configure step -... -== 2020-06-13 01:34:48,823 main.EB_HPL INFO Running method configure_step part of step configure -``` - -If you want to look at the start of the output produced by a particular command, -you can look for the log message that looks like this (this is from the installation -log for `HDF5`): - -``` -== 2020-06-09 13:11:19,968 run.py:222 INFO running cmd: make install -== 2020-06-09 13:11:25,186 run.py:538 INFO cmd " make install " exited with exit code 0 and output: -Making install in src -make[1]: Entering directory `/tmp/example/HDF5/1.10.7/gompi-2020b/hdf5-1.10.7/src' -... -``` - -It can be useful to look for the *first* error that occurred in the output of a command, since subsequent errors are -often fallout from earlier errors. You can do this by first navigating -to the start of the output for a command using "`INFO running cmd`" as a search pattern, and then looking for patterns -like "`error:`" from there. - -## Inspecting the build directory - -When an installation fails the corresponding build directory is *not* cleaned up -automatically, that is only done for successful installations. -This allows you to dive in and check for clues in the files that are stored there. - -The location of the build directory is mentioned in the EasyBuild error message: - -``` -== FAILED: Installation ended unsuccessfully (build directory: /tmp/example/example/1.0/GCC-10.2.0): ... -``` - -For software using a classic `configure` script, you may have to locate -and inspect the `config.log` file in the build directory to determine the underlying cause of an error. For software using CMake as a configuration tool you often have to check in -`CMakeOutput.log` or `CMakeError.log` for clues, which are sneakily hidden by CMake in a `CMakeFiles` subdirectory of the build directory. - -As a side note here: as EasyBuild does not clean out old and failed builds you will need to eventually manually remove these build directories from the `buildpath` directory. On the login nodes of LUMI this is currently not much of an issue as `$XDG_RUNTIME_DIR` is (ab)used for build and -temporary files and that directory is cleaned automatically. However, when building on compute nodes, where `$XDG_RUNTIME_DIR` is -not available, it is essential to manually clean that directory as the space is not automatically cleaned when your session -ends. The `EasyBuild-user` module does define the bash function `clear-eb` that can be used to clear that space. - -## Exercise - -Let's work our way through a less smooth software installation, -using the easyconfig file that is provided below. - -Don't worry if most of this is still unclear to you, we'll get -to writing easyconfig files from scratch [later in this tutorial](2_02_creating_easyconfig_files). - -```python -easyblock = 'MakeCp' - -name = 'Subread' -version = '2.0.1' - -homepage = 'http://subread.sourceforge.net' -description = "High performance read alignment, quantification and mutation discovery" - -toolchain = {'name': 'PrgEnv-gnu', 'version': '21.10'} - -# download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz -sources = ['subread-%(version)s-source.tar.gz'] -checksums = ['d808eb5b1823c572cb45a97c95a3c5acb3d8e29aa47ec74e3ca1eb345787c17b'] - -start_dir = 'src' - -# -fcommon is required to compile Subread 2.0.1 with GCC 10, -# which uses -fno-common by default (see https://www.gnu.org/software/gcc/gcc-10/porting_to.html) -buildopts = '-f Makefile.Linux CFLAGS="-fast -fcommon"' - -files_to_copy = ['bin'] - -sanity_check_paths = { - 'files': ['bin/featureCounts', 'bin/subread-align'], - 'dirs': ['bin/utilities'], -} - -sanity_check_commands = ["featureCounts --version"] - -moduleclass = 'bio' -``` - -Do you spot any potential problems yet with this easyconfig file? - -***Preparation*** - -Start by copying the text above in a file named `subread.eb` -(which does not follow the EasyBuild conventions but that is not a problem for this exercise), -so you can gradually fix the problem you'll encounter. - -Also make sure that the pre-installed software stack is available, -and that the EasyBuild-user module is loaded (unless you installed EasyBuild -yourself): - -``` -module load LUMI/21.12 -module load EasyBuild-user -``` - -This will configure EasyBuild correctly for this exercise, though if you already have -an existing EasyBuild user installation you may want to work in a different one -by pointing `$EBU_USER_PREFIX` to the desired work directory before loading -`LUMI/21.12`. - -Check your configuration via `eb --show-config`. - -Strictly speaking the configuration doesn't matter much for the sake of this -exercise, but it may help with the step-wise approach we'll take and -grasping the solutions. - -Remember though: *no peeking* before you tried to solve each step yourself! - ---- - -***Exercise T.1**** - Sources* - -Try to install the `subread.eb` easyconfig file, see what happens. - -Can you fix the problem you run into, perhaps without even changing -the easyconfig file? - -??? success "(click to show solution)" - - The installation fails because the source file `subread-2.0.1-source.tar.gz` - is not found: - ``` - $ eb subread.eb - ... - == FAILED: Installation ended unsuccessfully (build directory: /run/user/XXXXXXXX/easybuild/build/Subread/2.0.1/cpeGNU-21.12): build failed (first 300 chars): - Couldn't find file subread-2.0.1-source.tar.gz anywhere, and downloading it didn't work either... - Paths attempted (in order): ... - ``` - - In this case, the problem is that the easyconfig file does not specify - where the sources can be downloaded from. Not automatically at least, - but there is a helpful comment included: - ```python - # download from https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz - sources = ['subread-%(version)s-source.tar.gz'] - ``` - - We can download the source tarball ourselves, - and move it to the location where EasyBuild expects to find it - (in the `sourcepath` directory): - ``` - curl -OL https://download.sourceforge.net/subread/subread-2.0.1-source.tar.gz - mkdir -p $EBU_USER_PREFIX/sources/s/Subread - mv subread-2.0.1-source.tar.gz $EBU_USER_PREFIX/sources/s/Subread/ - ``` - - Or, we can change the easyconfig file to specify the location where - the easyconfig file can be downloaded from: - ```python - source_urls = ['https://download.sourceforge.net/subread/'] - sources = ['subread-%(version)s-source.tar.gz'] - ``` - Note that the `source_urls` value is a *list* of candidate download URLs, - *without* the filename of the source file itself. - - This way, EasyBuild will download the source file when running `eb subread.eb`. - - The source tarball is fairly large (23MB), so don't be alarmed if the download takes a little while. - - ```shell - $ ls -lh $EBU_USER_PREFIX/sources/s/Subread - total 23M - -rw-rw-r-- 1 XXXXXXXX XXXXXXXX 23M Mar 30 16:08 subread-2.0.1-source.tar.gz - ``` - ---- - -***Exercise T.2**** - Toolchain* - -After fixing the problem with missing source file, try the installation again. - -What's wrong now? How can you fix it quickly? - -Take into account that we just want to get this software package installed, -we don't care too much about details like the version of the dependencies or -the toolchain here... - - -??? success "(click to show solution)" - - The installation fails because the easyconfig specifies that GCC 8.5.0 - should be used as toolchain: - - ``` - $ eb subread.eb - ... - ERROR: Failed to process easyconfig /pfs/lustrep3/users/kurtlust/easybuild-tutorial/Troubleshooting/subread.eb: Toolchain PrgEnv-gnu not found, - available toolchains: ... - ... - ``` - - `PrgEnv-gnu` is an HPE Cray PE module that may look like a toolchain - it certainly has - the same function: provide compiler, MPI and basic math libraries - but it is not - recognised as a toolchain by EasyBuild. EasyBuild prefers to manage its own modules so that it knows - well what is in it which is not the case with the `PrgEnv-*` modules from the Cray PE - as the content may differe between systems and as the versions of the compilers etc. that - are loaded differ on other modules that are loaded. Hence we created Cray-specific toolchains. - You'll actually find two series of Cray toolchains in the list of available toolchains. The - `CrayGNU`, `CrayIntel`, `CrayPGI` and `CrayCCE` are included with the EasyBuild distribution - and where developed by CSCS for their systems using Environment Modules. These were not compatible - with the initial releases of the Cray PE with Lmod modules so new ones were developed on which we - also built for the LUMI toolchains. Those are called `cpeCray`, `cpeGNU`, `cpeAOCC` and `cpeAMD` - and are maintained by LUST and available via the LUMI repositories. - - Changing the toolchain name to `cpeGNU` is not enough to solve all problems though: - - ``` - $ eb subread.eb - ... - ERROR: Build of /pfs/lustrep3/users/kurtlust/easybuild-tutorial/Troubleshooting/subread.eb failed (err: 'build failed (first 300 chars): - No module found for toolchain: cpeGNU/21.10') - ... - ``` - - We don't have this `cpeGNU` version installed, but we do have GCC 21.12: - - ```shell - $ module avail cpeGNU/ - ----- Infrastructure modules for the software stack LUMI/21.12 on LUMI-L ----- - cpeGNU/21.12 - ... - ``` - - So let's try using that instead. - - Edit the easyconfig file so it contains this: - - ```python - toolchain = {'name': 'cpeGNU', 'version': '21.12'} - ``` ---- - -***Exercise T.3**** - Build step* - -With the first two problems fixed, now we can actually try to build the software. - -Can you fix the next problem you run into? - -??? success "(click to show solution)" - - The compilation fails, but the error message we see is incomplete due to - EasyBuild truncating the command output (only the 300 first characters of the output are shown): - ``` - == FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12): build failed - (first 300 chars): cmd " make -j 256 -f Makefile.Linux CFLAGS="-fast -fcommon"" exited with exit code 2 and output: - gcc -mtune=core2 -O3 -DMAKE_FOR_EXON -D MAKE_STANDALONE -D SUBREAD_VERSION=\""2.0.1"\" -D_FILE_OFFSET_BITS=64 -fmessage-length=0 - -ggdb -fast -fcommon -I/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86 (took 4 secs) - ``` - - If you open the log file (e.g., with `view $(eb --last-log)`) and scroll to the end, - the error is pretty obvious: - ``` - gcc: error: unrecognized command line option -fast; did you mean -Ofast? - make: *** [core.o] Error 1 - ``` - - The easyconfig file hard specifies the `-fast` compiler flag via the `CFLAGS` argument to the build command: - ```python - # -fcommon is required to compile Subread 2.0.1 with GCC 10, - # which uses -fno-common by default (see https://www.gnu.org/software/gcc/gcc-10/porting_to.html) - buildopts = '-f Makefile.Linux CFLAGS="-fast -fcommon"' - ``` - - EasyBuild sets up the build environment, so there should be no need - to hard specify compiler flags (certainly not incorrect ones). - The comment above the `buildopts` definition makes it clear that the `-fcommon` - flag *is* required though, because GCC 10 became a bit stricter by - using `-fno-common` by default (and we're using GCC 11 in `cpeGNU/21.12`). - Note that we are using `-fcommon` - as an escape mechanism here: it would be better to fix the source code - and create a patch file instead. - - An easy way to fix this problem is to replace the `-fast` with `-Ofast`, - as the compiler error suggests. - - In this case it is advised to change the `CFLAGS` argument that is added - to be build command to replace the `-fast` with `$CFLAGS`, - which is defined in the build environment by EasyBuild. - - ```python - buildopts = '-f Makefile.Linux CFLAGS="$CFLAGS -fcommon"' - ``` - - Note that we need to be careful with quotes here: we use inner double quotes - to ensure that `$CFLAGS` will be expanded to its value when the build command is run. - As you can see after the change by doing a dry-run: - - ``` - $ eb subread.eb -x - ... - - Defining build environment... - - export BLAS_INC_DIR='/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/include' - ... - export CC='cc' - export CFLAGS='-O2 -ftree-vectorize -fno-math-errno' - ... - [build_step method] - running command "make -j 256 -f Makefile.Linux CFLAGS="$CFLAGS -fcommon"" - (in /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12/Subread-2.0.1/src) - ... - ``` - - EasyBuild will launch the command - ``` - make -j 256 -f Makefile.Linux CFLAGS="$CFLAGS -fcommon" - ``` - in a shell where `CFLAGS` is defined and set to an appropriate value (determined by - defaults in EasyBuild, settings in the EasyBuild configuration and settings in the - easyconfig file that we shall discuss later). - ---- - -***Exercise T.4**** - Sanity check* - -After fixing the compilation issue, you're really close to getting the installation working, we promise! - -Don't give up now, try one last time and fix the last problem that occurs... - -??? success "(click to show solution)" - - Now the installation itself works but the sanity check fails, - and hence the module file does not get generated: - ``` - $ eb subread.eb - ... - == FAILED: Installation ended unsuccessfully (build directory: /run/user/10012026/easybuild/build/Subread/2.0.1/cpeGNU-21.12): - build failed (first 300 chars): Sanity check failed: sanity check command featureCounts --version exited with code 255 - (output: featureCounts: unrecognized option '--version' - ... - ... - ``` - - If you look at the full output in the log file you can see - that the correct option to check the version of the `featureCounts` command - is "`-v`" rather than "`--version`", so we need to fix this in the easyconfig file. - - Make the following change in the easyconfig file: - ```python - sanity_check_commands = ["featureCounts -v"] - ``` - - After doing so, **you don't have to redo the installation - from scratch**, you can use the `--module-only` option to only run the - sanity check and generate the module file again: - ``` - eb subread.eb --module-only - ``` - ---- - -In the end, you should be able to install Subread 2.0.1 with the cpeGNU 21.12 toolchain by fixing the problems with the `subread.eb` easyconfig file. - -Check your work by manually loading the module and checking the version -via the `featureCounts` command, which should look like this: - -```shell -$ module load Subread/2.0.1-cpeGNU-21.12 -... -$ featureCounts -v -featureCounts v2.0.1 -``` - ---- - -*[[next: Creating easyconfig files]](2_02_creating_easyconfig_files.md)* diff --git a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md b/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md deleted file mode 100644 index f9bec25a..00000000 --- a/docs/2022-CSC_and_LO/2_02_creating_easyconfig_files.md +++ /dev/null @@ -1,1027 +0,0 @@ -# Creating easyconfig files - -*[[back: Troubleshooting]](2_02_troubleshooting)* - ---- - -We already know how to install easyconfig files that are provided by LUMI, -but what about installing software for which no easyconfig is available yet, -or for which we need to adapt an existing easyconfig? - -To do this we will need to **create additional easyconfig files**, -since every software installation performed by EasyBuild is done based on an easyconfig file. - -In this part of the tutorial we will look at the guts of easyconfig files and even create some ourselves! - -## Easyconfigs vs easyblocks - -Before we dive into writing [easyconfig files](../1_05_terminology/#easyconfig-files), -let us take a brief look at how they relate to [easyblocks](../1_05_terminology/#easyblocks). - -As we discussed [earlier](../1_05_terminology/#terminology), an easyconfig file (`*.eb`) is required -for each installation -performed by EasyBuild which specifies the details of the installation (which software -version, toolchain, etc.), while the installation procedure is implemented -in an easyblock (a Python module). - -When can we leverage a *generic easyblock*, perhaps via a "fat" easyconfig file that includes -a lot of carefully defined easyconfig parameters, and when should we use a minimal easyconfig file -together with a custom *software-specific* easyblock? - -This is not an easy question to answer in a general sense, since it depends on several factors: -the complexity of the software you want to get installed, how much flexibility you want, -how "intelligent" the installation procedure should be with respect to the compiler toolchain and dependencies -that are used for the installation, whether you want an easyconfig that is more user-focused (with easy-to-understand -parameters) or one that is more focused towards people who understand installation commands, etc. - -In a nutshell, custom software-specific easyblocks are "do once and forget": they are central solution to peculiarities in the installation procedure of a particular software package. However, they also hide a lot from direct view, so if things go -wrong, it is often harder to debug the exact problem. And since a single easyblock has to cover -multiple software versions, multiple toolchains and multiple possible configurations of the package, -they are often harder to develop and certainly harder to test. As a result many of the easyblocks -included with EasyBuild work poorly on HPE Cray systems, e.g., because they want to add compiler flags -specific for a certain compiler and don't recognise the Cray compilers. - -Reasons to consider implementing a software-specific easyblock rather than using a generic easyblock include: - -- 'critical' values for easyconfig parameters required to make installation succeed; -- toolchain-specific aspects of the build and installation procedure (e.g., configure options); -- interactive commands that need to be run; - *For example: The easyblock to [install maple](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/m/maple.py) - responds to a number of interactive questions.* -- custom (configure) options for dependencies; - *For example, the [easyblock for VMD](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/v/vmd.py) - will add configure options depending on the dependency list of the package.* -- having to create or adjust specific (configuration) files; -- 'hackish' usage of a generic easyblock; -- complex or very non-standard installation procedure; - *For example, the [easyblock to install the gcc compilers from source](https://github.com/easybuilders/easybuild-easyblocks/blob/develop/easybuild/easyblocks/g/gcc.py), - bootstrapping with the system compiler and then re-installing with itself.* - -For implementing easyblocks we refer to the -["Implementing easyblocks" section of this tutorial](2_03_implementing_easyblocks) -and the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Implementing-easyblocks.html). - -## Writing easyconfig files - -Writing an easyconfig file boils down to defining a set of easyconfig parameters in a text file, -which we give a '`.eb`' extension by convention. The name of an easyconfig file doesn't matter -when using it directly to install software, but it does matter when EasyBuild needs to find it -to resolve a dependency for example (as we [discussed earlier](../1_08_basic_usage/#easyconfig-filenames)). - -The syntax for easyconfig files is *Python syntax*: you are basically defining a bunch of Python variables -that correspond to easyconfig parameters. - -The order in which the easyconfig parameters are defined doesn't matter, but we generally try to stick to a particular -order which roughly corresponds to the order in which the easyconfig parameters are used during the installation. -That is mostly helpful for humans staring at easyconfig files or comparing them. - -### Mandatory parameters - -A limited number of easyconfig parameters are *mandatory*, they must be defined in every easyconfig file: - -- `name`: the name of the software to install; -- `version`: the version of the software to install; -- `homepage`: a URL to the website of the software; -- `description`: a description of the software; -- `toolchain`: the compiler toolchain to use for the installation; - -#### `name`, `version` - -It should be no surprise that specifying the name and version of the software you want to install is mandatory. -This information may influence the value of several other easyconfig parameters (like the name of the source file), and is also used to the determine the name of the module file to install. - -```python -name = 'example' -version = '1.0' -``` - -#### `homepage`, `description` - -The homepage and description are included in the generated module file for the installation. -That way the "`module show`" command provides some useful high-level information about the installation. - -```python -homepage = 'https://example.org' -description = "This is just an example." -``` - -Usually it does not matter whether you use single or double quotes to specify string values. -There are cases where it is important to use -the right type of quotes however, we will get back to that later (keep it in mind for the exercises!). - -For multi-line descriptions, you will need to use "triple quoting" (which is standard Python syntax): - -```python -description = """This is an example - of a multi-line description. - It is spread across multiple lines.""" -``` - -If no homepage is known for the package, the convention in the EasyBuild community is to use -`(none)` as the value. - -The `description` field is used in two different places in the LUA module file generated by -EasyBuild: -- In the help block, for the first section of the help information -- If there is no `whatis` parameter in the easyconfig file, it is also used to generate - a whatis-command with the description in the module file. - -*Note:* We ask people who contribute easyconfigs to LUMI to instead use a decent description -that is useful in the help information, and to specify a short description via the `whatis` -parameter: - -```python -whatis = ['Description: Blosc is an extremely fast, multi-threaded, meta-compressor library'] -``` - -Note that in this case the word "Description:" has to be explicitly specified. - - -#### `toolchain` - -EasyBuild also requires that the [compiler toolchain](../1_05_terminology/#toolchains) is specified, via the `toolchain` -easyconfig parameter. - -This can either be the [`system` toolchain](../1_05_terminology/#system-toolchain), for which a constant named `SYSTEM` is available: - -```python -toolchain = SYSTEM -``` - -Usually we specify a 'proper' toolchain like the `cpeGNU/21.12` toolchain we have used bvefore. -The name and version of the toolchain can be specified using a small Python dictionary, -for example: - -```python -toolchain = {'name': 'cpeGNU', 'version': '21.12'} -``` - -### Commonly used parameters - -You will often need to specify additional easyconfig parameters to get something useful done. -We will cover the most commonly used ones here, but keep in mind that these are *not* mandatory. - -A full overview of all known easyconfig parameters can be obtained via "`eb --avail-easyconfig-params`" -or just "`eb -a`" for short, or can be consulted in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_parameters.html). - -#### Sources, patches, and checksums - -In most easyconfig files you will see that a list of source files is specified via the `sources` -easyconfig parameter, usually combined -with one or more URLs where these sources can be downloaded specified via `source_urls`. -There also may be patch files listed (specified via `patches`), -and checksums for both the source files and patches (specified via `checksums`). - -The `sources` easyconfig parameter is commonly defined but it is *not* mandatory, -because some easyconfig files only specify bundles of software packages and hence only -serve to generate a module file. - -Here is an example of how these easyconfig parameters can be specified: - -```python -source_urls = [ - 'https://example.org/download/', - 'https://example.org/download/archive/', -] -sources = ['example-1.0-src.tar.gz'] -patches = ['example-fix.patch'] -checksums = [ - '9febae18533d035ac688d977cb2ca050e6ca8379311d7a14490ad1ef948d45fa', - '864395d648ad9a5b75d1a745c8ef82b78421d571584037560a22a581ed7a261c', -] -``` - -Each of these require a *list* of values, so even if there is only a single source file or download URL -you must use square brackets as shown in the example. The default value for each of these is an empty list (`[]`). - -Some things worth pointing out here: - -* The download URLs specified via `source_urls` do *not* include the name of the file, that is added - automatically by EasyBuild when it tries to download the file (only if it's not available already.) -* If multiple download URLs are specified, they are each tried once in order until the download of the source file was - successful. This can be useful to include backup locations where source files can be downloaded from. -* Names of source files and patches should not include hardcoded software versions, they usually use a - template value like `%(version)s` instead: - ```python - sources = ['example-%(version)s-src.tar.gz'] - ``` - EasyBuild will use the value of the `version` easyconfig parameter to determine the actual name of the source - file. This way the software version is only specified in one place and the easyconfig file is easier to - update to other software versions. A list of template values can be consulted via the EasyBuild command - line via the `--avail-easyconfig-templates` option, or in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/easyconfig_templates.html). -* Source files can also be specified in ways other than just using a filename, see the - [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#common-easyconfig-param-sources-alt) for more information. - It is also possible to download a given commit from a GitHub repository. -* Specified checksums are usually SHA256 checksum values, but - [other types are also supported](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html?highlight=checksums#checksums). - - -#### Easyblock - -The easyblock that should be used for the installation can be specified via the `easyblock` easyconfig parameter. - -This is not mandatory however, because by default EasyBuild will determine the easyblock to use based on the -name of the software. If '`example`' is specified as software name, EasyBuild will try to locate a -software-specific easyblock named `EB_example` (in a Python module named `example.py`). Software-specific -easyblocks follow the convention that the class name starts with `'EB_`', followed by the software name -(where some characters are replaced, like '`-`' with '`_minus_`'). It is possible to use different -naming conventions for software-specific easyblocks, but then EasyBuild will not automatically detect -that there is one for the package and it will also need to be specified via the `easyblock` parameter. - -**Generic easyblocks** - -Usually the `easyblock` value is the name of a *generic* easyblock, if it is specified. The name of -a generic easyblock does *not* start with '`EB_`', so you can easily distinguish it from a software-specific -easyblock. - -Here are a couple of commonly used generic easyblocks: - -* `ConfigureMake`: implements the standard `./configure`, `make`, `make install` installation procedure; -* `CMakeMake`: same as `ConfigureMake`, but with `./configure` replaced with `cmake` for the configuration step; -* `PythonPackage`: implements the installation procedure for a single Python package, by default using - "`python setup.py install`" but other methods like using "`pip install`" are also supported; -* `Bundle`: a simple generic easyblock to bundle a set of software packages together in a single installation directory; -* `PythonBundle`: a customized version of the `Bundle` generic easyblock to install a bundle of Python packages - in a single installation directory; - -A full overview of the available generic easyblocks is available in the [EasyBuild documentation](https://docs.easybuild.io/en/latest/version-specific/generic_easyblocks.html). You can also consult the output of -`eb --list-easyblocks`, which gives an overview of *all* known easyblocks, and how they relate to each other. - -**Custom easyconfig parameters** - -Most generic easyblocks provide additional easyconfig parameters to steer their behaviour. -You can consult these via "`eb -a --easyblock`" or just "`eb -a -e`", which results in an -additional "`EASYBLOCK-SPECIFIC`" section to be added. See the (partial) output of this command for example: - -```shell -$ eb -a -e ConfigureMake -Available easyconfig parameters (* indicates specific to the ConfigureMake easyblock): -... -EASYBLOCK-SPECIFIC ------------------- -build_cmd* Build command to use [default: "make"] -build_type* Value to provide to --build option of configure script, e.g., x86_64-pc-linux-gnu (determined by config.guess shipped with EasyBuild if None, False implies to leave it up to the configure script) [default: None] -configure_cmd* Configure command to use [default: "./configure"] -configure_cmd_prefix* Prefix to be glued before ./configure [default: ""] -host_type* Value to provide to --host option of configure script, e.g., x86_64-pc-linux-gnu (determined by config.guess shipped with EasyBuild if None, False implies to leave it up to the configure script) [default: None] -install_cmd* Build command to use [default: "make install"] -prefix_opt* Prefix command line option for configure script ('--prefix=' if None) [default: None] -tar_config_opts* Override tar settings as determined by configure. [default: False] -``` - -#### Dependencies - -You will often need to list one or more [dependencies](../1_05_terminology/#dependencies) that are required -to install or run the software. -We distinguish between two main different types of dependencies: runtime dependencies and build dependencies. - -*Runtime dependencies* are required for using the installed software, and may also have to be available -during the installation. These dependencies can be specified via the `dependencies` easyconfig parameter. -EasyBuild will load the modules for these dependencies when setting up the build environment, -and will include load statements for them in the generated module file. - -*Build dependencies* are only required during the installation of the software, not for using the -software once it is installed. The modules for these dependencies will be loaded in the build environment -set up by EasyBuild during the installation, but they will *not* be loaded by the generated module file. -You can specify build dependencies via the `builddependencies` easyconfig parameter. -One typical example of a build dependency is `CMake`, which is only needed for configuring -the build. On LUMI we define a `buildtools` module for each version of the `LUMI` software stack -which contains most popular build tools. This ensures that we use a consistent set of build tools -and that they do not change over the life time of a stack, also in case of OS upgrades, to -increase the reproducibility of the build process - -Here is a simple example of specifying dependencies: - -```python -builddependencies = [ - ('buildtools', '%(toolchain_version)s', '', True) -] - -dependencies = [ - ('Python', '3.8.2'), - ('HDF5', '1.10.6'), - ('SciPy-bundle', '2020.03', '-Python-%(pyver)s'), -] -``` - -Both `builddependencies` and `dependencies` require a list of tuples, -each of which specifying one dependency. -The name and version of a dependency is specified with a 2-tuple (a tuple with two string values). - -In some cases additional information may have to be provided, as is shown in the example above for the `SciPy-bundle` -dependency where a 3rd value is specified corresponding to the `versionsuffix` value of this dependency. -If this is not specified, it is assumed to be the empty string (`''`). - -Note how we use the '`%(pyver)s'` template value in the `SciPy-bundle` dependency -specification, to avoid hardcoding the Python version in different places. (Though this -specific parameter is less useful on LUMI as we currently try to build on top of `cray-python`.) - -The `buildtools` build dependency shows that there is a fourth parameter specifying the toolchain -used for that dependency and is needed if that toolchain is different from the one used in the example. -As it is not possible to load several Cray toolchains together (they are not in a hierachical relation) -the only useful value on LUMI is `True` which tells that `buildtools` is build with the `SYSTEM` -toolchain. Here also we use a template, `%(toolchain_version)s` which - as its name suggests - expands -to the version of the toolchain, as we version our `buildtools` modules after the version of the Cray -toolchains for which they are intended. - -See also the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html#dependencies) -for additional options on specifying dependencies. That page specifies two more dependency types: - -* `hiddendependencies` are currently not used on LUMI, and if we would use them in the future, it will likely - be through a way that does not require this parameter. -* `osdependencies` can be used to let EasyBuild check if certain needed OS packages are installed. - See, e.g., the [easyconfigs for the `buildtools` package](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/easybuild/easyconfigs/b/buildtools) - on LUMI. - - -#### Version suffix - -In some cases you may want to build a particular software package in different configurations, -or include a label in the module name to highlight a particular aspect -of the installation. - -The `versionsuffix` easyconfig parameter can be used for this purpose. -The name of this parameter implies that this label will be added after the -software version (and toolchain label) in the standard module naming scheme. - -If you are configuring the software to build with a particular non-default value, -you can indicate this as follows for example: - -```python -versionsuffix = '-example-label' -``` - -This mechanism is used frequently to indicate that a software installation depends -on a particular version of Python, where the `%(pyver)s` template comes in useful again: - -```python -versionsuffix = '-Python-%(pyver)s' -... -dependencies = [('Python', '2.7.18')] -``` - -Even though Python 2 is officially dead and -buried some scientific software still requires it, and mixing modules where -some use Python 2 and other use Python 3 doesn't work well. -The `versionsuffix` label is helpful to inform the user that a particular Python version is -required by the installation. - -#### Customizing configure, build, test, and install commands - -When using a generic easyblock like `ConfigureMake` or `CMakeMake`, you will often -find yourself having to specify options to the configure, build, test, or install commands, or to inject additional commands right before them. - -For this the following standard easyconfig parameters are available: - -* `preconfigopts`: string value that is glued *before* the configure command; -* `configopts`: string value that is added *after* the configure command, which can be used to specify configuration options; - -Equivalent easyconfig parameters are available for the `build`, `test` and `install` steps: `prebuildopts`, `buildopts`, `pretestopts`, `testopts`, `preinstallopts`, and `installopts`. - -Here is a fictitious example of how they can be used: - -```python -easyblock = 'ConfigureMake' -... -dependencies = [('HDF5', '1.10.6')] -... -configopts = '--enable-hdf5-support' - -prebuildopts = 'export HDF5_PREFIX="$EBROOTHDF5" && ' - -installopts = "PREFIX='%(installdir)s'" -``` -(again an imaginary example as on LUMI we advise to use the Cray-provided HDF5 modules.) - -Here we are: - -* Adding the `--enable-hdf5-support` configure option, to convince the `ConfigureMake` easyblock to run the following - command during the configure step: - ```shell - ./configure --prefix ... --enable-hdf5-support - ``` - (where the '`...`' represents the path to installation directory where the software should be installed). - -* Specifying that an additional command has to be run before running `make` in the build step. We use '`&&`' to glue the - command to the `make` command, so `make` will only be run if the command we specified ran correctly. So, the build step will run something like: - ```shell - export HDF5_PREFIX="$EBROOTHDF5" && make -j 4 - ``` - The '`4`' value passed to the `-j` option shown here, which specifies how many commands `make` can run in parallel, is automatically determined by EasyBuild based on the number of available cores (taking into account `ulimit` settings, and cpuset and cgroup restrictions). - -* Passing the location where the software should be installed via the `PREFIX` argument to the `make install` command during the installation step. This results in the - following command being run: - ```shell - make install PREFIX=... - ``` - (where the '`...`' again represents the path to installation directory). - Even though the - installation directory is already specified in the configure command, it is - apparently blatantly ignored by the software we are installing here, and we are expected to specify it - this way instead. How rude! - -The `$EBROOTHDF5` environment variable that we are using in `prebuildopts` corresponds to the path of -the installation directory of the HDF5 dependency. EasyBuild includes a statement -to define an `$EBROOT*` environment variable -like this in every environment module file it generates (see the output of "`module show HDF5`"). - -#### Sanity check - -One seemingly trivial yet important aspect of the installation procedure that EasyBuild performs -is the sanity check step. - -By default EasyBuild does a simple sanity check that verifies whether there is a non-empty `bin` subdirectory -in the installation, next to a non-empty `lib` or `lib64` directory (either is sufficient). - -It is recommended to customize the sanity check and check for something more specific, like a particular -binary or directory, or making sure that a trivial command (like `example -V` or `example --help`) -runs correctly. - -To specify a custom set of files and/or directories to check, -you can use the `sanity_check_paths` easyconfig parameter. The expected value is Python dictionary -with two keys: `files` and `dirs`. For example: - -```python -sanity_check_paths = { - 'files': ['bin/example'], - 'dirs': ['examples/one', 'examples/two'], -} -``` - -In addition, you can specify one or more commands that should be working without a problem (that is, have a zero exit status) via the `sanity_check_commands` easyconfig parameter. -These commands will be run just like a user would: after loading the module that was generated for this installation. -Here is an example: - -```python -sanity_check_commands = [ - "example --version", - "example --help", -] -``` - - -#### Module class - -Finally, you will usually see the `moduleclass` easyconfig parameter to be defined as well, for example: - -```python -moduleclass = 'lib' -``` - -This is done to categorize software, and it is used to group the generated module files into smaller sets -([remember what we saw when installing software earlier](../1_08_basic_usage/#using-installed-software)). - -This is currently not used on LUMI since we feel that (a) it is not easy to explain to users how they can -then only make certain classes that are useful to them visible and more importantly (b) since it is not -always intuitive to decide which moduleclass should be used for a package, or from a user 's perspective, -in which category to look for a package. - -## Generating tweaked easyconfigs - -Sometimes you may want to install software that differs only slightly from an -existing easyconfig file, like a newer software version or using a different -compiler toolchain. Do we need to create an easyconfig file for this too? - -We do, but EasyBuild does provide some help so you don't need to *manually* -create the easyconfig file. You can use one of the `--try-*` options provided -by the `eb` command to make EasyBuild *generate* a new easyconfig file based on -an existing one. - -For example, to try installing a different software version you can use the `--try-software-version` option: - -```shell -eb example-1.2.3.eb --try-software-version 1.2.4 -``` - -Or, to try using a different compiler toolchain you can use `--try-toolchain`: - -```shell -eb example-1.2.3-foss-2020b.eb --try-toolchain cpeCray,21.12 -``` - -It is important to keep in mind the *"try"* aspect here: while easyconfigs that -are generated by EasyBuild via a `--try-*` option often do work fine, there is -no strong guarantee they will. Newer software versions may come with changes to -the installation procedure, additional dependencies that are required, etc. -Using a different compiler toolchain may be as simple as just switching one for -another, but it may require additional changes to be made to configure options, for example. - -## Copying easyconfigs - -One additional handy command line option we want to highlight is `--copy-ec`, which can be used to -copy easyconfig files to a specific location. That may sound trivial, but -keep in mind that you can specify easyconfigs to the `eb` command using only -the filename, and letting the robot search mechanism locate them. - -So to copy an easyconfig file, we would have to use `eb --search` first to -get the full location to it, copy-paste that, and then use the `cp` command. - -It is a lot easier with `--copy-ec`: - -```shell -$ eb --copy-ec SAMtools-1.11-GCC-10.2.0.eb SAMtools.eb -... -SAMtools-1.10-GCC-10.2.0.eb copied to SAMtools.eb -``` - -If you omit the target location, the easyconfig file will simply be copied -to the current working directory, retaining the original filename. - -You can copy multiple easyconfig files at once, as long as the target location -is an existing directory. - -## Example - -By means of example, we are going to puzzle together an easyconfig file to install the -example software package `eb-tutorial`. - -The sources for `eb-tutorial` version 1.0.1 are available at: - -``` -https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.0.1.tar.gz -``` - -You can consult the unpacked sources at [https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1). - -### Preparation - -Make sure that a proper version of the `LUMI` software stack is loaded -(we recommend `LUMI/21.12` for the session for which this tutorial was designed) -and also `EasyBuild-user` is loaded to configure EasyBuild to install in -`$HOME/EasyBuild` or `$EBU_USER_PREFIX`. - -``` shell -module load LUMI/21.12 -module load EasyBuild-user` -``` - - -### Easyblock - -Let's start by getting the mandatory easyconfig parameters defined in the easyconfig file: - -```python -name = 'eb-tutorial' -version = '1.0.1' - -homepage = 'https://easybuilders.github.io/easybuild-tutorial' - -whatis = [ 'Description: EasyBuild tutorial example'] - -description = """ -This is a short C++ example program that can be buid using CMake. -""" -``` - -Let's see what EasyBuild does with this very minimal easyconfig file: - -```shell -$ eb eb-tutorial.eb -== Temporary log file in case of crash /run/user/XXXXXXXXX/easybuild/tmp/eb-k_82wotb/easybuild-wg0k_reb.log -ERROR: Failed to process easyconfig /pfs/lustrep4/users/XXXXXXXX/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v1.eb: -No software-specific easyblock 'EB_eb_minus_tutorial' found for eb-tutorial -``` - -It is not mandatory to specify an easyblock in the easyconfig. However, in the absense of that -specification, EasyBuild goes looking for an application-specific easyblock with the standard name, -in this case `EB_eb_minus_tutorial`, which it does not have. Does that mean we have to implement an easyblock? - -In this simple case it doesn't, since we can leverage one of the available *generic easyblocks*. -But, which one? - -Build instructions are usually included in a `README` file, or in the documentation. -In this case, there's indeed a minimal [`README` -file](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1/README) available, -which tells us that we should use the `cmake` command to configure the installation, followed by `make` and `make install`. - -[We briefly discussed](#easyblock) a generic easyblock that does exactly this: `CMakeMake`. - -```python -easyblock = 'CMakeMake' -``` - -The "`easyblock =`" line is usually at the top of the easyconfig file, but strictly speaking -the order of the parameter definitions doesn't matter (unless one is defined in terms of another one). - - -### Mandatory parameters - -When trying this improved easyconfig file, EasyBuild will inform us that we failed to specify -one of the mandatory easyconfig parameters: `toolchain`: - -``` -$ eb example.eb -== Temporary log file in case of crash /run/user/XXXXXXXXX/easybuild/tmp/eb-22om7hut/easybuild-ipozjipf.log -ERROR: Failed to process easyconfig /pfs/lustrep4/users/XXXXXXXX/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v2.eb: -mandatory parameters not provided in pyheader: toolchain -``` - -We will use `cpeCray/21.12` as toolchain, so we also define the `toolchain` easyconfig parameter: - -```python -toolchain = {'name': 'cpeCray', 'version': '21.12'} -``` - -In addition, we'll also specify the `moduleclass`. -This is not required, but it is usually set to a sensible value: - -```python -moduleclass = 'tools' -``` - -The default value is '`base`', at least '`tools`' has *some* meaning. - -### CMake build dependency - -The `CMakeMake` easyblock needs the `cmake` command. On LUMI we are lucky as -`cmake` is already installed in the OS. It may be a somewhat older version, but for this -program is is probably enough. Yet in general it is better to use sufficiently recent -build tools, and `cmake` is one of those tools that is typically entered as a build -dependency. After all, the less you rely on the OS, the more likely it becomes that -your easyconfig is useful for other sites also. - -In the [section on Lmod](../1_02_Lmod#module-extensions) we've already seen that on LUMI -the `cmake` command is available through the `buildtools` modules, and as discussed in -other examples on this page, LUMI has one for every `LUMI` software stack with its version -number the same as the stack and corresponding toolchains. It is a good practice to -add this module as a build dependency: - -```python -builddependencies = [ - ('buildtools', '%(toolchain_version)s', '', True) -] -``` - -In a more traditional EasyBuild setup with the common toolchains, there is usually no need to specify -the toolchain for (build) dependencies. EasyBuild will automatically consider -[subtoolchains](../1_05_terminology/#toolchains) compatible with the specified toolchain to locate -modules for the dependencies. However, the Cray PE toolchains on LUMI are currently not part of -such a hierarchy and the `SYSTEM` toolchain we used for `buildtools` is not automatically considered -which is why we need the 4-element version of the dependency specification. - -You can verify that EasyBuild now locates the dependency via `eb -D` (equivalent with `eb --dry-run`): - -``` -$ eb eb-tutorial.eb -D - ... - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-common/buildtools/buildtools-21.12.eb (module: buildtools/21.12) - * [x] /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/cpeCray/cpeCray-21.12.eb (module: cpeCray/21.12) - * [ ] /pfs/lustrep4/users/kulust/easybuild-tutorial-exercises/2022-CSC_and_LO-examples/2_02_example_eb-tutorial/eb-tutorial_v4.eb (module: eb-tutorial/1.0.-cpeCray-21.12) -``` - -### Sources - -If you try again after adding `buildtools` as a build dependency, you will see the installation fail again in the -configuration step. Inspecting the log file reveals this: - -``` -CMake Error: The source directory "/run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12" does not appear to contain CMakeLists.txt. -``` - -Wait, but there *is* a `CMakeLists.txt`, we can see it in the -[unpacked sources](https://github.com/easybuilders/easybuild-tutorial/tree/main/docs/files/eb-tutorial-1.0.1)! - -Let's inspect the build directory: - -``` -$ ls /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12 -easybuild_obj -$ ls /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12/easybuild_obj -$ -``` - -There's nothing there at all! And that's not strange because we didn't actually -specify any sources in our easyconfig file... - -The `sources` easyconfig parameter is commonly defined but it is *not* mandatory, -because some easyconfig files only specify bundles of software packages and hence only -serve to generate a module file. - -So we need to specify one or more source files that should be used, -via the `sources` easyconfig parameter which specifies a *list* of -names of source files: - -```python -sources = ['eb-tutorial-1.0.1.tar.gz'] -``` - -We can avoid hardcoding the version number here by using a *template value*: - -```python -sources = ['eb-tutorial-%(version)s.tar.gz'] -``` - -And since this is a standard way of naming software files, there's -even a constant available that we can use: - -```python -sources = [SOURCE_TAR_GZ] -``` - -That way, we only have the software version specified *once* in the easyconfig file, -via the `version` easyconfig parameter. That will come in useful later (see [Exercise 7.2](#exercises))... - -If now we try installing the easyconfig file again, EasyBuild complains -that it can't find the specified source file anywhere: - -``` -Couldn't find file eb-tutorial-1.0.1.tar.gz anywhere, and downloading it didn't work either... -Paths attempted (in order):... -``` - -To let EasyBuild automatically download the source file if it is not available yet, -we have to specify *where* it can be downloaded. This is done via `source_urls`: - -```python -source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] -sources = [SOURCE_TAR_GZ] -``` - -### Required configure option - -With `sources` and `source_urls` defined, we can try again. Yet again we see the configure step fail. -Is this a ruse to make you hate CMake with a passion? Maybe... - -Here's what we find in the log file: - -``` -CMake Error at CMakeLists.txt:7 (message): - EBTUTORIAL_MSG is not set! -``` - -Apparently the `eb-tutorial` software has a required configure option. It's almost as if that -was done on purpose, how silly! - -Options to the configure command can be specified by the `configopts` easyconfig parameter. -To define the value of a CMake option, we need to use `-DNAME_OF_OPTION`, so: - -```python -configopts = "-DEBTUTORIAL_MSG='Hello from the EasyBuild tutorial!' " -``` - -We need to be a little bit careful with quotes here. If we use outer double quotes, -we have to use single quotes to specify the actual value for the `EBTUTORIAL_MSG` configure option. -That works fine here, but that's not always the case! -In some cases we will have to use inner doubles quotes, for example to get environment variables -expanded when the configure command is run (see [Exercise 7.1](#exercises)). - -### Sanity check - -Hopefully that brings us closer to getting the installation to work... - -``` -$ eb eb-tutorial.eb -.... -== sanity checking... -== ... (took 2 secs) -== FAILED: Installation ended unsuccessfully (build directory: /run/user/XXXXXXXXX/easybuild/build/ebtutorial/1.0.1/cpeCray-21.12): -build failed (first 300 chars): Sanity check failed: no (non-empty) directory found at 'lib' or 'lib64' in -/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 (took 7 secs) -``` - -It got all the way to the sanity check step, that's great! - -The sanity check failed because no '`lib`' or `'lib64'` directory was found. -Indeed: - -``` -$ ls /users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 -bin -$ ls /users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 -eb-tutorial -``` - -There is only a binary named `eb-tutorial` in the `bin` subdirectory. -So we need to customize the standard sanity check: - -```python -sanity_check_paths = { - 'files': ['bin/eb-tutorial'], - 'dirs': [], -} -``` - -Since we want to obtain a *working* installation, we might as well try to run this `eb-tutorial` command as well: - -```python -sanity_check_commands = ['eb-tutorial'] -``` - -Let us now retry, but use `--module-only` rather than redoing the whole installation. -`--module-only` still sanity checks the installation, so if it creates -a module, we know it will work as expected. -By enabling trace mode via `--trace` we can get some more information too: - -```shell -$ eb eb-tutorial.eb --module-only --trace -... -== sanity checking... - >> file 'bin/eb-tutorial' found: OK - >> loading modules: eb-tutorial/1.0.1-cpeCray-21.12... - >> running command 'eb-tutorial' ... - >> result for command 'eb-tutorial': OK -... -== COMPLETED: Installation ended successfully (took 4 sec) -``` - -Yes, great success! - -To convince yourself that the installation works as intended, try to load the `eb-tutorial` module and -run the `eb-tutorial` command yourself: - -``` -$ module load eb-tutorial -$ eb-tutorial -Hello from the EasyBuild tutorial! -``` - -### Complete easyconfig - -Here is the complete easyconfig we puzzled together for this example: - -```python -easyblock = 'CMakeMake' - -name = 'eb-tutorial' -version = '1.0.1' - -homepage = 'https://easybuilders.github.io/easybuild-tutorial' -description = "EasyBuild tutorial example" - -toolchain = {'name': 'GCC', 'version': '10.2.0'} - -source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] -sources = [SOURCE_TAR_GZ] -checksums = ['d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9'] - -builddependencies = [('CMake', '3.18.4')] - -configopts = "-DEBTUTORIAL_MSG='Hello from the EasyBuild tutorial!' " - -sanity_check_paths = { - 'files': ['bin/eb-tutorial'], - 'dirs': [], -} - -sanity_check_commands = ['eb-tutorial'] - -moduleclass = 'tools' -``` - -One additional easyconfig parameter we defined here is `checksums`, -which specifies SHA256 checksums for source (and patch) files, -so EasyBuild can verify them before performing an installation. - -You can let EasyBuild determine *and* inject these SHA256 checksums -automatically via `eb --inject-checksums`: - -``` -$ eb eb-tutorial.eb --inject-checksums -... -== injecting sha256 checksums for sources & patches in example.eb... -== * eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 -``` - -Finally, we should consider changing the name of the easyconfig that we just developed -to align with the EasyBuild conventions as otherwise it would not be found when used as -a dependency of another package. In this case, the name should be -`eb-tutorial-1.0.0-cpeCray-21.12.eb`. In fact, EasyBuild stored a processed version -of our easyconfig with that name in the repository: - -``` -$ ls $EASYBUILD_REPOSITORYPATH -eb-tutorial -$ ls $EASYBUILD_REPOSITORYPATH/eb-tutorial -eb-tutorial-1.0.1-cpeCray-21.12.eb -$ cat $EASYBUILD_REPOSITORYPATH/eb-tutorial/eb-tutorial-1.0.1-cpeCray-21.12.eb -... -``` - -Note that EasyBuild has added an additional parameter to the easyconfig file, `buildstats`, -with a lot of information about how easybuild was called and properties of the node on which -it was run. This file is still a valid easyconfig file though from which we can build the -program again. - -Let's also inspect the installation directory when the whole build process has finished successfully. -This is very easy after loading the module, as EasyBuild-generated modules define a number of environment -variables for each module: - -``` -$ module load eb-tutorial -$ env | grep TUTORIAL -EBVERSIONEBMINTUTORIAL=1.0.1 -EBDEVELEBMINTUTORIAL=/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12/easybuild/eb-tutorial-1.0.1-cpeCray-21.12-easybuild-devel -EBROOTEBMINTUTORIAL=/users/XXXXXXXX/EasyBuild/SW/LUMI-21.12/L/eb-tutorial/1.0.1-cpeCray-21.12 -``` -The most interesting one of those variables is the `EBROOT` variable which points to the installation directory. -As variable names cannot contain minus signs, the minus in the module name is replaced with `MIN` in the name -of the variable (which is not the most consistent thing however as in the name of an easyblock it is replaced -with `minus`). - -Let's have a look in that directory: - -``` -$ ls $EBROOTEBMINTUTORIAL -bin easybuild -$ ls EBROOTEBMINTUTORIAL/easybuild -easybuild-eb-tutorial-1.0.1-20220401.184518.log -easybuild-eb-tutorial-1.0.1-20220401.184518_test_report.md -eb-tutorial-1.0.1-cpeCray-21.12-easybuild-devel -eb-tutorial-1.0.1-cpeCray-21.12.eb -reprod -$ ls EBROOTEBMINTUTORIAL/easybuild/reprod -easyblocks -eb-tutorial-1.0.1-cpeCray-21.12.eb -eb-tutorial-1.0.1-cpeCray-21.12.env -hooks -ls $EBROOTEBMINTUTORIAL/easybuild/reprod/easyblocks -cmakemake.py configuremake.py -``` - -As you can see, EasyBuild has also created the `easybuild` subdirectory (and it actually told us about that -at the end of the installation) which contains a lot of information about the build, also to make it easier -to reproduce a build process afterwards. - - ---- - -## Exercises - ---- - -***Exercise E.1**** - Making `eb-tutorial` a bit more personal* - -Change the easyconfig file for `eb-tutorial` to make the message printed by the `eb-tutorial` command -a bit more personal: include the username of the account that was used to install the software in it -(using the `$USER` environment variable). - -??? success "(click to show solution)" - For this we need to change the value that is passed to the `EBTUTORIAL_MSG` configure option: - ```python - configopts = '-DEBTUTORIAL_MSG="Hello from the EasyBuild tutorial! I was installed by $USER." ' - ``` - Here we have to use inner double quotes, to ensure that the `$USER` environment variable is expanded - by the shell when running the `cmake` configure command. - - When you run the `eb-tutorial` command yourself, you should get output like this (not a message that - includes a literal '`$USER`' string): - - ```shell - Hello from the EasyBuild tutorial! I was installed by example. - ``` - - To re-install the `example.eb` easyconfig, you will need to use `eb --rebuild` or `eb --force`. - EasyBuild will also complain about modules that are loaded already if you tested the program above. - A good way to get rid of all those modules on LUMI is to simply use `module purge`. You don't need - to reload the software stack, but you will need to load `EasyBuild-user` again. - ---- - -***Exercise E.2**** - Installing eb-tutorial version 1.1.0* - -Install version 1.1.0 of the `eb-tutorial` example software, -which is a trivial version bump compared to version 1.0.1. - -The sources are available via: - -``` -https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.1.0.tar.gz -``` - -You can leverage the `eb-tutorial` easyconfig file we have composed in the example above, -**but you should not make any manual changes to it!** - -??? success "(click to show solution)" - You can use the `--try-software-version` option for this: - ```shell - $ eb eb-tutorial.eb --try-software-version 1.1.0 - ... - == building and installing eb-tutorial/1.1.0-cpeCray-21.12... - ... - == COMPLETED: Installation ended successfully (took 4 sec) - ``` - To test: - ``` - $ module load eb-tutorial/1.1.0-cpeCray-21.12 - $ eb-tutorial - I have a message for you: - Hello from the EasyBuild tutorial! I was installed by XXXXXXXX. - ``` - (`eb-tutorial` version 1.0.1 doesn't print "`I have a message for you:`") - - EasyBuild has also created a new easyconfig for this configuration and stored - in the repository and the `easybuild` subdirectory from the installation - directory. As on LUMI the repository is in the search path we can actually copy - the file back to the current directory: - ``` - eb --copy-ec eb-tutorial-1.1.0-cpeCray-21.12. ``` - Some of the formatting is lost though and the checksum is still missing, so you may want - to do some cleaning up. - ``` - eb eb-tutorial-1.1.0-cpeCray-21.12.eb --inject-checksum - ``` - - ---- - -*[[next: Using external modules from the Cray PE]](2_03_external_modules)* diff --git a/docs/2022-CSC_and_LO/2_03_external_modules.md b/docs/2022-CSC_and_LO/2_03_external_modules.md deleted file mode 100644 index 80a29712..00000000 --- a/docs/2022-CSC_and_LO/2_03_external_modules.md +++ /dev/null @@ -1,85 +0,0 @@ -# Using external modules from the Cray PE - -*[[back: Creating easyconfig files]](2_02_creating_easyconfig_files)* - ---- - -## What are external modules? - -EasyBuild supports the use of modules that were not installed via EasyBuild. -We refer to such modules as [external modules](https://docs.easybuild.io/en/latest/Using_external_modules.html). -External modules do not define the `EBROOT*` and `EBVERSION*` environment variables that EasyBuild would define -and uses internally in several easyblocks and some easyconfig files. - -External modules are used extensively on Cray systems to interface with the Cray PE (which comes with its own -modules and cannot be installed via EasyBuild): -[external modules can be used as dependencies](https://docs.easybuild.io/en/latest/Using_external_modules.html#using-external-modules-as-dependencies), -by including the module name in the dependencies list, -along with the `EXTERNAL_MODULE` constant marker. - -For example, to specify the module `cray-fftw` as a dependency, you should write the following in your easyconfig file: -``` python -dependencies = [('cray-fftw', EXTERNAL_MODULE)] -``` - -For such dependencies, EasyBuild will: - -* load the module before initiating the software build and install procedure - -* include a `module load` statement in the generated module file (for runtime dependencies) - -!!! Note - The default version of the external module will be loaded unless a specific version is given as dependency, - and here that version needs to be given as part of the name of the module and not as the second element in the - tuple. - - ```python - dependencies = [('cray-fftw/3.3.8.12', EXTERNAL_MODULE)] - ``` - -If the specified module is not available, EasyBuild will exit with an error message stating that the dependency -can not be resolved because the module could not be found, without searching for a matching easyconfig file -from which it could generate the module. - - ---- - -## EasyBuild Metadata for external modules - -[Metadata](https://docs.easybuild.io/en/latest/Using_external_modules.html#metadata-for-external-modules) -can be supplied to EasyBuild for external modules: using the `--external-modules-metadata` -configuration option, the location of one or more metadata files can be specified. - -The files are expected to be in INI format, with a section per module name -and key-value assignments specific to that module. - -The external modules metadata file can be also defined with the corresponding environment variable: -``` -echo $EASYBUILD_EXTERNAL_MODULES_METADATA -/apps/common/UES/jenkins/production/easybuild/cpe_external_modules_metadata-21.04.cfg -``` - -The following keys are -[supported by EasyBuild](https://docs.easybuild.io/en/latest/Using_external_modules.html#supported-metadata-values): - -* name: software name(s) provided by the module -* version: software version(s) provided by the module -* prefix: installation prefix of the software provided by the module - -For instance, the external module version loaded by the dependency `cray-fftw` can be specified as follows: -```ini -[cray-fftw] -name = FFTW -prefix = FFTW_DIR/.. -version = 3.3.8.10 -``` - -The environment variable `$EBROOTFFTW` will also be defined according to the `prefix` specified in the metadata file. - -On LUMI, users in generally don't need to be too concerned about the metadata file as the EasyBuild-user (and other hidden -EasyBuild configuration modules) take care of pointing to the right metadata file, which is specific for each version of the -Cray PE and hence each version of the LUMI software stack. - ---- - -*[[next: Implementing easyblocks]](2_04_implementing_easyblocks.md)* diff --git a/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md deleted file mode 100644 index a91b0dc8..00000000 --- a/docs/2022-CSC_and_LO/2_04_implementing_easyblocks.md +++ /dev/null @@ -1,754 +0,0 @@ -# Implementing easyblocks - -*[[back: Using external modules from the Cray PE]](2_03_external_modules.md)* - ---- - -## The basics - -An *easyblock* is a Python module that implements a software build and installation procedure. - -This concept is essentially implemented as a Python script that plugs into the EasyBuild framework. - -EasyBuild will leverage easyblocks as needed, depending on which software packages it needs to install. Which easyblock is required is determined by the ``easyblock`` easyconfig parameter, if it is present, or by the software name. - - -## Generic vs software-specific easyblocks - -Easyblocks can either be *generic* or *software-specific*. - -*Generic* easyblocks implement a "standard" software build and installation procedure that is used by multiple different -software packages. -A commonly used example is the -[``ConfigureMake``](https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/generic/configuremake.py) -generic easyblock, which implements the standard ``configure`` - ``make`` - ``make install`` installation procedure used -by most GNU software packages. - -*Software-specific* easyblocks implement the build and installation procedure for a particular software package. -Typically this involves highly customised steps, for example specifying dedicated configuration options, creating -or adjusting specific files, executing non-standard shell commands, etc. Usually a custom implementation of the -sanity check is also included. Much of the work done in software-specific easyblocks can often also be done -in generic easyblocks using parameters such as ``confdigopts`` etc., but a software-specific easyblock can -hide some of that complexity from the user. Other software-specific easyblocks implement very specific -installation procedures that do not fit in one of the generic ones. - -Using a generic easyblock requires specifying the ``easyblock`` parameter in the easyconfig file. -If it is not specified, EasyBuild will try and find the software-specific easyblock derived from the software name. - -The distinction between generic and software-specific easyblocks can be made based on the naming scheme that is used -for an easyblock (see below). - - -## Naming - -Easyblocks need to follow a strict naming scheme, to ensure that EasyBuild can pick them up automatically as needed. -This involves two aspects: - -* the name of the Python class; -* the name and location of the Python module file. - -### Python class name - -The name of the Python class is determined by the *software name* for software-specific easyblocks. -It consists of a prefix '``EB_``', followed by the (encoded) software name. - -Because of limitations in Python on characters allowed in names of Python classes, -only alphanumeric characters and underscores (``_``) are allowed. Any other characters are replaced following an encoding scheme: - -* spaces are replaced by underscores (``_``); -* dashes ``-`` are replaced by ``_minus_`` (note the inconsistency with the naming of ``EBROOT`` and ``EBVERSION`` variables); -* underscores are replaced by ``_underscore_``; - -The ``encode_class_name`` function provided in ``easybuild.tools.filetools`` returns the expected class name -for a given software name; for example: - -```shell -$ python3 -c "from easybuild.tools.filetools import encode_class_name; print(encode_class_name('netCDF-Fortran'))" -EB_netCDF_minus_Fortran -``` - -**Python class name for *generic* easyblocks** - -For *generic* easyblocks, the class name does *not* include an ``EB_`` prefix (since there is no need for an escaping -mechanism) and hence the name is fully free to choose, taking into account the restriction to alphanumeric characters -and underscores. - -For code style reasons, the class name should start with a capital letter and use CamelCasing. - -Examples include ``Bundle``, ``ConfigureMake``, ``CMakePythonPackage``. - -### Python module name and location - -The *filename* of the Python module is directly related to the name of Python class it provides. - -It should: - -* *not* include the ``EB_`` prefix of the class name for software-specific easyblocks; -* consists only of lower-case alphanumeric characters (``[a-z0-9]``) and underscores (``_``); - * dashes (``-``) are replaced by underscores (``_``); - * any other non-alphanumeric characters (incl. spaces) are simply dropped; - -Examples include: - -* ``gcc.py`` (for *GCC*) -* ``netcdf_fortran.py`` (for *netCDF-Fortran*) -* ``gamess_us.py`` (for *GAMESS (US)*) - -The ``get_module_path`` function provided by the EasyBuild framework in the -``easybuild.framework.easyconfig.easyconfig`` module returns the (full) -module location for a particular software name or easyblock class name. For example: - -```python ->>> from easybuild.framework.easyconfig.easyconfig import get_module_path ->>> get_module_path('netCDF-Fortran') -'easybuild.easyblocks.netcdf_fortran' ->>> get_module_path('EB_netCDF_minus_Fortran') -'easybuild.easyblocks.netcdf_fortran' -``` - -The location of the Python module is determined by whether the easyblock is generic or software-specific. -Generic easyblocks are located in the ``easybuid.easyblocks.generic`` namespace, while software-specific easyblocks -live in the ``easybuild.easyblocks`` namespace directly. - -To keep things organised, the actual Python module files -for software-specific easyblocks are kept in 'letter' subdirectories, -rather than in one large '``easyblocks``' directory -(see -[https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/](https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/)), -but this namespace is collapsed transparently by EasyBuild (you don't need to import from letter subpackages). - -To let EasyBuild pick up one or more new or customized easyblocks, you can use the [``--include-easyblocks``](https://docs.easybuild.io/en/latest/Including_additional_Python_modules.html#including-additional-easyblocks-include-easyblocks) -configuration option. As long as both the filename of the Python module and the name of the Python class -are correct, EasyBuild will use these easyblocks when needed. - -On LUMI, the EasyBuild configuration modules take care of setting this parameter (using the corresponding environment -variable), pointing to custom easyblocks in the LUMI software stack itself and a repo (with a fixed name) that users -can create themselves. At this moment it does not yet include possible other easyblock repositories in other repositories. - - -## Structure of an easyblock - -The example below shows the overal structure of an easyblock: - -```python -from easybuild.framework.easyblock import EasyBlock -from easybuild.tools.run import run_cmd - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def configure_step(self): - """Custom implementation of configure step for Example""" - - # run configure.sh to configure the build - run_cmd("./configure.sh --install-prefix=%s" % self.installdir) -``` - - -Each easyblock includes an implementation of a ``class`` that (directly or indirectly) derives from the abstract -``EasyBlock`` class. - -Typically some useful functions provided by the EasyBuild framework are imported at the top of the Python module. - -In the class definition, one or more '``*_step``' methods (and perhaps a couple of others) are redefined, -to implement the corresponding step in the build and installation procedure. - -Each easyblock *must* implement the ``configure``, ``build`` and ``install`` steps, since these are not implemented -in the abstract ``EasyBlock`` class. This could be done explicitly by redefining the corresponding ``*_step`` methods, -or implicitly by deriving from existing (generic) easyblocks. - -The full list of methods that can be redefined in an easyblock can be consulted in -the [API documentation](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyblock.html). - -## Deriving from existing easyblocks - -When implementing an easyblock, it is common to derive from an existing (usually generic) easyblock, -and to leverage the functionality provided by it. This approach is typically used when only a specific part -of the build and installation procedure needs to be customised. - -In the (fictitious) example below, we derive from the generic ``ConfigureMake`` easyblock to redefine the ``configure`` -step. In this case, we are *extending* the ``configure`` step as implemented by ``ConfigureMake`` rather than -redefining it entirely, since we call out to the original ``configure_step`` method at the end. - -```python -from easybuild.easyblocks.generic.configuremake import ConfigureMake -from easybuild.tools.filetools import copy_file - - -class EB_Example(ConfigureMake): - """Custom easyblock for Example""" - - def configure_step(self): - """Custom implementation of configure step for Example""" - - # use example make.cfg for x86-64 - copy_file('make.cfg.x86', 'make.cfg') - - # call out to original configure_step implementation of ConfigureMake easyblock - super(EB_Example, self).configure_step() -``` - -## Easyconfig parameters - -All of the easyconfig parameters that are defined in an easyconfig file -are available via the ``EasyConfig`` instance that can be accessed through ``self.cfg`` in an easyblock. -For instance, if the easyconfig file specifies - -```python -name = 'example' -version = '2.5.3' -versionsuffix = '-Python-3.7.4' -``` - -then these three parameters are accessible within an easyblock via ``self.cfg['name']``, ``self.cfg['version']`` -and ``self.cfg['versionsuffix']``. - -A few of the most commonly used parameters can be referenced directly: - -* **``self.name``** is equivalent with ``self.cfg['name']``; -* **``self.version``** is equivalent with ``self.cfg['version']``; -* **``self.toolchain``** is equivalent with ``self.cfg['toolchain']``; - - -### Updating parameters - -You will often find that you need to *update* some easyconfig parameters in an easyblock, -for example ``configopts`` which specifies options for the configure command. - -Because of implementation details (related to -how template values like ``%(version)s`` are handled), you need to be a bit careful here... - -To completely redefine the value of an easyconfig parameter, you can use simple assignment. For example: - -```python -self.cfg['example'] = "A new value for the example easyconfig parameter." -``` - -If want to *add* to the existing value however, you *must* use the ``self.cfg.update`` method. For example: - -```python -self.cfg.update('some_list', 'example') -``` - -This will ***not*** work (because ``self.cfg['some_list']`` does not return a reference to the original value, -but to a *temporary copy* thereof): - -```python -# anti-pattern, this does NOT work as expected! -self.cfg['some_list'].append('example') -``` - -### Custom parameters - -Additional custom easyconfig parameters can be defined in an easyblock to steer its behaviour. -This is done via the ``extra_options`` *static* method. Custom parameters can be specified to be mandatory. - -The example below shows how this can be implemented: - -```python -from easybuild.easyblocks.generic.configuremake import ConfigureMake -from easybuild.framework.easyconfig import CUSTOM, MANDATORY - - -class EB_Example(ConfigureMake): - """Custom easyblock for Example""" - - @staticmethod - def extra_options(): - """Custom easyconfig parameters for Example""" - extra_vars = { - 'required_example_param': [None, "Example required custom parameter", MANDATORY], - 'optional_example_param': [None, "Example optional custom parameter", CUSTOM], - } - return ConfigureMake.extra_options(extra_vars) -``` - -The first element in the list of a defined custom parameter corresponds to the default value for that parameter -(both ``None`` in the example above). The second element provides some informative help text -(which can then be displayed with ``eb -a -e ``, eg, ``eb -a -e EB_GCC``), -and the last element -indicates whether the parameter is mandatory (``MANDATORY``) or just an optional custom parameter (``CUSTOM``). - -## Easyblock constructor - -In the ``class`` constructor of the easyblock, i.e. the ``__init__`` method, one or more class variables -can be initialised. These can be used for sharing information between different ``*_step`` methods in the easyblock. - -For example: - -```python -from easybuild.framework.easyblock import EasyBlock - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def __init__(self, *args, **kwargs): - """Constructor for Example easyblock, initialises class variables.""" - - # call out to original constructor first, so 'self' (i.e. the class instance) is initialised - super(EB_Example, self).__init__(*args, **kwargs) - - # initialise class variables - self.example_value = None - self.example_list = [] -``` - -## File operations - -File operations is a common use case for implementing easyblocks, hence the EasyBuild framework provides a -number of useful functions related to this, including: - -* ``read_file()``: read file at a specified location and returns its contents; - -* ``write_file(, )`` at a specified location with provided contents; - to append to an existing file, use ``append=True`` as an extra argument; - -* ``copy_file(, )`` to copy an existing file; - -* ``apply_regex_substitutions(, )`` to patch an existing file; - -All of these functions are provided by the [``easybuild.tools.filetools``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html) module. - -## Executing shell commands - -For executing shell commands two functions are provided by the -[``easybuild.tools.run``](https://docs.easybuild.io/en/latest/api/easybuild.tools.run.html) module: - -* ``run_cmd()`` to run a non-interactive shell command; - -* ``run_cmd_qa(, )`` to run an interactive shell command; - -Both of these accept a number of optional arguments: - -* ``simple=True`` to just return ``True`` or ``False`` to indicate a successful execution, - rather than the default return value, i.e., a tuple that provides the command output and the exit code (in that - order); - -* ``path=`` to run the command in a specific subdirectory; - -The ``run_cmd_qa`` function takes two additional specific arguments: - -* ``no_qa=`` to specify a list of patterns to recognize non-questions; - -* ``std_qa=`` to specify regular expression patterns for common questions, and the matching answer; - -## Manipulating environment variables - -To (re)define environment variables, the ``setvar`` function provided by the -[``easybuild.tools.environment``](https://docs.easybuild.io/en/latest/api/easybuild.tools.environment.html) -module should be used. - -This makes sure that the changes being made to the specified environment variable are kept track of, -and that they are handled correctly under ``--extended-dry-run``. - -## Logging and errors - -It is good practice to include meaningful log messages in the ``*_step`` methods being customised in the easyblock, -to enrich the EasyBuild log with useful information for later debugging or diagnostics. - -For logging, the provided ``self.log`` logger class should be used. -You can use the ``self.log.info`` method to log an informative message. -Similar methods are available for logging debug messages (``self.log.debug``), which are -only emitted when ``eb`` is run with debugging mode enabled (``--debug`` or ``-d``), -and for logging warning messages (``self.log.warning``). - -If something goes wrong, you can raise an ``EasyBuildError`` instance to report the error. - -For example: - -```python -from easybuild.framework.easyblock import EasyBlock -from easybuild.tools.build_log import EasyBuildError -from easybuild.tools.run import run_cmd - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def configure_step(self): - """Custom implementation of configure step for Example""" - - cmd = "./configure --prefix %s" % self.installdir) - out, ec = run_cmd(cmd) - - success = 'SUCCESS' - if success in out: - self.log.info("Configuration command '%s' completed with success." % cmd) - else: - raise EasyBuildError("Pattern '%s' was not found in output of '%s'." % (success, cmd)) -``` - -## Custom sanity check - -For software-specific easyblocks, a custom sanity check is usually included to verify that the installation was -successful or not. - -This is done by redefining the ``sanity_check_step`` method in the easyblock. For example: - -```python -from easybuild.framework.easyblock import EasyBlock - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def sanity_check_step(self): - """Custom sanity check for Example.""" - - custom_paths = { - 'files': ['bin/example'], - 'dirs': ['lib/examples/'], - } - custom_commands = ['example --version'] - - # call out to parent to do the actual sanity checking, pass through custom paths and commands - super(EB_Example, self).sanity_check_step(custom_paths=custom_paths, custom_commands=custom_commands) -``` - -You can both specify file paths and subdirectories to check for, which are specified relative to the installation directory, -as well as simple commands that should execute successfully after completing the installation and loading the generated module file. - -It is up to you how extensive you make the sanity check, but it is recommended to make the check as complete -as possible to catch any potential build or installation problems that may occur, while ensuring that it can -run relatively quickly (in seconds, or at most a couple of minutes). - -## Version-specific parts - -In some cases version-specific actions or checks need to be included in an easyblock. -For this, it is recommended to use ``LooseVersion`` rather than directly comparing version numbers using string values. - -For example: - -```python -from distutils.version import LooseVersion -from easybuild.framework.easyblock import EasyBlock - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def sanity_check_step(self): - """Custom sanity check for Example.""" - - custom_paths = { - 'files': [], - 'dirs': [], - } - - # in older versions, the binary used to be named 'EXAMPLE' rather than 'example' - if LooseVersion(self.version) < LooseVersion('1.0'): - custom_paths['files'].append('bin/EXAMPLE') - else: - custom_paths['files'].append('bin/example') - - super(EB_Example, self).sanity_check_step(custom_paths=custom_paths) -``` - -## Compatibility with ``--extended-dry-run`` and ``--module-only`` - -Some special care must be taken to ensure that an easyblock is fully compatible with ``--extended-dry-run`` / ``-x`` -(see [Inspecting install procedures](../basic_usage/#inspecting-install-procedures)) and ``--module-only``. - -For compatibility with ``--extended-dry-run``, you need to take into account that specified operations -like manipulating files or running shell commands will not actually be executed. You can check -whether an easyblock is being run in dry run mode via ``self.dry_run``. - -For example: - -```python -from easybuild.framework.easyblock import EasyBlock -from easybuild.tools.build_log import EasyBuildError -from easybuild.tools.run import run_cmd - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def configure_step(self): - """Custom implementation of configure step for Example""" - - cmd = "./configure --prefix %s" % self.installdir) - out, ec = run_cmd(cmd) - - success = 'SUCCESS' - if success in out: - self.log.info("Configuration command '%s' completed with success." % cmd) - - # take into account that in dry run mode we won't get any output at all - elif self.dry_run: - self.log.info("Ignoring missing '%s' pattern since we're running in dry run mode." % success) - - else: - raise EasyBuildError("Pattern '%s' was not found in output of '%s'." % (success, cmd)) -``` - -For ``--module-only``, you should make sure that both the ``make_module_step``, including the ``make_module_*`` submethods, -and the ``sanity_check_step`` methods do not make any assumptions about the presence of certain environment variables, or that class variables have been defined already. - -This is required because under ``--module-only`` the large majority of the ``*_step`` functions are -simply *skipped* entirely. So, if the ``configure_step`` method is responsible for defining class variables that are -picked up in ``sanity_check_step``, the latter may run into unexpected initial values like ``None``. -A possible workaround is to define a separate custom method to define the class variables, and to call out to this -method from ``configure_step`` and ``sanity_check_step`` (for the latter, conditionally, i.e., only if the class -variables still have the initial values). - -For example: - -```python -from easybuild.framework.easyblock import EasyBlock - - -class EB_Example(EasyBlock): - """Custom easyblock for Example""" - - def __init__(self, *args, **kwargs): - """Easyblock constructor.""" - super(EB_Example, self).__init__(*args, **kwargs) - - self.command = None - - def set_command(self): - """Initialize 'command' class variable.""" - # $CC environment variable set by 'prepare' step determines exact command - self.command = self.name + '-' + os.getenv('CC') - - def configure_step(self): - """Custom configure step for Example.""" - - self.set_command() - self.cfg.update('configopts', "COMMAND=%s" % self.command) - - super(EB_Example, self).configure_step() - - def sanity_check_step(self): - """Custom implementation of configure step for Example""" - - if self.command is None: - self.set_command() - - super(EB_Example, self).sanity_check_step(custom_commands=[self.command]) -``` - - -## Easyblocks in the Cray ecosystem - -The generic easyblocks are usually rather independent of compilers etc. and tend to work well with all toolchains. -However, software-specific easyblocks may contain code that is specific for certain toolchains and are often only -tested with the common toolchains (foss and intel and their subtoolchains). Many of those easyblocks will fail -on Cray systems (or any system that uses other toolchains) as they don't recognise the compiler and rather than -implementing some generic behaviour that may or may not work, produce an error message instead that the compiler -toolchain is not supported. - -Several packages on LUMI therefore use generic easyblocks rather than the software-specific easyblocks that may -exist for those applications. Adapting those software-specific easyblocks for LUMI poses an interesting maintenance -problem. Either one could decide to not contribute back to the community, but this implies then that all modifications -made to the corresponding easyblocks in the EasyBuild distribution should be monitored and implemented in the custom -easyblocks for Cray also. On the other hand, contributing back to the community also poses two problems. First it -would also require to implement the Cray toolchains as used on LUMI in the core of EasyBuild (which already contains -a different set of toolchains targeted more at how the Cray PE works with the regular environment modules), and that -only makes sense if these toolchains are first extended to not only cover the programming environments supported on -LUMI but also the Intel and NVIDIA programming environments. Second, the EasyBuild community has no easy way of testing -any modification made to such an easyblock on a Cray PE system. Hence every update made in the community may break -the Cray PE support again. - - -## Exercise - -### Exercise I.1 - -Try implementing a new custom easyblock for ``eb-tutorial``, which derives directly -from the base ``EasyBlock`` class. - -Your easyblock should: - -* define a custom mandatory easyconfig parameter named ``message``; -* run `cmake` to configure the installation, which includes at least: - * specifying the correct installation prefix (using the `-DCMAKE_INSTALL_PREFIX=...` option); - * passing down the value of ``message`` easyconfig parameter via `-DEBTUTORIAL_MSG=...` -* run `make` to build `eb-tutorial`; -* run `make install` to install the generated binary; -* perform a custom sanity check to ensure the installation is correct; -* pick up on commonly used easyconfig parameters like `configopts` and `preinstallopts` where appropriate; - -??? success "(click to show solution)" - - Here's a complete custom easyblock for ``eb-tutorial`` that derives from the base ``EasyBlock`` class, - which should be included in a file named ``eb_tutorial.py``. - - We need to implement the ``configure_step``, ``build_step``, and ``install_step`` methods in - order to have a fully functional easyblock. - - The configure, build, and install steps take into account the corresponding easyconfig - parameters that allow customizing these commands from an easyconfig file. - - ```python - from easybuild.framework.easyblock import EasyBlock - from easybuild.framework.easyconfig import MANDATORY - from easybuild.tools.run import run_cmd - - - class EB_eb_minus_tutorial(EasyBlock): - """Custom easyblock for eb-tutorial.""" - - @staticmethod - def extra_options(): - extra = EasyBlock.extra_options() - extra.update({ - 'message': [None, "Message that eb-tutorial command should print", MANDATORY], - }) - return extra - - def configure_step(self): - """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" - - cmd = ' '.join([ - self.cfg['preconfigopts'], - 'cmake', - '-DCMAKE_INSTALL_PREFIX=\'%s\'' % self.installdir, - '-DEBTUTORIAL_MSG="%s"' % self.cfg['message'], - self.cfg['configopts'], - ]) - run_cmd(cmd) - - def build_step(self): - """Build step for eb-tutorial""" - - cmd = ' '.join([ - self.cfg['prebuildopts'], - 'make', - self.cfg['buildopts'], - ]) - run_cmd(cmd) - - def install_step(self): - """Install step for eb-tutorial""" - - cmd = ' '.join([ - self.cfg['preinstallopts'], - 'make install', - self.cfg['installopts'], - ]) - run_cmd(cmd) - - def sanity_check_step(self): - custom_paths = { - 'files': ['bin/eb-tutorial'], - 'dirs': [], - } - custom_commands = ['eb-tutorial'] - return super(EB_eb_minus_tutorial, self).sanity_check_step(custom_paths=custom_paths, - custom_commands=custom_commands) - ``` - - We also need to adapt our easyconfig file for ``eb-tutorial``: - - - The ``easyblock`` line is no longer needed as we will rely on the automatic selection of the - software-specific easyblock. - - We don't need to define the message through ``configopts`` but via the easyblock-specific - configuration parameter ``message``. In fact, we were so careful when implementing the ``configure_step`` - that even variable expansion will still work so we can still include ``$USER`` in the message. - - The sanity check is also no longer needed as it is done by the software-specific easyblock. - - So the easyconfig file simplifies to: - - ```python - name = 'eb-tutorial' - version = "1.1.0" - - homepage = 'https://easybuilders.github.io/easybuild-tutorial' - - whatis = [ 'Description: EasyBuild tutorial example'] - - description = """ - This is a short C++ example program that can be buid using CMake. - """ - - toolchain = {'name': 'cpeCray', 'version': '21.12'} - - builddependencies = [ - ('buildtools', '%(toolchain_version)s', '', True) - ] - - source_urls = ['https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/'] - sources = [SOURCE_TAR_GZ] - checksums = ['def18b69b11a3ec34ef2a81752603b2118cf1a57e350aee41de9ea13c2e6a7ef'] - - message = 'Hello from the EasyBuild tutorial! I was installed by $USER.' - - moduleclass = 'tools' - - ``` - - Running this example on LUMI is a little tricky as using ``--include-easyblocks`` to point EasyBuild to - our new easyblock interfers with settings already made by the EasyBuild configuration modules (``EasyBuild-user``) - and causes error messages about the toolchains. So either the easyblock needs to be copied to the user location - that can be found by looking at the output of ``eb --show-config`` or we simply need to extend the list of - easyblocks that EasyBuild searches with the easyblocks in the current directory: - - ``` shell - EASYBUILD_INCLUDE_EASYBLOCKS="$EASYBUILD_INCLUDE_EASYBLOCKS,./*.py" - ``` - - - - -### Exercise I.2 - -Try implementing another new custom easyblock for ``eb-tutorial``, -which derives from the generic ``CMakeMake`` easyblock. - -Your easyblock should only: - -* define a custom mandatory easyconfig parameter named ``message``; -* pass down the value of ``message`` easyconfig parameter via `-DEBTUTORIAL_MSG=...` -* perform a custom sanity check to ensure the installation is correct; - -??? success "(click to show solution)" - - When deriving from the ``CMakeMake`` generic easyblock, there is a lot less to worry about. - - We only need to customize the ``configure_step`` method to ensure that the ``-DEBTUTORIAL_MSG`` configuration - option is specified; the ``CMakeMake`` easyblock already takes care of specifying the location of - the installation directory (and a bunch of other configuration options, like compiler commands and flags, etc.). - - Implementing the ``build_step`` and ``install_step`` methods is no longer needed, - the standard procedure that is run by the ``CMakeMake`` generic easyblock is fine, - and even goes beyond what we did in the previous exercise (like building in parallel with ``make -j``). - - ```python - from easybuild.easyblocks.generic.cmakemake import CMakeMake - from easybuild.framework.easyconfig import MANDATORY - from easybuild.tools.run import run_cmd - - class EB_eb_minus_tutorial(CMakeMake): - """Custom easyblock for eb-tutorial.""" - - @staticmethod - def extra_options(): - extra = CMakeMake.extra_options() - extra.update({ - 'message': [None, "Message that eb-tutorial command should print", MANDATORY], - }) - return extra - - def configure_step(self): - """Custom configure step for eb-tutorial: define EBTUTORIAL_MSG configuration option.""" - self.cfg.update('configopts', '-DEBTUTORIAL_MSG="%s"'% self.cfg['message']) - - super(EB_eb_minus_tutorial, self).configure_step() - - def sanity_check_step(self): - custom_paths = { - 'files': ['bin/eb-tutorial'], - 'dirs': [], - } - custom_commands = ['eb-tutorial'] - return super(EB_eb_minus_tutorial, self).sanity_check_step(custom_paths=custom_paths, - custom_commands=custom_commands) - ``` - - This is a much simpler easyblock as we already use all the logic that has been written for us to build - with CMake. - -*[[next: Part 3: Overview]](3_00_part3_advanced.md)* diff --git a/docs/2022-CSC_and_LO/3_00_part3_advanced.md b/docs/2022-CSC_and_LO/3_00_part3_advanced.md deleted file mode 100644 index 28a24990..00000000 --- a/docs/2022-CSC_and_LO/3_00_part3_advanced.md +++ /dev/null @@ -1,15 +0,0 @@ -# Part III: Advanced topics - -*[[back to start page]](index.md)* - -In this section we mostly cover "good to know that they exist" features as they are not used -on LUMI, or not really accessible to regular user installations that are performed with the -LUMI ``EasyBuild-user`` module. Hooks are used on LUMI, but it is not really advised to -overwrite the centrally defined hooks with a local file. And the whole structure of the -EasyBuild integration is also set up to make use of the GitHub integration in the future. - -* [Using EasyBuild as a library](3_01_easybuild_library.md) *(hands-on)* -* [Using hooks to customise EasyBuild](3_02_hooks.md) *(hands-on)* -* [Submitting installations as Slurm jobs](3_03_slurm_jobs.md) *(hands-on)* -* [Module naming schemes (incl. hierarchical)](3_04_module_naming_schemes.md) *(hands-on)* -* [GitHub integration to facilitate contributing to EasyBuild](3_05_github_integration.md) *(hands-on)* diff --git a/docs/2022-CSC_and_LO/3_01_easybuild_library.md b/docs/2022-CSC_and_LO/3_01_easybuild_library.md deleted file mode 100644 index f0c255eb..00000000 --- a/docs/2022-CSC_and_LO/3_01_easybuild_library.md +++ /dev/null @@ -1,301 +0,0 @@ -# Using EasyBuild as a library - -*[[back: Configuring EasyBuild]](1_07_configuration.md)* - ---- - -You can use EasyBuild as a Python library and implement a script to automate a particular task. - -All the functionality provided by the EasyBuild framework can be leveraged, -so you only have to glue things together and add the missing pieces. - - -## Setting up the EasyBuild configuration - -Before you call any of the functions provided by the EasyBuild framework, -you should initialize EasyBuild by **setting up the configuration**. -Some of the EasyBuild framework functions assume that EasyBuild has been initialized properly, -and calling them without doing so will result in errors. - -To set up the EasyBuild configuration, you should call the ``set_up_configuration`` function -that is provided by the ``easybuild.tools.options`` module. - -This function takes a couple of optional arguments that are useful in the context of -a Python script that uses EasyBuild as a library: - -* Via ``args`` you can provide a list of command line arguments that should be passed to the option parser. - By default, the standard command line arguments are picked up via ``sys.args``, which may not be what you want. -* By passing ``silent=True`` you can specify that the ``set_up_configuration`` function should not print - anything when it is called. By default, it will print the location to the temporary EasyBuild log file. - -### Cleaning up the temporary directory - -When EasyBuild is configured, a temporary directory specific to that EasyBuild session will be created automatically. -You should clean up that directory, especially if you will be creating temporary files, or if the script will be run -often. - -Note that cleaning up the temporary directory implies removes the temporary log files, -so you probably only want to do this if no errors occurred. - -### Minimal example script - -Here is a minimal (and pretty useless) example Python script, which sets up the EasyBuild configuration -and cleans up the temporary directory: - -```python -#!/usr/bin/env python3 -from easybuild.tools.filetools import remove_dir -from easybuild.tools.options import set_up_configuration - -opts, _ = set_up_configuration(args=[], silent=True) - -remove_dir(opts.tmpdir) -``` - - -## Example use cases - -Once the EasyBuild configuration has been set up, the functions provided by the EasyBuild framework -can be called from a Python script (or directly from the Python interpreter). - -A full overview of all functions is available via the [EasyBuild API documentation](https://docs.easybuild.io/en/latest/api/easybuild.html). - -We highlight a couple commonly used functions in the sections below. - -### File operations - -The [``easybuild.tools.filetools``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html) -module provides a bunch of functions for performing file operations. - -That includes straightforward things like reading, writing, and copying files -(see [``read_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.read_file), [``write_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.write_file), [``copy_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.copy_file)), but also more specific functionality like applying a patch file ([``apply_patch``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.apply_patch)), -computing different types of checksums for a file ([``compute_checksum``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.compute_checksum)), and downloading files -([``download_file``](https://docs.easybuild.io/en/latest/api/easybuild.tools.filetools.html#easybuild.tools.filetools.download_file)). - -Here is a trivial example where we download a file from a specified URL to a temporary directory, -compute the SHA256 checksum, and then copy the downloaded file to the current working directory: - -```python -#!/usr/bin/env python3 -import os -import sys -import tempfile - -from easybuild.tools.filetools import download_file, compute_checksum, copy_file, remove_dir -from easybuild.tools.options import set_up_configuration - -url = sys.argv[1] - -opts, _ = set_up_configuration() - -fn = os.path.basename(url) -tmp_path = os.path.join(opts.tmpdir, fn) -download_file(fn, url, tmp_path) - -sha256 = compute_checksum(tmp_path, checksum_type='sha256') -print("SHA256 checksum for %s: %s" % (fn, sha256)) - -target_dir = os.getcwd() -copy_file(tmp_path, target_dir) -print("%s copied to %s" % (fn, target_dir)) - -# cleanup temp dir -remove_dir(opts.tmpdir) -``` - -Example output: -```shell -$ export TEST_URL='https://github.com/easybuilders/easybuild-tutorial/raw/main/docs/files/eb-tutorial-1.0.1.tar.gz' -$ python3 download_and_copy.py $TEST_URL -== Temporary log file in case of crash /tmp/eb-or_xhaw8/easybuild-ewooa00c.log -SHA256 checksum for eb-tutorial-1.0.1.tar.gz: d6cec2ea298f4092cb1b880cb017220ab191561da941e9e480639cf3354b7ef9 -eb-tutorial-1.0.1.tar.gz copied to /home/example -``` -We are deliberately *not* specifying the `args` optional argument to the `set_up_configuration` call here, -so we can pass EasyBuild configuration options as command line arguments to this script. -Of course, only those configuration options that are taken into account by the EasyBuild -framework functions being called have any impact. For example: - -```shell -# wait for max. 1h for the download to complete -python3 download_and_copy.py $TEST_URL --download-timeout 3600 -``` - -### Running shell commands - -To run shell commands, we can leverage the ``run_cmd`` functions from the ``easybuild.tools.run`` module. - -Here is a simple example for running the '`make`' command via a Python script: - -```python -#!/usr/bin/env python3 -import sys - -from easybuild.tools.filetools import remove_dir, which -from easybuild.tools.run import run_cmd -from easybuild.tools.options import set_up_configuration - -opts, _ = set_up_configuration(args=[]) - -cmd = 'make' -cmd_path = which(cmd) -if cmd_path: - print(">>> '%s' command found at %s" % (cmd, cmd_path)) -else: - sys.stderr.write("ERROR: '%s' command not found!\n" % cmd) - sys.exit(1) - -cmd = ' '.join(["make"] + sys.argv[1:]) -out, ec = run_cmd(cmd) - -print("\n>>> Output of '%s' (exit code %s):\n\n%s" % (cmd, ec, out)) - -remove_dir(opts.tmpdir) -``` - -Note that in this case it is important to use `args=[]` when calling `set_up_configuration`, -to avoid that the additional script arguments which are options for the '`make`' command -are picked up by the EasyBuild option parser. - -Example usage: - -```shell -$ python3 run_make.py --version -== Temporary log file in case of crash /tmp/eb-zfvbceg6/easybuild-7tynetaj.log - ->> Output of 'make --version' (exit code 0): - -GNU Make 3.81 -``` - -### Interacting with the modules tool - -You can interact with the environment modules tool in a Python script using the -[``easybuild.tools.modules``](https://docs.easybuild.io/en/latest/api/easybuild.tools.modules.html) -Python module that is a part of the EasyBuild framework. -The modules tool that will be used is determined by the active EasyBuild configuration. - -Here is an example script that checks which modules are available and currently loaded, -loads the default module file for ``bzip2``, and inspects the resulting changes to the environment. - -```python -#!/usr/bin/env python3 -import os -import sys - -from easybuild.tools.filetools import remove_dir -from easybuild.tools.modules import get_software_root_env_var_name, modules_tool -from easybuild.tools.options import set_up_configuration - -opts, _ = set_up_configuration() - -# obtain ModulesTool instance for preferred modules tool (determined by active EasyBuild configuration) -mod_tool = modules_tool() -print("Active modules tool: %s version %s" % (mod_tool.NAME, mod_tool.version)) - -avail_modules = mod_tool.available() -print("Found %d available modules in total" % len(avail_modules)) - -avail_eb_modules = mod_tool.available('EasyBuild') -print("Found %d available modules for EasyBuild: %s" % (len(avail_eb_modules), ', '.join(avail_eb_modules))) - -loaded_modules = mod_tool.loaded_modules() -print("%d modules are currently loaded: %s" % (len(loaded_modules), ', '.join(loaded_modules))) - -# load default module for bzip2, check changes to environment -name = 'bzip2' -env_var_name = get_software_root_env_var_name(name) -if any(m.startswith(name + '/') for m in avail_modules): - - print("Current $%s value: %s" % (env_var_name, os.getenv(env_var_name, '(no set)'))) - print("Loading (default) '%s' module..." % name) - - mod_tool.load([name]) - print("Loaded modules: %s" % ', '.join(mod_tool.loaded_modules())) - - # inspect $_LMFILES_ environment variable to determine path to loaded bzip2 module file - for mod_file_path in os.getenv('_LMFILES_').split(':'): - if name in mod_file_path: - print("Path to loaded %s module: %s" % (name, mod_file_path)) - break - - # $EBROOTBZIP2 should be set now (if the bzip2 module was installed with EasyBuild) - print("Current $%s value: %s" % (env_var_name, os.getenv(env_var_name, '(no set)'))) -else: - sys.stderr.write("No modules available for %s\n" % name) - sys.exit(1) - -remove_dir(opts.tmpdir) -``` - -### Parsing easyconfig files - -Here is another small example Python script, which uses the EasyBuild framework functionality -to locate and parse an easyconfig file, and inspect the value of specific easyconfig parameters. - -We define a small helper function named ``parse_easyconfig``, because the EasyBuild framework API -is a bit awkward to use for this simple use case. - -```python -#!/usr/bin/env python3 -import sys - -from easybuild.framework.easyconfig.tools import det_easyconfig_paths, parse_easyconfigs -from easybuild.tools.options import set_up_configuration - - -def parse_easyconfig(ec_fn): - """ - Helper function: find and parse easyconfig with specified filename, - and return parsed easyconfig file (an EasyConfig instance). - """ - # determine path to easyconfig file - ec_path = det_easyconfig_paths([ec_fn])[0] - - # parse easyconfig file; - # the 'parse_easyconfigs' function expects a list of tuples, - # where the second item indicates whether or not the easyconfig file was - # automatically generated or not - ec_dicts, _ = parse_easyconfigs([(ec_path, False)]) - - # only retain first parsed easyconfig, ignore any others (which are unlikely - # anyway) - return ec_path, ec_dicts[0]['ec'] - - -# check whether required arguments are provided -if len(sys.argv) < 3: - sys.stderr.write("ERROR: Usage: %s ") - sys.exit(1) - -ec_fn = sys.argv[1] -keys = sys.argv[2:] - -set_up_configuration(args=[], silent=True) - -ec_path, ec = parse_easyconfig(ec_fn) - -print("Inspecting %s ..." % ec_path) -for key in keys: - print("%s: %s" % (key, ec[key])) -``` - -Example usage (ensure that ``EasyBuild-user`` is loaded as EasyBuild should be -configured properly!): - -``` -$ ./inspect_easyconfig.py GMP-6.2.1-cpeCray-21.12.eb name version sources sanity_check_paths -Inspecting /appl/lumi/mgmt/ebrepo_files/LUMI-21.12/LUMI-L/GMP/GMP-6.2.1-cpeCray-21.12.eb ... -name: GMP -version: 6.2.1 -sources: ['gmp-6.2.1.tar.bz2'] -sanity_check_paths: {'files': ['include/gmp.h', 'include/gmpxx.h', 'lib/libgmp.a', 'lib/libgmp.la', 'lib/libgmp.so', 'lib/libgmpxx.a', 'lib/libgmpxx.la', 'lib/libgmpxx.so', 'lib/pkgconfig/gmp.pc', 'lib/pkgconfig/gmpxx.pc'], 'dirs': ['share/info']} -``` - -This feature can be used, e.g., to extract information from easyconfig files to generate documentation -for the library of easyconfig files. - - ---- - -*[[next: Using hooks to customise EasyBuild]](3_02_hooks.md)* diff --git a/docs/2022-CSC_and_LO/3_02_hooks.md b/docs/2022-CSC_and_LO/3_02_hooks.md deleted file mode 100644 index f91f7d7c..00000000 --- a/docs/2022-CSC_and_LO/3_02_hooks.md +++ /dev/null @@ -1,213 +0,0 @@ -# Using hooks to customise EasyBuild - -*[[back: Using EasyBuild as a library]](3_01_easybuild_library.md)* - ---- - -Sometimes you may have the need to customise the behaviour of EasyBuild, -while you want to avoid copying easyconfig files in order to make small changes -(like adding an additional configuration option), having to maintain a set -of modified easyblocks, or using a patched version of the EasyBuild framework. - -EasyBuild provides support for implementing so-called *hooks*, -which are small functions that are triggered automatically at specific times. - -Through these hook functions you can take additional actions, or even tweak -internal data structures, to change the software installation procedure that EasyBuild -would normally execute. - -This is a very powerful feature, but it is important to aware of some details in order -to make good use of it. - -Support for hooks is also -[covered in detail in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Hooks.html). - -## Why hooks? - -Hooks can be implemented for various reasons. Some examples used by some sites in the EasyBuild -community: - -- Enforce some site policies on easyconfig files. A practical example for LUMI could be imposing - the use of the `whatis` parameter and ensuring that it contains a `Description` line to have a - more consistent format for module files. -- Ensuring that some information is always in the module file. E.g., on LUMI the `site_contacts` - parameter is added automatically when installing in the central stack if it is not already in - the easyconfig file to refer to the LUST support pages. -- Modify the behaviour of standard easyconfig files that come with EasyBuild to adapt them to the system - while allowing users to simply use the standard easyconfig files. E.g., we could consider this option - to offer the common EasyBuild FOSS toolchain and its subtoolchains to users on LUMI-C, while using hooks to - adapt, e.g., the easyconfig files for Open MPI to work on LUMI. - -Some site have really large hook files to implement policies and modify standard EasyBuild build recipes, e.g., -- [The hooks file from Jülich Supercomputing Centre](https://github.com/easybuilders/JSC/blob/2022/Custom_Hooks/eb_hooks.py) -- ComputeCanada [cc_hooks_common.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_common.py) - and [cc_hooks_gento.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_gentoo.py) -- [The hooks file from the EESSI software stack](https://github.com/EESSI/software-layer/blob/main/eb_hooks.py). - [EESSI](https://www.eessi-hpc.org/) is an effort to build a software stack distributed via - [CernVM-FS](https://cernvm.cern.ch/fs/) using EasyBuild to build all software. - -## Implementing and using hooks - -Using hooks is done by implementing functions with specific names in a Python module, -and configuring EasyBuild to use them by specifying the path to the file that contains them -via the ``hooks`` configuration option. - -For example, if the file named ``my_eb_hooks.py`` in your home directory contains the implementation -of your hooks, you can configure EasyBuild to use them by defining the ``$EASYBUILD_HOOKS`` environment -variable: - -```shell -export EASYBUILD_HOOKS=$HOME/my_eb_hooks.py -``` - -## Available hooks - -EasyBuild supports various hooks, the full list can be consulted via ``eb --avail-hooks``. - -There are 3 different types of hooks: - -- start/end hooks -- parse hook -- pre/post-step hooks - -These are discussed in more details in the sections below. - -### Start/end hooks - -The first type of hooks are the ``start`` and ``end`` hooks, which are respectively triggered -at the very start of the EasyBuild session (right after setting up the EasyBuild configuration), -and at the very end of the session (right before stopping the logger and doing final cleanup). - -These hooks are only called *once* for every EasyBuild session, no matter how many installations -are performed during that session. - -The ``end`` hook is *not* triggered if an error occurred during one of the installations, -only on successful completion of the EasyBuild session. - -These hooks can be implementing by defining a function named ``start_hook`` and ``end_hook`` -in the Python module file that is provided via the ``hooks`` configuration option. -No arguments are provided when calling these hooks. - -### Parse hook - -The ``parse`` hook is triggered right after an easyconfig file is being parsed, -*before* EasyBuild sets up its internal data structures based on the parsed easyconfig. - -If you want to dynamically change one or more easyconfig parameters without changing the corresponding -easyconfig files, using this hook may be appropriate. - -Note that parsing an easyconfig file can happen for a variety of reasons, -not only when the easyconfig file will actually be installed. EasyBuild will also -parse easyconfig files to check whether they resolve required dependencies, -to check whether the corresponding module file is already installed, etc. - -This hook can be implemented via a function named ``parse_hook``, and exactly one -argument is provided when it is called: the [``EasyConfig``](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyconfig.easyconfig.html#easybuild.framework.easyconfig.easyconfig.EasyConfig) -instance that represents the parsed easyconfig file. - -### Pre/post-step hooks - -The third type of hooks are *pre/post-step* hooks, which are triggered right before or -after a particular installation step is executed. - -EasyBuild performs each software installation by stepping through over a dozen different methods, -and for each of these steps there a pre- and post-hook is triggered, which results in over 30 -additional hooks. - -To use any of these hooks, you need to implement a function that follow a strict naming scheme: -``__hook``. For example, the hook that is triggered right before the ``configure`` -step is run is a function named ``pre_configure_hook``. - -Every time these hooks are called, a single argument is provided: an [``EasyBlock``](https://docs.easybuild.io/en/latest/api/easybuild.framework.easyblock.html#easybuild.framework.easyblock.EasyBlock) -instance that represents the easyblock that is being used to perform the installation. -The parsed easyconfig file can be accessed via the ``cfg`` class variable of the ``EasyBlock`` instance. - -These hooks are useful for influencing the installation procedure at a particular stage. - -## Caveats - -There are a couple of important caveats to take into account when implementing hooks. - -### Breaking EasyBuild with hooks - -Since hooks allow you to inject custom code into EasyBuild at runtime, -it is also easy to break EasyBuild by using hooks... - -Make sure to carefully test your hook implementations, and constrain the actions -you take a much as possible, for example by adding conditions to control for which -software names you will actually modify the installation procedure, etc. - -Any errors that are triggered or raised while a hook function is running -will interrupt the EasyBuild session. - -So don't forget: with great power comes great responsibility! - -### Template values - -Depending on the type of hook, you may observe "raw" values of easyconfig parameters where -template values have not been resolved yet, or values in which template values have been resolved already. - -In the ``parse`` hook, you will always see unresolved template values. - -In the pre/post-step hooks you will see resolved template values, -unless you explicitly disable templating. - -To obtain easyconfig parameter values with unresolved template values in step hooks, -you can use the ``disable_templating`` [context manager](https://docs.python.org/3/reference/compound_stmts.html#with). -For example: - -```python -from easybuild.framework.easyconfig.easyconfig import disable_templating -from easybuild.tools.build_log import print_warning - -def pre_source_hook(eb): - """Print warning when software version was found in 'raw' name of source file.""" - with disable_templating(eb.cfg): - for src in eb.cfg['sources']: - if eb.version in src: - msg = "Software version '%s' found in name of source file (%s), " % (eb.version, src) - msg += "please use %(version)s template value instead!" - print_warning(msg) -``` - - -### Manipulating easyconfig parameters - -If you want update a particular easyconfig parameter without overwriting the existing value, -a bit of care has to be taken: you should use the ``update`` method of the ``EasyConfig`` instance -for this, unless you disable template resolution. This is particularly important when -updating easyconfig parameters that have *mutable* value (like a ``list`` or ``dict``). - -Here's a correct example of a pre-install hook: - -```python -def pre_install_hook(eb): - if eb.name == 'pigz': - # always copy the README directory too when installing pigz - eb.cfg.update('files_to_copy', 'README') -``` - -This seemingly equivalent implementation will ***not*** work (the value of the `files_to_copy` -easyconfig parameter will *not* be updated): - -```python -def pre_install_hook(eb): - if eb.name == 'pigz': - # incorrect way of adding 'README' to 'files_to_copy' (DON'T USE THIS!) - eb.cfg['files_to_copy'].append('README') -``` - -To use this coding style successfully, you have to disable the templating mechanism -when updating the easyconfig parameter: - -```python -def pre_install_hook(eb): - if eb.name == 'pigz': - # this works, but it is better to use the 'update' method instead... - with disable_templating(eb.cfg): - eb.cfg['files_to_copy'].append('README') -``` - ---- - -*[[next: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* diff --git a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_03_slurm_jobs.md deleted file mode 100644 index 283a53a0..00000000 --- a/docs/2022-CSC_and_LO/3_03_slurm_jobs.md +++ /dev/null @@ -1,333 +0,0 @@ -# Submitting installations as Slurm jobs - -*[[back: Using hooks to customise EasyBuild]](3_02_hooks.md)* - ---- - -EasyBuild can submit jobs to different backends including Slurm to install software, -to *distribute* the often time-consuming installation of a set of software applications and -the dependencies they require to a cluster. Each individual package is installed in a separate -job and job dependencies are used to manage the dependencies between package so that no build -is started before the dependencies are in place. - -This is done via the ``--job`` command line option. - -It is important to be aware of some details before you start using this, which we'll cover here. - -!!! Warning "This section is not supported on LUMI, use at your own risk" - - EasyBuild on LUMI is currently not fully configured to support job submission via Slurm. Several - changes would be needed to the configuration of EasyBuild, including the location of the - temporary files and build directory. Those have to be made by hand. - - Due to the setup of the central software stack, this feature is currently useless to install - the central stack. For user installations, there are also limitations as the enviornment - on the compute nodes is different from the login nodes so, e.g., different locations for - temporary files are being used. These would only be refreshed if the EasyBuild configuration - modules are reloaded on the compute nodes which cannot be done currently in the way Slurm - job submission is set up in EasyBuild. - - Use material in this section with care; it has not been completely tested. - - -## Configuration - -The EasyBuild configuration that is active at the time that ``eb --job`` is used -will be *passed down* into the submitted job automatically, via command line options to the ``eb`` -command that is run in the job script. - -This includes not only command line options used next to ``--job``, but also configuration settings -that are specified via an [EasyBuild configuration file](1_07_configuration#configuration-files) or through -[``$EASYBUILD_*`` environment variables](1_07_configuration#easybuild_-environment-variables). - -This implies that any EasyBuild configuration files or ``$EASYBUILD_*`` environment variables -that are in place in the job environment are most likely *irrelevant*, since configuration settings -they specify they will most likely be overruled by the corresponding command line options. -It does also imply however that the EasyBuild configuration that is in place when ``eb --job`` is used -does also work on the compute nodes to which the job is submitted. - - -## Using ``eb --job`` - -### Job backend - -The default job backend in EasyBuild v4.x is [``GC3Pie``](https://gc3pie.readthedocs.io). -To let EasyBuild submit jobs to Slurm instead, you should set the ``job-backend`` configuration setting -to ``Slurm``, for example by setting the corresponding environment variable: - -```shell -export EASYBUILD_JOB_BACKEND='Slurm' -``` - -On LUMI this is taken care of in the EasyBuild configuration modules such as ``EasyBuild-user``. - - -### Job resources - -To submit an installation as a job, simply use ``eb --job``: - -```shell -eb example.eb --job -``` - -By default, EasyBuild will submit single-core jobs requesting for 24 hours of walltime. -You can tweak the requested resources via the ``job-cores`` and ``job-max-walltime`` configuration options. -For example: - -```shell -# submit job to install example, using 5 cores and 2 hours of max. walltime -eb example.eb --job --job-cores 5 --job-max-walltime 2 -``` - -Note that not all ``job-*`` configuration settings apply to all job backends, -see the [EasyBuild documentation](https://docs.easybuild.io/en/latest/Submitting_jobs.html) for more details. - -### Controlling Slurm submission options - -When using Slurm as a job backend, EasyBuild will automatically generate job scripts which -use the ``eb`` command to perform a single installation. These scripts will be submitted -using the ``sbatch`` command. - -EasyBuild currently doesn't provide away to customize the Slurm submission options, -for example to submit to a particular partition, or to use a particular account, -build you can set the corresponding ``$SBATCH_*`` environment variables prior to running ``eb --job``. - -For example, to specify a particular account that should be used for the jobs submitted by EasyBuild -(equivalent with using the ``-A`` or ``--account`` command line option for ``sbatch``): - -```shell -export SBATCH_ACCOUNT='project_XXXXXXXXX' -``` - -Or to submit to a particular Slurm partition (equivalent with the ``-p`` or ``--partition`` option for ``sbatch``): - -```shell -export SBATCH_PARTITION='small' -``` - -For more information about supported ``$SBATCH_*`` environment variables, -see the [Slurm documentation](https://slurm.schedmd.com/sbatch.html#lbAJ). - -## Combining ``--job`` and ``--robot`` - -If one or more dependencies are still missing for the software you want to install, -you can combine ``--job`` and ``--robot`` to get EasyBuild to submit a *separate* job -for each of the installations. These jobs will *not* ``--robot``, they will each only -perform a single installation. - -Dependencies between jobs will be "registered" at submission time, so Slurm will put jobs -on hold until the jobs that install the required (build) dependencies have completed successfully, -and cancel jobs if the job to install a dependency failed for some reason. - -## Attention points - -There are a couple of important things to keep an eye on when submitting installations as jobs... - -### Differences on cluster workernodes - -Sometimes the resources available on the login nodes and cluster workernodes are slightly different, -and you may need to take this into account in your EasyBuild configuration. - -For example, plenty of disk space may be available in the `/tmp` temporary filesystem on a login node, -while the workernodes require you to use a different location for temporary files and directories. -As a result, you may need to slightly change your EasyBuild configuration when submitting installations -as jobs, to avoid that they fail almost instantly due to a lack of disk space. - -Keep in mind that the active EasyBuild configuration is passed down into the submitted jobs, -so any configuration that is present on the workernodes may not have any effect. - -For example, on LUMI it is possible to use ``$XDG_RUNTIME_DIR`` on the login nodes which has -the advantage that any leftovers of failed builds will be cleaned up when the user ends their last -login session on that node, but it is not possible to do so on the compute nodes. - -```shell -# EasByuild is configured to use /tmp/$USER on the login node -uan01 $ eb --show-config | grep buildpath -buildpath (E) = /run/user/XXXXXXXX/easybuild/build - -# use /dev/shm/$USER for build directories when submitting installations as jobs -login01 $ eb --job --buildpath /dev/shm/$USER/easybuild example.eb --robot -``` - - -### Temporary log files and build directories - -The problems for the temporary log files are twofold. First, they may end up in a place -that is not available on the compute nodes. E.g., for the same reasons as for the build -path, the LUMI EasyBuild configuration will place the temporary files in a subdirectory of -``$XDG_RUNTIME_DIR`` on the loginnodes but a subdirectory of ``/dev/shm/$USER`` on the -compute nodes. The second problem however is that if an installation fails, those log files are -not even accessible anymore which may leave you wondering about the actual cause of the failing -installation... - -To remedy this, there are a couple of EasyBuild configuration options you can use: - -* You can use ``--tmp-logdir`` to specify a different location where EasyBuild should store temporary log files, - for example: - ```shell - $ eb --job example.eb --tmp-logdir $HOME/eb_tmplogs - ``` - This will move at least the log file to a suitable place. - -* If you prefer having the entire log file stored in the Slurm job output files, - you can use ``--logtostdout`` when submitting the jobs. This will result in extensive logging - to your terminal window when submitting the jobs, but it will also make EasyBuild - log to ``stdout`` when the installation is running in the job, and hence the log messages will be - captured in the job output files. - -The build directory of course also suffers from the problem of being no longer accessible if the -installation fails, but there it is not so easy to find a solution. Building on a shared file system -is not only much slower, but in particular on parallel file systems like GPFS/SpectrumScale, Lustre -or BeeGFS buiding sometimes fails in strange ways. One thing you can consider if you cannot do the -build on a login node (e.g., because the code is not suitable for cross-compiling or the configure -system does tests that would fail on the login node), is to rety the installation in an -interactive job, so you can inspect the build directory after the installation fails. - -### Lock files - -EasyBuild creates [locks](https://docs.easybuild.io/en/latest/Locks.html) -to prevent that the same installation is started multiple times on -different system to the same installation directory. - -If an installation fails or gets interrupted, EasyBuild cleans up those locks automatically. - -However, if a Slurm job that is using EasyBuild to install software gets cancelled (because it -ran out of walltime, tried to consume too much memory, through an ``scancel`` command, etc.), -EasyBuild will not get the chance to clean up the lock file. - -If this occurs you will need to either clean up the lock file (which is located in the `software/.locks` -subdirectory of ``installpath``) manually, or re-submit the job with ``eb --job --ignore-locks``. - -## Example - -As an example, we will let EasyBuild submit jobs to install ``AUGUSTUS`` with the ``foss/2020b`` toolchain. - -!!! Warning "This example does not work on LUMI" - - Note that this is an example using the FOSS common toolchain. For this reason it does not work on - LUMI. - -### Configuration - -Before using ``--job``, let's make sure that EasyBuild is properly configured: - -```shell -# Load the EasyBuild-user module (central installations will not work at all -# using job submission) -module load LUMI/21.12 -module load partition/C -module load EasyBuild-user - -# use ramdisk for build directories -export EASYBUILD_BUILDPATH=/dev/shm/$USER/build -export EASYBUILD_TMPDIR=/dev/shm/$USER/tmp - -# use Slurm as job backend -export EASYBUILD_JOB_BACKEND=Slurm -``` - - -We will also need to inform Slurm that jobs should be submitted into a particular account, and -in a particular partition: - -```shell -export SBATCH_ACCOUNT=project_XXXXXXXXX -export SBATCH_PARTITION='small' -``` - -This will be picked up by the ``sbatch`` commands that EasyBuild will run to submit the software installation jobs. - - -### Submitting jobs to install AUGUSTUS - -Now we can let EasyBuild submit jobs for AUGUSTUS. - -Let's first check what is still missing: - -```shell -$ eb AUGUSTUS-3.4.0-foss-2020b.eb --missing -... -11 out of 61 required modules missing: - -* HTSlib/1.11-GCC-10.2.0 (HTSlib-1.11-GCC-10.2.0.eb) -* lpsolve/5.5.2.11-GCC-10.2.0 (lpsolve-5.5.2.11-GCC-10.2.0.eb) -* Boost/1.74.0-GCC-10.2.0 (Boost-1.74.0-GCC-10.2.0.eb) -* GSL/2.6-GCC-10.2.0 (GSL-2.6-GCC-10.2.0.eb) -* SAMtools/1.11-GCC-10.2.0 (SAMtools-1.11-GCC-10.2.0.eb) -* BCFtools/1.11-GCC-10.2.0 (BCFtools-1.11-GCC-10.2.0.eb) -* METIS/5.1.0-GCCcore-10.2.0 (METIS-5.1.0-GCCcore-10.2.0.eb) -* BamTools/2.5.1-GCC-10.2.0 (BamTools-2.5.1-GCC-10.2.0.eb) -* MPFR/4.1.0-GCCcore-10.2.0 (MPFR-4.1.0-GCCcore-10.2.0.eb) -* SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0.eb) -* AUGUSTUS/3.4.0-foss-2020b (AUGUSTUS-3.4.0-foss-2020b.eb) -``` - -Several dependencies are not installed yet, so we will need to use ``--robot`` to ensure that -EasyBuild also submits jobs to install these first. - -To speed up the installations a bit, we will request 8 cores for each submitted job (via ``--job-cores``). -That should be sufficient to let each installation finish in (well) under 1 hour, -so we only request 1 hour of walltime per job (via ``--job-max-walltime``). - -In order to have some meaningful job output files, we also enable trace mode (via ``--trace``). - -``` -$ eb AUGUSTUS-3.4.0-foss-2020b.eb --job --job-cores 8 --job-max-walltime 1 --robot --trace -... -== resolving dependencies ... -... -== List of submitted jobs (11): Boost-1.74.0-GCC-10.2.0 (Boost/1.74.0-GCC-10.2.0): 1000011; GSL-2.6-GCC-10.2.0 (GSL/2.6-GCC-10.2.0): 1000004; SAMtools-1.11-GCC-10.2.0 (SAMtools/1.11-GCC-10.2.0): 1000005; HTSlib-1.11-GCC-10.2.0 (HTSlib/1.11-GCC-10.2.0): 1000006; BCFtools-1.11-GCC-10.2.0 (BCFtools/1.11-GCC-10.2.0): 1000001; lpsolve-5.5.2.11-GCC-10.2.0 (lpsolve/5.5.2.11-GCC-10.2.0): 1000007; BamTools-2.5.1-GCC-10.2.0 (BamTools/2.5.1-GCC-10.2.0): 1000008; METIS-5.1.0-GCCcore-10.2.0 (METIS/5.1.0-GCCcore-10.2.0): 1000009; MPFR-4.1.0-GCCcore-10.2.0 (MPFR/4.1.0-GCCcore-10.2.0): 1000010; SuiteSparse-5.8.1-foss-2020b-METIS-5.1.0 (SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0): 1000002; AUGUSTUS-3.4.0-foss-2020b (AUGUSTUS/3.4.0-foss-2020b): 1000003 -== Submitted parallel build jobs, exiting now -``` - -### Inspecting the submitted jobs - -Once EasyBuild has submitted the jobs, we can inspect them via Slurm's ``squeue`` command: - -``` -$ squeue -u $USER -la - JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON) -1000001 small BCFtools user123 PENDING 0:00 2:00:00 1 (Dependency) -1000002 small SuiteSpa user123 PENDING 0:00 2:00:00 1 (Dependency) -1000003 small AUGUSTUS user123 PENDING 0:00 2:00:00 1 (Dependency) -1000004 small GSL-2.6- user123 RUNNING 0:21 2:00:00 1 node003 -1000005 small SAMtools user123 RUNNING 0:21 2:00:00 1 node007 -1000006 small HTSlib-1 user123 RUNNING 0:21 2:00:00 1 node007 -1000007 small lpsolve- user123 RUNNING 0:21 2:00:00 1 node011 -1000008 small BamTools user123 RUNNING 0:21 2:00:00 1 node011 -1000009 small METIS-5. user123 RUNNING 0:21 2:00:00 1 node013 -1000010 small MPFR-4.1 user123 RUNNING 0:21 2:00:00 1 node029 -1000011 small Boost-1. user123 RUNNING 0:24 2:00:00 1 node029 -``` - -Note that 3 jobs can not be started yet, because those installations require on one or more -missing dependencies. As soon as the jobs for those dependencies (successfully) complete, -these jobs will be able to start. - -### Final result - -After about 20 minutes, AUGUSTUS and all missing dependencies should be installed: - -``` -$ ls -lrt $HOME/EasyBuild/modules/.../*.lua | tail -11 --rw-rw----. 1 example example 1634 Mar 29 10:13 /users/example/easybuild/modules/all/HTSlib/1.11-GCC-10.2.0.lua --rw-rw----. 1 example example 1792 Mar 29 10:13 /users/example/easybuild/modules/all/SAMtools/1.11-GCC-10.2.0.lua --rw-rw----. 1 example example 1147 Mar 29 10:13 /users/example/easybuild/modules/all/BamTools/2.5.1-GCC-10.2.0.lua --rw-rw----. 1 example example 957 Mar 29 10:13 /users/example/easybuild/modules/all/lpsolve/5.5.2.11-GCC-10.2.0.lua --rw-rw----. 1 example example 1549 Mar 29 10:13 /users/example/easybuild/modules/all/METIS/5.1.0-GCCcore-10.2.0.lua --rw-rw----. 1 example example 1525 Mar 29 10:14 /users/example/easybuild/modules/all/GSL/2.6-GCC-10.2.0.lua --rw-rw----. 1 example example 1221 Mar 29 10:15 /users/example/easybuild/modules/all/MPFR/4.1.0-GCCcore-10.2.0.lua --rw-rw----. 1 example example 1678 Mar 29 10:15 /users/example/easybuild/modules/all/BCFtools/1.11-GCC-10.2.0.lua --rw-rw----. 1 example example 1292 Mar 29 10:21 /users/example/easybuild/modules/all/Boost/1.74.0-GCC-10.2.0.lua --rw-rw----. 1 example example 1365 Mar 29 10:28 /users/example/easybuild/modules/all/SuiteSparse/5.8.1-foss-2020b-METIS-5.1.0.lua --rw-rw----. 1 example example 2233 Mar 29 10:30 /users/example/easybuild/modules/all/AUGUSTUS/3.4.0-foss-2020b.lua - -$ module avail AUGUSTUS - --- EasyBuild managed user software for software stack ... -- - AUGUSTUS/3.4.0-foss-2020b -``` - ---- - -*[[next: Module naming schemes]](3_04_module_naming_scheme)* diff --git a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md b/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md deleted file mode 100644 index d0b4ebe8..00000000 --- a/docs/2022-CSC_and_LO/3_04_module_naming_schemes.md +++ /dev/null @@ -1,569 +0,0 @@ -# Module naming schemes - -*[[back: Submitting installations as Slurm jobs]](3_03_slurm_jobs.md)* - ---- - -Up until now we have used the naming scheme in use on LUMI which is for many practical purposes -is indistinguishable from the default EasyBuild naming scheme (`EasyBuildMNS`). -It produces module files with names that closely resemble to the names of the -corresponding easyconfig files. -For example, when installing `zlib-1.2.11-cpeGNU-21.12.eb` the generated module was named -`zlib/1.2.11-cpeGNU-21.12`. - -EasyBuild supports several different module naming schemes: - -```shell -$ eb --avail-module-naming-schemes -List of supported module naming schemes: - EasyBuildMNS - LUMI_FlatMNS - MigrateFromEBToHMNS - HierarchicalMNS - CategorizedHMNS - CategorizedModuleNamingScheme -``` - -In this part of the tutorial we will take a closer look at `HierarchicalMNS`, -which is the standard **hierarchical** module naming scheme included with EasyBuild. - -We will also take a quick look at implementing our own custom module naming scheme. - -## Flat vs hierarchical - -!!! Note - - This text is very much based on the generic EasyBuild tutorials as we cannot demonstrate - all aspects on LUMI as it is configured today. - - On LUMI, the complete module scheme is partly hierarchical, but EasyBuild uses a flat naming - scheme. The two levels in the hierarchy that are present to deal with different versions of - the software stack and the various types of compute nodes, are not managed by EasyBuild. - -The default module naming scheme `EasyBuildMNS` -and the custom `LUMI_FlatMNS` naming scheme are both examples of regular *"flat"* -module naming schemes, which are characterized by: - -- all module files are directly available for loading; -- each module name uniquely identifies a particular installation; - -In contrast, a *hierarchical* module naming scheme -consists of a *hierarchy* of module files. -A fairly typical 3-level schme (``Core``, ``Compiler`` and ``MPI``) has been -discussed in the [section on Lmod](../1_02_Lmod#lmod-hierarchy). -This typical Lmod hierarcny would map very well on the EasyBuild common toolchains. - -In fact, for the example - -
- -software at the ``Core`` level would be installed with the ``GCCcore`` and ``SYSTEM`` -toolchains. Software at the ``Compiler`` level would be installed with the ``GCC`` -toolchain, while software at the ``MPI`` level after loading the ``OpenMPI`` module -would be installed using the regular ``foss`` toolchain or the ``gompi`` toolchain -(see the diagram in the -["Common toolchains" section of the page on terminology](../1_05_terminology/#common-toolchains)). - -On LUMI, where software is installed through the Cray Programming Environment with no real choice of -MPI implementation, a two-level arrangement would still make a lot of sense, with at the ``Core`` level -all software compiled with the SYSTEM toolchain while there could be a ``PrgEnv`` level for software -compiled with a particula programming enviornment aka cepGNU/cpeCray/cpeAOCC toolchain. Such a scheme -is used on the Cray systems at CSCS. - -To recap, the characteristics of a module hierarchy are: - -* not all module files are directly available for loading; -* some modules serve as a gateway to more modules; -* to access some software installations you will first need to load one or more gateway modules in order - to use them; - -You can probably think of other ways to organize module files in a hierarchical module tree, but here -we will stick to the standard core / compiler / MPI hierarchy. - -### Pros & cons - -So why go through all this trouble of organizing modules hierarchically? - -There are a couple of advantages to this approach: - -* shorter module names (or at least for the version part of the name); -* less overwhelming list of available modules; -* only compatible modules can be loaded together; - -However, the are some minor disadvantages too: - -* not all existing modules are directly visible, so users have to learn how to find - modules using ``module spider`` etc.; -* gateway modules may have little meaning to end users; - -#### Length of module names - -When using a flat module naming scheme, module names can be fairly long and perhaps confusing. For a `HDF5` installation -with the EasyBuild common toolchains for example, -one might have `HDF5/1.10.6-gompi-2020a` as module name. The `-gompi-2020a` part of the name refers to the toolchain that was -used for this installation, but it may be confusing to some people (what kind of Pokémon is a "gompi"?!). - -In the example module hierarchy shown above, the module for `HDF5` could simply be named `HDF5/1.10.6` which is basically the bare -essentials: software name and version. That's way better, nice and clean! - -#### Amount of available modules - -The output of "`module avail`" can be quite overwhelming if lots of module files -are installed and a flat module naming scheme is used, since *all* modules are -*always* available. -EasyBuild makes it very easy to install lots of software, -so the number of installed modules can easily grow into the hundreds or even thousands... - -This often explosive growth of modules is less of an issue when using a hierarchical module naming scheme, since -initially only a small set of modules are available, and relatively limited -groups of additional modules become available as gateway modules are loaded. - -#### Loading compatible modules - -Since all modules are available at once when using a flat module naming scheme, you can easily load -modules together that are not compatible with each other. - -Imagine loading two modules that were built with a different compiler toolchain (different compiler, -different MPI library). That's likely to end in tears, unless you have the necessary technical expertise -to fully understand what is going on, *and* you are being very careful... - -In a module hierarchy this can be prevented, since modules for software that were installed with a -different compiler and/or a different MPI library are located in a different part of the module hierarchy, -and so these modules can not be loaded together easily. - - -#### Visibility of existing modules - -One downside of a module hierarchy is that not all existing modules are directly available for loading -or are even *visible* to the user, since the output of "`module avail`" only shows a subset of all modules. - -As we have discussed in [the Lmod section](../LMOD). -[Lmod](https://lmod.readthedocs.io) provides a powerful solution to search for modules through the -``module spider`` and to some extent also the ``module keyword`` commands. -So as long as the end users are -aware of this additional command, it should not be difficult to discover which software installations exist -and how they can be accessed. The "`module spider`" command will inform the user which of the gateway modules -need to be loaded in order to load a specific module file. - -#### Semantics of gateway modules - -An additional potential problem of a module hierarchy is that the semantics of the gateway modules may not be clear -to end users. They may wonder why they need to pick a specific compiler and MPI library, or which of the -available options is the best one. Maybe they are not even aware what exactly a "compiler" is, or how it is -relevant to the software they need in their bioinformatics pipeline... - -This can be partially resolved by loading a default compiler and MPI module so a particular set of modules -is available right after login, which could be the ones used in the most recent toolchain, or the -recommended versions. More experienced users could then leverage the "`module spider`" command to navigate -the module hierarchy. - -On LUMI this would mean loading a default software stack, but due to the the default modules are currently -loaded on LUMI this was not possible to accomplish without losing other functionality of the module tree. - -## Using a custom module naming scheme - -Next to the module naming schemes that are included with EasyBuild, -you can also define your own module naming scheme (MNS), and configure EasyBuild to use it -(which is eactly what has been done on LUMI to remove a feature of the default ``EasyBuildMNS`` scheme -that we do not use). - -### Implementation - -To use a custom module naming scheme, you need to implement a Python module file, -where you define a Python class that derives from the general [``ModuleNamingScheme``](https://docs.easybuild.io/en/latest/api/easybuild.tools.module_naming_scheme.mns.html#easybuild.tools.module_naming_scheme.mns.ModuleNamingScheme) class. - -For a flat module naming scheme, it is sufficient to implement the ``det_full_module_name`` method, -which should return a string value (the full module name). -You may also need to customize the ``is_short_modname_for`` method, which verifies whether -a given (short) module name is for the software with a particular given name (or not). - -The argument provided to ``det_full_module_name`` can be a small Python dictionary which only specifies -the value of a handful of easyconfig parameters (`name`, `version`, `toolchain`, and `versionsuffix`), -or an ``EasyConfig`` instance which represents a parsed easyconfig file and contains values for *all* -known easyconfig parameters. - -For simple module naming schemes, just have ``name``, ``version``, ``toolchain``, and ``versionsuffix`` -available is sufficient. If it is not, you can list which additional easyconfig parameters are required for -your module naming scheme via a class constant ``REQUIRED_KEYS``. - -A fairly straightforward example of a *flat* module name scheme is the default EasyBuild module naming scheme -[``EasyBuildMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/easybuild_mns.py). - -For a *hierarchical* module naming scheme, various additional methods have to be implemented. - -Some of these, like ``det_module_subdir`` and ``det_short_module_name``, determine which part of -the (full) module name determines the location of the module in the module hierarchy, and which part -is the user-facing (or "short") module name. Others, like ``det_modpath_extensions``, -determine which modules are *gateway* modules that open up an additional level of the hierarchy -(by *extending* the ``$MODULEPATH``). - -A typical example of a hierarchical module naming scheme is [``HierarchicalMNS``](https://github.com/easybuilders/easybuild-framework/blob/main/easybuild/tools/module_naming_scheme/hierarchical_mns.py) that comes with EasyBuild (which is quite complex because it includes quite a bit of special handling for particular compiler -toolchains), which implements the traditional ``Core``-``Compiler``-``MPI`` module hierarchy we discussed -above. - -### Configuring EasyBuild - -To let EasyBuild use a custom module naming scheme, you need to: - -* specify the path to the Python module file implementing it via the ``include-module-naming-schemes`` - EasyBuild configuration option; - -* indicate that you also want to *use* this custom module naming scheme via the ``module-naming-scheme`` - EasyBuild configuration option. - -For example: - -```shell -export EASYBUILD_INCLUDE_MODULE_NAMING_SCHEMES=$HOME/easybuild/example_mns.py -export EASYBUILD_MODULE_NAMING_SCHEME=ExampleMNS -``` - -#### Example custom module naming scheme - -Here is an example of a custom module naming scheme, where: - -* the ``versionsuffix`` goes directly after the ``version``; -* all dashes are replaced by underscores; -* all module names are lowercase; - -Note that we also need to customise the ``is_short_modname_for`` method, -to make sure it returns ``True`` when the EasyBuild framework checks whether -``scipy_bundle/2020.11_foss_2020b`` is a module name for ``SciPy-bundle``. - -```python -import os - -from easybuild.tools.module_naming_scheme.mns import ModuleNamingScheme - - -class ExampleMNS(ModuleNamingScheme): - - REQUIRED_KEYS = ['name', 'version', 'versionsuffix', 'toolchain'] - - def det_full_module_name(self, ec): - """ - Determine full module name: - - all lowercase - - replace all dashes with underscores - """ - parts = [ec['version']] - - # versionsuffix directly after version (but only if it's not empty) - if ec['versionsuffix']: - parts.append(ec['versionsuffix']) - - # only add toolchain name/version for non-system toolchain - tc = ec['toolchain'] - if tc['name'].lower() != 'system': - parts.extend([tc['name'], tc['version']]) - - modname = ec['name'] + '/' + '_'.join(parts) - - modname = modname.replace('-', '_').replace('__', '_') - - return modname.lower() - - def is_short_modname_for(self, short_modname, name): - """Determine whether short module name is a module for the software with specified name.""" - return short_modname.startswith(name.lower().replace('-', '_') + '/') -``` - -We can see what the module names with this module naming scheme would like like via ``eb -D``. -E.g., for the common toolchains (the example will not work on LUMI as the default easyconfig -files are not in the robot- and the search path): - -``` -$ eb SciPy-bundle-2020.11-foss-2020b-Python-2.7.18.eb -D - ... - * [ ] $CFGS/g/GCC/GCC-10.2.0.eb (module: gcc/10.2.0) - ... - * [ ] $CFGS/p/Python/Python-2.7.18-GCCcore-10.2.0.eb (module: python/2.7.18_gcccore_10.2.0) - ... - * [ ] $CFGS/o/OpenMPI/OpenMPI-4.0.5-GCC-10.2.0.eb (module: openmpi/4.0.5_gcc_10.2.0) - ... - * [ ] $CFGS/s/SciPy-bundle/SciPy-bundle-2020.03-foss-2020a-Python-2.7.18.eb (module: scipy_bundle/2020.03_python_2.7.18_foss_2020a) -``` - -## Example module hierarchy: HDF5 - -!!! Warning "Example not suitable for LUMI" - **This exercise is meant for a system where the common toolchains can be used and requires an - indpendent EasyBuild installation in your personal file space**, - because EasyBuild will try to copy the installation log file to each installation directory. - -Now that we know more about hierarchical module naming schemes, -let us see how EasyBuild can help us with generating a hierarchical module tree. - -In this example we will use EasyBuild to generate modules organised in a hierarchy -for some of the software that is already installed in the prepared environment. - -The good news is that the existing installations can be reused. There is absolutely no need -to reinstall the software, we are just creating a different "view" on these software installations. - -### Preparing the environment - -Before running EasyBuild to generate a hierarchical module tree, we have to be a bit careful -with preparing our environment. - -**We must absolutely avoid mixing modules from a flat and hierarchical module naming scheme!** - -Some module files will have the same name in both module trees (like `GCC/10.2.0` for example), -but their contents will be different. -Mixing modules from a flat and hierarchical module tree *will* trigger problems... - -So we have to make sure that the module files we already have in `/easybuild` are *not* visible. -The easiest way to do this is to unload all modules (using "`module purge`") -and resetting the module search path to be empty, which we can do with "`module unuse $MODULEPATH`". - -```shell -module purge -module unuse $MODULEPATH -``` - -In this part of the tutorial, we are assuming you are *not* using an EasyBuild installation provided through -a module. We have just made all modules unavailable, so we would have to first -install EasyBuild again in our hierarchical module tree before we can continue. - -**We strongly recommend using an EasyBuild installation that was [installed via "`pip install`" -or "`pip3 install`"](../1_06_installation#method-1-using-pip) in this part of the tutorial.** - -An easy way to do this is in the prepared environment is to run: - -```shell -pip3 install --user easybuild -export PATH=$HOME/.local/bin:$PATH -export EB_PYTHON=python3 -``` - -### Configuring EasyBuild - -First of all, we need to make sure that EasyBuild is properly configured. -We can do this by defining this set of environment variables: - -```shell -export EASYBUILD_PREFIX=$HOME/easybuild -export EASYBUILD_BUILDPATH=/tmp/$USER -export EASYBUILD_INSTALLPATH_SOFTWARE=/easybuild/software -export EASYBUILD_MODULE_NAMING_SCHEME=HierarchicalMNS -export EASYBUILD_INSTALLPATH_MODULES=$HOME/hmns/modules -``` - -To make sure we didn't make any silly mistakes, we double check using `eb --show-config`: - - -```shell -$ eb --show-config -# -# Current EasyBuild configuration -# (C: command line argument, D: default value, E: environment variable, F: configuration file) -# -buildpath (E) = /tmp/example -containerpath (E) = /home/example/easybuild/containers -installpath (E) = /home/example/easybuild -installpath-modules (E) = /home/example/hmns/modules -installpath-software (E) = /easybuild/software -module-naming-scheme (E) = HierarchicalMNS -packagepath (E) = /home/example/easybuild/packages -prefix (E) = /home/example/easybuild -repositorypath (E) = /home/example/easybuild/ebfiles_repo -robot-paths (D) = /home/example/.local/easybuild/easyconfigs -sourcepath (E) = /home/example/easybuild/sources -``` - -There are a couple of things worth pointing out here: - -* We have defined the `module-naming-scheme` configuration setting to `HierarchicalMNS`, - which makes EasyBuild use the included standard hierarchical module naming scheme (the classic - core / compiler / MPI one we discussed above). -* We have specified *different* locations for the software (via `installpath-software`) - and the module files (via `installpath-modules`). This is important because we want to - reuse the software that is already installed in `/easybuild/software` while we want to - generate an entirely new module tree for it (in `$HOME/hmns/modules`). - -The other configuration settings are the same as before, and mostly irrelevant for this example. - -### Generating modules for HDF5 - -Let us now generate a hierarchical module tree for `HDF5` and all of its dependencies, -including the toolchain. That sounds complicated, and it sort of is since there are -a lot of details you have to get right for the module hierarchy to works as intended, -but EasyBuild can do all the hard work for us. - -The steps we will have to go through are: - -* Tell EasyBuild we want to "install" the `HDF5-1.10.7-gompi-2020b.eb` easyconfig file; -* Enable dependency resolution via `--robot`; -* Assuming the software would have been installed already with the default naming scheme - in a different module directory, instruct EasyBuild to only generate the module files, - not to install the software (since it is - there already in `/easybuild/software`), via the `--module-only` option. - -These steps translate to this single `eb` command: - -``` -$ eb HDF5-1.10.7-gompi-2020b.eb --robot --module-only -... -== building and installing MPI/GCC/10.2.0/OpenMPI/4.0.5/HDF5/1.10.7... -... -== sanity checking... -== cleaning up [skipped] -== creating module... -... -== COMPLETED: Installation ended successfully (took 9 sec) -... -== Build succeeded for 41 out of 41 -``` - -This should take a couple of minutes in total, for generating 41 modules. -Remember that this also includes generating module files for the toolchain and -all of its components. - -In addition, there is a bit more going on one that just generating module files, -since the sanity check step is still being run for each of the installations -when using `--module-only` to ensure the installation is actually functional. -After all, there is no point in generating a module for an obviously broken -installation... - -### Loading the HDF5 module - -After generating the hierarchical module tree for HDF5, how do we access the HDF5 installation through it? - -Here's what the module tree looks like on disk: - -``` -$ ls $HOME/hmns/modules/all -Compiler Core MPI -``` - -Those are basically the 3 levels in the module hierarchy we showed in our example earlier. - -The starting point is the top level of the module hierarchy named `Core`: - -``` -module use $HOME/hmns/modules/all/Core -``` - -Let us see what that gives us in terms of available modules: - -``` -$ module avail - ---------------------- /home/easybuild/hmns/modules/all/Core --------------------- - Bison/3.5.3 GCCcore/10.2.0 flex/2.6.4 help2man/1.47.4 - Bison/3.7.1 (D) M4/1.4.18 gettext/0.21 ncurses/6.2 - GCC/10.2.0 binutils/2.35 gompi/2020b zlib/1.2.11 -``` - - -Nice and short module names, but only a limited set of them. - -We know a module file exists for `HDF5`, but we can't see it yet (and hence -we can't load it either). - -``` -$ module avail HDF5 -No module(s) or extension(s) found! -Use "module spider" to find all possible modules and extensions. -``` - -Let us see if `module spider` is of any help, as "`module avail`" so kindly suggests: - -``` -$ module spider HDF5 -... - - You will need to load all module(s) on any one of the lines below - before the "HDF5/1.10.7" module is available to load. - - GCC/10.2.0 OpenMPI/4.0.5 -``` - -This tells us we need to load two gateway modules before we can load the module -for HDF5. - -Let us start with loading the `GCC` compiler module: - -``` -module load GCC/10.2.0 -``` - -And then check again which modules are available: - -``` -$ module avail - --------------- /home/easybuild/hmns/modules/all/Compiler/GCC/10.2.0 -------------- - OpenMPI/4.0.5 - ------------- /home/easybuild/hmns/modules/all/Compiler/GCCcore/10.2.0 ------------ - Autoconf/2.69 UCX/1.9.0 libpciaccess/0.16 - ... - Szip/2.1.1 libfabric/1.11.0 zlib/1.2.11 (L,D) - Autoconf/2.69 XZ/5.2.5 libtool/2.4.6 - ---------------------- /home/easybuild/hmns/modules/all/Core --------------------- - Bison/3.5.3 GCCcore/10.2.0 (L) flex/2.6.4 help2man/1.47.4 - Bison/3.7.1 (D) M4/1.4.18 gettext/0.21 ncurses/6.2 - GCC/10.2.0 (L) binutils/2.35 gompi/2020b zlib/1.2.11 -``` - -Good news, we now have additional modules available! - -The compiler level of our hierarchy actually consists of two directories here: `Compiler/GCCcore/10.2.0` -and `Compiler/GCC/10.2.0`. The modules in the `GCCcore` directory are ones we can use in other compiler -toolchains that use GCC 10.2.0 as a base compiler (the details of that are out of scope here). - -The module we are interested in is `OpenMPI/4.0.5`, which is another gateway module. - -Remember that the "`module spider`" output told us that there does indeed exist a module for `HDF5`, but that -we need to load *both* the `GCC/10.2.0` and `OpenMPI/4.0.5` modules first. - -So, let us do exactly that (remember that `GCC/10.2.0` is already loaded): - -``` -module load OpenMPI/4.0.5 -``` - -If you now check the output of "`module avail`" again, you should see the `HDF5/1.10.7` module: - -``` -$ module avail - --------- /home/easybuild/hmns/modules/all/MPI/GCC/10.2.0/OpenMPI/4.0.5 ------- - HDF5/1.10.7 - ------------- /home/easybuild/hmns/modules/all/Compiler/GCC/10.2.0 ------------ - OpenMPI/4.0.5 (L) - -... -``` - -To use HDF5, we need to load this `HDF5/1.10.7` module. We can verify that the installation works -using one of the commands provided by HDF5, `h5dump` for example: - -``` -module load HDF5/1.10.7 -``` - -``` -$ h5dump --version -h5dump: Version 1.10.7 -``` - -If you now check which modules are loaded via "`module list`", you will notice that all module names -and nice and short now, which is one of the advantages of using a hierarchical module tree: - -```shell -$ module list - -Currently Loaded Modules: - 1) GCCcore/10.2.0 5) numactl/2.0.13 9) hwloc/2.2.0 13) PMIx/3.1.5 - 2) zlib/1.2.11 6) XZ/5.2.5 10) libevent/2.1.12 14) OpenMPI/4.0.5 - 3) binutils/2.35 7) libxml2/2.9.10 11) UCX/1.9.0 15) Szip/2.1.1 - 4) GCC/10.2.0 8) libpciaccess/0.16 12) libfabric/1.11.0 16) HDF5/1.10.7 -``` - - ---- - -*[[next: GitHub integration]](3_05_github_integration.md)* diff --git a/docs/2022-CSC_and_LO/3_05_github_integration.md b/docs/2022-CSC_and_LO/3_05_github_integration.md deleted file mode 100644 index 6e19cb9a..00000000 --- a/docs/2022-CSC_and_LO/3_05_github_integration.md +++ /dev/null @@ -1,460 +0,0 @@ -# GitHub integration to facilitate contributing to EasyBuild - -*[[back: Module naming schemes]](3_04_module_naming_schemes.md)* - ---- - -!!! Warning "Preliminary not for LUMI and many other Cray systems" - - Not everything in this section is as useful for EasyBuild installations that build - on the Cray Programming Environment. E.g., the default easyconfig repository does - not contain easyconfigs for the Cray PE. - - Moreover, due to the restricted operating system on the compute nodes of big - Cray systems such as the EX series (COS instead of the full SUSE linux), - the GitHub integration does not work on the compute nodes of a typical setup as - one of the required Python packages fails to work as it needs the DBus daemon. - - Since GitHub integration is not yet really used on LUMI, this section is not fully - tested but left in the tutorial for completeness. - -To contribute changes to the EasyBuild code (framework or easyblocks) or easyconfigs, -you will need to be a bit familiar with Git and GitHub. Or maybe not? - -## Manual contribution procedure - -0) Create and setup a [GitHub account](https://github.com/join) (and register your SSH public key); - -1) Clone and fork the appropriate GitHub repository, for example when contributing an easyconfig file: - -```shell -git clone git@github.com:easybuilders/easybuild-easyconfigs.git -cd easybuild-easyconfigs -git remote add my_fork git@github.com:your_github_account/easybuild-easyconfigs.git -``` - -**You should change '`your_github_account`' in the last line to your own GitHub user name!** - -2) Create and check out a new branch, starting from the (up-to-date) ``develop`` branch: - -``` -git checkout develop -git pull origin develop -git checkout -b example -``` - -3) Stage the changes you want to contribute, after you make sure that your easyconfig file has the -[correct filename](../basic_usage/#easyconfig-filenames), and that it's located in the appropriate directory. - -```shell -mkdir -p easybuild/easyconfigs/e/example/ -mv example.eb easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb -git add easybuild/easyconfigs/e/example/example-1.2.3-GCC-9.3.0.eb -``` - -4) Commit those changes with a sensible commit message: - -```shell -git commit -m "This is just an example" -``` - -5) Push your branch to your fork of the repository on GitHub: - -```shell -git push my_fork example -``` - -6) Open the pull request through the GitHub web interface, making sure that: - -* the target branch is correct (should be `develop`); -* an appropriate title is used; -* a short description of the changes is provided; -* the changes are indeed the ones you want to propose; -* clicking the (correct) green button; - -
Pfft
- -That didn't exactly motivate you to contribute, did it... - -## Github integration features - -Over the years we noticed that some people were keen on contributing to EasyBuild, -but they were not very familiar with Git or GitHub. That meant they had to overcome a -relatively steep learning curve before they could contribute... - -
Gandalf vs Git
- -In addition, the contribution workflow can be a bit daunting and time consuming, -even if you're already familiar with the procedure. You will have dozens of -branches flying around in no time, and if you get stuck in a weird corner -with `git` you may quickly end up demotivated. - -This is frustrating not only for the people who wanted to contribute but -also for the EasyBuild maintainers, and it doesn't agree with the philosophy of -a project that aims to *automate* tedious software installation procedures. - -At the end of 2015 efforts were made to tackle this issue by implementing -GitHub integration features in EasyBuild, which automate the contribution -workflow by running `git` commands and interacting with the [GitHub API](https://developer.github.com/v3/). - -We will briefly go over some of these features here, but they are also covered in detail [in the EasyBuild documentation](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html). - -### Requirements & configuration - -First of all, the GitHub integration features impose a couple of additional [requirements](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) -and configuration. - - -**Additional dependencies** - -Both the `GitPython` and `keyring` Python packages as well as the `keyrings.cryptfile` add-on package must be installed. - -!!! Note - You may experiences problems installing the ``cryptography`` Python packages, - which is a dependency of keyring. The underlying cause is that you need to have - the [``Rust``](https://www.rust-lang.org/) compiler installed to install the latest version - of ``cryptography`` (see [here](https://github.com/pyca/cryptography/issues/5771)). - - You can work around this issue using: - - ```shell - pip3 install --user 'cryptography<3.4' - ``` - -**SSH public key in GitHub account** - -You need to have a GitHub account that has your SSH public key registered in it -(via [https://github.com/settings/keys](https://github.com/settings/keys)). - -If you need to generate an SSH key pair, you can run the following command: - -```shell -ssh-keygen -t rsa -b 4096 -``` - -You can copy the SSH public key from the output of this command: - -```shell -cat .ssh/id_rsa.pub -``` - - -**Forked repository in GitHub** - -In addition, you must have *forked* the EasyBuild repository you want to contribute to -(for example [https://github.com/easybuilders/easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs)). - -**EasyBuild configuration, incl. GitHub token** - -You also have to configure EasyBuild a bit more, so it knows about your -GitHub user name *and* has a GitHub token available in order to perform actions -in GitHub with your credentials. - -To do this, you should define the `github-user` configuration option and -run the "`eb --install-github-token`" command: - -```shell -# replace 'ebtutorial' with your own GitHub username! -$ export EASYBUILD_GITHUB_USER=ebtutorial -$ eb --install-github-token -``` - -To create a GitHub token: - -* Visit [https://github.com/settings/tokens](https://github.com/settings/tokens). -* Click *"Personal access tokens"*. -* Click followed by *"Generate new token"*. -* Give the token a name (for example *"Token for EasyBuild"*). -* Select both the '`repo`' and '`gist`' scopes. -* Click the green *"Generate token"* button. -* Copy the generated token. -* Paste the token when asked by `--install-github-token` (and hit *Enter*). -* Enter a password to encrypt your GitHub token. - -The output should look something like this: - -```shell -$ eb --install-github-token -== temporary log file in case of crash /tmp/eb-9z0bdve9/easybuild-hfpti62w.log -Token: -Validating token... -Token seems to be valid, installing it. -Please set a password for your new keyring: -Please confirm the password: -Token 'fed..987' installed! -``` - - -**Checking status of GitHub integration** - -You can check the status of the GitHub integration using "`eb --check-github`": - -```shell -$ eb --check-github -== temporary log file in case of crash /tmp/eb-4ckdlyfy/easybuild-gp69ev2w.log - -Checking status of GitHub integration... - -Making sure we're online...OK - -* GitHub user...ebtutorial => OK -Please enter password for encrypted keyring: -* GitHub token...fed..987 (len: 40) => OK (validated) -* git command...OK ("git version 1.8.3.1; ") -* GitPython module...OK (GitPython version 3.1.3) -* push access to ebtutorial/easybuild-easyconfigs repo @ GitHub...OK -* creating gists...OK -* location to Git working dirs... not found (suboptimal) - -All checks PASSed! - -Status of GitHub integration: -* --from-pr: OK -* --new-pr: OK -* --review-pr: OK -* --update-pr: OK -* --upload-test-report: OK -``` - -If you see '`OK`' for each of the status checks, you're all set -to try out the GitHub integration features! - -!!! Note - If your SSH private key is protected with a password, you may need - to enter your password a couple of times when running "`eb --check-github`". - - You can avoid this by [using an SSH agent](https://help.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent). - -!!! Note - If you see the ``push access`` check fail with ``Failed to fetch branch 'main'``, - you will need to rename the ``master`` branch in your fork of the ``easybuild-easyconfigs`` - repository from ``master`` to ``main`` (this is required since EasyBuild v4.3.3). - - This can be done via the *pencil* icon at - [https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches](https://github.com/YOUR_GITHUB_ACCOUNT/easybuild-easyconfigs/branches) - (replace ``YOUR_GITHUB_ACCOUNT`` with the name of your GitHub account in this URL). - -### Creating pull requests - -The GitHub integration in EasyBuild allows you to **create pull requests -using the `eb` command**, without even leaving your shell environment. -How cool is that‽ - -To create a pull request to the `easybuild-easyconfigs` repository, -you can either do it in a single go by -running "`eb --new-pr`" and passing it one or more easyconfig files to add -into the pull request. - -The more detailed option is to first create a branch in your repository fork -in GitHub via "`eb --new-branch-github`" and then later open the pull request -via "`eb --new-pr-from-branch`". This method can be useful when preparing multiple -interdependent pull requests, or to check whether your changes pass the unit tests -(which are run automatically in the GitHub Actions CI environment for -all branches pushed to your fork). - -The `--new-pr` option can also be used to open pull requests to the easyblocks -and framework repositories, and it will even automatically determine the target -repository based on the contents of the files you provide. Whoa! - -You can control the target repository for your pull request using -`--pr-target-account` (default is `easybuilders`) and `--pr-target-repo`. - -If you want you can carefully double check your work before actually -opening the pull request by doing a dry run via "`eb --dry-run --new-pr`" -or "`eb -D --new-pr`". - -Finally, you can use "`eb --preview-pr`" to see how the easyconfig files -for which you plan to create a pull request differ from existing easyconfig -files. - -### Updating pull requests - -To update an existing pull request with additional changes -you can use "`eb --update-pr`" and pass the pull request ID, -alongside the paths to the updated files. - -If you have only created a branch (for example via `eb --new-branch-github`) -you can update it via `--update-branch-github` in the same way, -passing the branch name instead of a pull request ID. - -### Using a pull request - -Next to creating and updating branches and pull requests -you can also *use* easyconfig files and easyblocks from a pull request, -regardless of its status (open, merged, or closed). This is particularly -useful when testing contributions, or to install software for which -support is not yet included in the latest EasyBuild release. - -Using the `--from-pr` option you can install easyconfig files from the -pull request with specified ID. By default all easyconfig files that are -touched by the pull request will be installed, but you can specify -particular ones to use as well. It is generally advised to also use the -`--robot` option to ensure that the easyconfig files are installed in the -correct order with respect to dependencies. - -Similarly, using a new or updated easyblock from a pull request is as simple -as using the `--include-easyblocks-from-pr` option. And of course you can -combine it with `--from-pr`! - -Via `--upload-test-report` you can let EasyBuild submit a comment into the -easyconfig pull request to show that the installation worked on your system. This is -useful for others to know, in particular EasyBuild maintainers, since the comment -will include information about your system (OS, processor, etc.) and your EasyBuild configuration. - -## Demo - -!!! Warning "Not currently suited for LUMI" - - This is text from a previous (non-Cray) version of the tutorial. The demo does require - a number of files not installed on LUMI after going through this tutorial, - and someone who can master the repository used to clean up again. - However, having a look at the output (which was generated in the spring of 2021 on - the CSC system puhti) still offers some information, so the demo was left in this - version of the tutorial. - -That is a lot to digest, so let us make this a bit more concrete with an example: -we will open a pull request for the [`eb-tutorial` example software](../adding_support_software/#example) to *a fork* of the [`easybuild-easyconfigs` repository](https://github.com/easybuilders/easybuild-easyconfigs) using the `eb` command, -and submit a test report in it. - -!!! Note - Make sure that you have correctly configured the GitHub integration, - [see above](#requirements-configuration). - -### Creating pull request - -We first configure EasyBuild to target the `ebtutorial` GitHub account rather -than the default `easybuilders` GitHub organisation, -by defining the `pr-target-account` configuration setting: - -```shell -export EASYBUILD_PR_TARGET_ACCOUNT=ebtutorial -``` - -In the output of "`eb --show-config`" you should see a line like this: - -``` -pr-target-account (E) = ebtutorial -``` - -We only do this to avoid that lots of pull requests for the `eb-tutorial` -example software are opened in the [central easyconfigs repository](https://github.com/easybuilders/easybuild-easyconfigs). - -Opening a pull request is as simple as running "`eb --new-pr`" and passing -the easyconfig file: - -```shell -$ eb --new-pr example.eb -== temporary log file in case of crash /tmp/eb-ggr6scbq/easybuild-hnk271xj.log -== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... -== fetching branch 'develop' from https://github.com/ebtutorial/easybuild-easyconfigs.git... -== copying files to /tmp/eb-ggr6scbq/git-working-dirxwk1fzaw/easybuild-easyconfigs... -== pushing branch '20200622095415_new_pr_eb-tutorial100' to remote 'github_ebtutorial_qgtfU' (git@github.com:ebtutorial/easybuild-easyconfigs.git) -Enter passphrase for key '/home/example/.ssh/id_rsa': -Please enter password for encrypted keyring: - -Opening pull request -* target: ebtutorial/easybuild-easyconfigs:develop -* from: ebtutorial/easybuild-easyconfigs:20200622095415_new_pr_eb-tutorial100 -* title: "{tools}[GCC/10.2.0] eb-tutorial v1.0.1" -* labels: new -* description: -""" -(created using `eb --new-pr`) - -""" -* overview of changes: - easybuild/easyconfigs/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb | 26 ++++++++++++++++++++++++++ - 1 file changed, 26 insertions(+) - -Opened pull request: https://github.com/ebtutorial/easybuild-easyconfigs/pull/ -== Temporary log file(s) /tmp/eb-ggr6scbq/easybuild-hnk271xj.log* have been removed. -== Temporary directory /tmp/eb-ggr6scbq has been removed. -``` - -Take a moment to grasp what we did here: we ran **a single `eb` command** which -took care of the **[whole contribution procedure](#contribution-procedure)** for us, including: - -* Cloning the `easybuilders/easybuild-easyconfigs` repository and checking out the `develop` branch (in a temporary - directory); -* Picking a sensible name for a branch and creating it; -* Adding the `eb-tutorial` easyconfig file to the branch, in the correct location - (`easybuild/easyconfigs/e/eb-tutorial/`) and with the correct filename (`eb-tutorial-1.0.1-GCC-10.2.0.eb`); -* Pushing the branch to our fork (`example/easybuild-easyconfigs`); -* Actually opening the pull request, using an informative title. - -That is so... easy! - -This feature not only *significantly* lowers the bar for contributing, -it also saves quite a bit of time since you don't need to double check -various details (like targeting the `develop` branch) or spend time on -coming up with a nice looking title or funny branch name (although you -still can if you really want to). - -There are a couple of nice side effects too, like not having any local branches -to tidy up on once the pull request gets merged (since `--new-pr` created the -branch only in a temporary directory). - -If many contributions are made via `--new-pr` it also simplifies the task -of EasyBuild maintainers, since pull requests opened this way have a particular -structure to them and thus are easier to digest because they look familiar. - -### Uploading test report - -After opening the pull request, we should also upload a test report to show that the installation is working. -This is just as easy as creating the pull request. - -First make sure that the pre-installed software in the prepared environment -is available, since the required dependencies for `eb-tutorial` are already -installed there: - -```shell -module use /easybuild/modules/all -``` - -You can verify which dependencies are still missing using `--from-pr` combined with `--missing`: - -```shell -# change '1' to the ID of your own pull request (see output of --new-pr) -$ eb --from-pr 1 --missing -== temporary log file in case of crash /tmp/eb-ioi9ywm1/easybuild-e3v0xa1b.log -Please enter password for encrypted keyring: -== found valid index for /home/example/.local/easybuild/easyconfigs, so using it... - -1 out of 20 required modules missing: - -* eb-tutorial/1.0.1-GCC-10.2.0 (eb-tutorial-1.0.1-GCC-10.2.0.eb) -``` - -Uploading a test report boils down to combining `--from-pr` with `--upload-test-report`: - -```shell -# change '1' to the ID of your own pull request (see output of --new-pr) -$ eb --rebuild --from-pr 1 --upload-test-report -Please enter password for encrypted keyring: -... -== processing EasyBuild easyconfig /tmp/eb-bnb1pv3n/files_pr65/e/eb-tutorial/eb-tutorial-1.0.1-GCC-10.2.0.eb -== building and installing eb-tutorial/1.0.1-GCC-10.2.0... -... -== COMPLETED: Installation ended successfully (took 2 sec) -... -Adding comment to easybuild-easyconfigs issue #65: 'Test report by @ebtutorial -**SUCCESS** -Build succeeded for 1 out of 1 (1 easyconfigs in this PR) -example - Linux centos linux 7.8.2003, x86_64, Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz (haswell), Python 3.6.8 -See https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 for a full test report.' -== Test report uploaded to https://gist.github.com/f7c74159c809029afd99e30e4d994ef1 and mentioned in a comment in easyconfigs PR#1 -``` - -Note that we may need to use `--rebuild` here since `eb-tutorial` may already be installed. - -This results in a comment being added to the pull request: - -
Test report comment
- -The gist linked from this comment provides more detailed information: - -
Test report gist
- ---- - -*[[next: Additional reading]](4_00_additional_reading.md)* diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md b/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md index 74f3f82a..1e1de7c7 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md @@ -40,7 +40,6 @@ community: adapt, e.g., the easyconfig files for Open MPI to work on LUMI. Some site have really large hook files to implement policies and modify standard EasyBuild build recipes, e.g., - - [The hooks file from Jülich Supercomputing Centre](https://github.com/easybuilders/JSC/blob/2022/Custom_Hooks/eb_hooks.py) - ComputeCanada [cc_hooks_common.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_common.py) and [cc_hooks_gento.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_gentoo.py) diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md index d49a7925..f5c4ef39 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md @@ -332,4 +332,8 @@ $ module avail AUGUSTUS --- +<<<<<<< HEAD *[[next: Module naming schemes]](3_04_module_naming_schemes.md)* +======= +*[[next: Module naming schemes]](3_04_module_naming_scheme)* +>>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) From 806738ee42e74787f8f6011d5ddd87264db62468 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 7 Apr 2022 21:14:52 +0200 Subject: [PATCH 153/159] Correction of links. --- .../1_Intro/1_01_what_is_easybuild.md | 38 ++++++++++++++++++- .../3_Advanced/3_03_slurm_jobs.md | 4 ++ 2 files changed, 41 insertions(+), 1 deletion(-) diff --git a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md index 421c94cd..d268e40e 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md @@ -1,5 +1,6 @@ # What is EasyBuild? +<<<<<<< HEAD <<<<<<< HEAD *[[back: Course overview]](../index.md)* @@ -8,8 +9,11 @@ **[EasyBuild](https://easybuild.io)** is a **software build and installation framework** ======= *[[back: Overview]](index.md)* +======= +*[[back: Course overview]](../index.md)* +>>>>>>> ae27a051 (Correction of links.) -

EasyBuild logo

+

EasyBuild logo

**[EasyBuild](http://easybuild.io)** is a **software build and installation framework** >>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) @@ -24,8 +28,12 @@ and provides a **command-line interface** to facilitate the installation and man ======= +<<<<<<< HEAD >>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) +======= + +>>>>>>> ae27a051 (Correction of links.) ## Elevator pitch @@ -215,11 +223,15 @@ Since EasyBuild is primarily intended for installing (scientific) software on HP systems running some flavor of **GNU/Linux**. EasyBuild also works on *macOS*, but significant effort would be required to use it extensively there. +<<<<<<< HEAD <<<<<<< HEAD ======= >>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) +======= + +>>>>>>> ae27a051 (Correction of links.) EasyBuild is implemented in [**Python**](https://www.python.org), and is compatible with Python 2.7 and 3.5+. Since [Python 2 is no longer maintained](https://blog.python.org/2020/04/python-2718-last-release-of-python-2.html), @@ -235,11 +247,15 @@ On LUMI Lmod is used as the module tool. ## Development +<<<<<<< HEAD <<<<<<< HEAD ======= >>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) +======= + +>>>>>>> ae27a051 (Correction of links.) EasyBuild is **actively developed** by a worldwide community under the [**GPLv2 open source license**](https://www.gnu.org/licenses/old-licenses/gpl-2.0.html) via the [**``easybuilders``** organisation on GitHub](https://github.com/easybuilders), in the following repositories: @@ -277,6 +293,7 @@ sites leverage it, and integrate it with other tools like [Jenkins](https://www. HPC sites, consortia, and companies that use EasyBuild include (but are not limited to): +<<<<<<< HEAD <<<<<<< HEAD a href="https://www.vscentrum.be">VSC logo JSC logo @@ -289,12 +306,25 @@ a href="https://www.vscentrum.be">VSC <a href=UiO logo BEAR logo SNIC logo +======= +VSC logo +JSC logo +HPCNow! logo +Fred Hutch logo +Vienna BioCenter logo +CÉCI logo +Compute Canada logo +UiO logo +BEAR logo +SNIC logo +>>>>>>> ae27a051 (Correction of links.) National University of Singapore logo IT4Innovations logo Texas A&M logo TU Dresden logo University of Melbourne logo University of Luxembourg logo +<<<<<<< HEAD EPCC logo Cyprus Institute logo University of Basel logo @@ -321,6 +351,12 @@ a href="https://www.vscentrum.be">VSC <a href=University of Basel logo Microsoft logo >>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) +======= +EPCC logo +Cyprus Institute logo +University of Basel logo +Microsoft logo +>>>>>>> ae27a051 (Correction of links.) --- diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md index f5c4ef39..7eee7d54 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md @@ -332,8 +332,12 @@ $ module avail AUGUSTUS --- +<<<<<<< HEAD <<<<<<< HEAD *[[next: Module naming schemes]](3_04_module_naming_schemes.md)* ======= *[[next: Module naming schemes]](3_04_module_naming_scheme)* >>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) +======= +*[[next: Module naming schemes]](3_04_module_naming_schemes.md)* +>>>>>>> ae27a051 (Correction of links.) From a1b93c908c20ea2cc75d5a2e07b4ea7d499ee9ac Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 8 Apr 2022 18:35:45 +0200 Subject: [PATCH 154/159] Corrected a number of spelling mistakes. --- .../1_Intro/1_01_what_is_easybuild.md | 106 --------------- docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md | 32 ----- .../1_Intro/1_06_installation.md | 4 - .../3_Advanced/3_03_slurm_jobs.md | 8 -- .../4_00_additional_reading.md | 4 - docs/2022-CSC_and_LO/index-lust.md | 123 ++++++++++++++++++ 6 files changed, 123 insertions(+), 154 deletions(-) create mode 100644 docs/2022-CSC_and_LO/index-lust.md diff --git a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md index d268e40e..422d3b85 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md @@ -1,17 +1,10 @@ # What is EasyBuild? -<<<<<<< HEAD -<<<<<<< HEAD *[[back: Course overview]](../index.md)*

EasyBuild logo

**[EasyBuild](https://easybuild.io)** is a **software build and installation framework** -======= -*[[back: Overview]](index.md)* -======= -*[[back: Course overview]](../index.md)* ->>>>>>> ae27a051 (Correction of links.)

EasyBuild logo

@@ -23,17 +16,8 @@ and provides a **command-line interface** to facilitate the installation and man --- -<<<<<<< HEAD -======= - -<<<<<<< HEAD - ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) -======= - ->>>>>>> ae27a051 (Correction of links.) ## Elevator pitch @@ -89,11 +73,7 @@ as it is performed. EasyBuild supports using a **custom module naming scheme**, allows for *hierarchical* module naming schemes, and **integrates with various other tools** ranging from resource managers ([Slurm](https://slurm.schedmd.com) and [GC3Pie](https://github.com/gc3pie/gc3pie)), -<<<<<<< HEAD container tools ([Singularity](https://github.com/apptainer/singularity) and [Docker](https://www.docker.com)), -======= -container tools ([Singularity](https://github.com/hpcng/singularity) and [Docker](https://www.docker.com)), ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) packaging tools ([FPM](https://fpm.readthedocs.io)), and so on. --- @@ -120,11 +100,7 @@ By default EasyBuild tries to run tests if the build process of a package provid functionality, or can run simple tests of its own, and of course these tests will fail if the host CPU does not support the full instruction set of the target CPU. Secondly, more and more software has broken build processes where developers thought that it was a good -<<<<<<< HEAD idea to hard-code options such as ``-march=native`` or ``-xHost`` in a way that EasyBuild -======= -idea to hard-code options such as ``-march=native`` or ``--xHost`` in a way that EasyBuild ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) cannot easily overwrite those. @@ -132,24 +108,14 @@ cannot easily overwrite those. In addition to performance, **reproducibility of installations** is a core aspect of EasyBuild. -<<<<<<< HEAD Most software installations performed with EasyBuild use a **particular compiler toolchain**, -======= -TODO correct link in the next lines. - -Most software installations performed with EasyBuild use a **particular compiler toolchain**, ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) with which we aim to be in control over the build environment and avoid relying on tools and libraries provided by the operating system. For similar reasons, we try to **provide all required dependencies through EasyBuild** as well, with a few notable exceptions, like ``OpenSSL`` for security reasons, and Infiniband and GPU drivers which are too closely intertwined with the operating system. For both toolchains and dependencies, **fixed software versions** are specified in the -<<<<<<< HEAD easyconfig files. That way, easyconfig files can easily be shared with others: -======= -easyconfig files. That way, easyconfig files can easily be shared with others: ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) if they worked for you it is very likely that they will work for others too, because the vast majority of the software stack is controlled by EasyBuild. @@ -166,30 +132,17 @@ and to [**contribute back**](https://docs.easybuild.io/en/latest/Contributing.ht by opening pull requests to the [GitHub repositories](https://github.com/easybuilders) or the [documentation](https://docs.easybuild.io). The EasyBuild community also provides packages for two standard toolchains, called the -<<<<<<< HEAD [**common toolchains**](https://docs.easybuild.io/en/latest/Common-toolchains.html). The so-called ``foss`` is based on free and open software (GNU compilers, FlexiBlas with OpenBLAS backend, LaPACK, FFTW and Open MPI) while the ``intel`` toolchain uses the full stack of Intel compilers, libraries and MPI. Currently these toolchains cannot be supported on LUMI though as getting Open MPI to run on LUMI is challenging and as the Intel libraries and some MPI versions also produce slow or incorrect results on AMD CPUs (and it is not entirely clear if Intel MPI is compatible with SlingShot 11, though in principle it should). -======= -[**common toolchains**](https://easybuild.readthedocs.io/en/latest/Common-toolchains.html). -The so-called ``foss`` is based on free and open software (GNU compilers, FlexiBlas with OpenBLAS backend, LaPACK, FFTW and Open MPI) -while the ``intel`` toolchain uses the full stack of Intel compilers, libraries and MPI. Currently these toolchains -cannot be supported on LUMI though as getting Open MPI to run on LUMI is challenging and as the Intel libraries and -some MPI versions also produce slow or incorrect results on AMD CPUs (and it is not clear if Intel MPI is compatible -with SlingShot 11). ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) Focusing the efforts on those two popular common toolchains makes it possible to support a lot of build recipes that are collected in the [central repository](https://github.com/easybuilders/easybuild-easyconfigs). They can still serve as a basis to develop build recipes for the HPE Cray Programming Environment. -<<<<<<< HEAD Last but not least, EasyBuild provides various [**GitHub integration features**](https://docs.easybuild.io/en/latest/Integration_with_GitHub.html) -======= -Last but not least, EasyBuild provides various [**GitHub integration features**](https://easybuild.readthedocs.io/en/latest/Integration_with_GitHub.html) ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) that greatly facilitate the contribution process: opening, updating, and testing pull requests, reviewing incoming contributions, and much more can all be done directly from the EasyBuild command line. This not only saves time, effort, brain cycles, and mouse clicks for contributors, @@ -223,15 +176,7 @@ Since EasyBuild is primarily intended for installing (scientific) software on HP systems running some flavor of **GNU/Linux**. EasyBuild also works on *macOS*, but significant effort would be required to use it extensively there. -<<<<<<< HEAD -<<<<<<< HEAD - -======= - ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) -======= ->>>>>>> ae27a051 (Correction of links.) EasyBuild is implemented in [**Python**](https://www.python.org), and is compatible with Python 2.7 and 3.5+. Since [Python 2 is no longer maintained](https://blog.python.org/2020/04/python-2718-last-release-of-python-2.html), @@ -247,15 +192,7 @@ On LUMI Lmod is used as the module tool. ## Development -<<<<<<< HEAD -<<<<<<< HEAD - -======= - ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) -======= ->>>>>>> ae27a051 (Correction of links.) EasyBuild is **actively developed** by a worldwide community under the [**GPLv2 open source license**](https://www.gnu.org/licenses/old-licenses/gpl-2.0.html) via the [**``easybuilders``** organisation on GitHub](https://github.com/easybuilders), in the following repositories: @@ -293,8 +230,6 @@ sites leverage it, and integrate it with other tools like [Jenkins](https://www. HPC sites, consortia, and companies that use EasyBuild include (but are not limited to): -<<<<<<< HEAD -<<<<<<< HEAD a href="https://www.vscentrum.be">VSC logo JSC logo HPCNow! logo @@ -306,57 +241,16 @@ a href="https://www.vscentrum.be">VSC <a href=UiO logo BEAR logo SNIC logo -======= -VSC logo -JSC logo -HPCNow! logo -Fred Hutch logo -Vienna BioCenter logo -CÉCI logo -Compute Canada logo -UiO logo -BEAR logo -SNIC logo ->>>>>>> ae27a051 (Correction of links.) National University of Singapore logo IT4Innovations logo Texas A&M logo TU Dresden logo University of Melbourne logo University of Luxembourg logo -<<<<<<< HEAD EPCC logo Cyprus Institute logo University of Basel logo Microsoft logo -======= -VSC logo -JSC logo -HPCNow! logo -Fred Hutch logo -Vienna BioCenter logo -CÉCI logo -Compute Canada logo -UiO logo -BEAR logo -SNIC logo -National University of Singapore logo -IT4Innovations logo -Texas A&M logo -TU Dresden logo -University of Melbourne logo -University of Luxembourg logo -EPCC logo -Cyprus Institute logo -University of Basel logo -Microsoft logo ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) -======= -EPCC logo -Cyprus Institute logo -University of Basel logo -Microsoft logo ->>>>>>> ae27a051 (Correction of links.) --- diff --git a/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md b/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md index fd894324..8eb10e2c 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_03_CPE.md @@ -6,24 +6,14 @@ On LUMI, the main programming environment is the HPE Cray Programming Environment (further abbreviated as Cray PE). The environment provides several tools, including compilers, communication libraries, -<<<<<<< HEAD optimised math libraries and various other libraries, analyzers and debuggers. The Cray PE is made available through *environment modules* that allow to select particular versions of -======= -optmized math libraries and various other libraries, analyzers and debuggers. - -The Cray PE is made available through *environment modules* tha allow to select particular versions of ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) tools and to configure the environment in a flexible way. --- -<<<<<<< HEAD ## HPE Cray PE components -======= -## Cray PE components ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) ### Cray compiler environments @@ -132,12 +122,8 @@ LUMI environment and the integration with the Cray environment still seems to be The Cray PE also provides a programming environment module to enable the Intel® oneAPI compiler and tools. -<<<<<<< HEAD The documentation is available in the [Intel® oneAPI Programming Guide](https://www.intel.com/content/www/us/en/develop/documentation/oneapi-programming-guide/top/oneapi-development-environment-setup.html) -======= -The documentation is available in the [Intel® oneAPI Programming Guide](https://software.intel.com/content/www/us/en/develop/documentation/oneapi-programming-guide/top/oneapi-development-environment-setup.html) ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) Cray provides a bundled package of support libraries to install into the Cray PE to enable the Intel compiler, allowing utilities such as debuggers and performance tools to work with it. @@ -160,11 +146,7 @@ These libraries satisfy dependencies for many commonly used applications on Cray When the module for a CSML package (such as `cray-libsci` or `cray-fftw`) is loaded, all relevant headers and libraries for these packages are added to the compile and link lines of the `cc`, `ftn`, and `CC` compiler wrappers, so linking with them is -<<<<<<< HEAD completely transparent (to the extent that users wonder where the libraries are). -======= -completely transparant (to the extent that users wonder where the libraries are). ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) The CSML collection contains the following Scientific Libraries: @@ -185,11 +167,7 @@ The CSML collection contains the following Scientific Libraries: * Manpages: `intro_mpi` -<<<<<<< HEAD * Website: [http://www.mpi-forum.org](https://www.mpi-forum.org) -======= -* Website: [http://www.mpi-forum.org](http://www.mpi-forum.org) ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) MPI is a widely used parallel programming model that establishes a practical, portable, efficient, and flexible standard for passing messages between ranks in parallel processes. @@ -207,13 +185,7 @@ versions of the library are supported, the finalised LUMI system will only suppo ### DSMML * Modules: `cray-dsmml` -<<<<<<< HEAD -* Manpages: `intro_dsmml` -======= -* * Manpages: `intro_dsmml` -* ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) * Website: [https://pe-cray.github.io/cray-dsmml](https://pe-cray.github.io/cray-dsmml) Distributed Symmetric Memory Management Library (DSMML) is a HPE Cray proprietary memory management library. @@ -353,11 +325,7 @@ can be set through target modules: ``craype-network-ucx`` module. - The ``craype-hugepages*`` modules enable Cray Huge Pages support. To fully enable this support they have to -<<<<<<< HEAD be used at link-time and at run-time. At link time, support is compiled into the binary while at run-time they -======= - be used at link-time and at run-time. At link time, support is compiled into the binary wile at run-time they ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) are used to set the actual size of the huge pages. The ``craype-hugepages*`` modules are not supported by all compilers. E.g., the AOCC compiler does not support diff --git a/docs/2022-CSC_and_LO/1_Intro/1_06_installation.md b/docs/2022-CSC_and_LO/1_Intro/1_06_installation.md index 47cb5448..4d1ac6c5 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_06_installation.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_06_installation.md @@ -426,11 +426,7 @@ We do so only for the framework and easyblocks files as the easconfig files are install EasyBuild. Instead we create our own EasyConfig file for EasyBuild which contains some additional packages that enable extra features in EasyBuild and also provide more information to Lmod. Next the configuration module for EasyBuild (see the next section, -<<<<<<< HEAD ["Configuring EasyBuild"](../1_07_configuration), for more information) and use the temporary -======= -["Configuring EasyBuild"](1_07_configuration), for more information) and use the temporary ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) installation of EasyBuild with our own EasyConfig file to do a proper installation of EasyBuild with module in the final location. diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md index 7eee7d54..d49a7925 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_03_slurm_jobs.md @@ -332,12 +332,4 @@ $ module avail AUGUSTUS --- -<<<<<<< HEAD -<<<<<<< HEAD *[[next: Module naming schemes]](3_04_module_naming_schemes.md)* -======= -*[[next: Module naming schemes]](3_04_module_naming_scheme)* ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) -======= -*[[next: Module naming schemes]](3_04_module_naming_schemes.md)* ->>>>>>> ae27a051 (Correction of links.) diff --git a/docs/2022-CSC_and_LO/4_00_additional_reading.md b/docs/2022-CSC_and_LO/4_00_additional_reading.md index 6e6066fe..14f8454c 100644 --- a/docs/2022-CSC_and_LO/4_00_additional_reading.md +++ b/docs/2022-CSC_and_LO/4_00_additional_reading.md @@ -11,11 +11,7 @@ EasyBuild. - [EasyBuild YouTube channel](https://www.youtube.com/c/EasyBuilders) - This tutorial is an evolution of the -<<<<<<< HEAD [EasyBuild tutorial prepared for the LUMI User Support Team, spring '21](https://easybuilders.github.io/easybuild-tutorial/2021-lust/) -======= - [EasyBuild tutorial prepared for the LUMI User Suppport Team, spring '21](https://easybuilders.github.io/easybuild-tutorial/2021-lust/) ->>>>>>> eafaa943 (Additional reading section.) given by Kenneth Hoste (UGent, EasyBuild lead developer) and Luca Marsella (CSCS) - [Recordings are available on YouTube](https://www.youtube.com/watch?v=JTRw8hqi6x0&list=PLhnGtSmEGEQh573bk3BeOj_KCRBBiA5OT) - The EasyBuild setup on LUMI is partly insprired on the setup used at CSCS on their Cray systems diff --git a/docs/2022-CSC_and_LO/index-lust.md b/docs/2022-CSC_and_LO/index-lust.md new file mode 100644 index 00000000..b8cee2ab --- /dev/null +++ b/docs/2022-CSC_and_LO/index-lust.md @@ -0,0 +1,123 @@ +<<<<<<< HEAD +# EasyBuild tutorial for CSC and the LUMI consortium +======= +# EasyBuild tutorial for LUST + +Overview page of the introductory tutorial on [EasyBuild](https://easybuild.io) for the CSC +and Local Organizations who want to contribute to the LUMI EasyBuild setup. +>>>>>>> 5573462b (Initial commit, already contains the structure to work on the first tutorial.) + +This tutorial is based extensively on the more generic EasyBuild tutorials build by +the EasyBuild community and maintained on +[the EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). +See that site for [credits to the contributors of those tutorials](https://easybuilders.github.io/easybuild-tutorial/#contributors). + +<<<<<<< HEAD +## Scope + +This is an introductory tutorial to [EasyBuild](https://easybuild.io), +a command line tool for installing (scientific) software on High Performance Computing (HPC) systems. +This tutorial is specifically for EasyBuild as implemented on [LUMI](https://lumi-supercomputer.eu) +and has been developed for CSC and the Local Organisations within the LUMI consortium. Yet +much of the material is useful to a broader community of EasyBuild users on Cray systems +or even EasyBuild users in general. +For more generic EasyBuild tutorials, see the [EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). + +This tutorial aims to explain the core concepts of EasyBuild, +get you started with using it, make you familiar with some of the features it provides, +and show how it is used on LUMI to maintain the central software stacks and offer the users +an easy environment to install packages on top of the central stack and thus create their own +customised environment. + +Through hands-on exercises and demos, you will learn how EasyBuild can help you +to get scientific software installed in an efficient way. + + +## Intended audience + +This tutorial is primarily intended for people new to EasyBuild, but even if you're already familiar +with the project it could be interesting to step through it. + +Our main target audience includes: + +- application experts in LUST and the local organizations who want to contribute to the + software stack on LUMI or support their users; +- developers who want to make their developments available to LUMI users; +- advanced users who want to customize available build recipes or develop their own recipes. + + +## Prerequisites + +We expect you to be (a little bit) familiar with: + +- using a Linux command line interface; +- the (absolute) basics of compiling software from source. + +EasyBuild requires: + +- GNU/Linux (any distribution), +- Python 2.7 or 3.5+, though a fairly recent version of Python 3 is highly recommended, +- an environment modules tool (see the ``module`` command). On LUMI we use [Lmod](https://lmod.readthedocs.io), + a modern environment modules tool implemented in Lua. + +However, the LUMI version of the tutorial is currently specifically for the Cray Programming Environment which is not +freely available, so unless you have access to a system with this environment you cannot really do local development. + + +## Contents + +- [Part I: **Introduction to EasyBuild on Cray systems**](1_Intro/index.md) + - [What is EasyBuild?](1_Intro/1_01_what_is_easybuild.md) + - [The Lmod module system](1_Intro/1_02_Lmod.md) + - [The HPE Cray Programming Environment](1_Intro/1_03_CPE.md) + - [LUMI software stacks](1_Intro/1_04_LUMI_software_stack.md) + - [Terminology](1_Intro/1_05_terminology.md) + - [Installation](1_Intro/1_06_installation.md) + - [Configuration](1_Intro/1_07_configuration.md) + - [Basic usage](1_Intro/1_08_basic_usage.md) *(hands-on)* +- [Part II: **Using EasyBuild**](2_Using/index.md) + - [Troubleshooting](2_Using/2_01_troubleshooting.md) *(hands-on)* + - [Creating easyconfig files](2_Using/2_02_creating_easyconfig_files.md) *(hands-on)* + - [Using external modules from the Cray PE](2_Using/2_03_external_modules.md) + - [Implementing easyblocks](2_Using/2_04_implementing_easyblocks.md) *(hands-on)* +- [Part III: **Advanced topics**](3_Advanced/index.md) + - [Using EasyBuild as a library](3_Advanced/3_01_easybuild_library.md) + - [Using hooks to customise EasyBuild](3_Advanced/3_02_hooks.md) + - [Submitting installations as Slurm jobs](3_Advanced/3_03_slurm_jobs.md) + - [Module naming schemes (incl. hierarchical)](3_Advanced/3_04_module_naming_schemes.md) + - [GitHub integration to facilitate contributing to EasyBuild](3_Advanced/3_05_github_integration.md) +- [**Additional reading**](4_00_additional_reading.md) +======= +## Contents + +TODO + + + + + + +## OLD TEXT + +- [Part I: **Introduction to EasyBuild**](part1_intro.md) *(Tue March 9th 2021, 9am-12 CET)* + * [What is EasyBuild?](what_is_easybuild.md) + * [Terminology](terminology.md) + * [Installation](installation.md) *(hands-on)* + * [Configuration](configuration.md) *(hands-on)* + * [Basic usage](basic_usage.md) *(hands-on)* +- [Part II: **Using EasyBuild**](part2_using.md) *(Tue March 23rd 2021, 9am-12 CET)* + * [Troubleshooting](troubleshooting.md) *(hands-on)* + * [Creating easyconfig files](creating_easyconfig_files.md) *(hands-on)* + * [Implementing easyblocks](implementing_easyblocks.md) *(hands-on)* +- [Part III: **Advanced topics**](part3_advanced.md) *(Tue March 30th 2021, 9am-12 CEST)* + * [Using EasyBuild as a library](easybuild_library.md) *(hands-on)* + * [Using hooks to customise EasyBuild](hooks.md) *(hands-on)* + * [Submitting installations as Slurm jobs](slurm_jobs.md) *(hands-on)* + * [Module naming schemes (incl. hierarchical)](module_naming_schemes.md) *(hands-on)* + * [GitHub integration to facilitate contributing to EasyBuild](github_integration.md) *(hands-on)* +- [Part IV: **EasyBuild on Cray systems**](part4_cray.md) *(Friday June 18th 2021, 09-12 CEST)* + * [Introduction to Cray Programming Environment](cray/introduction.md) *(hands-on)* + * [Cray External Modules](cray/external_modules.md) *(hands-on)* + * [Cray Custom Toolchains](cray/custom_toolchains.md) *(hands-on)* + * [EasyBuild at CSCS](cray/easybuild_at_cscs.md) *(hands-on)* +>>>>>>> 5573462b (Initial commit, already contains the structure to work on the first tutorial.) From 8492956e72291b12207068a1995e0352bab74107 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Thu, 21 Apr 2022 12:38:27 +0200 Subject: [PATCH 155/159] Corrected and updated several links. --- docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md index 422d3b85..d6ed3f61 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md @@ -5,11 +5,6 @@

EasyBuild logo

**[EasyBuild](https://easybuild.io)** is a **software build and installation framework** - -

EasyBuild logo

- -**[EasyBuild](http://easybuild.io)** is a **software build and installation framework** ->>>>>>> 549d7d34 (Restructuring for nicer navigation bar on the left.) that manages (scientific) software in an efficient way, without compromising on performance. Easybuild is especially tailored towards [**High Performance Computing (HPC)**](https://en.wikipedia.org/wiki/Supercomputer) systems, and provides a **command-line interface** to facilitate the installation and management of a scientific software stack. From 5bb56bd460beb968654d7da3ff3fba066ad84611 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Tue, 10 May 2022 17:50:47 +0200 Subject: [PATCH 156/159] Multiple minor corrections. --- docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md b/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md index 1e1de7c7..74f3f82a 100644 --- a/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md +++ b/docs/2022-CSC_and_LO/3_Advanced/3_02_hooks.md @@ -40,6 +40,7 @@ community: adapt, e.g., the easyconfig files for Open MPI to work on LUMI. Some site have really large hook files to implement policies and modify standard EasyBuild build recipes, e.g., + - [The hooks file from Jülich Supercomputing Centre](https://github.com/easybuilders/JSC/blob/2022/Custom_Hooks/eb_hooks.py) - ComputeCanada [cc_hooks_common.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_common.py) and [cc_hooks_gento.py](https://github.com/ComputeCanada/easybuild-computecanada-config/blob/main/cc_hooks_gentoo.py) From aabc649be0adc55b942c02509820363e040d90d5 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 4 Nov 2022 17:32:12 +0100 Subject: [PATCH 157/159] Updated the first page of the CSC course to integrate with the regular EasyBuild tutorials. --- docs/2022-CSC_and_LO/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/2022-CSC_and_LO/index.md b/docs/2022-CSC_and_LO/index.md index b45a740b..1de6a052 100644 --- a/docs/2022-CSC_and_LO/index.md +++ b/docs/2022-CSC_and_LO/index.md @@ -2,7 +2,7 @@ This tutorial is based extensively on the more generic EasyBuild tutorials build by the EasyBuild community and maintained on -[the EasyBuild tutorial site](https://easybuilders.github.io/easybuild-tutorial/). +[this site](https://easybuilders.github.io/easybuild-tutorial/). See that site for [credits to the contributors of those tutorials](https://easybuilders.github.io/easybuild-tutorial/#contributors). ## Scope From d897f35e89bbe8bc54bcb14060c05ed568af5a85 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Fri, 4 Nov 2022 17:38:15 +0100 Subject: [PATCH 158/159] Corrected two typos. --- docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md b/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md index 5322678f..aab1dcaf 100644 --- a/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md +++ b/docs/2022-CSC_and_LO/2_Using/2_04_implementing_easyblocks.md @@ -128,7 +128,7 @@ can create themselves. At this moment it does not yet include possible other eas ## Structure of an easyblock -The example below shows the overal structure of an easyblock: +The example below shows the overall structure of an easyblock: ```python from easybuild.framework.easyblock import EasyBlock @@ -685,7 +685,7 @@ Your easyblock should: ``` Running this example on LUMI is a little tricky as using ``--include-easyblocks`` to point EasyBuild to - our new easyblock interfers with settings already made by the EasyBuild configuration modules (``EasyBuild-user``) + our new easyblock interferes with settings already made by the EasyBuild configuration modules (``EasyBuild-user``) and causes error messages about the toolchains. So either the easyblock needs to be copied to the user location that can be found by looking at the output of ``eb --show-config`` or we simply need to extend the list of easyblocks that EasyBuild searches with the easyblocks in the current directory: From f5bf0a8a733d4a83333f0d240646450850f14372 Mon Sep 17 00:00:00 2001 From: Kurt Lust Date: Wed, 3 May 2023 14:37:51 +0200 Subject: [PATCH 159/159] Correction after rebase. --- docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md index d6ed3f61..fa5075f5 100644 --- a/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md +++ b/docs/2022-CSC_and_LO/1_Intro/1_01_what_is_easybuild.md @@ -225,7 +225,7 @@ sites leverage it, and integrate it with other tools like [Jenkins](https://www. HPC sites, consortia, and companies that use EasyBuild include (but are not limited to): -a href="https://www.vscentrum.be">VSC logo +VSC logo JSC logo HPCNow! logo Fred Hutch logo