From 291c7dbd55fb5622093afdabcc2c10650a2eb69b Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Thu, 31 Aug 2023 21:20:52 +0200 Subject: [PATCH 1/7] document pre_run_shell_cmd_hook and post_run_shell_cmd_hook --- docs/hooks.md | 112 ++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 90 insertions(+), 22 deletions(-) diff --git a/docs/hooks.md b/docs/hooks.md index 2138aad5d..0c0b1168b 100644 --- a/docs/hooks.md +++ b/docs/hooks.md @@ -43,7 +43,7 @@ eb ... ## Available hooks -Since EasyBuild v4.8.1, five different types of hooks are supported: +Since EasyBuild v4.8.1, six different types of hooks are supported: * `start_hook`, `pre_build_and_install_loop_hook`, `post_build_and_install_loop_hook`, and `end_hook` which are triggered *once* right after EasyBuild starts, *once* before looping over the easyconfigs to be built, *once* after completing the loop over the eayconfigs to be installed, @@ -56,6 +56,8 @@ Since EasyBuild v4.8.1, five different types of hooks are supported: also aptly named '`pre`'- and '`post`' hooks. * `cancel_hook` and `fail_hook` which are triggered when a `KeyboardInterrupt` or `EasyBuildError` is raised, respectively. +* `pre_run_shell_cmd_hook` and `post_run_shell_cmd_hook` which are triggered right before and after running + a shell command, respectively. The list of currently available hooks in order of execution, which can also be consulted using `eb --avail-hooks`, is: @@ -79,7 +81,9 @@ which can also be consulted using `eb --avail-hooks`, is: * `pre_postproc_hook`, `post_postproc_hook` * `pre_sanitycheck_hook`, `post_sanitycheck_hook` * `pre_cleanup_hook`, `post_cleanup_hook` -* `pre_module_hook`, `post_module_hook` +* `pre_module_hook` +* `module_write_hook` *(called multiple times per installation, available since EasyBuild v4.4.1)* +* `post_module_hook` * `pre_permissions_hook`, `post_permissions_hook` * `pre_package_hook`, `post_package_hook` * `pre_testcases_hook`, `post_testcases_hook` @@ -87,7 +91,10 @@ which can also be consulted using `eb --avail-hooks`, is: * `end_hook` *(only called once in an EasyBuild session)* * `cancel_hook` *(available since EasyBuild v4.8.1)* * `fail_hook` *(available since EasyBuild v4.8.1)* -* `module_write_hook` *(called multiple times per installation, available since EasyBuild v4.4.1)* + +`pre_run_shell_cmd_hook` and `post_run_shell_cmd_hook` *(available since EasyBuild v4.8.1)* are not included in +the list above because they can not be put in a particular order relative to other hooks, since these hooks +are triggered several times throughout an EasyBuild session. All functions implemented in the provided Python module for which the name ends with `_hook` are considered. @@ -112,23 +119,46 @@ Run 'eb --avail-hooks' to get an overview of known hooks To implement hooks, simply define one or more functions in a Python module (`*.py`), each named after an available hook. +In hooks you have access to the full functionality provided by the EasyBuild framework, +so do `import` from `easybuild.tools.*` (or other `easybuild.*` namespaces) to leverage +those functions. + Do take into account the following: -* for `start_hook` and `end_hook`, no arguments are provided -* for `cancel_hook` and `fail_hook`, the `KeyboardInterrupt` or `EasyBuildError` exception that was raised is provided -* for `parse_hook`, one argument is provided: the `EasyConfig` instance - that corresponds to the easyconfig file being parsed (usually referred to as `ec`) -* for `pre_build_and_install_loop`, a list of easyconfigs is provided -* for `post_build_and_install_loop`, a list of easyconfigs with build results is provided -* for `module_write_hook`, 3 arguments are provided: - * the `EasyBlock` instance used to perform the installation (usually referred to as `self`) - * the filepath of the module that will be written - * the module text as a string - The return value of this hook, when set, will replace the original text that is then written to the module file. -* for the step hooks, one argument is provided: - the `EasyBlock` instance used to perform the installation (usually referred to as `self`) -* the parsed easyconfig file can be accessed in the step hooks via the `EasyBlock` instance, - i.e., via `self.cfg` +* [Hook arguments](#hooks_args) +* [Return value of hooks](#hooks_return_value) +* [Parse hook specifics](#hooks_parse_hook_specifics) + + +### Hook arguments { #hooks_args } + +* For both `start_hook` and `end_hook` no arguments are provided. +* For `cancel_hook` and `fail_hook` the `KeyboardInterrupt` or `EasyBuildError` exception that was raised + is provided as an argument. +* For `parse_hook` the `EasyConfig` instance that corresponds to the easyconfig file being parsed + (usually referred to as `ec`) is passed as an argument. +* For `pre_build_and_install_loop_hook` a list of easyconfigs is provided as an argument. +* For `post_build_and_install_loop_hook` a list of easyconfigs with build results is provided as an argument. +* For `pre_run_shell_cmd_hook`, multiple arguments are passed: + * An unnamed argument (often called `cmd`) that corresponds to the shell command that will be run, + which could be provided either as a string value (like `"echo hello"`) or a list value (like `['echo', 'hello']`). + * A named argument `work_dir` that specifies the path of the working directory in which the command will be run. + * For interactive commands (which are run via the `run_cmd_qa` function), an additional named argument + `interactive` is set to `True`. +* For `post_run_shell_cmd_hook`, multiple arguments are passed: + * An unnamed argument (often called `cmd`) that corresponds to the shell command that was run, + which could be provided either as a string value (like `"echo hello"`) or a list value (like `['echo', 'hello']`). + * A named argument `work_dir` that specifies the working directory in which the shell command was run. + * A named argument `exit_code` that specifies the exit code of the shell command that was run. + * A named argument `output` that specifies the output of the shell command that was run. + * For interactive commands (which are run via the `run_cmd_qa` function), an additional named argument + `interactive` is set to `True`. +* For `module_write_hook`, 3 arguments are provided: + * The `EasyBlock` instance used to perform the installation (usually referred to as `self`). + * The filepath of the module that will be written. + * The module text as a string. +* For the step hooks, the `EasyBlock` instance used to perform the installation (usually referred to as `self`). + The parsed easyconfig file can be accessed in the step hooks via the `EasyBlock` instance, i.e., via `self.cfg`. It is recommended to anticipate possible changes in the provided (named) arguments, using the `*args` and `**kwargs` mechanism commonly used in Python. This @@ -140,11 +170,22 @@ def pre_configure_hook(self, *args, **kwargs): ... ``` -In hooks you have access to the full functionality provided by the EasyBuild framework, -so do `import` from `easybuild.tools.*` (or other `easybuild.*` namespaces) to leverage -those functions. -### Parse hook specifics +### Return value of hooks { #hooks_return_value } + +The return value of a hook is usually ignored by EasyBuild, except in particular cases: + +* If the `module_write_hook` returns a (string) value, it **replaces the original text that was going to be + written to the module file**. This way the `module_write_hook` can extend, change, or entirely replace the + module text that was provided as an argument. + +* If the `pre_run_shell_cmd_hook` returns a value, it **replaces the shell command that was going to be run**. + Hence, this hook can change or entirely replace particular shell commands right before they are executed. + Note that the value type of the return value of `pre_run_shell_cmd_hook` *must* match with the type of the + first (unnamed) argument that provides the shell command that would have been run originally. + + +### Parse hook specifics { #hooks_parse_hook_specifics } `parse_hook` is triggered right *after* reading the easyconfig file, before further parsing of some easyconfig parameters (like `*dependencies`) into @@ -289,3 +330,30 @@ def module_write_hook(self, filepath, module_txt, *args, **kwargs): replace = 'prepend_path("EBPYTHONPREFIXES", root)' return re.sub(search, replace, module_txt) ``` + +### Log running of shell commands + preprend `make install` with `sudo` + +```py +shell_cmds_log = '/tmp/eb_shell_cmds.log' + +def pre_run_shell_cmd_hook(cmd, work_dir=None, interactive=None): + """ + Log shell commands before they are run, + and replace 'make install' with 'sudo make install'. + """ + with open(shell_cmds_log, 'a') as fp: + cmd_type = 'interactive' if interactive else 'non-interactive' + fp.write("%s command '%s' will be run in %s ...\n" % (cmd_type, cmd, work_dir)) + + if cmd == "make install": + return "sudo make install" + + +def post_run_shell_cmd_hook(cmd, work_dir=None, interactive=None, exit_code=None, output=None): + """ + Log shell commands that were run. + """ + with open(shell_cmds_log, 'a') as fp: + cmd_type = 'interactive' if interactive else 'non-interactive' + fp.write("%s command '%s' in %s exited with %s - output: %s\n" % (cmd_type, cmd, work_dir, exit_code, output)) +``` From 396af1eb03799ba9664fa1875cce0bb718cb0e40 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Fri, 1 Sep 2023 09:33:58 +0200 Subject: [PATCH 2/7] fix typo --- docs/hooks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hooks.md b/docs/hooks.md index 0c0b1168b..d4c6c9daa 100644 --- a/docs/hooks.md +++ b/docs/hooks.md @@ -331,7 +331,7 @@ def module_write_hook(self, filepath, module_txt, *args, **kwargs): return re.sub(search, replace, module_txt) ``` -### Log running of shell commands + preprend `make install` with `sudo` +### Log running of shell commands + prepend `make install` with `sudo` ```py shell_cmds_log = '/tmp/eb_shell_cmds.log' From 07c44407d54f5f66f995cf8a366721962f979637 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Fri, 1 Sep 2023 09:39:17 +0200 Subject: [PATCH 3/7] fix links to subsections of 'Implementing hooks' section --- docs/hooks.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/hooks.md b/docs/hooks.md index d4c6c9daa..f049c4f4e 100644 --- a/docs/hooks.md +++ b/docs/hooks.md @@ -125,12 +125,12 @@ those functions. Do take into account the following: -* [Hook arguments](#hooks_args) -* [Return value of hooks](#hooks_return_value) -* [Parse hook specifics](#hooks_parse_hook_specifics) +* [Hook arguments](#hooks-arguments) +* [Return value of hooks](#hooks-return-value) +* [Parse hook specifics](#parse-hook-specifics) -### Hook arguments { #hooks_args } +### Hook arguments { #hooks-arguments } * For both `start_hook` and `end_hook` no arguments are provided. * For `cancel_hook` and `fail_hook` the `KeyboardInterrupt` or `EasyBuildError` exception that was raised @@ -171,7 +171,7 @@ def pre_configure_hook(self, *args, **kwargs): ``` -### Return value of hooks { #hooks_return_value } +### Return value of hooks { #hooks-return-value } The return value of a hook is usually ignored by EasyBuild, except in particular cases: @@ -185,7 +185,7 @@ The return value of a hook is usually ignored by EasyBuild, except in particular first (unnamed) argument that provides the shell command that would have been run originally. -### Parse hook specifics { #hooks_parse_hook_specifics } +### Parse hook specifics { #parse-hook-specifics } `parse_hook` is triggered right *after* reading the easyconfig file, before further parsing of some easyconfig parameters (like `*dependencies`) into From 2d27925ffdd3738932b9a915fc26f80ffcb2a2ee Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Fri, 1 Sep 2023 09:44:59 +0200 Subject: [PATCH 4/7] use '{: #example }' to define custom subsection anchors --- docs/hooks.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/hooks.md b/docs/hooks.md index f049c4f4e..b5091053a 100644 --- a/docs/hooks.md +++ b/docs/hooks.md @@ -130,7 +130,7 @@ Do take into account the following: * [Parse hook specifics](#parse-hook-specifics) -### Hook arguments { #hooks-arguments } +### Hook arguments {: #hooks-arguments } * For both `start_hook` and `end_hook` no arguments are provided. * For `cancel_hook` and `fail_hook` the `KeyboardInterrupt` or `EasyBuildError` exception that was raised @@ -171,7 +171,7 @@ def pre_configure_hook(self, *args, **kwargs): ``` -### Return value of hooks { #hooks-return-value } +### Return value of hooks {: #hooks-return-value } The return value of a hook is usually ignored by EasyBuild, except in particular cases: @@ -185,7 +185,7 @@ The return value of a hook is usually ignored by EasyBuild, except in particular first (unnamed) argument that provides the shell command that would have been run originally. -### Parse hook specifics { #parse-hook-specifics } +### Parse hook specifics {: #parse-hook-specifics } `parse_hook` is triggered right *after* reading the easyconfig file, before further parsing of some easyconfig parameters (like `*dependencies`) into From ce34bff4b792bb401506ef48144785c11e085ee7 Mon Sep 17 00:00:00 2001 From: Kenneth Hoste Date: Fri, 1 Sep 2023 10:08:41 +0200 Subject: [PATCH 5/7] disable MD051 rule in markdownlint, since it incorrectly complains about links to subsections with a custom anchor --- .markdownlint.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.markdownlint.yml b/.markdownlint.yml index a3524d917..f1022e9b6 100644 --- a/.markdownlint.yml +++ b/.markdownlint.yml @@ -11,6 +11,7 @@ "MD026": false # Trailing punctuation in header "MD036": false # Emphasis instead of header "MD046": false # Code block style. Doesn't like MkDocs admonitions or content tabs +"MD051": false # Link fragments should be valid. Reports false negatives when using custom anchor for subsections. "MD052": false # defined ref links. Doesn't like mkdocs-autorefs "MD024": From d6c6bc46c2d91342034439366b4d0e1e10de0ba5 Mon Sep 17 00:00:00 2001 From: Simon Branford <4967+branfosj@users.noreply.github.com> Date: Fri, 1 Sep 2023 09:42:01 +0100 Subject: [PATCH 6/7] fixing internal links --- .markdownlint.yml | 1 - docs/hooks.md | 6 +++--- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/.markdownlint.yml b/.markdownlint.yml index f1022e9b6..a3524d917 100644 --- a/.markdownlint.yml +++ b/.markdownlint.yml @@ -11,7 +11,6 @@ "MD026": false # Trailing punctuation in header "MD036": false # Emphasis instead of header "MD046": false # Code block style. Doesn't like MkDocs admonitions or content tabs -"MD051": false # Link fragments should be valid. Reports false negatives when using custom anchor for subsections. "MD052": false # defined ref links. Doesn't like mkdocs-autorefs "MD024": diff --git a/docs/hooks.md b/docs/hooks.md index b5091053a..7592503c9 100644 --- a/docs/hooks.md +++ b/docs/hooks.md @@ -125,9 +125,9 @@ those functions. Do take into account the following: -* [Hook arguments](#hooks-arguments) -* [Return value of hooks](#hooks-return-value) -* [Parse hook specifics](#parse-hook-specifics) +* [Hook arguments][hooks-arguments] +* [Return value of hooks][hooks-return-value] +* [Parse hook specifics][parse-hook-specifics] ### Hook arguments {: #hooks-arguments } From 98c01cf0cc28f6953da4d5d290ac39bb592345f9 Mon Sep 17 00:00:00 2001 From: Simon Branford <4967+branfosj@users.noreply.github.com> Date: Fri, 1 Sep 2023 09:43:50 +0100 Subject: [PATCH 7/7] fix link --- docs/hooks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hooks.md b/docs/hooks.md index 7592503c9..1c4977561 100644 --- a/docs/hooks.md +++ b/docs/hooks.md @@ -274,7 +274,7 @@ To achieve the intended effect, you can either: ``` A better approach for manipulating easyconfig parameters is to use the `parse_hook` that -was introduced in EasyBuild v3.7.0 (see [Parse hook specifics](#parse-hook-specifics)), +was introduced in EasyBuild v3.7.0 (see [Parse hook specifics][parse-hook-specifics]), where these kind of surprises will not occur (because templating is automatically disabled before `parse_hook` is called and restored immediately afterwards). See also [Example hook to inject a custom patch file](#inject-a-custom-patch-file).