Run test suite #466

casparvl · 2024-02-02T17:07:16Z

First attempt at actually trying to run the test suite in the test.sh step from the bot and having the result checked in the check-test.sh step.

My understanding of how the bot executes these steps is a bit limited though, so this will probably take a bit of trial and error before it does what it should do :)

eessi-bot · 2024-02-02T17:07:20Z

Instance eessi-bot-mc-aws is configured to build:

arch x86_64/generic for repo eessi-hpc.org-2023.06-compat
arch x86_64/generic for repo eessi-hpc.org-2023.06-software
arch x86_64/generic for repo eessi.io-2023.06-compat
arch x86_64/generic for repo eessi.io-2023.06-software
arch x86_64/intel/haswell for repo eessi-hpc.org-2023.06-compat
arch x86_64/intel/haswell for repo eessi-hpc.org-2023.06-software
arch x86_64/intel/haswell for repo eessi.io-2023.06-compat
arch x86_64/intel/haswell for repo eessi.io-2023.06-software
arch x86_64/intel/skylake_avx512 for repo eessi-hpc.org-2023.06-compat
arch x86_64/intel/skylake_avx512 for repo eessi-hpc.org-2023.06-software
arch x86_64/intel/skylake_avx512 for repo eessi.io-2023.06-compat
arch x86_64/intel/skylake_avx512 for repo eessi.io-2023.06-software
arch x86_64/amd/zen2 for repo eessi-hpc.org-2023.06-compat
arch x86_64/amd/zen2 for repo eessi-hpc.org-2023.06-software
arch x86_64/amd/zen2 for repo eessi.io-2023.06-compat
arch x86_64/amd/zen2 for repo eessi.io-2023.06-software
arch x86_64/amd/zen3 for repo eessi-hpc.org-2023.06-compat
arch x86_64/amd/zen3 for repo eessi-hpc.org-2023.06-software
arch x86_64/amd/zen3 for repo eessi.io-2023.06-compat
arch x86_64/amd/zen3 for repo eessi.io-2023.06-software
arch aarch64/generic for repo eessi-hpc.org-2023.06-compat
arch aarch64/generic for repo eessi-hpc.org-2023.06-software
arch aarch64/generic for repo eessi.io-2023.06-compat
arch aarch64/generic for repo eessi.io-2023.06-software
arch aarch64/neoverse_n1 for repo eessi-hpc.org-2023.06-compat
arch aarch64/neoverse_n1 for repo eessi-hpc.org-2023.06-software
arch aarch64/neoverse_n1 for repo eessi.io-2023.06-compat
arch aarch64/neoverse_n1 for repo eessi.io-2023.06-software
arch aarch64/neoverse_v1 for repo eessi-hpc.org-2023.06-compat
arch aarch64/neoverse_v1 for repo eessi-hpc.org-2023.06-software
arch aarch64/neoverse_v1 for repo eessi.io-2023.06-compat
arch aarch64/neoverse_v1 for repo eessi.io-2023.06-software

…e easystack files to be able to trigger the bot building pipeline and see if the testing works

casparvl · 2024-02-02T17:11:37Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2

eessi-bot · 2024-02-02T17:11:40Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 from casparvl
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 resulted in:
- submitted job 5214, for details & status see Run test suite #466 (comment)

eessi-bot · 2024-02-02T17:11:43Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.02/pr_466/5214

date	job status	comment
Feb 02 17:11:43 UTC 2024	submitted	job id `5214` awaits release by job manager
Feb 02 17:12:45 UTC 2024	released	job awaits launch by Slurm scheduler
Feb 02 17:17:53 UTC 2024	running	job `5214` is running
Feb 02 17:18:57 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-5214.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1706894306.tar.gz` size: 0 MiB (152417 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Feb 02 17:18:57 UTC 2024	test result	(no tests yet)

casparvl · 2024-02-02T17:26:03Z

Well, it did run. But clearly I have to change things :D

bot/test.sh script found in '/project/60006/SHARED/jobs/2024.02/pr_466/event_201af980-c1ee-11ee-826f-092bb38801c8/run_000/linux_x86_64_amd_zen2/eessi.io-2023.06-software', so running it!
bot/test.sh: line 18: module: command not found
FAILED to load the ReFrame module
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'hpctestlib'
FAILED to load hpctestlib
Cloning into 'EESSI-test-suite'...
bot/test.sh: line 38: reframe: command not found
FAILED to run ReFrame
bot/test.sh: line 42: reframe: command not found
FAILED to list ReFrame tests
bot/test.sh: line 45: reframe: command not found
bot/test.sh finished
bot/check-test.sh script found in '/project/60006/SHARED/jobs/2024.02/pr_466/event_201af980-c1ee-11ee-826f-092bb38801c8/run_000/linux_x86_64_amd_zen2/eessi.io-2023.06-software', so running it!
check test step finished

That's clearly my Friday afternoon brain. Of course, if I want to use the EESSI environment, I should execute this stuff in the EESSI container. Point of attention here: I'll need to start the container in such a way that it picks up the installations it just created in the build step. I.e. with the correct workdir and everything.

In the meantime: it should have reported failure for the test step, but I was grepping for too strict a pattern (\[\s+FAILED\s+\]). In the test script, we just echo FAILED if a step there fails. In this case, the module load ReFrame/4.3.3 failed. The test step should correctly report back that it failed (even if it was not a test from the test suite, but the whole workflow that failed). I've changed the pattern to grep for simply to FAILED.

casparvl · 2024-02-02T17:26:19Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2

eessi-bot · 2024-02-02T17:26:21Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 from casparvl
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 resulted in:
- submitted job 5215, for details & status see Run test suite #466 (comment)

eessi-bot · 2024-02-02T17:26:25Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.02/pr_466/5215

date	job status	comment
Feb 02 17:26:24 UTC 2024	submitted	job id `5215` awaits release by job manager
Feb 02 17:27:10 UTC 2024	released	job awaits launch by Slurm scheduler
Feb 02 17:28:14 UTC 2024	running	job `5215` is running
Feb 02 17:29:17 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-5215.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1706894901.tar.gz` size: 0 MiB (152421 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Feb 02 17:29:17 UTC 2024	test result	🤷 UNKNOWN (click triangle for detailed information) Job test file `_bot_job5215.test` does not exist in job directory or reading it failed.

casparvl · 2024-02-02T17:38:19Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2

eessi-bot · 2024-02-02T17:38:23Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 from casparvl
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 resulted in:
- submitted job 5216, for details & status see Run test suite #466 (comment)

eessi-bot · 2024-02-02T17:38:27Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.02/pr_466/5216

date	job status	comment
Feb 02 17:38:26 UTC 2024	submitted	job id `5216` awaits release by job manager
Feb 02 17:38:30 UTC 2024	released	job awaits launch by Slurm scheduler
Feb 02 17:39:33 UTC 2024	running	job `5216` is running
Feb 02 17:40:37 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-5216.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1706895580.tar.gz` size: 0 MiB (152419 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Feb 02 17:40:37 UTC 2024	test result	🤷 UNKNOWN (click triangle for detailed information) Job test file `_bot_job5216.test` does not exist in job directory or reading it failed.

casparvl · 2024-02-02T17:43:04Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2

eessi-bot · 2024-02-02T17:43:08Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 from casparvl
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 resulted in:
- submitted job 5217, for details & status see Run test suite #466 (comment)

eessi-bot · 2024-02-02T17:43:11Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.02/pr_466/5217

date	job status	comment
Feb 02 17:43:11 UTC 2024	submitted	job id `5217` awaits release by job manager
Feb 02 17:43:42 UTC 2024	released	job awaits launch by Slurm scheduler
Feb 02 17:44:45 UTC 2024	running	job `5217` is running
Feb 02 17:45:48 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-5217.out` ❌ found message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ❌ no message matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1706895881.tar.gz` size: 0 MiB (152417 bytes) entries: 3 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `.lmod/cache/spiderT.lua` `.lmod/cache/spiderT.luac_5.1` `.lmod/cache/timestamp`
Feb 02 17:45:48 UTC 2024	test result	Test suite failed

boegel · 2024-02-08T12:33:55Z

@casparvl How does this related to #467 exactly?

casparvl · 2024-02-09T09:16:00Z

Closing in favour of #467

Caspar van Leeuwen added 2 commits February 2, 2024 17:50

Do actual testing

d5fb876

Added a check for the test

c6a09ab

I don't really need to build anything, but need a change to one of th…

730820e

…e easystack files to be able to trigger the bot building pipeline and see if the testing works

Make sure it notices failures in the test.sh script too

c05bc8e

Maybe it doesn't like my attempt at emojis

41333a0

Make the reported pattern even simpler...

89f82a0

casparvl mentioned this pull request Feb 5, 2024

let bot run EESSI test suite during test phase (part of bot build procedure) #467

Merged

casparvl closed this Feb 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run test suite #466

Run test suite #466

casparvl commented Feb 2, 2024

eessi-bot bot commented Feb 2, 2024

casparvl commented Feb 2, 2024

eessi-bot bot commented Feb 2, 2024 •

edited

Loading

eessi-bot bot commented Feb 2, 2024 •

edited

Loading

casparvl commented Feb 2, 2024

casparvl commented Feb 2, 2024

eessi-bot bot commented Feb 2, 2024 •

edited

Loading

eessi-bot bot commented Feb 2, 2024 •

edited

Loading

casparvl commented Feb 2, 2024

eessi-bot bot commented Feb 2, 2024 •

edited

Loading

eessi-bot bot commented Feb 2, 2024 •

edited

Loading

casparvl commented Feb 2, 2024

eessi-bot bot commented Feb 2, 2024 •

edited

Loading

eessi-bot bot commented Feb 2, 2024 •

edited

Loading

boegel commented Feb 8, 2024

casparvl commented Feb 9, 2024

Run test suite #466

Run test suite #466

Conversation

casparvl commented Feb 2, 2024

eessi-bot bot commented Feb 2, 2024

casparvl commented Feb 2, 2024

eessi-bot bot commented Feb 2, 2024 • edited Loading

eessi-bot bot commented Feb 2, 2024 • edited Loading

casparvl commented Feb 2, 2024

casparvl commented Feb 2, 2024

eessi-bot bot commented Feb 2, 2024 • edited Loading

eessi-bot bot commented Feb 2, 2024 • edited Loading

casparvl commented Feb 2, 2024

eessi-bot bot commented Feb 2, 2024 • edited Loading

eessi-bot bot commented Feb 2, 2024 • edited Loading

casparvl commented Feb 2, 2024

eessi-bot bot commented Feb 2, 2024 • edited Loading

eessi-bot bot commented Feb 2, 2024 • edited Loading

boegel commented Feb 8, 2024

casparvl commented Feb 9, 2024

eessi-bot bot commented Feb 2, 2024 •

edited

Loading

eessi-bot bot commented Feb 2, 2024 •

edited

Loading

eessi-bot bot commented Feb 2, 2024 •

edited

Loading

eessi-bot bot commented Feb 2, 2024 •

edited

Loading

eessi-bot bot commented Feb 2, 2024 •

edited

Loading

eessi-bot bot commented Feb 2, 2024 •

edited

Loading

eessi-bot bot commented Feb 2, 2024 •

edited

Loading

eessi-bot bot commented Feb 2, 2024 •

edited

Loading