Add script to test S1 performance

[ocaisa]

Example (pretty-printed) output

{
	"2023-06-09": [{
			"http://rug-nl.stratum1.cvmfs.eessi-infra.org/cvmfs/@fqrn@": {
				"time": "23.44",
				"speed": "38163",
				"speed_unit": "SPEED(K/S)",
				"data": "1832350",
				"data_unit": "CACHEUSE(K)",
				"application": "TensorFlow",
				"arch": "x86_64/intel/haswell"
			}
		},
		{
			"http://bgo-no.stratum1.cvmfs.eessi-infra.org/cvmfs/@fqrn@": {
				"time": "25.90",
				"speed": "38163",
				"speed_unit": "SPEED(K/S)",
				"data": "1832350",
				"data_unit": "CACHEUSE(K)",
				"application": "TensorFlow",
				"arch": "x86_64/intel/haswell"
			}
		},
		{
			"http://azure-us-east1.stratum1.cvmfs.eessi-infra.org/cvmfs/@fqrn@": {
				"time": "36.11",
				"speed": "38163",
				"speed_unit": "SPEED(K/S)",
				"data": "1832350",
				"data_unit": "CACHEUSE(K)",
				"application": "TensorFlow",
				"arch": "x86_64/intel/haswell"
			}
		},
		{
			"http://aws-eu-west1.stratum1.cvmfs.eessi-infra.org/cvmfs/@fqrn@": {
				"time": "28.19",
				"speed": "38190",
				"speed_unit": "SPEED(K/S)",
				"data": "1834483",
				"data_unit": "CACHEUSE(K)",
				"application": "TensorFlow",
				"arch": "x86_64/intel/haswell"
			}
		}
	]
}

[ocaisa]

Should have a way to check the user has the required sudo rights

[boegel]

Wasn't the idea to run this in a container, with bind mounts to empty directories for /var/lib/cvmfs & co (so empty cache), which would totally alleviate the need for sudo?

[ocaisa]

It wasn't my idea...

[ocaisa]

It actually doesn't seem like running it in a container can work:

Apptainer> cvmfs_config stat pilot.eessi-hpc.org    
pilot.eessi-hpc.org not mounted
Apptainer> source /cvmfs/pilot.eessi-hpc.org/latest/init/bash 
Found EESSI pilot repo @ /cvmfs/pilot.eessi-hpc.org/versions/2021.12!
archspec says aarch64/graviton2
Using aarch64/graviton2 as software subdirectory.
Using /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton2/modules/all as the directory to be added to MODULEPATH.
Found Lmod configuration file at /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton2/.lmod/lmodrc.lua
Initializing Lmod...
Prepending /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton2/modules/all to $MODULEPATH...
Environment set up to use EESSI pilot software stack, have fun!
[EESSI pilot 2021.12] $ cvmfs_config stat pilot.eessi-hpc.org
pilot.eessi-hpc.org not mounted
[EESSI pilot 2021.12] $ cvmfs_config wipecache
root privileges required
[EESSI pilot 2021.12] $ cvmfs_config setup    
root privileges required

I can get away with not wiping the cache since I control it, and I guess I can just overwrite the EESSI configuration file with a bind mount, but I am only left with timing the command if I can't run cvmfs_config stat

[ocaisa]

Actually, more than that, I need to be able to run cvmfs_config setup to put the new configuration in place. Otherwise I need to bind mount a new configuration and run each S1 in a separate container execution step.

I really don't see the worth, we would lose a lot of valuable info and this implementation will run just fine as a CI job.

[terjekv]

Should we make this more self-contained to the point of running in a container that has CVMFS? Right now, we assume the machine doing the testing either has CVMFS installed or that we run this into a container. I'd love it if this was a job I could run on a server that has very little of anything installed, but rather pulls a container and uses that to do the work.

[terjekv]

For parsing reasons, could we do something like this instead?
json_array+=("{\"$s1server\": [{\"time\":\"$realtime\"},{\"speed\":\"${KB_per_sec}\",{\"speed_units\":\"KB/sec}\"}]}")

(I'm not sold on using JSON in command line environments though, I fear the parsing in general.)

[ocaisa]

@terjekv I tried to get this working inside apptainer but the relevant cvmfs_config commands don't work. Here's the patch in case someone has some inspiration: https://gist.github.com/ocaisa/3d429c6c77365d94aadcbcd960a2a6ec

[terjekv]

@terjekv I tried to get this working inside apptainer but the relevant cvmfs_config commands don't work. Here's the patch in case someone has some inspiration: https://gist.github.com/ocaisa/3d429c6c77365d94aadcbcd960a2a6ec

Meh, yeah, I was afraid of that. Another option is to dedicate a VM to run these tests and leave it at that.

[ocaisa]

I think we can do better than that, we will probably use our own GitHub runner soon, and can just run the tests in CI from this repo then and upload a json artifact.

[terjekv]

I'm included to take this as is, and go with the proposed solution of an action. At least that gives us a starting point and we can always work around that later if need be.

[ocaisa]

We still don't have any testing for measuring S1 performance, so I'm reopening this until it is replaced