Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dynamically create ReFrame config with correct amount of memory #407

Merged
merged 2 commits into from
Jun 14, 2024
Merged

Dynamically create ReFrame config with correct amount of memory #407

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d28e552
Merged Dynamically create ReFrame config with correct amount of memory
Jun 14, 2024
9e69d52
Merged Dynamically create ReFrame config with correct amount of memory
Jun 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 49 additions & 40 deletions eessi_container.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,46 +70,48 @@ export EESSI_REPOS_CFG_FILE="${EESSI_REPOS_CFG_DIR}/repos.cfg"
display_help() {
echo "usage: $0 [OPTIONS] [[--] SCRIPT or COMMAND]"
echo " OPTIONS:"
echo " -a | --access {ro,rw} - ro (read-only), rw (read & write) [default: ro]"
echo " -c | --container IMG - image file or URL defining the container to use"
echo " [default: docker://ghcr.io/eessi/build-node:debian11]"
echo " -f | --fakeroot - run the container with --fakeroot [default: false]"
echo " Note, currently this option is ignored."
echo " -g | --storage DIR - directory space on host machine (used for"
echo " temporary data) [default: 1. TMPDIR, 2. /tmp]"
echo " -h | --help - display this usage information [default: false]"
echo " -i | --host-injections - directory to link to for host_injections "
echo " [default: /..storage../opt-eessi]"
echo " -l | --list-repos - list available repository identifiers [default: false]"
echo " -m | --mode MODE - with MODE==shell (launch interactive shell) or"
echo " MODE==run (run a script or command) [default: shell]"
echo " -n | --nvidia MODE - configure the container to work with NVIDIA GPUs,"
echo " MODE==install for a CUDA installation, MODE==run to"
echo " attach a GPU, MODE==all for both [default: false]"
echo " -o | --lower-dirs DIRS - list of ':' separated directories that are used"
echo " in front of the default lower dir (CVMFS repo);"
echo " fuse-overlayfs will merge all lower directories;"
echo " the option can be used to make certain directories"
echo " in the CVMFS repo writable [default: none]"
echo " -r | --repository CFG - configuration file or identifier defining the"
echo " repository to use [default: EESSI via"
echo " default container, see --container]"
echo " -u | --resume DIR/TGZ - resume a previous run from a directory or tarball,"
echo " where DIR points to a previously used tmp directory"
echo " (check for output 'Using DIR as tmp ...' of a previous"
echo " run) and TGZ is the path to a tarball which is"
echo " unpacked the tmp dir stored on the local storage space"
echo " (see option --storage above) [default: not set]"
echo " -s | --save DIR/TGZ - save contents of tmp directory to a tarball in"
echo " directory DIR or provided with the fixed full path TGZ"
echo " when a directory is provided, the format of the"
echo " tarball's name will be {REPO_ID}-{TIMESTAMP}.tgz"
echo " [default: not set]"
echo " -v | --verbose - display more information [default: false]"
echo " -x | --http-proxy URL - provides URL for the env variable http_proxy"
echo " [default: not set]; uses env var \$http_proxy if set"
echo " -y | --https-proxy URL - provides URL for the env variable https_proxy"
echo " [default: not set]; uses env var \$https_proxy if set"
echo " -a | --access {ro,rw} - ro (read-only), rw (read & write) [default: ro]"
echo " -b | --extra-bind-paths - specify extra paths to be bound into the container."
echo " To specify multiple bind paths, separate by comma."
echo " Example: '/src:/dest:ro,/src2:/dest2:rw'"
echo " -c | --container IMG - image file or URL defining the container to use"
echo " [default: docker://ghcr.io/eessi/build-node:debian11]"
echo " -f | --fakeroot - run the container with --fakeroot [default: false]"
echo " -g | --storage DIR - directory space on host machine (used for"
echo " temporary data) [default: 1. TMPDIR, 2. /tmp]"
echo " -h | --help - display this usage information [default: false]"
echo " -i | --host-injections - directory to link to for host_injections "
echo " [default: /..storage../opt-eessi]"
echo " -l | --list-repos - list available repository identifiers [default: false]"
echo " -m | --mode MODE - with MODE==shell (launch interactive shell) or"
echo " MODE==run (run a script or command) [default: shell]"
echo " -n | --nvidia MODE - configure the container to work with NVIDIA GPUs,"
echo " MODE==install for a CUDA installation, MODE==run to"
echo " attach a GPU, MODE==all for both [default: false]"
echo " -o | --lower-dirs DIRS - list of ':' separated directories that are used"
echo " in front of the default lower dir (CVMFS repo);"
echo " fuse-overlayfs will merge all lower directories;"
echo " the option can be used to make certain directories"
echo " in the CVMFS repo writable [default: none]"
echo " -r | --repository CFG - configuration file or identifier defining the"
echo " repository to use [default: EESSI via"
echo " default container, see --container]"
echo " -u | --resume DIR/TGZ - resume a previous run from a directory or tarball,"
echo " where DIR points to a previously used tmp directory"
echo " (check for output 'Using DIR as tmp ...' of a previous"
echo " run) and TGZ is the path to a tarball which is"
echo " unpacked the tmp dir stored on the local storage space"
echo " (see option --storage above) [default: not set]"
echo " -s | --save DIR/TGZ - save contents of tmp directory to a tarball in"
echo " directory DIR or provided with the fixed full path TGZ"
echo " when a directory is provided, the format of the"
echo " tarball's name will be {REPO_ID}-{TIMESTAMP}.tgz"
echo " [default: not set]"
echo " -v | --verbose - display more information [default: false]"
echo " -x | --http-proxy URL - provides URL for the env variable http_proxy"
echo " [default: not set]; uses env var \$http_proxy if set"
echo " -y | --https-proxy URL - provides URL for the env variable https_proxy"
echo " [default: not set]; uses env var \$https_proxy if set"
echo
echo " If value for --mode is 'run', the SCRIPT/COMMAND provided is executed. If"
echo " arguments to the script/command start with '-' or '--', use the flag terminator"
Expand Down Expand Up @@ -141,6 +143,10 @@ while [[ $# -gt 0 ]]; do
ACCESS="$2"
shift 2
;;
-b|--extra-bind-paths)
EXTRA_BIND_PATHS="$2"
shift 2
;;
-c|--container)
CONTAINER="$2"
shift 2
Expand Down Expand Up @@ -453,6 +459,9 @@ fi
BIND_PATHS="${EESSI_CVMFS_VAR_LIB}:/var/lib/cvmfs,${EESSI_CVMFS_VAR_RUN}:/var/run/cvmfs,${HOST_INJECTIONS}:/opt/eessi"
# provide a '/tmp' inside the container
BIND_PATHS="${BIND_PATHS},${EESSI_TMPDIR}:${TMP_IN_CONTAINER}"
if [[ ! -z ${EXTRA_BIND_PATHS} ]]; then
BIND_PATHS="${BIND_PATHS},${EXTRA_BIND_PATHS}"
fi

[[ ${VERBOSE} -eq 1 ]] && echo "BIND_PATHS=${BIND_PATHS}"

Expand Down
2 changes: 1 addition & 1 deletion test_suite.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ if [[ "${cpuinfo}" =~ (Core\(s\) per socket:[^0-9]*([0-9]+)) ]]; then
else
fatal_error "Failed to get the number of cores per socket for the current test hardware with lscpu."
fi
cgroup_mem_bytes=$(cat /sys/fs/cgroup/memory/slurm/uid_${UID}/job_${SLURM_JOB_ID}/memory.limit_in_bytes)
cgroup_mem_bytes=$(cat /hostsys/fs/cgroup/memory/slurm/uid_${UID}/job_${SLURM_JOB_ID}/memory.limit_in_bytes)
if [[ $? -eq 0 ]]; then
# Convert to MiB
cgroup_mem_mib=$((cgroup_mem_bytes/(1024*1024)))
Expand Down
Loading