Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update alps #194

Merged
merged 2 commits into from
Aug 30, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 52 additions & 11 deletions docs/alps/uenvs.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,23 +38,59 @@ It can be used to
- run a single command with that uenv mounted
- access uenv modules

For now, `uenv` isn't available on Balfrin and requires (easy) [manual installation :material-open-in-new:](https://eth-cscs.github.io/uenv/#getting-uenv){:target="_blank"}.
For now, `uenv` isn't available on Balfrin and requires (easy) [manual installation :material-open-in-new:](https://eth-cscs.github.io/uenv/#getting-uenv){:target="_blank"}:
```shell
git clone https://github.com/eth-cscs/uenv.git && ./uenv/install --local
```

### Quickstart

!!! note "TODO: Quick start"
In the following, using `name/version:tag` as metadata to target a uenv can be replaced by the absolute path to the image if necessary.


#### Create a local images repository
In order to be able to use uenv images, you first need to `pull`them into a local repository. First create the repo with
```shell
uenv repo create
```

- [ ] `uenv image repo create`
- [ ] `uenv image find`
- [ ] `uenv image pull`
- [ ] `uenv start`
- [ ] `uenv run`
#### List available images
```shell
uenv image find
```

## `--uenv` SLURM plugin
#### Pull an image in the local repo
```shell
uenv image pull name/version:tag
```

#### Start a uenv for interactive use
This will spawn a new shell with the uenv mounted at `/mount-point` (defaulting to `/user-environment`)
```shell
uenv start name/version:tag [/mount-point]
```
Exit with a usual `exit` or `uenv stop`.

#### Use modules
Software in a uenv can be accessed by using it as a `spack` upstream (see [`spack-c2sm` integration](#spack-c2sm-integration)) but also by loading modules. For that, use the `--view=modules` option:
```shell
uenv start --view=modules name/version:tag [/mount-point]
```
A usual `module avail` command would then show you the different modules added by the uenv on top of the original environment

#### launch a subprocess with a uenv mounted
```shell
uenv run name/version:tag -- my_command
```
will run `my_command` (can be executing a script) with the uenv mounted. This is particularly useful when inside scripts.

## The `--uenv` SLURM plugin

CSCS developped the `--uenv` slurm plugin in order to mount uenvs at runtime. Either use the name of a uenv in your local repo or the full absolute path to the uenv.

```shell
sbatch --uenv uenv_name:/mount/point
sbatch --uenv /image/path:/mount/point
sbatch --uenv name/version:tag[:/mount/point]
sbatch --uenv /image/path[:/mount/point]
```

`:mount/point` can always be omitted and defaults to `:/user-environment`.
Expand All @@ -72,8 +108,13 @@ Uenvs cannot be mounted *anywhere*. They are generated with a predefined install

## `spack-c2sm` integration

User environments are used as a spack upstream with
User environments are supported in non-yet released `spack-c2sm` tags. For testing purposes, you can use `v0.21.1.2`, so
```shell
git clone --depth 1 --recurse-submodules --shallow-submodules -b v0.21.1.2 https://github.com/C2SM/spack-c2sm.git
```
Then start the user environment and use it as a spack upstream with
```shell
uenv start name/version:tag
source spack-c2sm/setup-env.sh /user-environment
```

Expand Down
12 changes: 12 additions & 0 deletions docs/alps/vclusters.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,15 @@ Santis is dedicated to Weather and Climate and is deployed on ~400 Grace-Hopper
!!! note "TODO"

- [ ] Storage


## Tödi

Tödi is the testing vCluster and is currently deployed on the entire machine.

### Uenvs

| uenv | activity |
|--------------------------|-------------------------------|
| icon-wcp/v1:rc4 | build and run icon |
| netcdf-tools/2024:v1-rc1 | pre and post processing tools |
35 changes: 0 additions & 35 deletions docs/models/icon.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,41 +54,6 @@ spack install cosmo-eccodes-definitions
srun -N 1 -c 12 --mem-per-cpu=20G spack install -v -j 12
```

### Temporary Tödi

!!! warning "These are only temporary guidelines during the acceptance phase of Alps"

- Santis will be the official W&C vcluster but Tödi has to be deployed on the entire Alps system for the acceptance phase. These guidelines are for the ones who have access to Tödi.
- The resulting executable is not tested.
- Anything can go wrong

- Get `spack-c2sm`
```shell
git clone --depth 1 --recurse-submodules --shallow-submodules https://github.com/C2SM/spack-c2sm.git
```

- Start user environment
```shell
uenv start /capstor/scratch/cscs/leclairm/uenvs/images/icon_c2sm_v1_rc3.sqfs
```

- Setup `spack-c2sm`
```shell
source spack-c2sm/setup-env.sh /user-environment
```

- Adapt your spack environment following `/capstor/scratch/cscs/leclairm/icon-nwp/config/cscs/spack/uenv/mch_gpu_double/spack.yaml` and activate it as usual with `spack env activate`.

- Build as usual from your icon root dir
```shell
spack install
```

- Submit job with uenv
```shell
sbatch --uenv /capstor/scratch/cscs/leclairm/uenvs/images/icon_c2sm_v1_rc3.sqfs my_run_script
```

## Run test case with ICON
In the *run* folder, you find many prepared test cases, which you can convert into run scripts. To generate the runscript of one of the experiment files, e.g. *mch_ch_lowres*, you can use the `make_runscripts` function.

Expand Down
Loading