From ec04c85168f700dca4f0854d1ff8edc062ea9278 Mon Sep 17 00:00:00 2001 From: pmitev Date: Mon, 6 May 2024 10:01:26 +0200 Subject: [PATCH] Gaussian and Singularity pages --- .../{directly-from-IG => }/gaussian.md | 108 ++++---- docs/software/singularity.md | 241 ++++++++++++++++++ 2 files changed, 304 insertions(+), 45 deletions(-) rename docs/software/{directly-from-IG => }/gaussian.md (56%) create mode 100644 docs/software/singularity.md diff --git a/docs/software/directly-from-IG/gaussian.md b/docs/software/gaussian.md similarity index 56% rename from docs/software/directly-from-IG/gaussian.md rename to docs/software/gaussian.md index 7d484bc55..9d88e397b 100644 --- a/docs/software/directly-from-IG/gaussian.md +++ b/docs/software/gaussian.md @@ -1,20 +1,19 @@ # Gaussian 09 user guide +> A short guide on how to run g09 on UPPMAX. -https://www.uppmax.uu.se/support/user-guides/gaussian-09-user-guide/ +## Access to Gaussian 09 +Gaussian 09 is available at UPPMAX. Uppsala University has an university license for all employees. If you want to be able to run g09 email [support@uppmax.uu.se](mailto:support@uppmax.uu.se) and ask to be added to the g09 group. -A short guide on how to run g09 on UPPMAX. - -Access to Gaussian 09 -Gaussian 09 is available at UPPMAX. Uppsala University has an university license for all employees. If you want to be able to run g09 email support@uppmax.uu.se and ask to be added to the g09 group. - -Running g09 +## Running g09 In order to run g09 you must first set up the correct environment. You do this with: -module load gaussian/g09.d01 -Running single core jobs in SLURM +`module load gaussian/g09.d01` + +### Running single core jobs in SLURM Here is an example of a submit script for SLURM: +```slurm #!/bin/bash -l #SBATCH -J g09test #SBATCH -p core @@ -25,12 +24,15 @@ Here is an example of a submit script for SLURM: module load gaussian/g09.d01 g09 mp2.inp mp2.out +``` + If you run a single core job on Rackham you can't use more than 6.4GB of memory. When specifying the memory requirements, make sure that you ask for some more memory in the submit-script than in g09 to allow for some memory overhead for the program. As a general rule you should ask for 200MB more than you need in the calculation. -The mp2.inp inputfile in the example above: +The `mp2.inp` input file in the example above: +``` %Mem=800MB #P MP2 aug-cc-pVTZ OPT @@ -39,30 +41,35 @@ test 0 1 Li F 1 1.0 -Scratch space -The g09 module sets the environment GAUSS_SCRDIR to /scratch/$SLURM_JOBID in slurm. These directories are removed after the job is finished. +``` -If you want to set GAUSS_SCRDIR, you must do it after module load gaussian/g09.a02 in your script. +## Scratch space +The g09 module sets the environment `GAUSS_SCRDIR` to `/scratch/$SLURM_JOBID` in slurm. These directories are removed after the job is finished. -If you set GAUSS_SCRDIR to something else in your submit script remember to remove all unwanted files after your job has finished. +If you want to set `GAUSS_SCRDIR`, you must do it after module load `gaussian/g09.a02` in your script. -If you think you will use a large amount of scratch space, you might want to set maxdisk in your input file. You can either set maxdisk directly on the command line in your input file: +If you set `GAUSS_SCRDIR` to something else in your submit script remember to remove all unwanted files after your job has finished. +If you think you will use a large amount of scratch space, you might want to set **maxdisk** in your input file. You can either set **maxdisk** directly on the command line in your input file: + +``` #P MP2 aug-cc-pVTZ SCF=Tight maxdisk=170GB +``` or you can put something like: +```bash MAXDISK=$( df | awk '/scratch/ { print $4 }' )KB sed -i '/^#/ s/ maxdisk=[[:digit:]]*KB//' inputfile sed -i '/^#/ s/$/ maxdisk='$MAXDISK'/'; inputfile -in your scriptfile. This will set maxdisk to the currently available size of the /scratch disk on the node you will run on. Read more on maxdisk in the online manual. - -Running g09 in parallel -Gaussian can be run in parallel on a single node using shared memory. +``` -This is the input file for the slurm example below: +in your scriptfile. This will set **maxdisk** to the currently available size of the /scratch disk on the node you will run on. Read more on maxdisk in the [online manual](https://gaussian.com/maxdisk/). -The dimer4.inp input: +## Running g09 in parallel +Gaussian can be run in parallel on a single node using shared memory. This is the input file for the slurm example below: +The `dimer4.inp` input: +``` %Mem=3800MB %NProcShared=4 #P MP2 aug-cc-pVTZ SCF=Tight @@ -82,11 +89,15 @@ methanol dimer MP2 1 2.062618 4.333044 1.344537 8 2.372298 2.640544 0.197416 1 2.702458 3.161614 -0.539550 -Running g09 in parallel in slurm -This can be done by asking for CPUs on the same node using the parallel node environments and telling Gaussian to use several CPUs using the NProcShared link 0 command. + +``` + +### Running g09 in parallel in slurm +This can be done by asking for CPUs on the same **node** using the parallel node environments and telling Gaussian to use several CPUs using the `NProcShared` link 0 command. An example submit-script: +```slurm #!/bin/bash -l #SBATCH -J g09_4 #SBATCH -p node -n 8 @@ -96,27 +107,25 @@ An example submit-script: module load gaussian/g09.d01 export OMP_NUM_THREADS=1 ulimit -s $STACKLIMIT -g09 dimer4.inp dimer4.out -Notice that 8 cores are requested from the queue-system using the line #SLURM -p node -n 8 and that Gaussian is told to use 4 cores with the link 0 command %NProcShared=4 -The example above runs about 1.7 times as fast on eight cores than on four, just change in the input file to %NProcShared=8. +g09 dimer4.inp dimer4.out +``` -Please benchmark your own inputs as the speedup depends heavily on the method and size of system. +Notice that 8 cores are requested from the queue-system using the line `#SLURM -p node -n 8` and that Gaussian is told to use 4 cores with the link 0 command `%NProcShared=4`. The example above runs about 1.7 times as fast on eight cores than on four, just change in the input file to `%NProcShared=8`. Please benchmark your own inputs as the speedup depends heavily on the method and size of system. In some cases Gaussian cannot use all the cpus you ask for. This is indicated in the output with lines looking like this: -In some cases Gaussian cannot use all the cpus you ask for. This is indicated in the output with lines looking like this: +_PrsmSu: requested number of processors reduced to: 1 ShMem 1 Linda._ -PrsmSu: requested number of processors reduced to: 1 ShMem 1 Linda. +The reason for specifying `OMP_NUM_THREADS=1` is to not use the parts of OpenMP in the Gaussian code, but to use Gaussians own threads. -The reason for specifying OMP_NUM_THREADS=1 is to not use the parts of OpenMP in the Gaussian code, but to use Gaussians own threads. - -Running g09 in parallel with linda +## Running g09 in parallel with linda In order to run g09 in parallel over several nodes we have acquired Linda TCP. -Running g09 in parallel with linda in slurm -This can be done by asking for CPUs on the same node using the parallel node environments and telling Gaussian to use several CPUs using the NProcLinda and NProcShared link 0 command. +### Running g09 in parallel with linda in slurm +This can be done by asking for CPUs on the same **node** using the parallel node environments and telling Gaussian to use several CPUs using the `NProcLinda` and `NProcShared` link 0 command. An example submit-script: +```slurm #!/bin/bash -l #SBATCH -J g09-linda # @@ -137,8 +146,11 @@ export GAUSS_LFLAGS='-nodefile tsnet.nodes.$SLURM_JOBID -opt "Tsnet.Node.lindars time g09 dimer20-2.inp dimer20-2.out rm tsnet.nodes.$SLURM_JOBID +``` + Here is the input file: +``` %NProcLinda=2 %NProcShared=20 %Mem=2800MB @@ -159,26 +171,32 @@ methanol dimer MP2 1 2.062618 4.333044 1.344537 8 2.372298 2.640544 0.197416 1 2.702458 3.161614 -0.539550 -Notice that 40 cores are requested from the queue-system using the line #SLURM -p node -n 40 and that g09 is told to use 2 nodes via linda with the %NProcLinda=2 link 0 command and 20 cores on each node with the link 0 command %NProcShared=20. + +``` + +Notice that 40 cores are requested from the queue-system using the line `#SLURM -p node -n 40` and that g09 is told to use 2 nodes via linda with the `%NProcLinda=2` link 0 command and 20 cores on each node with the link 0 command `%NProcShared=20`. Please benchmark your own inputs as the speedup depends heavily on the method and size of system. In some cases Gaussian cannot use all the cpus you ask for. This is indicated in the output with lines looking like this: -PrsmSu: requested number of processors reduced to: 1 ShMem 1 Linda. -Number of CPUs on the shared memory nodes +_ PrsmSu: requested number of processors reduced to: 1 ShMem 1 Linda._ + +## Number of CPUs on the shared memory nodes Use the information below as a guide to how many CPUs to request for your calculation: -On Rackham: +### On Rackham: + +- 272 nodes with two 10-core CPUs and 128GB memory +- 32 nodes with two 10-core CPUs and 256GB memory + +### On Milou: -272 nodes with two 10-core CPUs and 128GB memory -32 nodes with two 10-core CPUs and 256GB memory -On Milou: +- 174 nodes with two 8-core CPUs and 128GB memory +- 17 nodes with two 8-core CPUs and 256GB memory +- 17 nodes with two 8-core CPUs and 512GB memory -174 nodes with two 8-core CPUs and 128GB memory -17 nodes with two 8-core CPUs and 256GB memory -17 nodes with two 8-core CPUs and 512GB memory -Note on chk-files: +### Note on chk-files: You may experience difficulties if you mix different versions (g09 and g03) or revisions of gaussian. If you use a checkpoint file (.chk file) from an older revision (say g03 e.01), in a new calculation with revision a.02, g09 may not run properly. We recommend using the same revision if you want to restart a calculation or reuse an older checkpoint file. diff --git a/docs/software/singularity.md b/docs/software/singularity.md new file mode 100644 index 000000000..582be3072 --- /dev/null +++ b/docs/software/singularity.md @@ -0,0 +1,241 @@ +# Singularity User Guide + +Singularity [www.sylabs.io/docs](https://www.sylabs.io/docs) provide tools for running containters that are more suitable to traditional HPC environments when some other tools such as Docker or lxc. These containers can be portable and could be run both on your desktop machine and our clusters. + +One of the ways in which Singularity is more suitable for HPC is that it very actively restricts permissions so that you do not gain access to addtitional resources while inside the container. One consequence of this is that some common tools like ping or sudo do not work when run within a container (as a regular user). + +Singularity is installed and usable to run custom container images on the clusters bianca and rackham. + +## Running an existing image +It's possible to download and run pre-built images from the Singularity hub [https://singularity-hub.org](https://singularity-hub.org) and the singularity library ([https://cloud.sylabs.io](https://cloud.sylabs.io) library) using the singularity pull sub command such as: + +```bash +singularity pull library://ubuntu +``` +Which will download the requested image and place it in the current directory. You can also upload and run image directly your self. + +Once you have an image, you can "run" it with a command such as + +```bash +singularity run singularityhub-ubuntu-14.04.img +``` +which will try to execute a "run" target in the container. There are also the shell and exec subcommands for starting a shell and running a specific command respectively. + +## Access to UPPMAX file systems. +By default, singularity will try to help and map the UPPMAX file systems from the current cluster so that they can be accessed from within the container. For CentOS7 based clusters (snowy, rackham, bianca), this works as expected. + +Singularity is installed on the system (on each separate node) and does not require any module load to be available. + +It's possible to run Docker containers. You can try to run + +```bash +singularity shell docker://debian:stretch +``` + +but note that Docker containers are typically designed to run with more privileges than are allowed with Singularity, so it's quite possible things do not work as expected. + +## Not all images may work everywhere! +Images run with the same linux kernel as the rest of the system. For HPC we systems, the kernel used tend to be quite old for stability reasons (with milou using very old kernels). This is not normally a problem, but can cause issues if the libraries of the images you try to run expects functionality added in newer kernels. How and what works is difficult to know without trying, but we have successfully started a shell in an image for the currently most recent Ubuntu release (17.04). + +## Creating your own images +If you have singularity on your own machine, you can create your own images and upload and run them from UPPMAX. Creating images directly on UPPMAX is not possible as it requires administrator (root) privileges (but see below). + +Singularity provides funcitonality to create and bootstrap images, and the installation contains example definitions for bootstrapping various images you can use as a start (like Ubuntu, Scientific Linux and so on). If you normally run Linux and have administrative access, you can install Singularity and build images locally, typical usage would be: + +```bash +sudo singularity build myimage.img examples/ubuntu.def +``` + +See [https://www.sylabs.io/guides/2.6/user-guide/container_recipes.html](https://www.sylabs.io/guides/2.6/user-guide/container_recipes.html) for the recipe format. + +## Using the sylabs cloud remote builder, an example +Even if you don't run Linux on your desktop system, you can still build your own images. Typically software pages contain enough information as installation instructions to create a modified recipe to build images for software. + +[Sylabs](https://www.sylabs.io/) (the company currently developing singularity) offer an experimental cloud service for building images from recipies. Here's an example on how to use that to build an image for a software (for this example, we picked [sortmerna](https://github.com/biocore/sortmerna) as it was the first user software request that came up). + +- First, we need to sign in to the sylabs remote builder, available at https://cloud.sylabs.io. It allows sign in through various providers (currently Google, Microsoft, GitHub, GitLab). If you don't have an account with one of these, at least one of them is likely to offer free registration. + +- Once signed in, we want to create a project for this software, so we click on "Singularity Library" in the top menu and select "Create a new project", set ourselves as owner and enter sortmerna as project name. We also select to make images for this project public. + +- Once we click onwards, we can create our first image for the project. This can be done either by pushing an image from a client (commands are given) or by building in the cloud service. + +- Since we want to use the cloud service, we click remote builder and get a box to fill in an image recipe. We also get a box to enter a tag or version and the possibility to enter a description. + +- As we'll create an image for the currently latest release version (3.0.3), we enter 3.0.3 in the tag field. + +- Finally, we need to provide a recipe for the image, since this is not our first rodeo, we know roughly how it should look. + +```singularity +BootStrap: library +From: ubuntu:18.04 + +%runscript + sortmerna "$@" + +%post + echo "Hello from inside the container" + apt-get update + apt-get -y dist-upgrade + apt-get clean + apt-get -y install curl + curl -L https://github.com/biocore/sortmerna/releases/download/v3.0.3/sortmerna-3.0.3-Linux_C6.sh > /tmp/sortmerna.sh + bash /tmp/sortmerna.sh --skip-license +``` + +- We paste that recipe in the box (or upload a file with it) and click build. The service will try to find an available builder and It will work for a while. Output from the build process will be displayed in a console window on the same page. + +- Hopefully the build completes successfully (a notification is shown and build status is updated). We can when go to the the projects through the dropdown with our username in the upper right corner, select "My projects", select the sortmerna project. + +- Once on the project view, we select the tab "Images" to see what images we have built. We see that we have an image tagged 3.0.3 and are given the option to download the image file through the web browser as well as commands to let singularity pull it. + +- As it's a hassle to transfer the image from our desktop, we just copy the command to pull the image and run it. + +```bash +$ singularity pull library://pontus/default/sortmerna:3.0.3 +WARNING: Authentication token file not found : Only pulls of public images will succeed +INFO: Downloading library image + 65.02 MiB / 65.02 MiB [=========================================================================================================================================] 100.00% 30.61 MiB/s 2s + +``` + +which downloads the image and creates sortmerna_3.0.3.sif for us. We can then run that just as sortmerna. + +```bash +$ ./sortmerna_3.0.3.sif + + Program: SortMeRNA version 3.0.3 + Copyright: 2016-2019 Clarity Genomics BVBA: + Turnhoutseweg 30, 2340 Beerse, Belgium + 2014-2016 Knight Lab: + Department of Pediatrics, UCSD, La Jolla + 2012-2014 Bonsai Bioinformatics Research Group: + LIFL, University Lille 1, CNRS UMR 8022, INRIA Nord-Europe + Disclaimer: SortMeRNA comes with ABSOLUTELY NO WARRANTY; without even the + implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. + See the GNU Lesser General Public License for more details. + Contributors: Jenya Kopylova jenya.kopylov@gmail.com + Laurent Noé laurent.noe@lifl.fr + Pierre Pericard pierre.pericard@lifl.fr + Daniel McDonald wasade@gmail.com + Mikaël Salson mikael.salson@lifl.fr + Hélène Touzet helene.touzet@lifl.fr + Rob Knight robknight@ucsd.edu + + For help or more information on usage, type `./sortmerna -h' +``` + +We can also transfer the file to another system (e.g. move it to bianca where it can be pulled directly). + +Just like with the docker example, we can also run the image without pulling it explicitly. + +```bash +$ singularity run library://pontus/default/sortmerna:3.0.3 +WARNING: Authentication token file not found : Only pulls of public images will succeed + + Program: SortMeRNA version 3.0.3 + Copyright: 2016-2019 Clarity Genomics BVBA: + Turnhoutseweg 30, 2340 Beerse, Belgium + 2014-2016 Knight Lab: + Department of Pediatrics, UCSD, La Jolla + 2012-2014 Bonsai Bioinformatics Research Group: + LIFL, University Lille 1, CNRS UMR 8022, INRIA Nord-Europe + Disclaimer: SortMeRNA comes with ABSOLUTELY NO WARRANTY; without even the + implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. + See the GNU Lesser General Public License for more details. + Contributors: Jenya Kopylova jenya.kopylov@gmail.com + Laurent Noé laurent.noe@lifl.fr + Pierre Pericard pierre.pericard@lifl.fr + Daniel McDonald wasade@gmail.com + Mikaël Salson mikael.salson@lifl.fr + Hélène Touzet helene.touzet@lifl.fr + Rob Knight robknight@ucsd.edu + + For help or more information on usage, type `./sortmerna -h' +``` + +If we wanted to install [multiqc](https://github.com/ewels/MultiQC) instead, we only change the recipe to + +```singularity +BootStrap: library +From: ubuntu:18.04 + +%runscript + multiqc "$@" + +%post + echo "Hello from inside the container" + apt-get update + apt-get -y dist-upgrade + apt-get clean + apt-get -y install python-pip + pip install multiqc +``` + +## Using the sylabs cloud remote builder from the UPPMAX shell +The remote builder service provided by sylabs also supports remote builds through an API. This means you can call on it from the shell at UPPMAX. + +Using this service also requires you to register/log in to the Sylabs cloud service. To use this, simply run +```bash +singularity remote login SylabsCloud +``` +and you should see +``` +Generate an API Key at https://cloud.sylabs.io/auth/tokens, and paste here: +API Key: +``` + +if you visit that link and give a name, a text-token will be created for you. Copy and paste this to the prompt at UPPMAX. You should see +``` +INFO: API Key Verified! +``` +once you've done this, you can go on and build images almost as normal, using commands like +```bash +singularity build --remote testcontainer.sif testdefinition.def +``` +which will build the container from testdefinition.def remotely and transfer it to your directory, storing it as testcontainer.sif + +Running your container in jobs +A sample job script for running a tool provided in a container may look like + +```slurm +#!/bin/bash -l +#SBATCH -N 1 +#SBATCH -n 1 +#SBATCH -t 0:30:00 +#SBATCH -A your-project +#SBATCH -p core +cd /proj/something/containers + +singularity exec ./ubuntu.img echo "Hey, I'm running ubuntu" +singularity exec ./ubuntu.img lsb_release -a +singularity run ./anotherimage some parameters here +./yetanotherimage parameters +``` + +## Another example; building a container from conda +To build a container from a conda environment, here we demonstrate for `qiime2` + +```singularity +BootStrap: library +From: centos:7 + +%runscript + . /miniconda/etc/profile.d/conda.sh + PATH=$PATH:/miniconda/bin + conda activate qiime2-2019.7 + qiime "$@" + +%post + yum clean all + yum -y update + yum -y install wget python-devel + cd /tmp + wget https://repo.anaconda.com/miniconda/Miniconda2-latest-Linux-x86_64.sh + bash ./Miniconda2-latest-Linux-x86_64.sh -b -p /miniconda + /miniconda/bin/conda update -y conda + wget https://data.qiime2.org/distro/core/qiime2-2019.7-py36-linux-conda.yml + /miniconda/bin/conda env create -n qiime2-2019.7 --file qiime2-2019.7-py36-linux-conda.yml + # OPTIONAL CLEANUP + rm qiime2-2019.7-py36-linux-conda.yml + /miniconda/bin/conda clean -a +``` \ No newline at end of file