deploy: 550d9db

UPPMAX · Jan 14, 2024 · 23262cf · 23262cf
commit 23262cf
Show file tree

Hide file tree

Showing 179 changed files with 33,817 additions and 0 deletions.
diff --git a/.buildinfo b/.buildinfo
@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 9b73e76feb3affc408e7e6f8f7f34104
+tags: 645f666f9bcd5a90fca523b33c5a78b7
diff --git a/.doctrees/bianca.doctree b/.doctrees/bianca.doctree
diff --git a/.doctrees/clusters.doctree b/.doctrees/clusters.doctree
diff --git a/.doctrees/courses.doctree b/.doctrees/courses.doctree
diff --git a/.doctrees/environment.pickle b/.doctrees/environment.pickle
diff --git a/.doctrees/exercises.doctree b/.doctrees/exercises.doctree
diff --git a/.doctrees/hpc.doctree b/.doctrees/hpc.doctree
diff --git a/.doctrees/index.doctree b/.doctrees/index.doctree
diff --git a/.doctrees/intro.doctree b/.doctrees/intro.doctree
diff --git a/.doctrees/intro_online.doctree b/.doctrees/intro_online.doctree
diff --git a/.doctrees/linux.doctree b/.doctrees/linux.doctree
diff --git a/.doctrees/linux_basics.doctree b/.doctrees/linux_basics.doctree
diff --git a/.doctrees/linux_cases.doctree b/.doctrees/linux_cases.doctree
diff --git a/.doctrees/login.doctree b/.doctrees/login.doctree
diff --git a/.doctrees/login2.doctree b/.doctrees/login2.doctree
diff --git a/.doctrees/naiss.doctree b/.doctrees/naiss.doctree
diff --git a/.doctrees/overview.doctree b/.doctrees/overview.doctree
diff --git a/.doctrees/overview2.doctree b/.doctrees/overview2.doctree
diff --git a/.doctrees/rackham.doctree b/.doctrees/rackham.doctree
diff --git a/.doctrees/slurm_intro.doctree b/.doctrees/slurm_intro.doctree
diff --git a/.doctrees/snic.doctree b/.doctrees/snic.doctree
diff --git a/.doctrees/software.doctree b/.doctrees/software.doctree
diff --git a/.doctrees/specific.doctree b/.doctrees/specific.doctree
diff --git a/.doctrees/start.doctree b/.doctrees/start.doctree
diff --git a/.doctrees/summary.doctree b/.doctrees/summary.doctree
diff --git a/.doctrees/userguide.doctree b/.doctrees/userguide.doctree
diff --git a/.doctrees/working.doctree b/.doctrees/working.doctree
diff --git a/.nojekyll b/.nojekyll
diff --git a/_images/Bild1.png b/_images/Bild1.png
diff --git a/_images/IMG_5111.jpeg b/_images/IMG_5111.jpeg
diff --git a/_images/Mac_terminal.png b/_images/Mac_terminal.png
diff --git a/_images/Thinlinc2.jpg b/_images/Thinlinc2.jpg
diff --git a/_images/biancaorganisation-01.png b/_images/biancaorganisation-01.png
diff --git a/_images/cat.png b/_images/cat.png
diff --git a/_images/caution.png b/_images/caution.png
diff --git a/_images/cross.png b/_images/cross.png
diff --git a/_images/edit.png b/_images/edit.png
diff --git a/_images/flavours.png b/_images/flavours.png
diff --git a/_images/folders.png b/_images/folders.png
diff --git a/_images/fz3_osx_main.png b/_images/fz3_osx_main.png
diff --git a/_images/gnu.png b/_images/gnu.png
diff --git a/_images/head.png b/_images/head.png
diff --git a/_images/images.jfif b/_images/images.jfif
diff --git a/_images/mermaid-046a34d4a7c3df57cf3899fe2f612f27a718b9e9.png b/_images/mermaid-046a34d4a7c3df57cf3899fe2f612f27a718b9e9.png
diff --git a/_images/mermaid-34a9f1fb84f529a9f83dc50768b11e05430057fa.png b/_images/mermaid-34a9f1fb84f529a9f83dc50768b11e05430057fa.png
diff --git a/_images/mermaid-b59416dfac2620d7679ac3fa5aece5f05e93534a.png b/_images/mermaid-b59416dfac2620d7679ac3fa5aece5f05e93534a.png
diff --git a/_images/mermaid-cd1265279c453823b4cb3aeee063da2eec670533.png b/_images/mermaid-cd1265279c453823b4cb3aeee063da2eec670533.png
diff --git a/_images/mobax.jpg b/_images/mobax.jpg
diff --git a/_images/mobax_start.jpg b/_images/mobax_start.jpg
diff --git a/_images/mobax_start1.jpg b/_images/mobax_start1.jpg
diff --git a/_images/mv_inbox.png b/_images/mv_inbox.png
diff --git a/_images/node.png b/_images/node.png
diff --git a/_images/nodes.png b/_images/nodes.png
diff --git a/_images/permission.png b/_images/permission.png
diff --git a/_images/pingvin.png b/_images/pingvin.png
diff --git a/_images/plantuml-7e4c7f5a44bfa5548ada11af24e62e3e3d70a989.png b/_images/plantuml-7e4c7f5a44bfa5548ada11af24e62e3e3d70a989.png
diff --git a/_images/program_flags.png b/_images/program_flags.png
diff --git a/_images/putty.jpg b/_images/putty.jpg
diff --git a/_images/queue1.png b/_images/queue1.png
diff --git a/_images/queue2.png b/_images/queue2.png
diff --git a/_images/queue3.png b/_images/queue3.png
diff --git a/_images/screen.png b/_images/screen.png
diff --git a/_images/shell.jpg b/_images/shell.jpg
diff --git a/_images/tab.png b/_images/tab.png
diff --git a/_images/tail.png b/_images/tail.png
diff --git a/_images/terminal.png b/_images/terminal.png
diff --git a/_images/unix_architecture.jpg b/_images/unix_architecture.jpg
diff --git a/_images/uppmax-light2.jpg b/_images/uppmax-light2.jpg
diff --git a/_images/wildcards_bear.png b/_images/wildcards_bear.png
diff --git a/_images/xeyes.png b/_images/xeyes.png
diff --git a/_plantuml/7e/7e4c7f5a44bfa5548ada11af24e62e3e3d70a989.png b/_plantuml/7e/7e4c7f5a44bfa5548ada11af24e62e3e3d70a989.png
diff --git a/_sources/bianca.md.txt b/_sources/bianca.md.txt
@@ -0,0 +1,131 @@
+# NAISS SENS and Bianca
+```{objectives}
+- We'll briefly get an overview of kinds of sensitive data
+- ... and the **Bianca** system
+```
+
+```{Admonition} The Bianca workshop
+- [Bianca workshop material](https://uppmax.github.io/bianca_workshop/)
+- Next time given later this spring
+```
+
+```{instructor-note}
+- Approx timing: 15.05-15.30
+- Theory
+```
+
+```{seealso}
+- Bianca portal: http://docs.uppmax.uu.se/cluster_guides/bianca_portal/
+```
+
+## Sensitive personal data
+
+- <https://www.snic.se/allocations/snic-sens/>
+- Traced to now living persons, e.g.
+  - human genomic data
+  - images/videos containing persons
+  - health registry (health data records from healthcare providers)
+- More about sensitive data
+  - [GDPR](https://gdpr.eu/)
+  - [Data protection](https://ec.europa.eu/info/law/law-topic/data-protection_en)
+  - [Skydd av personuppgifter](https://ec.europa.eu/info/law/law-topic/data-protection_sv)
+
+- When in doubt, contact your university's [data protection officer](https://www.uppmax.uu.se/support/faq/general-miscellaneous-faq/sensitive+data+questions/).
+- Generally, there must be a [Data Processing Agreement](https://www.uppmax.uu.se/support/faq/general-miscellaneous-faq/how-to-establish-a-puba-with-uu/) between UU and the data controlling university.
+
+## Apply for project
+[Open NAISS SENS Rounds](https://supr.naiss.se/round/open_type/?type=NAISS+SENS)
+
+## Bianca
+- Bianca is a great platform for computationally intensive research on sensitive personal data. It can also be useful for:
+  - national and international collaboration on sensitive personal data (without a high compute need)
+  - other types of sensitive data
+- Bianca is not good for:
+  - storing data
+  - publishing data
+     - unless the dataset is very popular among Bianca users, e.g. [Swegen](https://snd.gu.se/en/catalogue/study/ext0285), [SIMPLER](https://www.simpler4health.se/)
+
+
+## Bianca's design
+
+- Bianca was designed to:
+  - make accidental data leaks difficult
+  - make correct data management as easy as possible
+  - emulate the HPC cluster environment that SNIC/NAISS users were familiar with
+  - provide a maximum amount of resources
+  - satisfy regulations.
+
+### Bianca has no Internet
+... but we have “solutions”
+
+![Image](./img/biancaorganisation-01.png)
+
+- Bianca is only accessible from within Sunet (i.e. from university networks).
+- Use VPN outside Sunet. [Link to VPN for UU](https://mp.uu.se/en/web/info/stod/it-telefoni/it-support/network-on-campus/vpn-service)
+  - You can get VPN credentials from all Swedish universities.
+
+<br>
+
+- The whole Bianca cluster (blue) contains hundreds of virtual project clusters (green), each of which is isolated from each other and the Internet.
+- Data can be transferred to or from a virtual project cluster through the Wharf, which is a special file area that is visible from the Internet.
+
+### The log in steps
+1. When you log in to [https://bianca.uppmax.uu.se](https://bianca.uppmax.uu.se), your SSH or ThinLinc client first meets the blue Bianca login node.
+    - `<username>-<projid>@bianca.uppmax.uu.se`
+    - like: `[email protected]`
+2. After checking your [2-factor authentication](https://www.uppmax.uu.se/support/user-guides/setting-up-two-factor-authentication/), this server looks for your virtual project cluster.
+3. If it's present, then you are transferred to a login prompt on your cluster's login node. If not, then the virtual cluster is started.
+4. Inside each virtual project cluster, by default there is just a one-core login node. When you need more memory or more CPU power, you submit a job (interactive or batch), and an idle node will be moved into your project cluster.
+
+
+### Data transfers:
+- <https://www.uppmax.uu.se/support/user-guides/bianca-user-guide/> 
+  - section 3: Transfer files to and from Bianca
+- wharf
+- NGI Deliver through SUPR
+- Transit server (SSH to transit.uppmax.uu.se)
+
+### Software
+
+- Modules library (almost same as Rackham)
+- Local Conda repository
+- Local Perl modules
+- Local R packages
+
+- More info at [Bianca user guide](https://www.uppmax.uu.se/support/user-guides/bianca-user-guide/)
+
+
+## ThinLinc
+
+- Bianca offers graphical login
+  - You need to be on SUNET or use VPN. 
+  - On web:
+    - [https://bianca.uppmax.uu.se](https://bianca.uppmax.uu.se)
+    - requires [2-factor authentication](https://www.uppmax.uu.se/support/user-guides/setting-up-two-factor-authentication/)
+
+
+![Image](./img/Thinlinc2.jpg)
+
+
+## Introduction course
+- [Overview](https://uppmax.github.io/bianca_workshop/overview/)
+- [Log in](https://uppmax.github.io/bianca_workshop/login_bianca/)
+- [Command-line intro](https://uppmax.github.io/bianca_workshop/commandline/)
+- [Module system](https://uppmax.github.io/bianca_workshop/modules1/)
+- [Transfering files](https://uppmax.github.io/bianca_workshop/transfer)
+- [More on NAISS-SENS](https://uppmax.github.io/bianca_workshop/naiss-sens-bianca/)
+- [The compute nodes](https://uppmax.github.io/bianca_workshop/slurm-intro/)
+- [Software and package installation](https://uppmax.github.io/bianca_workshop/install/)
+
+
+
+
+ ```{keypoints}
+ - If you handle sensitive data, apply for a NAISS-SENS project
+ - SENS projects will get accounts on Bianca
+ - Bianca has no internet itself but there are solutions like:
+   - wharf
+   - transit server
+   - many installed software
+ - Ask support if you need additional software tools
+ ```
diff --git a/_sources/clusters.md.txt b/_sources/clusters.md.txt
@@ -0,0 +1,34 @@
+
+
+# Clusters
+
+
+We have a number of compute clusters:
+
+    Rackham, reserved for SNIC projects
+    Snowy, reserved for SNIC projects
+    Miarka, reserved for Scilifelab production
+    Bianca, a part of SNIC-SENS
+    UPPMAX Cloud, a part of SNIC Science Cloud
+
+The storage systems we have provide a total volume of about 20 PB, the equivalent of nearly 15 billion 3.5-inch floppy disks or 40,000 years of 128-bit encoded music. Read more on the storage systems page.
+
+
+## Rackham
+[Rackham](https://www.uppmax.uu.se/resources/systems/the-rackham-cluster/)
+
+## Snowy
+[Snowy](https://www.uppmax.uu.se/resources/systems/the-snowy-cluster/)
+
+## Bianca
+[Bianca](https://www.uppmax.uu.se/resources/systems/the-bianca-cluster/)
+
+## Miarka
+[Miarka](https://www.uppmax.uu.se/resources/systems/miarka-cluster/)
+
+## The UPPMAX Cloud
+[UPPMAX cloud](https://www.uppmax.uu.se/resources/systems/the-uppmax-cloud/)
+
+## System usage
+[System usage](https://www.uppmax.uu.se/resources/system-usage/)
+
diff --git a/_sources/courses.md.txt b/_sources/courses.md.txt
@@ -0,0 +1,70 @@
+# Courses and workshops
+
+
+```{instructor-note}
+- Approx timing: 14.40-14:50
+- Theory
+```
+
+##  Introductory Linux courses
+### Intro to UPPMAX (This one)
+- [Intro to UPPMAX](https://www.uppmax.uu.se/support/courses-and-workshops/introductory-course)
+
+### Awk workshop
+- [Intro to Awk](https://www.uppmax.uu.se/support/courses-and-workshops/awk-workshop/)
+- Jan 22-23 
+
+##  Other UPPMAX courses
+### Intro to Bianca
+- [Introduction to Bianca](https://www.uppmax.uu.se/support/courses-and-workshops/bianca-workshop-2023/)
+- [Course material](https://uppmax.github.io/bianca_workshop/)
+- Once in spring and once in fall
+
+###  Basic Singularity
+- [Basic Singularity](https://www.uppmax.uu.se/support/courses-and-workshops/singularity-workshop-announcement/)
+- Feb 13
+
+### Introduction to running R, Python, and Julia in HPC
+- [Introduction to running R, Python, and Julia in HPC](https://www.uppmax.uu.se/support/courses-and-workshops/r-python-julia)
+- **Focus on packages and using Slurm for the jobs**
+- March 12-14
+
+### Git course
+- [Last year's material](https://www.hpc2n.umu.se/node/982)
+- Nov 13-17
+
+### Python at UPPMAX and HPC2N
+- [Using Python in an HPC environment](https://uppmax.github.io/HPC-python/index.html)
+- Also containing parallelism, Machine learning and GPU
+- May/June (Date not set)
+
+### Programming formalisms
+- [Programming formalisms](https://www.uppmax.uu.se/support/courses-and-workshops/programming-formalisms/)
+- A bit more focus on parallelism and GPUs and ML
+- May/June (date not set)
+
+## Courses at other NAISS/SNIC Centers
+- [SNIC training](http://snicdocs.nsc.liu.se/wiki/Training)
+
+## CodeRefinery
+- [Tools workshop](https://coderefinery.github.io/2023-09-19-workshop/)
+- March 12-14, 19-22?
+
+## ENCCS (EuroCC National Competence Centre Sweden
+- [Lessons](https://enccs.se/lessons/)
+- [Upcoming events](https://enccs.se/events)
+
+##    Links to extra material
+###        Data Management Seminar
+- [Data Management Seminar](https://www.uppmax.uu.se/support/courses-and-workshops/#tocjump_046728919125111124_0)
+
+###        Python
+- [Python at Codecademy](https://www.codecademy.com/)
+- [Python at Learn X in Y minutes](https://learnxinyminutes.com/docs/python/)
+- [Python village](https://rosalind.info/problems/list-view/?location=python-village)
+###        General Bioinformatics
+- [General Bioinformatics](https://rosalind.info/problems/list-view/)
+###        Swedish Science Cloud
+- [Swedish Science Cloud](https://github.com/SNICScienceCloud/technical-training)
+
+
diff --git a/_sources/exercises.rst.txt b/_sources/exercises.rst.txt
@@ -0,0 +1,6 @@
+Exercises
+=========
+
+.. exerciselist}::
+
+  :include: exercise challenge solution instructor-note
diff --git a/_sources/hpc.md.txt b/_sources/hpc.md.txt
@@ -0,0 +1,127 @@
+# High Performance Computing — HPC
+
+```{objectives}
+- Let's recap and go a little further into the UPPMAX hardware!
+```
+
+```{seealso}
+[Uppmax clusters](http://docs.uppmax.uu.se/cluster_guides/uppmax_cluster/)
+```
+
+```{instructor-note}
+- Approx timing: 11.40-11:50
+- Theory
+```
+
+## HPC, HTC and MTC
+
+- The Buzz word is **HPC or High Performance Computing**, but this is rather narrow focusing on fast calculation, i.e. processors and parallelism
+- Many of your projects are more focusing on high throughput, large memory demands and many tasks.
+- Here is a list of the three most common **Computing paradigms**:
+
+- **HPC**: High Performance Computing — Focus on floating point operations per second (**FLOPS**, flops or flop/s) 
+  - characterized as needing large amounts of computing power for short periods of time
+- **HTC**: High-Throughput Computing — 
+  - operations or **jobs per month or per year**. 
+  - more interested in how many jobs can be completed over a long period of time instead of how fast. 
+  - independent, sequential jobs that can be individually scheduled o
+- **MTC**: Many-task Computing — emphasis of using many computing resources over short periods of time to accomplish many computational tasks
+  - bridge the gap between HTC and HPC. 
+  - reminiscent of HTC, but including both dependent and independent tasks), where the primary metrics are measured in seconds (e.g. **FLOPS**, tasks/s, **MB/s** **I/O rates**), as opposed to operations (e.g. jobs) per month. 
+  - high-performance computations comprising multiple distinct activities, coupled via file system operations. 
+
+
+
+## What is a cluster?
+- A network of computers, each computer working as a node.
+
+- From small scale RaspberryPi cluster... 
+
+![RaspBerry](./img/IMG_5111.jpeg)
+
+- To supercomputers like Rackham.
+
+![Rackham](./img/uppmax-light2.jpg)
+
+- Each node contains several processor cores and RAM and a local disk called scratch.
+
+![Node](./img/node.png)
+
+- The user logs in to login nodes via Internet through ssh or Thinlinc.
+
+  - Here the file management and lighter data analysis can be performed.
+
+![RaspBerry](./img/nodes.png)
+
+![RaspBerry](./img/Bild1.png)
+
+- The calculation nodes has to be used for intense computing. 
+  - "Normal" softwares use one core.
+  - Parallelized software can utilize several cores or even several nodes. Keywords signalizing this are e.g.:
+    - "multi-threaded", "MPI", "distributed memory", "openMP", "shared memory".
+  - To let your software run on the calculation nodes
+    - start an "interactive session" or
+    - "submit a batch job".
+    - More about this in today's introduction to jobs.
+
+## Storage basics
+- All nodes can access:
+  - Your home directory on Domus or Castor
+  - Your project directories on Crex or Castor
+  - Its own local scratch disk (2-3 TB)
+
+- If you’re reading/writing a file once, use a directory on Crex or Castor
+- If you’re reading/writing a file many times...
+    - Copy the file to ”scratch”, the node local disk:
+    ```
+    cp myFile $SNIC_TMP
+    ```
+
+
+## The UPPMAX hardware   
+
+### Clusters
+
+- We have a number of compute clusters:
+
+  -  [Rackham](http://docs.uppmax.uu.se/cluster_guides/rackham_portal/)
+, reserved for SNIC projects
+  -  [Snowy](http://docs.uppmax.uu.se/cluster_guides/snowy_portal/), GPU, long jobs reserved for UPPMAX projects and Education
+  -  [Bianca](http://docs.uppmax.uu.se/cluster_guides/bianca_portal/)
+, a part of SNIC-SENS
+  -  [Miarka](https://www.uppmax.uu.se/resources/systems/miarka-cluster/), reserved for Scilifelab production
+  -  [UPPMAX cloud](https://www.uppmax.uu.se/resources/systems/the-uppmax-cloud/), a part of SNIC Science Cloud
+
+- [User guides](http://docs.uppmax.uu.se/)
+
+- The storage systems we have provide a total volume of about 25 PB, the equivalent of  50,000 years of 128-bit encoded music. Read more on the [storage systems page](https://www.uppmax.uu.se/resources/systems/storage-systems/).
+
+### UPPMAX storage system names (projects & home directories)
+- Rackham storage: Crex & Domus
+- Bianca storage: Castor & Cygnus
+- NGI production system (Miarka): Vulpes
+- NGI delivery server: Grus
+- Off-load storage: Lutra
+
+### System usage
+[System usage](https://www.uppmax.uu.se/resources/system-usage/)
+
+- More about the systems can be found at the [System resources page](https://www.uppmax.uu.se/resources/systems/)
+
+
+### A little bit more about Snowy
+
+- [User guide](http://docs.uppmax.uu.se/cluster_guides/snowy_portal/).
+  - There is a [local compute round](https://supr.naiss.se/round/uppmaxcompute2023/) for UU users applying for Snowy in SUPR.
+  - GU (courses) applications (including GU GPU usage) are not done in SUPR, but are supposed to be routed through the service desk.   
+  - The details can be found at the [Getting started page](https://www.uppmax.uu.se/support/getting-started/course-projects/).
+
+### About Bianca?
+- Wait for it!
+
+
+```{keypoints}
+- UPPMAX has several clusters 
+     - each having its focus and limitation or possibilites
+     - access is determined by type of project
+```