diff --git a/.wordlist.txt b/.wordlist.txt index 9024be690..86d0d7799 100644 --- a/.wordlist.txt +++ b/.wordlist.txt @@ -873,4 +873,36 @@ ies ReadTheDocs Klemming Svanslös - +Bilderbeek +bioinformaticians +Busybox +Eklund +Filezilla +gff +gnuplot +KRT +nd +oneliners +SSD +Söderberg +Tilda +vCPU +VirtualBox +virtualized +xNVIDIA +DNS +lookups +misconfiguration +bootstrapotp +Authy +twofactor +Lastpass +softwares +SOMETEXT +otpauth +totp +dns +whatismyip +ifconfig +LastPass +SWAMID diff --git a/docs/cluster_guides/bianca_login_node.md b/docs/cluster_guides/bianca_login_node.md new file mode 100644 index 000000000..7396004ae --- /dev/null +++ b/docs/cluster_guides/bianca_login_node.md @@ -0,0 +1,11 @@ +--- +tags: + - Bianca + - login node + - login + - node +--- + +# Bianca login node + +There are running up to 7 login nodes on a physical Bianca-node. diff --git a/docs/cluster_guides/cluster_speeds.md b/docs/cluster_guides/cluster_speeds.md new file mode 100644 index 000000000..9ce961922 --- /dev/null +++ b/docs/cluster_guides/cluster_speeds.md @@ -0,0 +1,79 @@ +--- +tags: + - UPPMAX + - cluster + - clusters + - speed + - fast + - slow +--- + +# Cluster speeds + +Sometimes you feel a cluster is slow. + +Below are some benchmark results, so you can compare with what you are +experiencing. + +Please [contact support](../support.md) when you find out that your favorite +cluster is slower than expected. + +???- question "What could cause such a slowdown?" + + + > When things are slow it is usually due to latency + > when many processes are accessing the same files + > and physical hard drives + + Examples: + + - 2024-12-06: Castor is still holding some file systems for Bianca + or a user that is running a lot of very short lived Perl jobs on Bianca, + that are running too hard on Castor. + +## Starting an interactive session with two cores for one hour + +In general: + +- It takes seconds if a free compute node is available +- It takes minutes to start a new node + +Date and time |Cluster|Command |You waited for `x` seconds|Complete time (secs) +---------------|-------|--------------------------------------------|--------------------------|-------------------- +2024-12-19 8:00|Bianca |`interactive -A sens2023036 -n 2 -t 1:00:00`|518 (8:38) |548 (9:08) + +## Loading the `R_packages/4.3.1` module + +For [a benchmark to solve a ticket](https://github.com/UPPMAX/ticket_304069/blob/master/module_load.md#answers), +the following command was run in multiple settings: + +```bash +time module load R_packages/4.3.1 +``` + +From the three resulting times, the 'Real' time is used. + +Here are some expected timings: + +Project |Setting |Real loading time +-----------|----------------------------|----------------- +Rackham |SSH |0m0.758s +Bianca |SSH |0m8.984s + +Here are some unexpected timings: + +Project |Setting |Real loading time +-----------|----------------------------|----------------- +sens2023598|SSH |6m1.265s +sens2023598|Website |6m20.234s +sens2017625|SSH |6m4.584s +sens2017625|Website, interactive session|7m41.433s +sens2017625|SSH, interactive session |7m13.111s + + +## Loading the `RStudio/2023.12.1-402` module + +Project |Setting |Real loading time +-----------|----------------------------|----------------- +Bianca |Website |2m3.184s + diff --git a/docs/cluster_guides/gorilla.md b/docs/cluster_guides/gorilla.md index 49188ba36..6fcd01b22 100644 --- a/docs/cluster_guides/gorilla.md +++ b/docs/cluster_guides/gorilla.md @@ -1,3 +1,4 @@ # Gorilla -Gorilla will be ceph +Gorilla is a future storage system, +using the ceph file system. diff --git a/docs/cluster_guides/img/bianca_system_usage_20241218.png b/docs/cluster_guides/img/bianca_system_usage_20241218.png new file mode 100644 index 000000000..76c1a1fd0 Binary files /dev/null and b/docs/cluster_guides/img/bianca_system_usage_20241218.png differ diff --git a/docs/cluster_guides/img/cpu_usage_sens2017625.png b/docs/cluster_guides/img/cpu_usage_sens2017625.png new file mode 100644 index 000000000..cbbb158ea Binary files /dev/null and b/docs/cluster_guides/img/cpu_usage_sens2017625.png differ diff --git a/docs/cluster_guides/img/cpu_usage_sens2023036.png b/docs/cluster_guides/img/cpu_usage_sens2023036.png new file mode 100644 index 000000000..76b4e4c1d Binary files /dev/null and b/docs/cluster_guides/img/cpu_usage_sens2023036.png differ diff --git a/docs/cluster_guides/pelle.md b/docs/cluster_guides/pelle.md index 7ac40bbcf..2ff019408 100644 --- a/docs/cluster_guides/pelle.md +++ b/docs/cluster_guides/pelle.md @@ -9,7 +9,6 @@ tags: ![Image of Pelle Svanslös, from https://www.comics.org/issue/105368/cover/4](./img/pelle.jpg) - ???- question "Why such a bad image?" Copyright. This is one of the few images that shows Pelle Svanslös @@ -24,6 +23,25 @@ by UPPMAX after [applying to a Pelle project](../getting_started/project_apply_p Non-Uppsala users of [Rackham](rackham.md) can move their data to Dardel, see [the Rackham to Dardel migration guide](../cluster_guides/dardel_migration.md). +!!! warning "Status" + + Pelle is in the process of being set up. Here is the current + status of Pelle. + + The delivery of the new UPPMAX system, Pelle, faces continued delays. UPPMAX now has a plan for provisioning Rackham to Uppsala University researchers until Pelle is ready. + + We have started to approve proposals submitted to the UPPMAX Local 2025 round. This will take a little time, but when your project is approved we ask you to examine the decision email closely and let us know if we’re missing any projects for data migration. + + New projects with Pelle and Gorilla resources will not be able to use Pelle or Gorilla until the systems are operational. The projects will have those resources represented in SUPR anyway. + + Approved projects will receive a Rackham allocation, in addition to the resources you asked for, which will let you log in and submit jobs to Rackham. + + Access to storage directories on Crex belonging to UU-affiliated projects will not expire, so you will have continued access to data in UU-affiliated projects on Crex until Pelle is ready. Therefore, most approved projects will not receive a Crex allocation. If we can not identify existing storage allocations for you, then we will create a Crex allocation for your storage needs. + + When Pelle is operational, we will move data from the NAISS projects you’ve mentioned in the proposals to the UPPMAX Local projects for you. Then, we will switch off Rackham and Crex and let you use Pelle and Gorilla. + + The “UPPMAX for Education” service on Snowy will, for now, continue as normal. At some point, we will move the GPUs from Snowy to newer hardware. At the same time, we will add Rackham and/or Pelle allocations to projects and shut down Snowy. We will inform you before we do this. + ## [How to apply to a Pelle project](../getting_started/project_apply_pelle.md) See [how to apply to a Pelle project](../getting_started/project_apply_pelle.md). @@ -31,10 +49,10 @@ See [how to apply to a Pelle project](../getting_started/project_apply_pelle.md) ## Features of Pelle, compared to Rackham Although we from UPPMAX cannot say exactly how Pelle will look, -we try to have Pelle be as similar to Rackham as possible. +we try to have Pelle be as similar to [Rackham](rackham.md) as possible. ## Migration from Rackham to Pelle As both clusters are UPPMAX clusters, -we will transfer your data from Rackham +we will transfer your data from [Rackham](rackham.md) to Pelle. Users will have to [apply to a Pelle project](../getting_started/project_apply_pelle.md). diff --git a/docs/cluster_guides/rackham_modules.md b/docs/cluster_guides/rackham_modules.md index afe0bbfb4..805d587a0 100644 --- a/docs/cluster_guides/rackham_modules.md +++ b/docs/cluster_guides/rackham_modules.md @@ -8,7 +8,3 @@ do their work independently and undisturbed. Because this is the same for nearly all UPPMAX clusters, there is a general page on modules [here](modules.md) - -## Sidebar - -This is to see if a sidebar appears. diff --git a/docs/getting_started/get_uppmax_2fa.md b/docs/getting_started/get_uppmax_2fa.md index 52b0e9076..c73a0a69d 100644 --- a/docs/getting_started/get_uppmax_2fa.md +++ b/docs/getting_started/get_uppmax_2fa.md @@ -7,18 +7,106 @@ tags: # Setting up two factor authentication for UPPMAX +Two factor authentication (abbreviated to '2FA') +increases the security of your UPPMAX account +and is mandatory is multiple contexts. + +???- question "Why is this important?" + + See [Why is 2FA important?](why_2fa.md) + +This page describes how to set this up. + +## Procedure + ???- question "Prefer a video?" Watch [the YouTube video 'Get your UPPMAX 2FA'](https://youtu.be/eSn0kLkU5Dc) -Follow the guide [Setting up two factor authentication for UPPMAX](https://www.uu.se/en/centre/uppmax/get-started/2-factor). +### 1. Install a 2FA app + +Install an app to use for 2FA. + +???- question "Which app do you recommend?" + + Any app that works for you. + + Search for '2FA' or 'OTP' (short for + 'one time password').. + + Wikipedia maintains a list of 2FA apps + [here](https://en.wikipedia.org/wiki/Comparison_of_OTP_applications). + +### 2. Go to + +In your web browser, go to . + +???- question "How does this look like?" + + Here is how + looks like: + + ![https://suprintegration.uppmax.uu.se/bootstrapotp](./img/suprintegration_uppmax_uu_se_bootstrapotp.png) + +This will take you to the UU page to request a second factor for your UPPMAX account. + +### 3. Click on 'Continue' + +At this page, click on 'Continue' to be sent to a 'Login to SUPR' page. + +Click on 'Continue' to be sent to a 'Login to SUPR' page. + +### 4. Log in to SUPR + +At the 'Login to SUPR' page, log in, in any way that works for you. + +???- question "How does this look like?" + + ![Login to SUPR page when requesting a 2FA](./img/supr_login_for_2fa.png) + +In case you are not logged in already, log in to SUPR. + +### 5. Press the button 'Prove My Identity to UPPMAX' + +Acknowledge to SUPR that they may tell UPPMAX who you are, +by pressing the button "Prove My Identity to UPPMAX" on the page. + +### 6. Scan the QR-code with your 2FA app + +Scan the QR-code with your 2FA app. + +???- question "How does that look like?" + + ![Getting an UPPMAX 2FA QR code](./img/get_uppmax_2fa_qr.png) + +More details [here](get_uppmax_2fa_qr.md). + +### 7. Enter the code on the webpage + +Your application will show you a code, enter this code on the same webpage. + +More details [here](get_uppmax_2fa_qr_code.md). + +### 8. See acknowledgement that the new two factor has been registered + +You should see an acknowledgement that the new two factor has been registered. + +???- question "How does that look like?" + + ![The final page](./img/get_uppmax_2fa_qr_code.png) + +### 9. Wait for a confirmation email After this procedure, it takes around 15 minutes before you can use the 2FA to log in. -![UPPMAX 2FA set up for a fictional UPPMAX user called `sven`](./img/uppmax_2fa.png) +## FAQ + +???- question "How does the use of a 2FA app looks like?" -> UPPMAX 2FA set up for a fictional UPPMAX user called `sven` + ![UPPMAX 2FA set up for a fictional UPPMAX user called `sven`](./img/uppmax_2fa.png) + + > UPPMAX 2FA set up for a fictional UPPMAX user called `sven` ???- question "How do I know I used my new 2FA too early?" @@ -31,6 +119,27 @@ the 2FA to log in. ![Authentication failed](./img/get_uppmax_2fa_authentication_failed_directly_after_new_2fa.png) -## Link +???- question "How long does it take before my 2FA is active?" + + This is a matter of minutes. + + It takes a little while before your newly registered factor is usable, + but this should be a matter of minutes, not days. + +???- question "Will I get an email when my 2FA is active?" + + No. + + There is no extra mail sent to let you know that the newly registered + factor is usable, just the confirmation mail that mentions + that it will be activated soon. + + +## Troubleshooting + +Some of the common problems we've seen include -- [Setting up two factor authentication for UPPMAX](https://www.uu.se/en/centre/uppmax/get-started/2-factor) +- Not having an account at UPPMAX. This is required to get the second factor for your account. +- Using a device having it's time set differently from our systems. There are services on the internet (e.g. ) you can visit from the device you try to manage the code on that will show you if your device settings are problematic. +- Noting the code given at first and trying to use it every time when asked for a code. The code to give will change every thirty second and you should give whatever code is shown at the time. +- Expecting something else to be sent to you. You register the new second factor as part of the process. A confirmation mail is sent as well, but this is mostly to let you know in case your account details in SUPR have gone astray and someone else has registered a second factor for your account. diff --git a/docs/getting_started/get_uppmax_2fa_qr.md b/docs/getting_started/get_uppmax_2fa_qr.md new file mode 100644 index 000000000..95c319dfc --- /dev/null +++ b/docs/getting_started/get_uppmax_2fa_qr.md @@ -0,0 +1,37 @@ +--- +tags: + - 2FA + - MFA + - UPPMAX + - QR +--- + +# Setting up the QR code for two factor authentication for UPPMAX + +Part of [setting up two factor authentication for UPPMAX](get_uppmax_2fa.md) +is to get a QR code. + +![Getting an UPPMAX 2FA QR code](./img/get_uppmax_2fa_qr.png) + +You need to scan this QR code to add your account to your software. +Most softwares call this "Add account" or similar +and will offer an option to scan a QR code using the smartphone camera +or select an area of the screen where the code is. + +Note that this must often be done from +within the app for two factor authentication. + +If you see a string similar to + +```text +otpauth://totp/username@UPPMAX?secret=SOMETEXT&issuer=UPPMAX +``` + +it didn't work and you probably need to do something different +(such as starting the app and select scan from within). + +Once you've scanned the code, you are often allowed to change the name the +software will use for the account before you add it. +You can change the name if you want - changing the name does not affect the +codes generated. +Finish adding the account to the software. diff --git a/docs/getting_started/get_uppmax_2fa_qr_code.md b/docs/getting_started/get_uppmax_2fa_qr_code.md new file mode 100644 index 000000000..4f4b89cde --- /dev/null +++ b/docs/getting_started/get_uppmax_2fa_qr_code.md @@ -0,0 +1,25 @@ +--- +tags: + - 2FA + - MFA + - UPPMAX + - QR + - code +--- + +# Setting up the QR code for two factor authentication for UPPMAX + +Once you have the new account, you should get one time codes for it when you +have it selected. +To finish the registration at UPPMAX, you need to enter the code +that is displayed in the field where it says "Code:" and submit. +The codes will change over time, don't worry about this, +you just need to use whatever code is current. + +Once you have entered and submitted the current code, +you should see a final page: + +![The final page](./img/get_uppmax_2fa_qr_code.png) + +When you see that page, it will take a little while and the token will +be activated (you should also receive an e-mail about the new token). diff --git a/docs/getting_started/img/accounts.jpg b/docs/getting_started/img/accounts.jpg new file mode 100644 index 000000000..bfb8e65ff Binary files /dev/null and b/docs/getting_started/img/accounts.jpg differ diff --git a/docs/getting_started/img/find_inst.PNG b/docs/getting_started/img/find_inst.PNG new file mode 100644 index 000000000..ab9d5b5a0 Binary files /dev/null and b/docs/getting_started/img/find_inst.PNG differ diff --git a/docs/getting_started/img/get_uppmax_2fa_qr.png b/docs/getting_started/img/get_uppmax_2fa_qr.png new file mode 100644 index 000000000..8b880b3b2 Binary files /dev/null and b/docs/getting_started/img/get_uppmax_2fa_qr.png differ diff --git a/docs/getting_started/img/get_uppmax_2fa_qr_code.png b/docs/getting_started/img/get_uppmax_2fa_qr_code.png new file mode 100644 index 000000000..7e6356608 Binary files /dev/null and b/docs/getting_started/img/get_uppmax_2fa_qr_code.png differ diff --git a/docs/getting_started/img/proj.PNG b/docs/getting_started/img/proj.PNG new file mode 100644 index 000000000..4bed0e4c6 Binary files /dev/null and b/docs/getting_started/img/proj.PNG differ diff --git a/docs/getting_started/img/proj_red.PNG b/docs/getting_started/img/proj_red.PNG new file mode 100644 index 000000000..c800f4110 Binary files /dev/null and b/docs/getting_started/img/proj_red.PNG differ diff --git a/docs/getting_started/img/proj_request.PNG b/docs/getting_started/img/proj_request.PNG new file mode 100644 index 000000000..70f199955 Binary files /dev/null and b/docs/getting_started/img/proj_request.PNG differ diff --git a/docs/getting_started/img/reg_form.PNG b/docs/getting_started/img/reg_form.PNG new file mode 100644 index 000000000..6bdaa3c83 Binary files /dev/null and b/docs/getting_started/img/reg_form.PNG differ diff --git a/docs/getting_started/img/reg_new_person.PNG b/docs/getting_started/img/reg_new_person.PNG new file mode 100644 index 000000000..1357a5858 Binary files /dev/null and b/docs/getting_started/img/reg_new_person.PNG differ diff --git a/docs/getting_started/img/request_account.jpg b/docs/getting_started/img/request_account.jpg new file mode 100644 index 000000000..b70bf3b62 Binary files /dev/null and b/docs/getting_started/img/request_account.jpg differ diff --git a/docs/getting_started/img/supr_login.PNG b/docs/getting_started/img/supr_login.PNG new file mode 100644 index 000000000..d22577734 Binary files /dev/null and b/docs/getting_started/img/supr_login.PNG differ diff --git a/docs/getting_started/img/supr_login_for_2fa.png b/docs/getting_started/img/supr_login_for_2fa.png new file mode 100644 index 000000000..0c99cfb18 Binary files /dev/null and b/docs/getting_started/img/supr_login_for_2fa.png differ diff --git a/docs/getting_started/img/supr_request_2fa.png b/docs/getting_started/img/supr_request_2fa.png new file mode 100644 index 000000000..842b11646 Binary files /dev/null and b/docs/getting_started/img/supr_request_2fa.png differ diff --git a/docs/getting_started/img/suprintegration_uppmax_uu_se_bootstrapotp.png b/docs/getting_started/img/suprintegration_uppmax_uu_se_bootstrapotp.png new file mode 100644 index 000000000..3c9ed7c9e Binary files /dev/null and b/docs/getting_started/img/suprintegration_uppmax_uu_se_bootstrapotp.png differ diff --git a/docs/getting_started/img/suprintegration_uppmax_uu_se_bootstrapotp_ubuntu.png b/docs/getting_started/img/suprintegration_uppmax_uu_se_bootstrapotp_ubuntu.png new file mode 100644 index 000000000..281307ffa Binary files /dev/null and b/docs/getting_started/img/suprintegration_uppmax_uu_se_bootstrapotp_ubuntu.png differ diff --git a/docs/getting_started/img/usr_agree.PNG b/docs/getting_started/img/usr_agree.PNG new file mode 100644 index 000000000..4488865f4 Binary files /dev/null and b/docs/getting_started/img/usr_agree.PNG differ diff --git a/docs/getting_started/project_apply.md b/docs/getting_started/project_apply.md index 85b63b65f..21bc1ce29 100644 --- a/docs/getting_started/project_apply.md +++ b/docs/getting_started/project_apply.md @@ -12,7 +12,35 @@ To use [UPPMAX](../cluster_guides/uppmax.md) resources, one needs: - [an active research project](project.md) - [an UPPMAX user account](user_account.md) -This page is about applying for an UPPMAX project: +Your user account is a personal log-in to our systems. Computer resources like CPU-hours and disk storage are allocated to projects. + +The workflow is like this: + + - Register in SUPR + - Accept SUPR user agreement + - Become a member of a project + - Apply for an account at UPPMAX (or other resources) + +!!! warning + + Note that you can only get a user account on a resource if you belong to a project with allocations there! + +## SUPR account + +???- question "Get a SUPR account" + + - You should visit the national project management platform [SUPR](https://supr.naiss.se/) and [register](https://supr.naiss.se/person/register/) there. + - Make sure that you don't already have an account at SUPR. You must not have more than one account in SUPR. + - All steps [here](supr_register.md). + +???- question "Accept SUPR user agreement" + + - You must accept the user agreement in SUPR, either online or in paper form. + - Details [here](supr_register.md#accept-the-user-agreement). + +## Applying for an UPPMAX project (PI) + +If you are a PI: apply for a project in SUPR. - [Apply for a Bianca project](project_apply_bianca.md), i.e. a project using sensitive data @@ -22,7 +50,11 @@ This page is about applying for an UPPMAX project: i.e. a project for the Swedish Infrastructure for Medical Population-based Life-course and Environmental Research - [Other applications](https://www.uu.se/en/centre/uppmax/get-started/create-account-and-apply-for-project/apply-for-projects) -Alternatively, one can [join an existing project](join_existing_project.md). +## Become a member of a project + +If you are not a PI: Apply for membership in a project you want to join in SUPR, Wait for the PI to accept your application. Alternatively, the PI can add you directly. + +- [join an existing project](join_existing_project.md). ???- question "I just got an UPPMAX project, yet I cannot login to UPPMAX?" @@ -31,3 +63,10 @@ Alternatively, one can [join an existing project](join_existing_project.md). If after one night you cannot login, please [contact support](../support.md). + +## Apply for an account at UPPMAX + +If you don't already have an account at UPPMAX you are ready by now! + +- [Apply for an UPPMAX user account](user_account.md) + diff --git a/docs/getting_started/setup_vpn.md b/docs/getting_started/setup_vpn.md index b0da18c4b..46844c421 100644 --- a/docs/getting_started/setup_vpn.md +++ b/docs/getting_started/setup_vpn.md @@ -7,9 +7,9 @@ Swedish university networks to connect to the UPPMAX clusters. To be able to use a VPN to [get inside of SUNET](get_inside_sunet.md): -- For Uppsala University: +- For Uppsala University: - Mac and Windows users: [go to this page](https://www.uu.se/en/staff/service-and-tools/tools-and-guides/connect-to-the-network-remotely/connect-using-the-universitys-vpn-service) - - Linux users: [go to this page](https://its.gmu.edu/knowledge-base/how-to-install-cisco-secure-client-on-linux/) (which is not a UU page) + - Linux users: [go to this page](https://its.gmu.edu/knowledge-base/how-to-install-cisco-secure-client-on-linux/) (which is not a UU page) - For Lund University: [go to this page](https://luservicedesk.service-now.com/support_en?id=kb_article_en&sys_id=a0d9ee21db74605020681ea605961927) - For other Swedish universities, search their websites to get the required VPN credentials. diff --git a/docs/getting_started/supr.md b/docs/getting_started/supr.md index 644a52780..115faa29b 100644 --- a/docs/getting_started/supr.md +++ b/docs/getting_started/supr.md @@ -5,6 +5,29 @@ tags: # SUPR +SUPR ('Swedish User and Project Repository') is a website +at [https://supr.naiss.se/](https://supr.naiss.se/) that +manages HPC accounts and projects. + +## Apply for an UPPMAX project + +See [the UPPMAX page on how to apply for an UPPMAX project](project_apply.md). + +## Setting up an UPPMAX 2FA token + +Go to [https://suprintegration.uppmax.uu.se/bootstrapotp/](https://suprintegration.uppmax.uu.se/bootstrapotp/) +to initiate the token +creation. This should take you to a landing page with some initial +information and let you know that you will be sent to SUPR to log in. + +???- question "How does that look like?" + + ![SUPR request 2FA](./img/supr_request_2fa.png) + +Once you click "Continue", you'll be sent to SUPR where you should log in. +Once you've done so SUPR will let you know that you will be sent back +and the identity you are logged in with. + ???- info "For staff only" [SUPR API documentation](https://sonc.swegrid.se/wiki/SUPR/API), diff --git a/docs/getting_started/supr_register.md b/docs/getting_started/supr_register.md new file mode 100644 index 000000000..855dba3c0 --- /dev/null +++ b/docs/getting_started/supr_register.md @@ -0,0 +1,32 @@ +# Register at SUPR + +- Go to +- If you already have an account you must use that account, otherwise you can register with or without SWAMID by clicking on the appropriate button. + +![register new person](./img/reg_new_person.PNG) + +- If you clicked on register via SWAMID you will have to choose the university that you belong to. Here as an example I choose Uppsala University: + +![Find institution](./img/find_inst.PNG) + +- Click on the University and then choose if you want to login via this SWAMID once +- or if your browser should remember your choice and use every time you visit SUPR +- Here again as an example is Uppsala University: + +![SUPR login](./img/supr_login.PNG) + +- You can now check the information your university has sent to SUPR and accept it to create a SUPR account. +- If you instead click on Register without SWAMID you will have to fill in this: + +![Registration form](./img/reg_form.PNG) + +- Fill in the form and click the button to create your SUPR account. +- You have to confirm your account by answering an email sent to the address you registered. + +## Accept the User Agreement + +- After logging into your SUPR account you must accept the user agreement. Click on Handle User Agreement + +![User agreement](./img/usr_agree.PNG) + +- Depending on how you take care of the User Agreement, it may be approved automatically or it may require manual checking (for example if you choose to use the paper form). You will get an email from SUPR when it has been approved. diff --git a/docs/getting_started/user_account.md b/docs/getting_started/user_account.md index 4edd666d2..576c8e834 100644 --- a/docs/getting_started/user_account.md +++ b/docs/getting_started/user_account.md @@ -10,15 +10,33 @@ To apply for an UPPMAX user account, you (the user) and the PI of the project (the researcher in charge of the research project) must complete the following steps: -- You should visit the national project management platform SUPR and register there. +- You should visit the national project management platform SUPR and register there. Instructions [here](supr_register.md). - Make sure that you don't already have an account at SUPR. You must not have more than one account in SUPR. -- You must accept the user agreement in SUPR, either online or in paper form. +- You must accept the user agreement in SUPR, either online or in paper form. Details [here](supr_register.md#accept-the-user-agreement). - Become a member of a project: - - If you are a PI: apply for a project in SUPR. - - If you are not a PI: Apply for membership in a project you want to join in SUPR, Wait for the PI to accept your application. Alternatively, the PI can add you directly. + - If you are a PI: apply for a project in SUPR. Details [here](project_apply.md#applying-for-an-uppmax-project-pi). + - If you are not a PI: Apply for membership in a project you want to join in SUPR, Wait for the PI to accept your application. Alternatively, the PI can add you directly. [Join an existing project](join_existing_project.md) - You must apply for an account at UPPMAX in SUPR. -After the 4 steps are completed your account will be created at UPPMAX -within 2 working days and you will receive two emails with information -on how to access the UPPMAX resources. +## Apply for an account at UPPMAX + +- When the PI has accepted your membership application. You will receive an email. +- Log in to SUPR and click on Accounts in the list to the left. + +- You will see the login accounts you already have at other NAISS centres if you have any. Under the "Possible Resource Account Requests" headings you find the UPPMAX resources you can apply for login account on. Just use the "Request Account on UPPMAX" button. + +![Accounts](./img/accounts.jpg) + +- You can then request a username. Then click Request Account + +![Accounts](./img/request_account.jpg) + +- After applying it might take up to 2 working days before you receive 2 emails with information on how to login to UPPMAX. + +- If you have any questions please contact us through the [Support Form](https://supr.naiss.se/support/) on how to access the UPPMAX resources. + +!!! note + + After the 4 steps are completed your account will be created at UPPMAX within 2 working days and you will receive two emails with information + diff --git a/docs/getting_started/why_2fa.md b/docs/getting_started/why_2fa.md new file mode 100644 index 000000000..1af782fa8 --- /dev/null +++ b/docs/getting_started/why_2fa.md @@ -0,0 +1,88 @@ +--- +tags: + - 2FA + - MFA + - UPPMAX + - Why +--- + +# Why is 2FA important? + +By requiring a second factor, typically a cell phone or other physical device, +it becomes much harder for an attacker to gain access to your account +if they somehow have gotten hold of your password +(which in this case is the 1:st factor). + +For security reasons you will have to use a two factor authentication system +if you are a) connecting to UPPMAX from outside of Sweden, or b) connecting +from a network within Sweden that does not support forward and reverse DNS +lookups (due to a misconfiguration in your network, you can ask your +internet service provider about this). More information about why can be +found below. + +## From outside Sweden + +If you try to connect directly to our resources from computers outside Sweden +you will most likely be required to set up and use two factor +authentication (you will be asked for a code from your second factor +automatically if required). + +Another alternative, if you need to access UPPMAX from outside Sweden, +may be to use a Swedish VPN service. +For example, if you're employed at Uppsala University, +then you can connect using the university's VPN service. + +## From within Sweden + +If you are required to use two factor authentication, +and are connecting from a computer in Sweden, this is typically caused by +your computer not having a proper DNS name, or the forward and reverse name +resolution do not match. + +???- question "Why is that important?" + + See [here](http://en.wikipedia.org/wiki/Forward_Confirmed_reverse_DNS) + +If this is the case, please contact your ISP and ask them to correct this. + +## Note + +You can check forward and reverse name resolution on this webpage: + +- [http://www.whatismyip.com/reverse-dns-lookup](http://www.whatismyip.com/reverse-dns-lookup) + +To see what address the other side thinks you come from (which will likely be what our systems see), services like + +- [https://ifconfig.co/](https://ifconfig.co/) + +can be helpful. + +On Linux, you can also use these commands: + +- Forward resolution: `host mycomputername.domain.tld`. + You have to replace `mycomputername.domain.tld` + with your computers actual name, for example: + +```bash +host rackham2.uppmax.uu.se +``` + +will give: + +```bash +rackham2.uppmax.uu.se has address 89.44.250.83 +``` + +- Reverse resolution: `host my_ipnumber`. + You have to replace `my_ipnumber` with your computers actual IP number, + for example: + +```bash +host 89.44.250.83t +``` + +which should give something similar to: + +```bash +89.44.250.83.in-addr.arpa domain name pointer tintin1.uppmax.uu.se +``` diff --git a/docs/hardware/clusters/bianca.md b/docs/hardware/clusters/bianca.md index 174f4c299..b83db40b7 100644 --- a/docs/hardware/clusters/bianca.md +++ b/docs/hardware/clusters/bianca.md @@ -1,5 +1,41 @@ +--- +tags: + - Bianca + - hardware + - specifications + - specs +--- + # Bianca hardware +## Technical Summary + +- 204 compute nodes with single or dual CPUs and one 4TB mechanical drive or 1TB SSD +- Each CPU has 8 cores +- 75 compute nodes, 256 GB memory each. +- 15 compute nodes, 512 GB memory each +- 10 compute nodes each equipped with 2xNVIDIA A100 (40GB) GPUs +- Total number of CPU cores is 4800 +- Login nodes have 2vCPU each and 16GB memory +- Dual 10 Gigabit Ethernet for all nodes + +## Parameters + +Parameter |Bianca +------------------------|----------------------------------- +**Purpose** |Sensitive data +**Reserved for** |NAISS-SENS projects +**Nodes (Intel)** |272 + 4 nodes á 2 NVIDIA A100 GPUs +**Cores per node** |16/64 +**Memory per node** |128GB +**Fat nodes** |256 & 512GB +**Local disk (scratch)**|4TB +**Network** |Dual 10Gbit/s +**Operating System** |CentOS 7 +**Login nodes** |Yes (2 cores and 15 GB) +**"Home" storage** |Castor +**"Project" Storage** |Castor + ## CPU ## GPU @@ -7,3 +43,20 @@ ## Network ## Storage + +## Security + +Since Bianca is designed to handle sensitive personal data security is a key aspect of the configuration. In order to ensure that the data is safe we have implemented a series of security measures including, but not limited to: + +- One virtualized cluster per project, no resources are shared between projects. +- Separate storage volumes per project. +- Detailed logging of file transfers in and out of the cluster. +- Two factor authentication +- No internet access inside the clusters. +- Locked racks for the hardware +- Destruction of broken hard drives + +Uppsala University has decided on the following KRT classifications for Bianca: + +- 321 for project directories +- 322 for home directories diff --git a/docs/hardware/overview.md b/docs/hardware/overview.md index f781fd5fa..e9809090b 100644 --- a/docs/hardware/overview.md +++ b/docs/hardware/overview.md @@ -1,3 +1,11 @@ +--- +tags: + - overview + - hardware + - specifications + - specs +--- + # Hardware overview This page describes the hardware architecture of the different compute clusters @@ -6,20 +14,20 @@ at UPPMAX as well as their storage systems. UPPMAX is part of the [National Academic Infrastructure for Supercomputing in Sweden (NAISS)](https://www.naiss.se/). -Parameter |Rackham |Snowy |Bianca |UPPMAX Cloud +Parameter |Rackham |Snowy |[Bianca](clusters/bianca.md) |UPPMAX Cloud ------------------------|-----------------------|---------------------------------------|-----------------------------------|----------------------------- **Purpose** |General-purpose |General-purpose |Sensitive data |IaaS -**Reserved for** |NAISS projects |Uppsala researchers and course projects|NAISS-SENS projects |NAISS and local projects -**Nodes (Intel)** |486+144 |228 + 50 N vidia T4 GPUs |272 + 4 nodes á 2 NVIDIA A100 GPUs |40 + 20 A2 and 4 T4 Nvidia GPUs -**Cores per node** |20/16 |16 |16/64 |16 -**Memory per node** |128GB |128GB |128GB |128/256GB -**Fat nodes** |256GB & 1TB |256, 512 GB & 4TB | 256 & 512GB |N/A -**Local disk (scratch)**|2/3TB |4TB | 4TB |N/A -**Network** |InfiniBand FDR 56Gbit/s|InfiniBand FDR 40Gbit/ s | Dual 10Gbit/s |10GbE -**Operating System** |CentOS 7 |CentOS 7 | CentOS 7 |[Linux cloud image](https://cloud.snic.se/instances/) -**Login nodes** |Yes |No (reached from Rackham) |Yes (2 cores and 15 GB) |N/A -**"Home" storage** |Domus |Domus |Castor |N/A -**"Project" Storage** |Crex, Lutra |Crex, Lutra |Castor |N/A +**Reserved for** |NAISS projects |Uppsala researchers and course projects|See [Bianca](clusters/bianca.md) |NAISS and local projects +**Nodes (Intel)** |486+144 |228 + 50 N vidia T4 GPUs |See [Bianca](clusters/bianca.md) |40 + 20 A2 and 4 T4 Nvidia GPUs +**Cores per node** |20/16 |16 |See [Bianca](clusters/bianca.md) |16 +**Memory per node** |128GB |128GB |See [Bianca](clusters/bianca.md) |128/256GB +**Fat nodes** |256GB & 1TB |256, 512 GB & 4TB |See [Bianca](clusters/bianca.md) |N/A +**Local disk (scratch)**|2/3TB |4TB |See [Bianca](clusters/bianca.md) |N/A +**Network** |InfiniBand FDR 56Gbit/s|InfiniBand FDR 40Gbit/ s |See [Bianca](clusters/bianca.md) |10GbE +**Operating System** |CentOS 7 |CentOS 7 |See [Bianca](clusters/bianca.md) |[Linux cloud image](https://cloud.snic.se/instances/) +**Login nodes** |Yes |No (reached from Rackham) |See [Bianca](clusters/bianca.md) |N/A +**"Home" storage** |Domus |Domus |See [Bianca](clusters/bianca.md) |N/A +**"Project" Storage** |Crex, Lutra |Crex, Lutra |See [Bianca](clusters/bianca.md) |N/A The storage systems we have provide a total volume of about 20 PB, the equivalent of nearly 15 billion 3.5-inch floppy disks or 40,000 years of diff --git a/docs/software/bianca_file_transfer_using_filezilla.md b/docs/software/bianca_file_transfer_using_filezilla.md index 4f4a566f5..df8a0dd58 100644 --- a/docs/software/bianca_file_transfer_using_filezilla.md +++ b/docs/software/bianca_file_transfer_using_filezilla.md @@ -96,6 +96,9 @@ type `[your password][2FA code]`, e.g. `VerySecret123456`. Now you can transfer files between your local computer and [your `wharf` folder](../cluster_guides/wharf.md). +NOTE: Filezilla will ask for your password and two-factor for each file you transfer. To avoid that, go to +Site Manager > Transfer Settings > Limit number of simultaneous connections to 1. + ???- tip "How does that look like?" It looks like this: diff --git a/docs/software/conda.md b/docs/software/conda.md index 70cad60aa..15f9bf134 100644 --- a/docs/software/conda.md +++ b/docs/software/conda.md @@ -104,7 +104,7 @@ We have mirrored all major non-proprietary Conda repositories (not ``main``, ``a !!! info "More info" - - [Installing Conda packages on Bianca](https://uppmax.github.io/bianca_workshop/extra/conda/) + - [Installing Conda packages on Bianca](https://uppmax.github.io/bianca_workshops/extra/conda/) ## Using Conda diff --git a/docs/software/img/rstudio_inresponsive_interpreter_and_no_files_pane.png b/docs/software/img/rstudio_inresponsive_interpreter_and_no_files_pane.png new file mode 100644 index 000000000..169e3cfa6 Binary files /dev/null and b/docs/software/img/rstudio_inresponsive_interpreter_and_no_files_pane.png differ diff --git a/docs/software/img/thinlinc_error_no_agent_server_available.png b/docs/software/img/thinlinc_error_no_agent_server_available.png new file mode 100644 index 000000000..71e75ba3c Binary files /dev/null and b/docs/software/img/thinlinc_error_no_agent_server_available.png differ diff --git a/docs/software/r_packages_bianca.md b/docs/software/r_packages_bianca.md index 8986776b9..db3df67bb 100644 --- a/docs/software/r_packages_bianca.md +++ b/docs/software/r_packages_bianca.md @@ -46,13 +46,13 @@ drwxrwsr-x 7 douglas sw 4096 May 25 10:46 glmnetr #### Methods - automatic download and install from CRAN - - + - - automatic download and install from GitHub - - + - - manual download and install - - + - - **NOTE** that if you install a package this way, you need to handle any dependencies yourself. - For instance you might get use of our modules diff --git a/docs/software/r_packages_bianca2.md b/docs/software/r_packages_bianca2.md index e9cc2a89a..126295e07 100644 --- a/docs/software/r_packages_bianca2.md +++ b/docs/software/r_packages_bianca2.md @@ -49,7 +49,7 @@ manual download. It is also possible to install from inside RStudio. ### setup (first time) -[https://uppmax.github.io/bianca_workshop/extra/rpackages/#setup](https://uppmax.github.io/bianca_workshop/extra/rpackages/#setup) +[https://uppmax.github.io/bianca_workshops/extra/rpackages/#setup](https://uppmax.github.io/bianca_workshops/extra/rpackages/#setup) - We need to create a place for the own-installed packages to be and to tell R where to find them. The initial setup only needs to be done once, but separate package directories need to be created for each R version used. @@ -91,7 +91,7 @@ version 4.0.4: ### Automatic download and install from CRAN -[https://uppmax.github.io/bianca_workshop/extra/rpackages/#automatic-download-and-install-from-cran](https://uppmax.github.io/bianca_workshop/extra/rpackages/#automatic-download-and-install-from-cran) +[https://uppmax.github.io/bianca_workshops/extra/rpackages/#automatic-download-and-install-from-cran](https://uppmax.github.io/bianca_workshops/extra/rpackages/#automatic-download-and-install-from-cran) !!! note @@ -119,7 +119,7 @@ In either case, the dependencies of the package will be downloaded and installed ### Automatic download and install from GitHub -[https://uppmax.github.io/bianca_workshop/extra/rpackages/#automatic-download-and-install-from-github](https://uppmax.github.io/bianca_workshop/extra/rpackages/#automatic-download-and-install-from-github) +[https://uppmax.github.io/bianca_workshops/extra/rpackages/#automatic-download-and-install-from-github](https://uppmax.github.io/bianca_workshops/extra/rpackages/#automatic-download-and-install-from-github) If you want to install a package that is not on CRAN, but which do have a GitHub page, then there is an automatic way of installing, but you need to handle prerequisites yourself by installing those first. @@ -141,7 +141,7 @@ This is how you install a package from GitHub, inside R: ### Manual download and install -[https://uppmax.github.io/bianca_workshop/extra/rpackages/#manual-download-and-install](https://uppmax.github.io/bianca_workshop/extra/rpackages/#manual-download-and-install) +[https://uppmax.github.io/bianca_workshops/extra/rpackages/#manual-download-and-install](https://uppmax.github.io/bianca_workshops/extra/rpackages/#manual-download-and-install) If the package is not on CRAN or you want the development version, or you for other reason want to install a package you downloaded, then this is how to install from the command line: diff --git a/docs/software/rstudio.md b/docs/software/rstudio.md index 571e5d12f..1f0eed771 100644 --- a/docs/software/rstudio.md +++ b/docs/software/rstudio.md @@ -16,7 +16,6 @@ Using RStudio differs per UPPMAX cluster: - [RStudio on Bianca](../software/rstudio_on_bianca.md) - [RStudio on Rackham](../software/rstudio_on_rackham.md) - ## RStudio versions ???- question "Which versions of RStudio are available?" @@ -51,9 +50,24 @@ RStudio module |RStudio Builds documentation -----------------------|----------------------- `RStudio/2023.06.2-561`|[here](https://dailies.rstudio.com/version/2023.06.2+561.pro1/) - ## Troubleshooting +### RStudio runs partially + +RStudio runs partially: + +- File content is displayed just fine +- The R interpreter does not respond +- The files pane at the bottom-right is loading forever + +![RStudio runs partially](./img/rstudio_inresponsive_interpreter_and_no_files_pane.png) + +In one case (see [ticket](https://github.com/UPPMAX/ticket_304069) for details), +the problem was caused by a process called `-bash` (yes, the first character +is a dash/minus). Killing it with `kill -s 1 [PID]` (for example, +`kill -s 1 11723`) and then restarting RStudio solved the +problem. + ### R encountered a fatal error Full error message: diff --git a/docs/software/rstudio_on_bianca.md b/docs/software/rstudio_on_bianca.md index a6c91398e..ccaff1825 100644 --- a/docs/software/rstudio_on_bianca.md +++ b/docs/software/rstudio_on_bianca.md @@ -100,21 +100,31 @@ interactive -A sens2016001 -n 2 -t 8:00:00 In the terminal of the interactive session, do: -In the terminal of the interactive session, do: ```bash -module load R/4.3.1 R_packages/4.3.1 RStudio/2023.06.2-561 +module load R_packages/4.3.1 RStudio/2023.12.1-402 ``` -???- question "What happens if I do not load `R` or `R_packages`?" +???- questions "Do all combinations of `R_packages` and `RStudio` work?" - Then you will have the sytem-wide R version 3.6.0 - without any packages installed. + No. -???- question "For UPPMAX staff" + Not all combination of `R_packages` and `RStudio` work equally well, + but this one is known to work (as + it was used in [this solved ticket](https://github.com/UPPMAX/ticket_304069)). There have been issues using `RStudio/2023.06.2-561` together with `R/4.3.1` +???- question "Shouldn't I load `R` first?" + + No. + + Loading `R_packages` will load the corresponding `R` module. + +???- question "What happens if I do not load `R_packages`?" + + Then you will have RStudio running without any R packages installed + ### 5. Start RStudio With the modules loaded, start RStudio from the terminal (on the diff --git a/docs/software/thinlinc.md b/docs/software/thinlinc.md index 773f8a5cb..9608025ae 100644 --- a/docs/software/thinlinc.md +++ b/docs/software/thinlinc.md @@ -67,3 +67,12 @@ Typically, users prefer to turn off full screen mode. Normally you don't have to change anything else here, and we have also disabled all "local devices" (USB-sticks, sound and printers) on server side. So no point to fiddle with these specific options. + + +## ThinLinc error: no agent server available + +![ThinLinc error: no agent server available](./img/thinlinc_error_no_agent_server_available.png) + +> ThinLinc error: no agent server available + +Try again :-) diff --git a/docs/software/whisper.md b/docs/software/whisper.md index 0c1b66d83..882e587db 100644 --- a/docs/software/whisper.md +++ b/docs/software/whisper.md @@ -1,6 +1,10 @@ --- tags: - Whisper + - transcriptions + - AI +date: + updated: 2024-12-20 --- # Whisper @@ -37,6 +41,12 @@ automatic speech recognition system. Whisper is available on Bianca. It can eith Whisper also tries to give separate sentences for different speakers. But it is not guaranteed. + !!! warning "Recordings from Dictaphone " + + If you record using dictaphone such as Olympus DS-9000, it would by default record in `.DS` or `.DS2` file formats which are NOT supported by Whisper. + Make sure to change the settings on the dictaphone to `.mp3` format before you start recording. + Follow this [guide](https://audiosupport.omsystem.com/wp-content/uploads/2021/11/DictationModule.pdf) to convert your `DS` or `.DS2` recording to `.mp3` using the software that comes with your dictaphone. Else, you can also download the sofware from [here](https://audiosupport.omsystem.com/en/product/odms-r8/) and then follow the same guide. + ## Glossary **SUPR account** : Gives access to project management account for submitting project proposals on SUPR. @@ -49,25 +59,35 @@ automatic speech recognition system. Whisper is available on Bianca. It can eith **Job**: A request for transcribing/translating one or many recordings. **Slurm**: "job" handler. - -## Accessing your project - -Following steps are derived from [UPPMAX User Accounts](https://www.uu.se/en/centre/uppmax/get-started/create-account-and-apply-for-project/user-account): - -!!! info inline end "Checklist for new project" +!!! info inline begin "Checklist for new project" * [x] SUPR account * [x] Submit project proposal * [x] UPPMAX username and password * [x] UPPMAX two factor authentication. -!!! info inline end "Checklist for existing project" +!!! info inline begin "Checklist for existing project" * [x] SUPR account * [ ] Submit project proposal * [x] UPPMAX username and password * [x] UPPMAX two factor authentication. +| + +| + +| + +| + +| + + +## Accessing your project + +Following steps are derived from [UPPMAX User Accounts](https://www.uu.se/en/centre/uppmax/get-started/create-account-and-apply-for-project/user-account): + 1. Register an [account on SUPR](https://supr.naiss.se/person/register/). 2. Apply for a project for [sensitive data at Bianca](https://supr.naiss.se/round/senssmall2024/create_proposal). @@ -283,7 +303,7 @@ Under the Basic Information section on NAISS SUPR, provide the following compuls To load the Whisper module, run the following command: ```console - [jayan@sens2024544-bianca jayan]$ module load Whisper/0.5.1 + [jayan@sens2024544-bianca jayan]$ module load Whisper ``` This will also load the necessary dependencies, including `python` @@ -292,8 +312,7 @@ Under the Basic Information section on NAISS SUPR, provide the following compuls ```console [jayan@sens2024544-bianca jayan]$ module list Currently Loaded Modules: - 1) uppmax 3) mp-tools/latest 5) FFmpeg/5.1.2 - 2) git/2.34.1 4) python/3.11.4 6) Whisper/0.5.1 + 1) uppmax 2) python/3.11.4 3) FFmpeg/5.1.2 4) Whisper/20240930 ``` ### Command-line @@ -304,6 +323,9 @@ Under the Basic Information section on NAISS SUPR, provide the following compuls [jayan@sens2024544-bianca jayan]$ whisper audio.flac audio.mp3 audio.wav --model medium ``` + For more ways to run whisper, for example on cpu node or do translations, check the correct flags by doing : `whisper --help` + You can also check the source code with arguments [here on the official GitHub repository](https://github.com/openai/whisper/blob/90db0de1896c23cbfaf0c58bc2d30665f709f170/whisper/transcribe.py#L528). + ### Python ```python title="example.py" diff --git a/docs/stylesheets/extra.css b/docs/stylesheets/extra.css index dcbd6bfb8..3f0d64c5d 100644 --- a/docs/stylesheets/extra.css +++ b/docs/stylesheets/extra.css @@ -5,16 +5,16 @@ --md-primary-fg-color: #E6E6E6; --md-primary-bg-color: #1d1d20; - --md-default-fg-color: #1d1d20; - --md-default-bg-color: #ffffff; + /* --md-default-fg-color: #1d1d20; */ + /* --md-default-bg-color: #ffffff; */ - --md-code-fg-color: #1d1d20; + /* --md-code-fg-color: #1d1d20; */ --md-accent-fg-color: #999; - --md-admonition-fg-color: #1d1d20; + /* --md-admonition-fg-color: #1d1d20; */ - --md-typeset-color: #1d1d20; + /* --md-typeset-color: #1d1d20; */ --md-typeset-a-color: #0645AD; --md-footer-bg-color: #E6E6E6; @@ -22,6 +22,10 @@ } +/* Maximum space for text block */ +.md-grid { + max-width: 70%; /* or 100%, if you want to stretch to full-width */ +} /* https://github.com/squidfunk/mkdocs-material/issues/4832#issuecomment-1374891676 */ .md-nav__link[for] { diff --git a/docs/workshops_courses/awk.md b/docs/workshops_courses/awk.md new file mode 100644 index 000000000..6507635d3 --- /dev/null +++ b/docs/workshops_courses/awk.md @@ -0,0 +1,125 @@ +# Awk workshop + +AWK is an interpreted programming language designed for text processing and typically used as a data extraction and reporting tool. + +This two-days workshop aims to promote and demonstrate the flexibility of the tool, where the overhead of more sophisticated approaches and programming languages is not worth the bother. + +## Learn how to + +- use Awk as **an advanced** grep command, capable of arithmetic selection rules with control over the content of the matched lines +- perform simple conversions, analysis or filter you data on the fly making it easy to plot or read in your favorite research tool +- handle and take advantage on data split over multiple file data sets. +- use Awk as simple function or data generator +- perform simple sanity checks on your results + +## Awk for bioinformaticians + +Use what you learn and dive into the basic concepts of bioinformatics with simple exercises on typical scientific problems and tasks. + +!!! note "Venue and registration:" + + **Date**: 16 and 17 January, 2025 + **Time**: 9:15 - 12:00 and 13:15 -16:00 + **Location**: Zoom: link will be sent to applicants + **Application**: [form](https://forms.gle/Hz1mQJSiA6ewfLqz8). + + +## Schedule + +--- + +### 1-st day 9:15 - 12:00 + +[Seminar session](https://pmitev.github.io/to-awk-or-not/) + +- Examples of typical problems suitable for Awk “treatment” +- Introduction to the basics of Awk scripting language +- Solving interactively simple problems + +#### 1-st day lunch break + +[Exercises](https://pmitev.github.io/to-awk-or-not/Exercises/Exercises/) 13:15 -16:00 + +- Solving interactively the exercise problems + +### 2-nd day 9:15 - 12:00 + +- Awk for bioinformaticians - seminar +- Case Study: [Manipulating the output from a genome analysis - vcf and gff](https://pmitev.github.io/to-awk-or-not/Case_studies/manipulating_vcf/) +- Filtering and formatting raw data +- Counting and piling features +- Indexing and hashing to compare variants and annotations + +#### 2-nd day lunch break + +Walk-through session on various topics: + +- Awk parsing “simultaneously” multiple input files +- Multiple input files - second approach scenario will be discussed. +- How to trick awk to accept options on the command line like regular program i.e. `$ script.awk filename parameter1 parameter2` [link](https://pmitev.github.io/to-awk-or-not/More_awk/Command_params/) +- Declaring and calling functions in awk - [link](https://pmitev.github.io/to-awk-or-not/More_awk/User_defined_functions/) +- Input/output to/from an external programs +- Learn how to send input to an external program (might be based on your data) and read the result back - [link](https://pmitev.github.io/to-awk-or-not/More_awk/Input_output/) +- Handy tips: awk oneliners use with Vim, gnuplot… + +Also: [Suggest topic](https://forms.gle/usYYkbWaZVkNceSK6) for discussion or see recently [suggested topics](https://docs.google.com/forms/d/1tQYWc504BQ-uYRA7MWgu1pNXM613r4Ua1wP_yBPlNDM/viewanalytics). + +## Prerequisites + +--- + +### MacOS + +The system provided awk version will work for most of the examples during the workshop with few exceptions, which are noted in the online material. + +_Tilda `~` sign on Mac with Swedish keyboard layout - Alt + ^_ + +### Linux + +Several distributions have other awk flavors installed by default. The easiest fix is to install the gnu version gawk i.e. for Ubuntu: `sudo apt install gawk` + +### Windows 10/11 + +- [Ubuntu for Windows 10](https://docs.microsoft.com/en-us/windows/wsl/install-win10) - it is better to read from the source, despite it might not be the easiest tutorial. To my experience, this is the best Linux environment without virtualization. +- [MobaXterm](https://mobaxterm.mobatek.net/) use the internal package manager to install gawk. The default is provided by Busybox and is not enough for the purpose of the workshop. + +### Linux computer center + +- Just login to your account and use the provided awk - any version newer than 4 will work. + +```bash +rackham3:[~] awk -V GNU Awk 4.0.2 Copyright (C) 1989, 1991-2012 Free Software Foundation. +``` + +### Virtual Linux Machine + +Just follow some tutorial on how to setup and use the virtual Linux environment. + +- [VirtualBox](https://www.virtualbox.org/) +- [Ubuntu on Public Clouds](https://ubuntu.com/public-cloud) +- [GitHub & Binder](https://pmitev.github.io/to-awk-or-not/Other/Binder/) (you need only a browser) +- [Singularity](https://sylabs.io/) `singularity run shub://pmitev/Teoroo-singularity:gawk 'BEGIN{ for(i=1;i<=4;i++) print i}'` + +??? "Feedback from previous workshops" + + - [2024.08](https://docs.google.com/forms/d/1GRELoVvXo975c4lQkx9a8k-93mxXe-WqQaZg_Zk5HeA/viewanalytics) | [2024.01](https://docs.google.com/forms/d/1eGzI8FxBlhP6SV8iPQPTfQn7CWJyBf4ZaCmvJ2srxDk/viewanalytics) + - [2023.09](https://docs.google.com/forms/d/16xCpKhhHqhcQpN-tD7yiFxoIhN3_7fp5-IpsUUOAgxM/viewanalytics) | 2023.01 (not enough data to be anonymous) + - [2022.09](https://docs.google.com/forms/d/1UUZP97qXq3rwxY7VGJsu1w-4QWfRCzEmO1xZWva-CVM/viewanalytics) | [2022.01](https://docs.google.com/forms/d/1mIboAG1nudj1yPN07-HZbQ6L9ghlZxrCLTFbAMJpARg/viewanalytics) + ​- [2021.09](https://docs.google.com/forms/d/1GILWudpKGoZSkyfkyBR-kRGTYieoXC1yPOz0Jn0UrcI/viewanalytics) | [2021.01](https://docs.google.com/forms/d/1be529TgFwsaNnsH_YQ-6qJWFNV15NTl510dWqrqzu1A/viewanalytics) + - [2020.08](https://docs.google.com/forms/d/1I6tMA-mXy5kIMEy5H1Nt2fbKcuMZpvxE_WYpJPkAJ5Q/viewanalytics) | [2020.01](https://docs.google.com/forms/d/1Wa9lCwxp0Pes38KFziilNbdcvYfHwxBiou9j3c3hNO0/viewanalytics) + - [2019.08](https://docs.google.com/forms/d/1-wha3xg_jkcZ03ljF6HmPnTFQGzGe08Jun5c0IAFfEU/viewanalytics) | [2019.01](https://docs.google.com/forms/d/1O1v8i3f1UDavfmntbEZ9cvm8_U-5Mj5P6GTEHUWyuuk/viewanalytics) + - [2018.08](https://docs.google.com/forms/d/1PG8dt0LSOdp9gv1rFCjEe1kiapx3a-SiSJkvl2MOlyA/viewanalytics) | [2018.01](https://docs.google.com/forms/d/1d85npGj6O5xuQEF9drBRhneqYKjW0yAZJOnTiI1QP0c/viewanalytics) + - [2017.01](https://docs.google.com/forms/d/1aTeYzOJTLNVkRYnXqOAOWFbtWIzgigqbt6hvuc4EBoE/viewanalytics) | [2017.08](https://docs.google.com/forms/d/1Y_D8kKDHsVCeu3Hli87iphnxp_ayNXfVJRcmFDiSe7Y/viewanalytics) + - [2016.08](https://docs.google.com/forms/d/1PXdyRsABx60Uq6mDwepKv8-0ztur8z9dEkoUOmmfqjg/viewanalytics) | [2016.01](https://docs.google.com/forms/d/11q4-HAOSy7LB8mla0EkP0PhkfuBVdyIpOKb9pSqCkb0/viewanalytics) + - [2015.10](https://docs.google.com/forms/d/1KSab3x3IlXdgtTScXPfHbFR81FrEpZ8j__hOgV8P5wU/viewanalytics) + + +## Contacts for the course + +--- +[Pavlin Mitev](https://katalog.uu.se/profile/?id=N3-1425) +[Jonas Söderberg](https://katalog.uu.se/empinfo/?id=N2-1277) +[Lars Eklund](https://katalog.uu.se/empinfo/?id=N5-89) +[Richel Bilderbeek](https://www.katalog.uu.se/empinfo/?id=N21-617) +[UPPMAX](https://www.uu.se/en/centre/uppmax) + diff --git a/docs/workshops_courses/workshops_courses.md b/docs/workshops_courses/workshops_courses.md index 538ac45ad..14ad53405 100644 --- a/docs/workshops_courses/workshops_courses.md +++ b/docs/workshops_courses/workshops_courses.md @@ -6,7 +6,7 @@ - [MkDocs style](https://uppmax.github.io/uppmax_intro_day_1/) - [Sphinx style](https://uppmax.github.io/uppmax_intro/) - [Intro to Python](https://uppmax.github.io/uppmax_intro_python/) -- [Introduction to Bianca](https://uppmax.github.io/bianca_workshop) +- [Introduction to Bianca](https://uppmax.github.io/bianca_workshops) - [HPC-Python](https://uppmax.github.io/HPC-python/index.html) - [Introduction to running R, Python, and Julia in HPC](https://uppmax.github.io/R-python-julia-matlab-HPC/) diff --git a/feedback/feedback.md b/feedback/feedback.md index 7f639bd29..1032b59a2 100644 --- a/feedback/feedback.md +++ b/feedback/feedback.md @@ -1,3 +1,7 @@ # Feedback Here we publish the anonymous feedback we get. + +## 2024-12-20 + +No feedback yet. diff --git a/mkdocs.yml b/mkdocs.yml index 291d5e32c..0abca76b0 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,6 +1,7 @@ site_name: UPPMAX Documentation repo_url: https://github.com/UPPMAX/UPPMAX-documentation +edit_uri: blob/main/docs/ nav: - Home: index.md @@ -115,9 +116,23 @@ theme: code: JetBrains Mono logo: assets/logo_color.png + palette: + + # Palette toggle for light mode + - scheme: default + toggle: + icon: material/weather-sunny + name: Switch to dark mode + + # Palette toggle for dark mode + - scheme: slate + toggle: + icon: material/weather-night + name: Switch to light mode features: - - toc.integrate + # - toc.integrate + - toc.follow - navigation.tabs - navigation.tabs.sticky - navigation.sections diff --git a/mkdocs_uppmax/home.html b/mkdocs_uppmax/home.html index 51e5b8bbb..5e1474bb2 100644 --- a/mkdocs_uppmax/home.html +++ b/mkdocs_uppmax/home.html @@ -8,9 +8,9 @@