Skip to content

Commit

Permalink
Remove trailing whitespace, improve markdown lint
Browse files Browse the repository at this point in the history
  • Loading branch information
richelbilderbeek committed May 14, 2024
1 parent af832f0 commit ff6b293
Show file tree
Hide file tree
Showing 16 changed files with 142 additions and 142 deletions.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ python3 scripts/sw_table_md_creator.py -i https://export.uppmax.uu.se/staff/soft

Dependencies:
```bash
pip install beautifulsoup4\>=4.11.1 gTTS\>=2.2.4
pip install beautifulsoup4\>=4.11.1 gTTS\>=2.2.4
```

The script `md_to_speech.py` takes an `.md` file, parses the text and generates
Expand All @@ -79,5 +79,5 @@ Filename |Descriptions
The website is created using
[mkdocs-material](https://squidfunk.github.io/mkdocs-material). The landing
page and layout was inspired by the documentation of the HPC cluster
[LUMI](https://docs.lumi-supercomputer.eu/).
[LUMI](https://docs.lumi-supercomputer.eu/).

10 changes: 5 additions & 5 deletions docs/cluster_guides/backup.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ As PI, you and your academic institution are ultimately responsible for your dat
While UPPMAX systems may have backup, these are not designed to act as the sole repository of primary data, e.g. raw data or originals.

Check failure on line 7 in docs/cluster_guides/backup.md

View workflow job for this annotation

GitHub Actions / check_markdown

Line length [Expected: 80; Actual: 135]

## What does "backup" mean for my data?

Check failure on line 9 in docs/cluster_guides/backup.md

View workflow job for this annotation

GitHub Actions / check_markdown

Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "## What does "backup" mean for my data?"]
The type of backup that is generally available for project storage at UPPMAX is incremental backup with 30 day retention. This means that any file that was deleted more than 30 days ago is irretrievably gone. Changes in a file are kept for 30 days, so we can potentially retrieve an old version up to a month after you edited it.
The type of backup that is generally available for project storage at UPPMAX is incremental backup with 30 day retention. This means that any file that was deleted more than 30 days ago is irretrievably gone. Changes in a file are kept for 30 days, so we can potentially retrieve an old version up to a month after you edited it.

Check failure on line 10 in docs/cluster_guides/backup.md

View workflow job for this annotation

GitHub Actions / check_markdown

Line length [Expected: 80; Actual: 329]

The backup service tries to backup all changes as often as they occur, but rapid changes will not register. Due to the large amounts of files in the file systems, a single backup session may take upwards of a week or more. This means that if you create a file and delete it the next day, it will probably not be backed up.

Check failure on line 12 in docs/cluster_guides/backup.md

View workflow job for this annotation

GitHub Actions / check_markdown

Line length [Expected: 80; Actual: 322]

Backups are sent off-site to either KTH or LiU, depending on the storage system.

To ensure timely backups, it is very important to reduce the workload of the backup system as much as possible. Create directories with "nobackup" in their name or use the pre-existing nobackup directory in /proj/XYZ to store data that does not need backup.
To ensure timely backups, it is very important to reduce the workload of the backup system as much as possible. Create directories with "nobackup" in their name or use the pre-existing nobackup directory in /proj/XYZ to store data that does not need backup.

Check failure on line 16 in docs/cluster_guides/backup.md

View workflow job for this annotation

GitHub Actions / check_markdown

Line length [Expected: 80; Actual: 257]

- It is especially important that temporary files and files that are changed often are placed in nobackup directories.

Check failure on line 18 in docs/cluster_guides/backup.md

View workflow job for this annotation

GitHub Actions / check_markdown

Unordered list style [Expected: asterisk; Actual: dash]

Check failure on line 18 in docs/cluster_guides/backup.md

View workflow job for this annotation

GitHub Actions / check_markdown

Line length [Expected: 80; Actual: 118]

Expand All @@ -32,11 +32,11 @@ Backup is done on:

## What should I not put in directories with backup?
- Directories where you are actively working, especially if you are creating or modifying many files.
The backup mechanisms cannot keep up with large amounts of files changing on a rapid basis.
The backup mechanisms cannot keep up with large amounts of files changing on a rapid basis.

## How robust is uppmax storage?
- All UPPMAX storage systems use RAID technology to make storage more robust through redundancy.
- This means that two or more disks must fail in the same "RAID volume" before there is a risk of data loss.
- All UPPMAX storage systems use RAID technology to make storage more robust through redundancy.
- This means that two or more disks must fail in the same "RAID volume" before there is a risk of data loss.

- However, this technology does not protect against user error (e.g. "rm -rf * in your project directory) or in case of a significant disaster (e.g. fire in computer hall).
- Off-site backup is crucial.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ In the 'Site Manager' dialog, click 'New site'

In the 'New Site' dialog, create a name for the site, e.g. `bianca-sens123456`.

## 6. Configure site
## 6. Configure site

In the 'New Site' dialog, use all standards, except:

Expand Down
2 changes: 1 addition & 1 deletion docs/cluster_guides/bianca_file_transfer_using_gui.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ one needs [to be inside of SUNET](../getting_started/get_inside_sunet.md).

See the 'get inside the university networks' page [here](../getting_started/get_inside_sunet.md)

When a tool is setup, one can only transfer files
When a tool is setup, one can only transfer files
between you local computer and [your Bianca `wharf` folder](wharf.md).

## Bianca's constraints
Expand Down
8 changes: 4 additions & 4 deletions docs/cluster_guides/lftp_with_bianca.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
`lftp` is a command-line program
to [transfer files to/from Bianca](transfer_bianca.md).

With the command line SFTP client `lftp`,
you need to "set net:connection_limit 1".
`lftp` may also defer the actual connection
With the command line SFTP client `lftp`,
you need to "set net:connection_limit 1".
`lftp` may also defer the actual connection
until it's really required unless you end your connect URL with a path.

[When inside of SUNET](../getting_started/get_inside_sunet.md)
Expand All @@ -15,7 +15,7 @@ until it's really required unless you end your connect URL with a path.
lftp sftp://[user_name]-[project_id]@bianca-sftp.uppmax.uu.se/[user_name]-[project_id]/
```

where
where

* `[project_id]` is the ID of your [NAISS project](../getting_started/project.md)
* `[user_name]` is the name of your [UPPMAX user account](../getting_started/user_account.md)
Expand Down
10 changes: 5 additions & 5 deletions docs/cluster_guides/project_management.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@

???- question "What is this 'glob' folder in my home folder?"

- The glob directory found in your home has been deprecated since early 2017.
- It is now a normal directory and shared your default 32GByte sized home.
- The glob directory found in your home has been deprecated since early 2017.
- It is now a normal directory and shared your default 32GByte sized home.
- The glob directory remains to not interfere with scripts who might reference ~/glob in the source code.

- Historically, the glob directory was the main storage area for storage of user data.
- It was shared by all nodes.
- The directory was used for files needed by all job instances and could house files exceeding the quota of the home directory.
- Historically, the glob directory was the main storage area for storage of user data.
- It was shared by all nodes.
- The directory was used for files needed by all job instances and could house files exceeding the quota of the home directory.
- Job input and output files was (and can still be) stored here.

## Members
10 changes: 5 additions & 5 deletions docs/cluster_guides/rsync_on_bianca.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# `rsync` on Bianca

[`rsync`](../software/rsync.md) is a command-line tool
[`rsync`](../software/rsync.md) is a command-line tool
for [file transfer](../cluster_guides/file_transfer.md).

This page describes how to use [`rsync`](../software/rsync.md) on [Bianca](bianca.md).
Expand All @@ -12,18 +12,18 @@ One cannot `rsync` directly to `wharf`.
One cannot `rsync` directly to `wharf`.

However, this is how it looks like:


```
richel@richel-N141CU:~$ rsync my_local_file.txt [email protected]:/richel-sens2016001

Hi!

You are connected to the bianca wharf (sftp service) at
You are connected to the bianca wharf (sftp service) at
bianca-sftp.uppmax.uu.se.

Note that we only support SFTP, which is not exactly the
same as SSH (rsync and scp will not work).
same as SSH (rsync and scp will not work).

Please see our homepage and the Bianca User Guide
for more information:
Expand All @@ -36,7 +36,7 @@ One cannot `rsync` directly to `wharf`.
Best regards,
UPPMAX

[email protected]'s password:
[email protected]'s password:
protocol version mismatch -- is your shell clean?
(see the rsync manpage for an explanation)
rsync error: protocol incompatibility (code 2) at compat.c(622) [sender=3.2.7]
Expand Down
12 changes: 6 additions & 6 deletions docs/cluster_guides/running_jobs/storage_compression.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,16 @@
???- question "How does automatic backup of project areas work at UPPMAX?"

[Backup](../backup.md)

???- question "What is this 'glob' folder in my home folder?"

- The glob directory found in your home has been deprecated since early 2017.
- It is now a normal directory and shared your default 32GByte sized home.
- The glob directory found in your home has been deprecated since early 2017.
- It is now a normal directory and shared your default 32GByte sized home.
- The glob directory remains to not interfere with scripts who might reference ~/glob in the source code.

- Historically, the glob directory was the main storage area for storage of user data.
- It was shared by all nodes.
- The directory was used for files needed by all job instances and could house files exceeding the quota of the home directory.
- Historically, the glob directory was the main storage area for storage of user data.
- It was shared by all nodes.
- The directory was used for files needed by all job instances and could house files exceeding the quota of the home directory.
- Job input and output files was (and can still be) stored here.

- You might also be interested in our [disk storage guide](../storage/disk_storage_guide.md).
Expand Down
16 changes: 8 additions & 8 deletions docs/cluster_guides/sftp_with_bianca.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ to [transfer files to/from Bianca](transfer_bianca.md).
sftp [user_name]-[project_id]@bianca-sftp.uppmax.uu.se:/[user_name]-[project_id]
```

where
where

* `[project_id]` is the ID of your [NAISS project](../getting_started/project.md)
* `[user_name]` is the name of your [UPPMAX user account](../getting_started/user_account.md)
Expand All @@ -30,12 +30,12 @@ sftp [email protected]:/sven-sens2016001
`sftp` will ask for a password:

```
[email protected]'s password:
[email protected]'s password:
```

The password is your normal UPPMAX password directly followed by
the six digits from the [the `UPPMAX` 2-factor authentication](https://www.uu.se/en/centre/uppmax/get-started/2-factor).
For example, if your password is `VerySecret` and the second factor code is `123456`
For example, if your password is `VerySecret` and the second factor code is `123456`
you would type `VerySecret123456` as the password in this step.

After typing in the password and 2FA one sees a welcome message
Expand All @@ -48,11 +48,11 @@ and the `sftp` prompt.
```
Hi!

You are connected to the bianca wharf (sftp service) at
You are connected to the bianca wharf (sftp service) at
bianca-sftp.uppmax.uu.se.

Note that we only support SFTP, which is not exactly the
same as SSH (rsync and scp will not work).
same as SSH (rsync and scp will not work).

Please see our homepage and the Bianca User Guide
for more information:
Expand All @@ -65,9 +65,9 @@ and the `sftp` prompt.
Best regards,
UPPMAX

[email protected]'s password:
[email protected]'s password:
Connected to bianca-sftp.uppmax.uu.se.
sftp>
sftp>
```

???- question "How do I get rid of the welcome message?"
Expand All @@ -81,7 +81,7 @@ and the `sftp` prompt.
The last line, `sftp> ` is the `sftp` prompt.


Once connected you will have to type the `sftp` commands to upload/download files.
Once connected you will have to type the `sftp` commands to upload/download files.
See [the UPPMAX page on `sftp`](../software/sftp.md) how to do so.

With `sftp` you only have access to [your wharf folder](wharf.md).
30 changes: 15 additions & 15 deletions docs/cluster_guides/software_on_transit.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
# Software on Transit

[Transit](../cluster_guides/transit.md)
[Transit](../cluster_guides/transit.md)
is an UPPMAX service that can be used to securely transfer files.

This page describes the software on [Transit](../cluster_guides/transit.md).

After [logging in to Transit](../cluster_guides/login_transit.md),
you cannot make lasting changes to anything,
except for mounted [wharf](../cluster_guides/wharf.md) directories.
However, anything you have added to your [Rackham](../cluster_guides/rackham.md) home directory
is available on [Transit](../cluster_guides/transit.md).
you cannot make lasting changes to anything,
except for mounted [wharf](../cluster_guides/wharf.md) directories.
However, anything you have added to your [Rackham](../cluster_guides/rackham.md) home directory
is available on [Transit](../cluster_guides/transit.md).

In addition, some modules are available.

Expand All @@ -24,18 +24,18 @@ In addition, some modules are available.
```
![dds-cli](../img/dds-cli.png)

To download data from TCGA,
[log in to Rackham](../getting_started/login_rackham.md)
and install the GDC client to your home directory.
Then [log in to Transit](../cluster_guides/login_transit.md),
mount the [wharf](../cluster_guides/wharf.md),
To download data from TCGA,
[log in to Rackham](../getting_started/login_rackham.md)
and install the GDC client to your home directory.
Then [log in to Transit](../cluster_guides/login_transit.md),
mount the [wharf](../cluster_guides/wharf.md),
and run `./gdc-client`.

!!! warning "2FA on transit"

If you connect from abroad and
you are asked for the **2FA** (_two factor authentication_),
there is a grace period (_about 5 minutes_) in which you can
`ssh`/`scp`/`rsync`/`sftp` to **transit** without the need for **2FA**.
This allows you to use these and other tools
If you connect from abroad and
you are asked for the **2FA** (_two factor authentication_),
there is a grace period (_about 5 minutes_) in which you can
`ssh`/`scp`/`rsync`/`sftp` to **transit** without the need for **2FA**.
This allows you to use these and other tools
that might experience problems with the **2FA**.
12 changes: 6 additions & 6 deletions docs/cluster_guides/storage/compress_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ We have several compression programs installed and you are free to chose whichev
gzip also has a parallel version (pigz) that will let the program use multiple cores, making it much faster. If you want to run multithreaded you should make a reservation in the queue system, as the login nodes will throttle your programs if they use too much resources.

```
# compress a file
# compress a file
$ gzip file.txt # single threaded
$ pigz -p 4 file.txt # using 4 threads
# decompress a file
Expand All @@ -23,7 +23,7 @@ $ unpigz -p 4 file.txt # using 4 threads (4 is max)
bzip2 also has a parallel version (pbzip2) that will let the program use multiple cores, making it much faster. If you want to run multithreaded you should make a reservation in the queue system, as the login nodes will throttle your programs if they use too much resources.

```
# compress a file
# compress a file
$ bzip2 file.txt # single threaded
$ pbzip2 -p4 file.txt # using 4 threads
# decompress a file
Expand All @@ -36,7 +36,7 @@ $ pbunzip2 -p4 file.txt.gz # using 4 threads
zstd has built in support for using multiple threads when compressing data only, making it much faster. If you want to run multithreaded you should make a reservation in the queue system, as the login nodes will throttle your programs if they use too much resources.

```
# compress a file
# compress a file
$ zstd --rm file.txt # single threaded
$ zstd --rm -T4 file.txt # using 4 threads
# decompress a file, only single threaded
Expand All @@ -48,7 +48,7 @@ The commands above work on a single file at a time, and if you have 1000s of fil

```
# to compress a folder (folder/)
# and all files/folder inside it,
# and all files/folder inside it,
# creating a archive file named files.tar.gz
$ tar -czvf files.tar.gz folder/
# to decompress the archive later
Expand Down Expand Up @@ -76,7 +76,7 @@ There are some compression algorithms that have become standard practice to use
### fastq files

```
# compress sample.fq
# compress sample.fq
$ gzip sample.fq # single threaded
$ pigz -p 4 sample.fq # using 4 threads
```
Expand All @@ -103,7 +103,7 @@ $ module load bioinfo-tools htslib
# compress sample.vcf / sample.g.vcf
$ bgzip sample.vcf # single threaded
$ bgzip -@ 4 sample.vcf # using 4 threads
# index sample.vcf.gz / sample.g.vcf.gz
# index sample.vcf.gz / sample.g.vcf.gz
$ tabix sample.vcf.gz
```

Expand Down
12 changes: 6 additions & 6 deletions docs/cluster_guides/transfer_bianca.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ flowchart LR
end
```

[File transfer](file_transfer.md) is the process of getting files
[File transfer](file_transfer.md) is the process of getting files
from one place to the other. This page shows how to do [file transfer](file_transfer.md) to/from
the [Bianca](bianca.md) UPPMAX cluster.

Expand Down Expand Up @@ -61,8 +61,8 @@ See [using `lftp` with Bianca](lftp_with_bianca.md).

## Transit server

To facilitate secure data transfers to, from,
and within the system for computing on sensitive data a special service is available
To facilitate secure data transfers to, from,
and within the system for computing on sensitive data a special service is available
via SSH at `transit.uppmax.uu.se`.

![A user that is logged in to Transit](./img/logged_in_transit.png)
Expand Down Expand Up @@ -103,10 +103,10 @@ rsync -avh [email protected]:path/my_files ~/sens2023531/

### Moving data between projects

- You can use transit to transfer data between projects
by mounting the wharfs for the different projects
- You can use transit to transfer data between projects
by mounting the wharfs for the different projects
and transferring data with `rsync`.
- Note that you may of course only do this if this is allowed
- Note that you may of course only do this if this is allowed
(agreements, permissions, etc.)

## Mounting `wharf` on your local computer
Expand Down
2 changes: 1 addition & 1 deletion docs/cluster_guides/transit.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Transit

[Transit](../cluster_guides/transit.md)
[Transit](../cluster_guides/transit.md)
is an UPPMAX service that can be used to securely transfer files.

???- question "Is Transit a file server?"
Expand Down
Loading

0 comments on commit ff6b293

Please sign in to comment.