Create page for wharf and rsync, move content to these, progress #63

docs/cluster_guides/file_transfer.md

@@ -0,0 +1,8 @@
+# File transfer
+
+[File transfer](file_transfer.md) is the process of getting files from one place to the other.
+
+ * [File transfer to/from Bianca](transfer_bianca.md)
+ * [File transfer to/from Dardel](transfer_dardel.md)
+ * [File transfer to/from Rackham](transfer_rackham.md)
+ * [File transfer to/from Transit](transfer_transfer.md)

docs/cluster_guides/transfer_bianca.md

@@ -6,37 +6,22 @@
 
 ![Bianca](../img/Bianca-transfer.png)
 
+[File transfer](file_transfer.md) is the process of getting files 
+from one place to the other. This page shows how to do [file transfer](file_transfer.md) to/from
+the [Bianca](bianca.md) UPPMAX cluster.
+
+## File transfer methods
+
 There are multiple ways to transfer files to/from Bianca:
 
 Method                                                        |Features
 --------------------------------------------------------------|---------------------------------------------
 [Using a graphical program](#gui-sftp-clients), see below     |Graphical interface, intuitive, for small amounts of data only
 Using standard command line SFTP client, see below            |Terminal, easy to learn, terminal-based, can use terminal commands to select files
-Transit Server from/to Rackham, see below                     |Terminal, can be used to transfer data between clusters in general
-Mounting the wharf on your local computer, see below          |Both graphical and terminal, need a computer with `sshfs` installed
-
-After discussing the `wharf`, each of these methods is discussed in detail.
-
-## The `wharf`
-
-The `wharf` is like a "postbox" :postbox: for data/file exchange between the Internet restricted Bianca cluster and the remaining of the World Wide Internet. This "postbox" is reachable to transfer data from two internal servers -
-`bianca-sftp.uppmax.uu.se` and `transit.uppmax.uu.se`.
-
----
-**The `wharf` location on Bianca**
-
-- The path to this special folder, once you are logged into your project's cluster, is:
-```
-/proj/<projid>/nobackup/wharf/<username>/<username>-<projid>
-```
-Example:
-```
-/proj/sens2023598/nobackup/wharf/myuser/myuser-sens2023598
-```
-
-- To transfer data from Bianca, copy/move the files/folders you want in that folder.
-- On Bianca you have full access to your `wharf` and readonly access to other's project users `wharf`s.
+Transit server from/to Rackham, see below                     |Terminal, can be used to transfer data between clusters in general
+[Mounting `wharf` on your local computer](#mounting-wharf)    |Both graphical and terminal, need a computer with `sshfs` installed
 
+All file transfer on Bianca uses [the `wharf` folder](wharf.md).
 
 ## GUI SFTP clients
 ---
@@ -162,6 +147,7 @@ my_user@transit:~$
 ### Moving data from transit to Rackham
 
 - **On Rackham:** (_or other computer_) copy files to Bianca via transit:
+
 ```bash
 # scp
 scp path/my_files [email protected]:sens2023531/
@@ -188,7 +174,6 @@ rsync -avh [email protected]:path/my_files ~/sens2023531/
 - You can use transit to transfer data between projects by mounting the wharfs for the different projects and transferring data with ``rsync``.
 - Note that you may of course only do this if this is allowed (agreements, permissions, etc.)
 
-
 ### Software on Transit
 
 - While logged in to Transit, you cannot make lasting changes to anything except for mounted wharf directories. However, anything you have added to your Rackham home directory is available on Transit. In addition, some modules are available.
@@ -208,6 +193,14 @@ rsync -avh [email protected]:path/my_files ~/sens2023531/
 !!! warning "2FA on transit"
     If you connect from abroad and you are asked for the **2FA** (_two factor authentication_), there is a grace period (_about 5 minutes_) in which you can `ssh`/`scp`/`rsync`/`sftp` to **transit** without the need for **2FA**. This allows you to use these and other tools that might experience problems with the **2FA**.
 
+## Mounting `wharf` on your local computer
+
+Mounting `wharf` means that a `wharf` folder is added to the
+filesystem of your local computer, after which you can use
+it like any other folder.
+
+See [the UPPMAX documentation of `wharf`](wharf.md) on how to do so.
+
 ---
 
 !!! info "Summary"
@@ -224,17 +217,3 @@ rsync -avh [email protected]:path/my_files ~/sens2023531/
         - transit server
         - rsync, scp/sftp
 
-## Mounting the SFTP-server with ``sshfs`` on you local machine
----
-**Mount the wharf on your machine**
-
-- This is only possible on your own system.
-- ``sshfs`` allows you to mount the ``wharf`` on your own machine.
-- You will be able to copy and work on the data using your own local tools such as ``cp`` or ``vim``.
-- Remember that you are neither logged in on the distant server, nor is the data physically on your local disk (until you have copied it).
-
-!!! warning
-    - UPPMAX doesn't have ``sshfs`` client package installed for security reasons.
-    - ``sshfs`` is available on most Linux distributions:
-        - install the package ``sshfs`` on Ubuntu,
-        - ``fuse-sshfs`` on Fedora, RHEL7/CentOS7 (enable EPEL repository) and RHEL8 (enable codeready-builder repository) / CentOS8 (enable powertools repository).

docs/cluster_guides/wharf.md

@@ -0,0 +1,76 @@
+# `wharf`
+
+`wharf` is a folder on [Bianca](bianca.md) used 
+for [file transfer on Bianca](tranfer_bianca.md).
+
+He it is described:
+
+ * [What is `wharf`?](#what-is-wharf)
+ * [The `wharf` location](#the-wharf-location)
+ * [`wharf` use](#wharf-use)
+ * [mounting `wharf`](#mounting-wharf)
+
+## What is `wharf`?
+
+The `wharf` is like a "postbox" :postbox: for data/file exchange 
+between the Internet restricted Bianca cluster 
+and the remaining of the World Wide Internet. 
+This "postbox" is reachable to transfer data from two internal servers -
+`bianca-sftp.uppmax.uu.se` and `transit.uppmax.uu.se`.
+
+## The `wharf` location
+
+The path to this special folder is:
+
+```
+/proj/[project_id]/nobackup/wharf/[user_name]/[user_name]-[project_id]
+```
+
+where 
+
+ * `[project_id]` is the ID of your [NAISS project](../getting_started/project.md)
+ * `[user_name]` is the name of your [UPPMAX user account](../getting_started/user_account.md)
+
+For example:
+
+```
+/proj/sens2023598/nobackup/wharf/sven/sven-sens2023598
+```
+
+## `wharf` use
+
+To [transfer data from/to Bianca](tranfer_bianca.md), 
+`wharf` is to folder where files are sent to/from.
+
+Do not keep files in `wharf`, as this folder is connected to the outside
+world and hence is a security risk. Instead, move your data to your project folder.
+
+You have full access to your `wharf` and read-only access 
+to other users' `wharf` folders in that same project.
+
+`wharf` is only accessible when [inside the university networks](../getting_started/get_inside_sunet.md).
+
+## Mounting `wharf`
+
+Mounting `wharf` means that a `wharf` folder is added to the
+filesystem of your local computer, after which you can use
+it like any other folder. The data shown in the folder is on Bianca,
+not on your local harddisk.
+
+One can mount `wharf` on your local computer using `sshfs`
+when [inside the university networks](../getting_started/get_inside_sunet.md).
+`sshfs` is available on most Linux distributions:
+
+Distro            |Package name
+------------------|-------------
+Ubuntu            | `sshfs`
+Fedora            | `fuse-sshfs`
+RHEL7/CentOS7 [1] | `fuse-sshfs`
+RHEL8 [2]         | `fuse-sshfs`
+CentOS8 [3]       | `fuse-sshfs`
+
+ * [1] Enable EPEL repository
+ * [2] Enable `codeready-builder` repository
+ * [3] Enable `powertools` repository
+
+UPPMAX does not have `sshfs` installed for security reasons.

docs/software/rsync.md

@@ -0,0 +1,67 @@
+# `rsync`
+
+`rsync` is a command-line tool for file transfer,
+with the goal of ensuring integrity of the data,
+as well as a minimal amount of data transfer.
+
+`rsync` can be used for copying, but also synchronizing files,
+such as is ideal for making a backup. At this page, we use the word 'copy',
+although `rsync` by default does a one-way synchronize: if the data is already
+there, it will do nothing.
+
+ * [Using `rsync` on Bianca](../cluster_guides/rsync_on_bianca.md)
+ * [Using `rsync` on Rackham](../cluster_guides/rsync_on_rackham.md)
+
+## Copy a folder from local to Rackham
+
+Copy a folder from a local computer to a Rackham home folder.
+
+On your local computer, do:
+
+```
+rsync --recursive [folder_name] [user_name]@rackham.uppmax.uu.se:/home/[user_name]/
+```
+
+For example:
+
+```
+rsync --recursive my_folder [email protected]:/home/sven/
+```
+
+The `--recursive` flag is used to
+copy a folder and all of its subfolders.
+
+???- question "Want to preserve timestamps?"
+
+    To preserve the files' timestamps, use the `--archive` flag, e.g.
+
+    ```
+    rsync --recursive --archive my_folder [email protected]:/home/sven/
+    ```
+
+## Copy a folder from Rackham to local
+
+Copy a folder from Rackham
+to your local computer.
+
+On your local computer, do:
+
+```
+rsync --recursive [user_name]@rackham.uppmax.uu.se:/home/[user_name]/[folder_name] [local_folder_destination]
+```
+
+For example:
+
+```
+rsync --recursive [email protected]:/home/sven/my_folder .
+```
+
+Where `.` means 'the folder where I am now'.
+
+???- question "Want to preserve timestamps?"
+
+    To preserve the files' timestamps, use the `--archive` flag, e.g.
+
+    ```
+    rsync --recursive --archive my_folder [email protected]:/home/sven/
+    ```