Merge branch 'main' into temporary_todi

.github/workflows/check_markdown_links.yml

@@ -1,4 +1,4 @@
-name: Check Markdown links
+name: Check Markdown Links
 
 on:
   push:
@@ -10,8 +10,10 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@master
+
     - name: Check for broken links (internal)
-      run: python check_md_links.py --path ./docs
+      run: python ./scripts/check_md_links.py --path ./docs
+
     - name: Check for broken links (external)
       run: |
         pip install mkdocs-linkcheck

.github/workflows/format_external_links.yml

@@ -22,7 +22,7 @@ jobs:
           python-version: '3.x'
 
       - name: Run Link Formatter Script
-        run: python format_external_links.py --path ./docs
+        run: python ./scripts/format_external_links.py --path ./docs
 
       - name: Commit changes
         uses: stefanzweifel/git-auto-commit-action@v4

.github/workflows/update_datasets.yml

@@ -0,0 +1,54 @@
+name: Update Datasets and Create PR
+
+on:
+  schedule:
+    - cron: '0 2 * * *' # Runs at 02:00 UTC every day
+  workflow_dispatch: # This line enables manual triggering of the workflow
+
+jobs:
+  update-datasets:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+        with:
+          fetch-depth: 0 # Fetches all history for all branches and tags
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.x' # Replace with the version you need
+
+      - name: Install dependencies
+        run: pip install requests pyyaml
+
+      - name: Run update_datasets.py
+        run: python ./scripts/update_datasets.py
+
+      - name: Check for changes
+        id: git-check
+        run: |
+          git diff
+          if [ -n "$(git status -s)" ]; then
+            echo "Changes detected"
+            echo "::set-output name=changes_detected::true"
+          fi
+
+      - name: Commit changes
+        if: steps.git-check.outputs.changes_detected == 'true'
+        run: |
+          git config --global user.name 'github-actions[bot]'
+          git config --global user.email 'github-actions[bot]@users.noreply.github.com'
+          git add .
+          git commit -m "GitHub Action: Update datasets"
+
+      - name: Create Pull Request
+        if: steps.git-check.outputs.changes_detected == 'true'
+        uses: peter-evans/create-pull-request@v4
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+          commit-message: "GitHub Action: Update datasets"
+          title: "[AUTO] Update Datasets"
+          body: "This is an auto-generated PR with dataset updates."
+          branch: "update-datasets-${{ github.run_number }}"
+          delete-branch: true

datasets.yaml

@@ -0,0 +1,7 @@
+cerra: CERRA
+cerra-land: CERRA-Land
+cmip6-ng: CMIP6-ng
+cordex: CORDEX
+cordex-reklies: CORDEX-ReKliEs
+era5_cds: ERA5
+era5-land: ERA5-Land

docs/blog/posts/2024-07-12_switching_to_Alps.md → docs/blog/posts/2024-07-02_switching_to_Alps.md

@@ -5,7 +5,7 @@ date:
 
 # Switching to Alps
 
-As you probably all know the new [Alps system :material-open-in-new:](https://www.cscs.ch/computers/alps) is currently being set up at CSCS. With this machine, CSCS introduced major changes, of course in the hardware, but also in the user experience.
+As you probably all know the new [Alps system :material-open-in-new:](https://www.cscs.ch/computers/alps){:target="_blank"} is currently being set up at CSCS. With this machine, CSCS introduced major changes, of course in the hardware, but also in the user experience.
 
 Piz Daint being scheduled to be decommissioned at the end of August, we are organizing a workshop to guide our users in this transition on **August, 12th from 10:00 to 12:00** at **ETHZ in CHN-L17.1** and over zoom at **https://ethz.zoom.us/j/63402117251.**
 

docs/datasets/climate_model_data.md

@@ -9,33 +9,45 @@
     "Resolution: native" means that each model output is available on the native grid on which it was run. Size and file information last updated in June 2021.
 
 ### CMIP2
+
+- Location: IAC
 - Size: 0.01 TB 
 - Number of files: 500
-- Location: IAC
 - Access: direct / rsync
 - Status: frozen (2010-10)
 - Resolution: native
 
 ### CMIP3
-- Size: 6 TB
-- Number of files: 40’000
+
 - Location: IAC
+- Size: 6 TB
+- Number of files: 40.000
 - Access: direct / rsync
 - Status: frozen (2016-12)
 - Resolution: native
 
 ### CMIP5
-- Size: 130 TB
-- Number of files: 700’000
+
 - Location: IAC
+- Size: 130 TB
+- Number of files: 700.000
 - Access: direct / rsync
 - Status: monthly updated
 - Resolution: native
 
 ### CMIP6
+
+=== "IAC"
+    ```console
+    /nfs/atmos/c2sm
+    ```
+=== "Euler"
+    ```console
+    /nfs/atmos/c2sm
+    ```
+
 - Size: 520 TB 
-- Number of Files: 5’500’000
-- Location: IAC, Euler: `/nfs/atmos/c2sm`
+- Number of Files: 5.500.000
 - Access: direct / rsync
 - Status: weekly updated
 - Resolution: native
@@ -46,35 +58,101 @@
     The datasets are checked, standardized, and regridded to common grid.
 
 !!! note
-    Next Generation (ng) archives were maintained by Reto’s group (Jan Sedlacek/Lukas Brunner). Ruth Lorenz / C2SM took over cmip6-ng in 2022. Contact: [[email protected]](mailto:[email protected]). Documentation: [https://doi.org/10.5281/zenodo.373412 :material-open-in-new:](https://doi.org/10.5281/zenodo.3734128){:target="_blank"}.
+    Next Generation (ng) archives were maintained by Reto's group (Jan Sedlacek/Lukas Brunner). Ruth Lorenz / C2SM took over cmip6-ng in 2022. Contact: [[email protected]](mailto:[email protected]). Documentation: [https://doi.org/10.5281/zenodo.373412 :material-open-in-new:](https://doi.org/10.5281/zenodo.3734128){:target="_blank"}.
 
 ### CMIP3-ng
-- Size: 0.5 TB
-- Number of files: 5’000
+
 - Location: IAC
+- Size: 0.5 TB
+- Number of files: 5.000
 - Access: direct / rsync
 - Status: frozen (2019-03)
 - Variables: n/a
 - Resolution: native and 2.5°x2.5°
 
 ### CMIP5-ng
-- Size: 33 TB
-- Number of files: 100’000
+
 - Location: IAC
+- Size: 33 TB
+- Number of files: 100.000
 - Access: direct / rsync
 - Status: frozen (2019-03)
 - Variables: n/a
 - Resolution: native and 2.5° x 2.5°
 
 ### CMIP6-ng
+
+=== "IAC"
+    ```console
+    /nfs/atmos/c2sm
+    ```
+=== "Euler"
+    ```console
+    /nfs/atmos/c2sm
+    ```
+
 - Size: 175 TB
-- Number of files: 500’000
-- Location: IAC, Euler: `/nfs/atmos/c2sm`
+- Number of files: 500.000
 - Access: direct / rsync
 - Status: frozen (2019-03)
-- Variables: n/a
-    - monthly: `areacella`, `clt`, `co2mass`, `evspsbl`, `evspsblsoi`, `evspsblveg`, `gpp`, `hfds`, `hfss`, `hfls`, `hurs`, `huss`, `lai`, `mrsol`, `mrsos`, `mrro`, `mrros`, `mrso`, `npp`, `nbp`, `pr`, `prw`, `ra`, `rh`, `rlus`, `rlut`, `rlutcs`, `rlds`, `rldscs`, `rsds`, `rsdscs`, `rsus`, `rsuscs`, `rsut`, `rsutcs`, `rtmt`, `sftlf`, `siconc`, `tas`, `tasmax`, `tasmin`, `tos`, `tran`, `treeFrac`, `zg500`, `zos`
-    - daily: `mrro`, `pr`, `tas`, `tasmax`, `tasmin`, `zg500`
+- Variables: 
+  `areacella`{ title="fx (native)" },
+  `clt`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `co2mass`{ title="mon (native)" },
+  `evspsbl`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `evspsblsoi`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `evspsblveg`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `gpp`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `hfds`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `hfls`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `hfss`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `hurs`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `huss`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `lai`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `masks`{ title="fx (native)" },
+  `mrro`{ title="ann (g025), ann (native), day (g025), day (native), mon (g025), mon (native)" },
+  `mrros`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `mrso`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `mrsol`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `mrsos`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `nbp`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `npp`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `pr`{ title="ann (g025), ann (native), day (g025), day (native), mon (g025), mon (native)" },
+  `prw`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `ps`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `psl`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `ra`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `rh`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `rlds`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `rldscs`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `rlus`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `rlut`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `rlutcs`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `rsds`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `rsdscs`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `rsdt`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `rsus`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `rsuscs`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `rsut`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `rsutcs`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `rtmt`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `sfcWind`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `sftlf`{ title="fx (native)" },
+  `siconc`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `ta`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `tas`{ title="ann (g025), ann (native), day (g025), day (native), mon (g025), mon (native)" },
+  `tasmax`{ title="ann (g025), ann (native), day (g025), day (native), mon (g025), mon (native)" },
+  `tasmin`{ title="ann (g025), ann (native), day (g025), day (native), mon (g025), mon (native)" },
+  `tauu`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `tauv`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `tos`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `tran`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `treeFrac`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `tsl`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `uas`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `vas`{ title="ann (g025), ann (native), mon (g025), mon (native)" },
+  `zg500`{ title="ann (g025), ann (native), day (g025), day (native), mon (g025), mon (native)" },
+  `zos`{ title="ann (g025), ann (native), mon (g025), mon (native)" }
 - Resolution: native and 2.5° x 2.5°
 
 ## Raw CORDEX
@@ -83,17 +161,31 @@
     The data is available as downloaded.
 
 ### CORDEX
+
+=== "IAC"
+    ```console
+    /nfs/atmos/c2sm
+    ```
+=== "Euler"
+    ```console
+    /nfs/atmos/c2sm
+    ```
+=== "CSCS"
+    ```console
+    /store/c2sm/c2sme
+    ```
+
 - Size: 370 TB
-- Number of files: 800’000
-- Location: IAC, Euler: `/nfs/atmos/c2sm`, CSCS: `/store/c2sm/c2sme`
+- Number of files: 800.000
 - Access: direct / rsync
 - Status: monthly updated
 - Resolution: 0.44° and 0.11°
 
 ### CORDEX-ReKliEs
-- Size: 23 TB
-- Number of files: 100’000
+
 - Location: IAC, Euler
+- Size: 23 TB
+- Number of files: 100.000
 - Access: direct
 - Status: monthly updated
 - Resolution: 0.11°
@@ -104,29 +196,88 @@
     The data is checked, and regridded to identical grids if necessary.
 
 ### CORDEX.ch2018
-- Size: 11 TB
-- Number of files: 1’800
+
 - Location: IAC, CSCS
+- Size: 11 TB
+- Number of files: 1.800
 - Access: direct
 - Status: frozen (2019-04)
-- Variables:
-    - daily: `hurs`, `huss`, `pr`, `rsds`, `sfcWind`, `sfcWindmax`, `snw`, `tas`, `tasmax`, `tasmin`
+- Variables: 
+  `hfls`{ title="historical: 3hr, day, mon, sem — rcp26: 3hr, day, mon, sem — rcp85: 3hr, day, mon, sem" },
+  `hfss`{ title="historical: 3hr, day, mon, sem — rcp26: 3hr, day, mon, sem — rcp85: 3hr, day, mon, sem" },
+  `hurs`{ title="historical: 3hr, day, mon, sem — rcp26: 3hr, day, mon, sem — rcp85: 3hr, day, mon, sem" },
+  `huss`{ title="historical: 3hr, day, mon, sem — rcp26: 3hr, day, mon, sem — rcp85: 3hr, day, mon, sem" },
+  `pr`{ title="historical: 3hr, day, mon, sem — rcp26: 3hr, day, mon, sem — rcp85: 3hr, day, mon, sem" },
+  `prc`{ title="historical: 3hr, day, mon, sem — rcp26: 3hr, day, mon, sem — rcp85: 3hr, day, mon, sem" },
+  `ps`{ title="historical: 3hr, day, mon, sem — rcp26: 3hr, day, mon, sem — rcp85: 3hr, day, mon, sem" },
+  `psl`{ title="historical: 3hr, day, mon, sem — rcp26: 3hr, day — rcp85: 3hr, day, mon, sem" },
+  `rlds`{ title="historical: 3hr, day, mon, sem — rcp26: 3hr, day, mon, sem — rcp85: 3hr, day, mon, sem" },
+  `rlus`{ title="historical: 3hr, day, mon, sem — rcp26: 3hr, day, mon, sem — rcp85: 3hr, day, mon, sem" },
+  `rsds`{ title="historical: 3hr, day, mon, sem — rcp26: 3hr, day, mon, sem — rcp85: 3hr, day, mon, sem" },
+  `rsus`{ title="historical: 3hr, day, mon, sem — rcp26: 3hr, day, mon, sem — rcp85: 3hr, day, mon, sem" },
+  `sfcWind`{ title="historical: 3hr, day, mon, sem — rcp26: 3hr, day, mon, sem — rcp85: 3hr, day, mon, sem" },
+  `sund`{ title="historical: 3hr, day, mon, sem — rcp26: 3hr, day, mon, sem — rcp85: 3hr, day, mon, sem" },
+  `tas`{ title="historical: 3hr, day, mon, sem — rcp26: 3hr, day, mon, sem — rcp85: 3hr, day, mon, sem" },
+  `evspsbl`{ title="historical: 6hr, day, mon, sem — rcp26: 6hr, day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `hus850`{ title="historical: 6hr, day, mon, sem — rcp26: day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `mrro`{ title="historical: 6hr, day, mon, sem — rcp26: 6hr, day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `mrso`{ title="historical: 6hr, day, mon, sem — rcp26: 6hr, day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `rlut`{ title="historical: 6hr, day, mon, sem — rcp26: 6hr, day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `rsdt`{ title="historical: 6hr, day, mon, sem — rcp26: 6hr, day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `rsut`{ title="historical: 6hr, day, mon, sem — rcp26: 6hr, day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `snd`{ title="historical: 6hr, day, mon, sem — rcp26: 6hr, day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `snw`{ title="historical: 6hr, day, mon, sem — rcp26: 6hr, day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `ta200`{ title="historical: 6hr, day, mon, sem — rcp26: day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `ta500`{ title="historical: 6hr, day, mon, sem — rcp26: day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `ta850`{ title="historical: 6hr, day, mon, sem — rcp26: day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `ts`{ title="historical: 6hr, day, mon, sem — rcp26: 6hr, day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `ua500`{ title="historical: 6hr, day, mon, sem — rcp26: day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `ua850`{ title="historical: 6hr, day, mon, sem — rcp26: day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `uas`{ title="historical: 6hr, day, mon, sem — rcp26: 6hr, day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `va500`{ title="historical: 6hr, day, mon, sem — rcp26: day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `va850`{ title="historical: 6hr, day, mon, sem — rcp26: day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `vas`{ title="historical: 6hr, day, mon, sem — rcp26: 6hr, day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `zg500`{ title="historical: 6hr, day, mon, sem — rcp26: day, mon, sem — rcp85: 6hr, day, mon, sem" },
+  `clt`{ title="historical: day, mon, sem — rcp26: day — rcp85: day, mon, sem" },
+  `prhmax`{ title="historical: day — rcp85: day" },
+  `prsn`{ title="historical: day, mon, sem — rcp26: day, mon, sem — rcp85: day, mon, sem" },
+  `sfcWindmax`{ title="historical: day, mon, sem — rcp26: day, mon, sem — rcp85: day, mon, sem" },
+  `snc`{ title="historical: day, mon, sem — rcp26: day, mon, sem — rcp85: day, mon, sem" },
+  `tasmax`{ title="historical: day, mon, sem — rcp26: day, mon, sem — rcp85: day, mon, sem" },
+  `tasmin`{ title="historical: day, mon, sem — rcp26: day, mon, sem — rcp85: day, mon, sem" },
+  `wsgsmax`{ title="historical: day — rcp26: day — rcp85: day" },
+  `orog`{ title="historical: fx — rcp26: fx — rcp85: fx" },
+  `sftlf`{ title="historical: fx — rcp26: fx — rcp85: fx" }
 - Resolution: 0.44° and 0.11°
 
 ### CORDEX.ch2025
+
+=== "IAC"
+    ```console
+    /net/ch4/data/cordex.ch2025/
+    ```
+
 - Size: 46 TB 
-- Number of files: 3’500
-- Location: IAC: `/net/ch4/data/cordex.ch2025/`
+- Number of files: 3.500
 - Access: direct
 - Status: ongoing
 - Variables: 
     - daily: `pr`, `tas`, `tasmax`, `tasmin`, `zg500`
 - Resolution: 0.44° and 0.11°
 
 ### CORDEX-FPSCONV
+
+=== "IAC"
+    ```console
+    /net/krypton/hyclimm_nobackup/CORDEX-FPSCONV/ALP-3/
+    ```
+=== "CSCS"
+    ```console
+    /store/c2sm/c2sme/CH202X/CORDEX-FPSCONV/ALP-3/
+    ```
+
 - Size: 30 TB
-- Number of files: 22'949
-- Location: CSCS: `/store/c2sm/c2sme/CH202X/CORDEX-FPSCONV/ALP-3/`, IAC: `/net/krypton/hyclimm_nobackup/CORDEX-FPSCONV/ALP-3/`
+- Number of files: 22.949
 - Access: direct / rsync
 - Status: ongoing
 - Variables: