From e626369724aaa80c86bf3336a96737b39a2fd1fe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Michael=20J=C3=A4hn?= Date: Thu, 15 Aug 2024 16:23:27 +0200 Subject: [PATCH 1/2] Update Link Formatter (#187) * Check partly formatted links * Push md to trigger action * Restore argparse * Fix argparse * Push md again * Fix formatter * Test new formatter * GitHub Action: Apply external link format * Test again * Restore function * GitHub Action: Apply external link format * Fix and test * GitHub Action: Apply external link format * Fix and test * GitHub Action: Apply external link format * Fix and test * GitHub Action: Apply external link format * Fix formatter * GitHub Action: Apply external link format * Cleanup * Revert "GitHub Action: Apply external link format" This reverts commit b4e04fdaae5490489eb091a3f0d1eb473f148f4c. * GitHub Action: Apply external link format * Revert "GitHub Action: Apply external link format" This reverts commit 0652faf9623b079f20dedf9d9e9fe388fa0e24b0. * Only check if pattern appears * Fix pattern check * Trigger test * GitHub Action: Apply external link format --------- Co-authored-by: mjaehn --- docs/best_practices/coding.md | 20 +++++++++++-- scripts/format_external_links.py | 49 ++++++++++++++++++++++---------- 2 files changed, 51 insertions(+), 18 deletions(-) diff --git a/docs/best_practices/coding.md b/docs/best_practices/coding.md index a0dc5484..6d9d9345 100644 --- a/docs/best_practices/coding.md +++ b/docs/best_practices/coding.md @@ -1,8 +1,9 @@ # Coding -Whether you're new to coding or already working on it, there are two important things to remember: always **use version control** and **add automatic testing**. These steps will make things easier for you and everyone you work with. Also, try out tools that can make coding easier for you. +Whether you are new to coding or already working on it, there are two important things to remember: always **use version control** and **add automatic testing**. These steps will make things easier for you and everyone you work with. Also, try out tools that can make coding easier for you. ## Version Control with Git + **Git** is a popular tool for managing source code. C2SM repositories are hosted on GitHub, which uses Git as its version control system. If you're new to Git or want to improve your Git skills, we recommend attending our annual **Git for Beginners** and/or **Git for Advanced** courses. @@ -28,29 +29,42 @@ When working together with Git, follow these simple steps for a smooth collabora 6. **Merge Time**: Once the revisions are done and the tests are green, merge your feature branch into the main branch. ## Automatic Testing + Code testing is a critical step in software development. It's about finding and fixing problems to make sure the software works well, is of high quality and reliable. Testing involves checking different parts of the code to make sure the software is strong and free of bugs. The specific tests you need will depend on your project and its requirements. Here is a list of tests that are usually very useful. + ### Unit Tests + These tests are for testing individual components or functions of your code to ensure they work correctly in isolation. -Find an [example for unit tests :material-open-in-new:](https://github.com/C2SM/spack-c2sm/blob/main/test/unit_test.py){:target="_blank"} in our spack-c2sm repository. +Find an [example for unit tests :material-open-in-new:](https://github.com/C2SM/spack-c2sm/blob/main/test/unit_test.py){:target="_blank"} in our spack-c2sm repository. + ### Integrations Tests + These tests are to check how different parts of your code work together and communicate with each other. Find an [example for integration tests :material-open-in-new:](https://github.com/C2SM/spack-c2sm/blob/main/test/integration_test.py){:target="_blank"} in our spack-c2sm repository. + ### System Tests + These tests are performed to ensure that all the components and modules of a software system work together as intended and that the system meets its specified requirements and functions correctly in its operational environment. Find an [example for system tests :material-open-in-new:](https://github.com/C2SM/spack-c2sm/blob/main/test/system_test.py){:target="_blank"} in our spack-c2sm repository. + ### Tolerance tests + These tests are used in the development of ICON, specifically when code is ported from CPU to GPU. The results when running on CPU and GPU are not bit identical, therefore a tolerance range is accepted when comparing a test case to the CPU reference. The accepted tolerance range is created by running an ensemble of the same test case with different perturbations. MeteoSwiss has developed [probtest :material-open-in-new:](https://github.com/MeteoSwiss/probtest){:target="_blank"} for handling everything related to tolerance tests with ICON. If you have a DKRZ account and are working with ICON-NWP, you can also check out the manual for [Generating tolerances for non-standard tests :material-open-in-new:](https://gitlab.dkrz.de/icon/wiki/-/wikis/GPU-development/Validating-with-probtest-without-buildbot-references-(Generating-tolerances-for-non-standard-tests){:target="_blank"}). + ### Git Hooks & GitHub Actions + Git Hooks are local scripts in Git that make sure things get done right when you work on your code. GitHub Actions, on the other hand, are integrated with GitHub and allow you to automate code management workflows. They can be run automatically on GitHub whenever something is committed. Check out our Git course for examples of [Custom Git Hooks :material-open-in-new:](https://github.com/C2SM/git-course/blob/main/advanced/Exercise_7_git-hooks.md){:target="_blank"} hooks, and read GitHub's [documentation on GitHub Actions :material-open-in-new:](https://docs.github.com/en/actions){:target="_blank"}. - ## Useful tools for coding + Two popular tools for coding are [Visual Studio (VS) Code :material-open-in-new:](https://code.visualstudio.com){:target="_blank"} and [PyCharm :material-open-in-new:](https://www.jetbrains.com/pycharm/){:target="_blank"}. Here are instructions for setting up and using Visual Studio Code with SSH on a CSCS machine. + ### VS Code + 1. [Download :material-open-in-new:](https://code.visualstudio.com/download){:target="_blank"} and install VS Code on your computer. 2. Install extensions: - Open VS Code. diff --git a/scripts/format_external_links.py b/scripts/format_external_links.py index be8c0d07..7f3b7d24 100755 --- a/scripts/format_external_links.py +++ b/scripts/format_external_links.py @@ -3,27 +3,44 @@ import argparse def modify_link(line): + replaced = False + + # Define icons and attributes + icon_external_link = ':material-open-in-new:' + icon_download = ':material-download:' + open_new_tab = '{:target="_blank"}' + # Define patterns for general and download links general_pattern = r'\[([^\]]+)\]\((http[s]?://[^\s\)]+)\)' download_pattern = r'\[([^\]]+)\]\((https://polybox\.ethz\.ch/index\.php/s/[^\s\)]+)\)' # Define replacements for general and download links - general_replacement = r'[\1 :material-open-in-new:](\2){:target="_blank"}' - download_replacement = r'[\1 :material-download:](\2){:target="_blank"}' + general_replacement = r'[\1 ' + icon_external_link + r'](\2)' + download_replacement = r'[\1 ' + icon_download + r'](\2)' - # Check if the line was already modified or doesn't need modification - if ':material-open-in-new:' in line and '{:target="_blank"}' in line: - return line, False - if ':material-download:' in line and '{:target="_blank"}' in line: - return line, False + # Check if the general pattern appears in the line + if re.search(general_pattern, line): + # Check for link icon + if icon_external_link not in line and icon_download not in line: + new_line = re.sub(download_pattern, download_replacement, line) + if new_line != line: + line = new_line + replaced = True + else: + new_line = re.sub(general_pattern, general_replacement, line) + if new_line != line: + line = new_line + replaced = True - # Apply the appropriate replacement based on the URL pattern - if re.search(download_pattern, line): - modified_line, num_subs = re.subn(download_pattern, download_replacement, line) - else: - modified_line, num_subs = re.subn(general_pattern, general_replacement, line) + # Check for new tab attribute + if open_new_tab not in line: + new_line = re.sub(r'(\[.*?\]\(.*?\))', r'\1' + open_new_tab, line) + if new_line != line: + line = new_line + replaced = True + + return line, replaced - return modified_line, num_subs > 0 def process_markdown_file(file_path): with open(file_path, 'r+', encoding='utf-8') as file: @@ -34,11 +51,13 @@ def process_markdown_file(file_path): if changed: modified = True lines[i] = new_line + print(f"Modifying line {i+1} in file: {file_path}") if modified: file.seek(0) file.writelines(lines) file.truncate() - print(f"Modified: {file_path}") + print(f"File modified: {file_path}") + def main(start_path): for root, dirs, files in os.walk(start_path): @@ -51,4 +70,4 @@ def main(start_path): parser.add_argument('-p', '--path', default=os.getcwd(), help='Base path to search for markdown files. Defaults to current working directory.') args = parser.parse_args() - main(args.path) \ No newline at end of file + main(args.path) From 4fb38b63ccc4beaec5806ef0010bcdb8588cfdab Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Michael=20J=C3=A4hn?= Date: Sun, 18 Aug 2024 04:22:10 +0200 Subject: [PATCH 2/2] GitHub Action: Update datasets (#188) Co-authored-by: github-actions[bot] --- docs/datasets/climate_model_data.md | 12 ++++++------ docs/datasets/obs_reanalysis_data.md | 16 ++++++++-------- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/datasets/climate_model_data.md b/docs/datasets/climate_model_data.md index bb0dad6f..8c227e31 100644 --- a/docs/datasets/climate_model_data.md +++ b/docs/datasets/climate_model_data.md @@ -111,8 +111,8 @@ /nfs/atmos/c2sm/cmip6-ng ``` -- Size: 200.63 TB :material-information-outline:{ title="last updated: 2024-08-11 01:47:43" } -- Number of files: 663,802 :material-information-outline:{ title="last updated: 2024-08-11 01:47:43" } +- Size: 200.63 TB :material-information-outline:{ title="last updated: 2024-08-18 01:47:09" } +- Number of files: 663,802 :material-information-outline:{ title="last updated: 2024-08-18 01:47:09" } - Access: direct / rsync - Status: frozen (2019-03) - Variables: @@ -195,8 +195,8 @@ /store/c2sm/c2sme/cordex ``` -- Size: 356.97 TB :material-information-outline:{ title="last updated: 2024-08-11 01:48:16" } -- Number of files: 552,595 :material-information-outline:{ title="last updated: 2024-08-11 01:48:16" } +- Size: 356.97 TB :material-information-outline:{ title="last updated: 2024-08-18 01:48:54" } +- Number of files: 552,595 :material-information-outline:{ title="last updated: 2024-08-18 01:48:54" } - Access: direct / rsync - Status: monthly updated - Resolution: 0.44° and 0.11° @@ -211,8 +211,8 @@ ```console /nfs/atmos/c2sm/cordex-reklies ``` -- Size: 22.41 TB :material-information-outline:{ title="last updated: 2024-08-11 01:45:25" } -- Number of files: 94,936 :material-information-outline:{ title="last updated: 2024-08-11 01:45:25" } +- Size: 22.41 TB :material-information-outline:{ title="last updated: 2024-08-18 01:45:22" } +- Number of files: 94,936 :material-information-outline:{ title="last updated: 2024-08-18 01:45:22" } - Access: direct - Status: monthly updated - Resolution: 0.11° diff --git a/docs/datasets/obs_reanalysis_data.md b/docs/datasets/obs_reanalysis_data.md index 3cbbcef8..abb77b34 100644 --- a/docs/datasets/obs_reanalysis_data.md +++ b/docs/datasets/obs_reanalysis_data.md @@ -157,8 +157,8 @@ /net/atmos/data/cerra/processed/v1/ ``` -- Size: 7.72 TB :material-information-outline:{ title="last updated: 2024-08-11 01:44:39" } -- Number of files: 1,824 :material-information-outline:{ title="last updated: 2024-08-11 01:44:39" } +- Size: 7.72 TB :material-information-outline:{ title="last updated: 2024-08-18 01:44:42" } +- Number of files: 1,824 :material-information-outline:{ title="last updated: 2024-08-18 01:44:42" } - Access: direct - Status: updated - Time period: 1985-2020 @@ -199,8 +199,8 @@ /net/atmos/data/cerra-land/processed/v1/ ``` -- Size: 1.78 TB :material-information-outline:{ title="last updated: 2024-08-11 01:44:39" } -- Number of files: 1,016 :material-information-outline:{ title="last updated: 2024-08-11 01:44:39" } +- Size: 1.78 TB :material-information-outline:{ title="last updated: 2024-08-18 01:44:42" } +- Number of files: 1,016 :material-information-outline:{ title="last updated: 2024-08-18 01:44:42" } - Access: direct - Status: updated - Time period: 1985-2020 @@ -253,8 +253,8 @@ /nfs/atmos/c2sm/era5/processed/v2 ``` -- Size: 45.76 TB :material-information-outline:{ title="last updated: 2024-08-11 01:45:18" } -- Number of files: 138,400 :material-information-outline:{ title="last updated: 2024-08-11 01:45:18" } +- Size: 45.76 TB :material-information-outline:{ title="last updated: 2024-08-18 01:45:24" } +- Number of files: 138,400 :material-information-outline:{ title="last updated: 2024-08-18 01:45:24" } - Access: direct - Status: updated - Time period: v1: 1940-present, v2: 1980-present @@ -294,8 +294,8 @@ /net/atmos/data/era5-land_cds/processed/v1/ ``` -- Size: 6.64 TB :material-information-outline:{ title="last updated: 2024-08-11 01:44:40" } -- Number of files: 1,504 :material-information-outline:{ title="last updated: 2024-08-11 01:44:40" } +- Size: 6.64 TB :material-information-outline:{ title="last updated: 2024-08-18 01:44:44" } +- Number of files: 1,504 :material-information-outline:{ title="last updated: 2024-08-18 01:44:44" } - Access: direct - Status: updated - Time period: 1950-present