Making code changes in the UFS weather model and its subcomponents

UFS-Model Code Development

Introduction

The development process for the UFS-Model has been structured to make best use of the tools git/github provides for code development. Here is a brief summary of whats included in this page:

• Intro
  • Build Structure
  • Repository Structure
• Obtaining code and committing changes
  • Forking
  • Cloning and setup
  • Committing
• Developing in subcomponents
  • Bringing in your changes from subcomponents
• Regression Testing
  • Steps for proper regression testing
• Pushing and pull requests
  • Setting up remotes
  • Pushing Changes
  • Creating a Pull Request
• Specifics for NOAA Staff

Build structure and build tools

The UFS uses a CMake build environment which parallels well with the hierarchical code structure used in the repository.

• UFS Structure:
  • Model build files:
    • CMakeLists.txt -- Top level build instructions for CMake.
    • cmake/ -- Files used by CMake in building UFS.
  • System setup:
    • conf/ -- model specific flag and variables for multiple machine types.
    • modulefiles/ -- machine specific variables for build.
  • Testing:
    • tests/* -- Regression testing files.
  • GIT specific files:
    • .gitmodules -- Defines the submodules needed for proper build of UFS.

Repository Structure

GIT version control using GIT submodules for UFS subcomponents

UFS-Weather-Model Submodules

• FMS
• NEMS
  • NCEPLIBS-pyprodutil
• fv3atm
  • GFDL_atmos_cubed_sphere
  • ccpp-framework
  • ccpp-physics
• stochastic_physics
• WW3
• MOM6-interface
  • MOM6
• CICE-interface
  • CICE6
• CMEPS-interface
  • CMEPS

Notes

• The model and its subcomponents are housed in authoritative (official) repositories. The UFS weather model uses git submodules to manage its subcomponents.
• All the authoritative repositories are read-only for users.
• Branches under the authoritative repositories include main develop (master) branch, production branches, and public release branches.
• Branches under the authoritative repositories are protected.

Obtaining UFS code (Using ufs-weather-model repository as an example)

Before you start:

*If you do not have a GitHub account, please get one at github.org. EMC staff can also refer to Kate Friedman’s document: “EMC GitHub Transition”:

• Set up a Github account
• Get added to the NOAA-EMC org people list

Forking the authoritative repository

• Why Choose Forking?
  • A fork is a copy of a repository. You manage your own fork.
  • Forks let you make changes to a project without affecting the original repository.
  • You can fetch updates from the original repository.
  • You can submit changes to the original repository with pull requests.
  • For more details, please see Working with forks.

Cloning and setup of local repository

• Create a fork via github:
  • Login to your personal GitHub account
  • Go to the official code repository website (e.g. https://github.com/ufs-community/ufs-weather-model), click on “fork” on the top right corner. You will see the repository in your own GitHub account.
• Cloning your fork:

  git clone https://github.com/<your_github_username>/ufs-weather-model
  cd ufs-weather-model

• Checking out the develop branch:

  git checkout develop

  • NOTE: You can check which branch you are using at any time by typing git branch in the top level of your repository.
• Retrieve submodule files:

  git submodule update --init --recursive

  • NOTE: --init will initialize any submodule you have not already initialized.
  • NOTE: --recursive will make sure to get submodules inside the submodules listed in your .gitmodules file.

Making changes & Committing

• Creating a feature branch to make changes in:

  git checkout -b feature/newfeaturename

  • NOTE: We typically use feature and bugfix (i.e. feature/addfunction or bugfix/missingtags)
  • You can confirm your using the new branch name by typing git branch in the top level of your repository
• Make your change or edit or fix.
• After each change or edit or fix:

  git commit -m "Type an explanation of the change or edit or fix here"

  • NOTE: Committing after each change or edit or fix gives the code maintainers a very detailed understanding of what you're doing after you submit your pull request.

Developing in subcomponents

While developing in subcomponents, the process of forking into your account, cloning it, committing and pushing is identical to what is represented here. The steps that follow pertain to what needs to be done inside your forked ufs-weather-model reposotiry to bring in the changes you have made to subcomponents.

Bringing in your changes from subcomponents

• If you haven't cloned your fork yet using git clone:
  • On your fork in GitHub edit your .gitmodules file.
  • If you made changes in fv3atm that you want to bring in to test in ufs-weather-model you would edit:

    [submodule "FV3"]
    path = FV3
    url = https://github.com/NOAA-EMC/fv3atm
    branch = develop
    

  • such that you comment out the url and branch lines and add the appropriate information from your personal fork of fv3atm to look like this:

    [submodule "FV3"]
    path = FV3
    #url = https://github.com/NOAA-EMC/fv3atm
    #branch = develop
    url = https://github.com/<your_github_username>/fv3atm
    branch = <your branch name>
    

  • then clone, checkout and submodule update as previously instructed.
• If you have already used clone and submodule update locally:
  • You should update your .gitmodules file as instructed above.
  • cd into your FV3: cd FV3
  • update your remotes

    git remote rename origin upstream
    git remote add origin https://github.com/<your_github_username>/fv3atm
    

  • checkout your branch

    git fetch origin
    git checkout origin/<your_branch_name>
    

    • NOTE: You can verify your branch by typing in git branch.

Regression Testing

• Regression testing is your way to prove to us that the changes you are proposing through a pull request do not change the build or operation of the model in a negative way. When development work is done, users need to sync their feature branch with the latest develop/master branch in official repository and run the regression tests:

  setenv NEMS_COMPILER GNU
  cd tests
  ./rt.sh -fe -l rt_gnu.conf
  

  • NOTE: This example is for those that are using the GNU compiler.
• For an in-depth tutorial on Regression Testing please see the Regression Testing section.

Pushing code and pull requests

• Once all of your commits have been completed and you've passed all the regression tests (if applicable) you will need to push those changes to your fork on github inside the branch you created

Setting up remotes

• First you need to setup your remotes to add the authoritative repository:

  git remote add upstream https://github.com/ufs-community/ufs-weather-model

• Next you should confirm it was added to your remotes by issuing the command:

  git remote -v

  • You should see something like:

  upstream        https://github.com/ufs-community/ufs-weather-model (fetch)  
  upstream        https://github.com/ufs-community/ufs-weather-model (push)  
  origin  https://github.com/your_github_account/ufs-weather-model (fetch)  
  origin  https://github.com/your_github_account/ufs-weather-model (push)

Pushing Changes

• Push all your commits to your fork.

  git push origin feature/newfeaturename

  • NOTE: Make sure you choose the remote of origin as that is where your personal fork of the authoritative repository is.
  • NOTE: Make sure to match the branch name you created earlier (i.e. feature/newfeaturename).

Creating a Pull Request

Our methodology

• Pull requests (PR) tells people that your code changes from your feature branch are ready to be merged into a repository (develop branch in official repository)
• An issue needs to be created associated with the pull request. The issue should cover all the features in detail.
• Before making a pull request, users’ feature branch should be merged with the corresponding branch in the authoritative repository. All the conflicts should be resolved. Regression test needs to be passed on at least one of the supported platforms.
• Users submit a pull request (PR) for code commit. If PRs for several repositories are cross-referenced in the description of the PR, and a note needs to be placed in each PR.
• Code manager(s) will run the regression test on the required platforms that users don’t have access to.
• Code managers will add reviewers.
• At least one reviewer with write permission needs to give approval before the code can be committed.

Steps

• You need to make a pull request for each of the repository that you changed in the .gitmodule. You can start from the subcomponent repository, then make PR to each repository until the UFS application repository
• To make a PR, first go to the your_github_account repository, choose your branch, then click on “new pull request”.
• In the next page, choose base repository and base branch. By default, pull requests are based on the parent repository default branch.
• Write down a title and some in the “write comments” description for your pull request in the “write” box. If you know the PR number of other associated repository in your commit, add them in the “write” box.
• click on “Create Pull Request” button.
• Making PR for other repositories in your commit. Remember to write down all the associated PRs in each of the PR. To do this, go to your PR in the official repository. In the box with title “your_github_account commented on…”, on the right, there is a button with “...”, click on it, and choose “edit”, at the end of the description, add” “associated PRs”, then list all the PRs.

Code review

• Reviews allow collaborators to comment on the changes proposed in pull requests, approve the changes, or request further changes before the pull request is merged.
• Anyone with read access can review and comment on the changes it proposes.
• Repository collaborators can assign a pull request review to any person that has been explicitly granted read access to a user-owned repository. * *
• Organization members can assign a pull request review to any person or team with read access to a repository.
• To assign reviewers:
  • Go to the PR from repository GitHub webpage, click on “Reviewers” on the right sidebar, type in the GitHub account of the reviewer.
• Review a PR
  • Go to the PR from repository GitHub webpage, click on “files changed”.
  • To comment the line of code, and click the blue comment icon in front of the line. To add a comment on multiple lines, click and drag to select the range of lines, then click the blue comment icon.
  • In the comment field, type your comment.
• Marking a file as viewed
  • After you finish reviewing a file, click on “viewed” on the right of the header of the file. The file will collapse, and it will be unmarked as viewed
• Submit code review
  • After reviewing the files in the PR, your can submit your review.
  • On the pull request, click on “Files changed”
  • Above the changed code, click Review changes
  • Type a comment summarizing your feedback on the proposed changes.

Specifics for NOAA staff

• Add specific configurations or setup steps for NOAA staff that use NOAA HPC systems

Questions/suggestions

Please create an issue at: https://github.com/ufs-community/ufs-weather-model/issues if you have any questions/suggestions.