Merging in changes from main

docs/setup.rst

@@ -1,14 +1,14 @@
 =====
 Setup
 =====
-fre-cli is conda-installable from the “noaa-gfdl” anaconda channel (https://anaconda.org/NOAA-GFDL/fre-cli)
+FRE-cli is conda-installable from the “noaa-gfdl” anaconda channel (https://anaconda.org/NOAA-GFDL/fre-cli)
 and is deployed on GFDL systems as Environment Modules.
 
 On GFDL systems
 ========================
 If you are at GFDL (gaea, PP/AN, workstations), you may skip installation::
 
-  module load fre/2024.01
+  module load fre/2025.01
 
   fre --help
 
@@ -26,7 +26,7 @@ Once you have conda available, install the latest fre-cli from the NOAA-GFDL ana
 
 To install a specific version::
 
-  conda create --name fre-202401 --channel noaa-gfdl --channel conda-forge fre-cli::2024.01
+  conda create --name fre-202501 --channel noaa-gfdl --channel conda-forge fre-cli::2025.01
 
 and activate it::
 

docs/tools/cmor.rst

@@ -75,7 +75,7 @@ background
 ~~~~~~~~~~
 
 The bulk of this routine is housed in ``fre/cmor/cmor_mixer.py``, which is a rewritten version of
-Sergey Malyshev's original ``CMORcommander.py`` script, utilized during GFDL's CMIP6 publishing run.
+Sergey Nikonov's original ``CMORcommander.py`` script, utilized during GFDL's CMIP6 publishing run.
 
 This code is dependent on two primary json configuration files- a MIP
 variable table and another containing experiment (i.e. model) specific metdata (e.g. grid) to append

docs/usage.rst

@@ -1,7 +1,7 @@
 =============
 Usage
 =============
-Using a set of YAML configuration files, ``fre make`` compiles a FMS-based model, and ``fre pp`` postprocesses the history output and runs diagnostic analysis scripts. Please note that model running is not yet supported in FRE 2024; continue to use FRE Bronx frerun.
+Using a set of YAML configuration files, ``fre make`` compiles a FMS-based model, and ``fre pp`` postprocesses the history output and runs diagnostic analysis scripts. Please note that model running is not yet supported in FRE 2025; continue to use FRE Bronx frerun.
 
 YAML Framework
 ========================

docs/usage/yaml_framework.rst

@@ -1,4 +1,4 @@
-In order to utilize FRE 2024.01 tools, a distrubuted YAML structure is required. This framework includes a main model yaml, a compile yaml, a platforms yaml, and post-processing yamls. Throughout the compilation and post-processing steps, combined yamls that will be parsed for information are created. Yamls follow a dictionary-like structure with ``[key]: [value]`` fields. 
+In order to utilize these FRE tools, a distrubuted YAML structure is required. This framework includes a main model yaml, a compile yaml, a platforms yaml, and post-processing yamls. Throughout the compilation and post-processing steps, combined yamls that will be parsed for information are created. Yamls follow a dictionary-like structure with ``[key]: [value]`` fields. 
 
 Yaml Formatting
 ----------
@@ -53,14 +53,15 @@ Where each dash indicates a list.
 
   *ReusableVariable
 
-7. If the reusable variable must be combined with other strings, the **`!join`** constructor is used. Example: 
+7. If the reusable variable must be combined with other strings, the **`!join`** constructor is used. Simplified example: 
 
 .. code-block:: 
 
-  &version "2024.01"
-  &stem !join [FRE/, *version]
+  &name "experiment-name"
+  ...
+  pp_dir: !join [/archive/$USER/, *name, /, pp]
 
-In this example, the reuasble variable ``stem`` will be parsed as ``FRE/2024.01``.
+In this example, the variable ``pp_dir`` will be parsed as ``/archive/$USER/experiment-name/pp``.
 
 Model Yaml
 ----------

docs/what-is-fre.rst

@@ -4,7 +4,7 @@ What is FRE?
 
 FRE, the FMS Runtime Environment, is the companion runtime workflow for FMS-based climate and earth system models, and contains scripts and batch job handlers to compile models, run experiments, and postprocess and analyze the output. Developed around 2004 by GFDL's Modeling System Division, FRE was developed primarily in one repository ("fre-commands", https://github.com/NOAA-GFDL/FRE), used subtools in another repository (FRE-NCtools, https://github.com/NOAA-GFDL/fre-nctools), and was deployed using a set of Environment Modules (https://gitlab.gfdl.noaa.gov/fre/modulefiles). Originally, the major releases of FRE were rivers (Arkansas, Bronx) and the minor releases were numbers. In practice, though, the "Bronx" release name was retained and the number has been incremented over the years. e.g. Bronx-23 is the latest release.
 
-Over the last couple years, MSD's workflow team has reengineered the compiling and postprocessing parts of FRE, in a modern python and Cylc-based ecosystem (running experiments is not yet possible with this new FRE; stay tuned). Following a semantic versioning adopted in other FMS repositories, the reengineered FRE is versioned with a year and incrementing two-digit number. e.g. the first release of 2024 is 2024.01, the second 2024.02, and the first release next year will be 2025.01. (Optional minor releases are also available in the scheme; e.g. 2024.01.01 would be the first minor/patch release after 2024.01.) This version is used as tags in FRE repositories and in the corresponding conda (and in the future, container) release, and can be retrieved from ``fre --version``.
+Over the last couple years, MSD's workflow team has reengineered the compiling and postprocessing parts of FRE, in a modern python and Cylc-based ecosystem (running experiments is not yet possible with this new FRE; stay tuned). Following a semantic versioning adopted in other FMS repositories, the reengineered FRE is versioned with a year and incrementing two-digit number. e.g. the first release of 2025 is 2025.01, the second 2025.02, etc. (Optional minor releases are also available in the scheme; e.g. 2025.01.01 would be the first minor/patch release after 2025.01.) This version is used as tags in FRE repositories and in the corresponding conda (and in the future, container) release, and can be retrieved from ``fre --version``.
 
 fre-cli (this repository) can be considered a successor to the FRE Bronx “fre-commands” repository, which primarily contains user-facing tools and subtools. fre-workflows (https://github.com/NOAA-GFDL/fre-workflows) is a companion repository containing workflow definitions that can be run by the Cylc workflow engine. It contains workflow-specific elements previously in FRE Bronx, and allows flexibility to support multiple and more complex workflows. The two new FRE repositories are versioned with the same approach, and updates will be released together for some time to ensure compatibility.
 

fre/cmor/__init__.py

@@ -1,2 +1,3 @@
 ''' for fre.cmor imports '''
 from .cmor_mixer import cmor_run_subtool
+from .cmor_lister import cmor_list_subtool

fre/cmor/cmor_lister.py

@@ -0,0 +1,120 @@
+''' fre cmor list 
+because ian got tired of typing things like the following in bash...
+
+varname=sos; \
+table_files=$(ls fre/tests/test_files/cmip6-cmor-tables/Tables/CMIP6_*.json); \
+for table_file in $table_files; do \
+    echo $table_file; \
+    cat $table_file | grep -A 10 "\"$varname\""; \
+done;
+
+'''
+
+#import os
+import glob
+import json
+#import shutil
+#import subprocess
+from pathlib import Path
+
+import click
+
+DO_NOT_PRINT_LIST=[ 'comment',
+                    'ok_min_mean_abs', 'ok_max_mean_abs',
+                    'valid_min', 'valid_max' ]
+
+def print_var_content( table_config_file = None, var_name = None):
+    ''' one variable printing routine- looks for info regarding var_name in table_config_file '''
+    try:
+        proj_table_vars=json.load(table_config_file)
+    except Exception as exc:
+        raise Exception(f'problem getting proj_table_vars... WHY')
+
+    var_content = None
+    try:
+        var_content = proj_table_vars["variable_entry"].get(var_name)
+    except:
+        #print(f'(cmor_list_subtool) WARNING no "variable_entry" key. for {json_table_config}.'
+        #       '                    not the right json file probably. moving on!')
+        return
+
+    if var_content is None:
+        #print(f'(cmor_list_subtool) variable {var_name} not found in {Path(json_table_config).name}, moving on!')
+        return
+
+    table_name = None
+    try:
+        #print(f'(print_var_content) trying to get table_name from proj_table_vars...')
+        #print(f'                    table header is {proj_table_vars["Header"]}')
+        table_name = proj_table_vars["Header"].get('table_id').split(' ')[1]
+        #print(f'                    table_name = {table_name}')
+    except:
+        print(f'print_var_content) WARNING couldnt get header and table_name field')
+        pass
+
+    if table_name is not None:
+        print(f'(print_var_content) found {var_name} data in table {table_name}!')
+    else:
+        print(f'(print_var_content) found {var_name} data in table, but not its table_name!')
+
+    print(f'              variable key: {var_name}')
+    for content in var_content:
+        if content in DO_NOT_PRINT_LIST:
+            continue
+        print(f'              {content}: {var_content[content]}')        
+    print('\n')
+
+    return
+
+def cmor_list_subtool( json_var_list = None, json_table_config_dir = None, opt_var_name = None):
+    ''' 
+    finds tables in the CMIP json config directory containing variable data of interest. prints it 
+    out to screen, intended largely as a helper tool for cli users.
+    '''
+    if not Path(json_table_config_dir).exists():
+        raise OSError(f'(cmor_list_subtool) ERROR directory {json_table_config_dir} does not exist! exit.')
+
+    print(f'(cmor_list_subtool) attempting to find and open files in dir: \n {json_table_config_dir} ')
+    json_table_configs=glob.glob(f'{json_table_config_dir}/CMIP6_*.json')
+    if json_table_configs is None:
+        raise OSError(f'ERROR directory {json_table_config_dir} contains no JSON files, exit.')
+    else:
+        print(f'(cmor_list_subtool) found content in json_table_config_dir')#: {json_table_configs}')
+
+    var_list = None
+    if json_var_list is not None:
+        with open( json_var_list, "r", encoding = "utf-8") as var_list_file :
+            var_list=json.load(var_list_file)
+
+    if opt_var_name is None and var_list is None:
+        raise ValueError(f'(cmor_list_subtool) ERROR: no opt_var_name given but also no content in variable list!!! exit!')
+
+    if opt_var_name is not None:
+        print(f'(cmor_list_subtool) opt_var_name is not None: looking for only ONE variables worth of info!')
+        for json_table_config in json_table_configs:
+            #print(f'(cmor_list_subtool) attempting to open {json_table_config}')        
+            with open( json_table_config, "r", encoding = "utf-8") as table_config_file:
+                print_var_content(table_config_file, opt_var_name)
+
+    elif var_list is not None:
+        print(f'(cmor_list_subtool) opt_var_name is None, and var_list is not None, looking for many variables worth of info!')
+        for var in var_list:            
+            for json_table_config in json_table_configs:
+                #print(f'(cmor_list_subtool) attempting to open {json_table_config}')        
+                with open( json_table_config, "r", encoding = "utf-8") as table_config_file:
+                    #print(f'    var = {var}, var_list[{var}]={var_list[var]}')
+                    print_var_content(table_config_file, str(var_list[var]))
+    else:
+        print(f'(FATAL) this line should be unreachable!!!')
+
+    return
+
+
+@click.command()
+def _cmor_list_subtool( json_var_list = None, json_table_config_dir = None, opt_var_name = None):
+    ''' entry point to fre cmor run for click. see cmor_list_subtool for argument descriptions.'''
+    return cmor_list_subtool(json_var_list, json_table_config_dir, opt_var_name)
+
+
+if __name__ == '__main__':
+    cmor_list_subtool()