Varlist

.github/workflows/mopper-conda.yaml

@@ -1,14 +1,14 @@
-name: xmhw-conda-install-test
+name: mopper-conda-install-test
 
 #on: [push]
 on: 
   push:
     branches:
-      - main
-      - newrelease
+      - prerelease
   pull_request:
     branches:
       - main
+      - prerelease
 
 
 jobs:
@@ -38,8 +38,8 @@ jobs:
         # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
         flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
         #    - name: Install package 
-      #      run: |
-              #        conda run python setup.py install
+            run: |
+                      conda build conda/meta.yaml
     - name: Test with pytest
       run: |
         conda install pytest coverage codecov

conda/meta.yaml

@@ -1,4 +1,4 @@
-{% set version = "0.6.1" %}
+{% set version = "1.0.0" %}
 package:
     name: mopper 
     version: {{ version }}

docs/gettingstarted.rst

@@ -3,46 +3,38 @@ Starting with MOPPeR
 
 A typical workflow to post-process an ACCESS or UM model output requires three steps.
 
-Step1: get a list of variables from the raw output
---------------------------------------------------
-
-     *mopdb varlist -i <path-to-raw-output> -d <date-pattern>*
-
-`mopdb varlist` will output one or more `csv` files with a detailed list of variables, one list for each pattern of output files.
-
-.. code-block:: console
-
-   $ mopdb varlist -i /scratch/../exp -d 20120101
-   Opened database ~/.local/lib/python3.10/site-packages/data/access.db successfully
-   Variable list for ocean_scalar.nc- successfully written
-   Variable list for ocean_month.nc- successfully written
-   Variable list for ocean_daily.nc- successfully written
-
-.. csv-table:: Example of varlist output 
-   :file: varlist_example.csv
-   :delim: ;
-
-The <date-pattern> argument is used to reduce the number of files to check. The tool will recognise anyway a repeated pattern and only add a list of variable for the same pattern once.
-
 
-Step2: create a template for a mapping file
+Step1: create a template for a mapping file
 -------------------------------------------
 
-   *mopdb template -i <varlist.csv> -v <access-version> -a <alias>*
+   *mopdb template -f <path-to-model-output> -v <access-version> -a <alias>*
 
 .. code-block:: console 
 
-   $ mopdb template -f ocean.csv -v OM2 -a ocnmon
-   Opened database ~/.local/lib/python3.10/site-packages/data/access.db successfully
-   Derived variables: {'msftyrho', 'msftmrho', 'hfds', 'msftmz', 'msftyz'}
-   Changing advectsweby-CM2_mon units from Watts/m^2 to W m-2
-   Changing areacello-CMIP6_Ofx units from m^2 to m2
-   Variable difvho-CM2_Omon not found in cmor table
+   $ mopdb template -f /scratch/.../exp1/atmos -m 095101 -v CM2 -a exp1
+   Opened database /home/581/pxp581/.local/lib/python3.10/site-packages/data/access.db successfully
+   Found more than 1 definition for fld_s16i222:
+   [('psl', 'AUS2200', 'AUS2200_A10min', '10minPt'), ('psl', 'AUS2200', 'AUS2200_A1hr', '1hr')]
+   Using psl from AUS2200_A10min
+   Variable list for cw323a.pm successfully written
+   Opened database /home/581/pxp581/.local/lib/python3.10/site-packages/data/access.db successfully
+   Derived variables: {'treeFracBdlEvg', 'grassFracC4', 'shrubFrac', 'prc', 'mrsfl', 'landCoverFrac', 'mmrbc', 'mmrso4', 'theta24', 'sftgif', 'treeFracNdlEvg', 'snw', 'rtmt', 'nwdFracLut', 'sifllatstop', 'prw', 'mrfso', 'rlus', 'mrsll', 'baresoilFrac', 'c4PftFrac', 'wetlandFrac', 'mrro', 'c3PftFrac', 'treeFracBdlDcd', 'od550lt1aer', 'treeFracNdlDcd', 'residualFrac', 'wetss', 'sbl', 'vegFrac', 'rsus', 'cropFrac', 'mmrdust', 'grassFrac', 'mmrss', 'od550aer', 'hus24', 'dryss', 'fracLut', 'mrlso', 'mc', 'od440aer', 'grassFracC3', 'nep', 'mmroa', 'cropFracC3', 'snm', 'agesno'}
+   Changing cl-CMIP6_Amon units from 1 to %
+   Changing cli-CMIP6_Amon units from 1 to kg kg-1
+   Changing clt-CMIP6_Amon units from 1 to %
+   Changing clw-CMIP6_Amon units from 1 to kg kg-1
+   Variable husuvgrid-CM2_mon not found in cmor table
+   ...
 
 `mopdb template` takes as input:
- * the output/s of `varlist` - To get one template for the all variable concatenate the output on `varlist` into one file first.
- * the access version to use as preferred
- * an optional alias, if omitted the varlist filename will be used. Based on the example: `map_ocnmon.csv` or `map_ocean.csv` if omitted.
+ * -f/--fpath : the path to the model output
+ * -m/--match : used to identify files' patterns. The tool will only add a list of variables for the same pattern once.
+ * -v/--version : the access version to use as preferred mapping. ESM1.5, CM2, OM2 and AUS2200 are currently available.
+ * -a/--alias : an optional alias, if omitted default names will be used for the output files. 
+
+Alternatively a list of variables can be created separately using the *varlist* command and this can be passed directly to template using the *fpath* option.
+
+   *mopdb template -f <varlist.csv> -v <access-version> -a <alias>*
 
 It produces a csv file with a list of all the variables from raw output mapped to cmip style variables. These mappings also take into account the frequency and include variables that can be potentially calculated with the listed fields. The console output lists these, as shown above.
 
@@ -51,18 +43,20 @@ The mappings can be different between different version and/or configurations of
 
 Starting with version 0.6 the list includes matches based on the standard_name, as these rows often list more than one option per field, it's important to either edit or remove these rows before using the mapping file. 
 The :doc:`Customing section <customising>` covers what to do for an experiment using a new configuration which is substantially different from the ones which are available.
+It also provides an intermediate varlist_<alias>.csv file that shows the information derived directly from the files. This can be useful to debug in case of issues with the mapping. This file is checked before the mapping step to make sure the tool has detected sensible frequency and realm, if the check fails the mapping won't proceed but the varlist file can be edited appropriately.
 
 .. warning:: 
    Always check that the resulting template is mapping the variables correctly. This is particularly true for derived variables. Comment lines are inserted to give some information on what assumptions were done for each group of mappings.
+   The se
 
 
-Step3: Set up the working environment 
+Step2: Set up the working environment 
 -------------------------------------
 
    *mop -c <conf_exp.yaml> setup*
 
 .. code-block:: console 
-
+https://climate-cms.org/posts/2023-05-31-vscode-are.html
    $ mop -c exp_conf.yaml setup
    Simulation to process: cy286
    Setting environment and creating working directory

docs/mopdb_command.rst

@@ -54,29 +54,17 @@ e.g. use aus2200 for mappings related to the AUS2200 configuration:
 
 A user that wants to create a mapping table for another AUS2200 simulation can use this value to select appropriate mappings (see how to do that below).
 
-Get a list of variables from the model output
----------------------------------------------
+Create a mapping file
+---------------------
 .. code-block::
 
-    mopdb varlist -i <output-path> -d <start-date>
-
-this will create for each output file a list of variables with useful attributes
-These can be concatenated into one or used to create separate mappings.
-
-.. _varlist example:
-.. dropdown:: Example output of varlist
+This can be done by providing the model output path and a pattern to match or directly a varlist file
 
-   name;cmor_var;units;dimensions;frequency;realm;cell_methods;cmor_table;vtype;size;nsteps;filename;long_name;standard_name
-   fld_s00i004;theta;K;time model_theta_level_number lat lon;mon;atmos;area: time: mean;CM2_mon;float32;9400320;12;cw323a.pm;THETA AFTER TIMESTEP;air_potential_temperature
-   fld_s00i010;hus;1;time model_theta_level_number lat lon;mon;atmos;area: time: mean;CMIP6_Amon;float32;9400320;12;cw323a.pm;SPECIFIC HUMIDITY AFTER TIMESTEP;specific_humidity
-   fld_s00i024;ts;K;time lat lon;mon;atmos;area: time: mean;CMIP6_Amon;float32;110592;12;cw323a.pm;SURFACE TEMPERATURE AFTER TIMESTEP;surface_temperature
-   fld_s00i030;;1;time lat lon;mon;atmos;area: time: mean;;float32;110592;12;cw323a.pm;LAND MASK (No halo) (LAND=TRUE);land_binary_mask
-   fld_s00i031;siconca;1;time lat lon;mon;atmos;area: time: mean;CMIP6_SImon;float32;110592;12;cw323a.pm;FRAC OF SEA ICE IN SEA AFTER TSTEP;sea_ice_area_fraction
-   ...
+From output path:
+
+    mopdb template  -f <output-path> -m <string-to-match> -v <access-version>
 
-Create a mapping file starting from variable list
--------------------------------------------------
-.. code-block::
+From varlist file:
 
     mopdb template  -f <varlist-out> -v <access-version>
 
@@ -119,6 +107,29 @@ The other groups of records require checking, as either the version or the frequ
    ...
 
 
+Get a list of variables from the model output
+---------------------------------------------
+.. code-block::
+
+    mopdb varlist -f <output-path> -m <string-to-match>
+
+this will create a list of variables with useful attributes
+
+.. _varlist example:
+.. dropdown:: Example output of varlist
+
+   name;cmor_var;units;dimensions;frequency;realm;cell_methods;cmor_table;vtype;size;nsteps;filename;long_name;standard_name
+   #cw323a.pm
+   fld_s00i004;theta;K;time model_theta_level_number lat lon;mon;atmos;area: time: mean;CM2_mon;float32;9400320;12;cw323a.pm;THETA AFTER TIMESTEP;air_potential_temperature
+   fld_s00i010;hus;1;time model_theta_level_number lat lon;mon;atmos;area: time: mean;CMIP6_Amon;float32;9400320;12;cw323a.pm;SPECIFIC HUMIDITY AFTER TIMESTEP;specific_humidity
+   fld_s00i024;ts;K;time lat lon;mon;atmos;area: time: mean;CMIP6_Amon;float32;110592;12;cw323a.pm;SURFACE TEMPERATURE AFTER TIMESTEP;surface_temperature
+   fld_s00i030;;1;time lat lon;mon;atmos;area: time: mean;;float32;110592;12;cw323a.pm;LAND MASK (No halo) (LAND=TRUE);land_binary_mask
+   fld_s00i031;siconca;1;time lat lon;mon;atmos;area: time: mean;CMIP6_SImon;float32;110592;12;cw323a.pm;FRAC OF SEA ICE IN SEA AFTER TSTEP;sea_ice_area_fraction
+   ...
+
+Doing this step separately can be useful if the model output is using a random directory structure, as it's more likely in such a case that important attributes like frequency and realm which are used for the mapping might be incorrect or missing. In such a case it might be more efficient processing different kind of files separately first, making sure frequency and realm are correct and then combining them into one file to pass to template.
+The template command will stop execution if detects potentially wrong values for these fields and save 
+
 Check which variables aren't yet defined
 ----------------------------------------
 .. code-block:: console

docs/overview.rst

@@ -1,25 +1,16 @@
 Install
 =======
 
-You can install the latest version of `mopper` directly from conda (accessnri channel)::
+We are planning to release ACCESS-MOPPeR in conda soon and then it will be available at NCI on our conda environments.
+In the meantime, you can icreate a custom conda environment and install mopper following these steps:
 
-   conda install -c accessnri mopper 
+1. module load conda/analysis3
+2. python -m venv mopper_env --system-site-packages
+3. source  <path-to-mopper-env>/mopper_env/bin/activate
+4. pip install git+https://github.com/ACCESS-Community-Hub/ACCESS-MOPPeR@main
+
+The source command will activate the conda env you just created.
+Any time you want to use the tool in a new session repeat the first and third steps.
 
-If you want to install an unstable version or a different branch:
+The `pip` command above will install from the main branch, you can also indicate a different branch.
 
-    * git clone 
-    * git checkout <branch-name>   (if installing a a different branch from master)
-    * cd mopper 
-    * pip install ./ 
-      use --user flag if you want to install it in ~/.local
-
-Working on the NCI server
--------------------------
-
-MOPPeR is pre-installed into a Conda environment at NCI. Load it with::
-
-    module use /g/data3/hh5/public/modules
-    module load conda/analysis3-unstable
-
-.. note::
-   You need to be a member of the hh5 project to load the modules.