Miscellaneous improvements for 2024-W14

.github/codecov.yml

@@ -8,6 +8,11 @@ coverage:
     project:
       default:
         if_ci_failed: success
+        # Currently, CI runs on PR branches do not include tests of (a) snapshot
+        # handling or (b) legacy reporting; these are only run on schedule
+        # triggers. This resuls in an apparent coverage drop of about 90 lines,
+        # or 0.46 percent. Allow this.
+        threshold: 0.5%
     patch:
       default:
         if_ci_failed: success

.github/workflows/pytest.yaml

@@ -13,7 +13,17 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
+  warm-lfs-cache:
+    strategy:
+      matrix:
+        os: [ macos-latest, ubuntu-latest, windows-latest ]
+    runs-on: ${{ matrix.os }}
+    steps:
+    - uses: nschloe/action-cached-lfs-checkout@v1
+
   pytest:
+    needs: warm-lfs-cache
+
     strategy:
       matrix:
         os:
@@ -67,10 +77,7 @@ jobs:
         key: ${{ matrix.os }}
 
     - name: Check out message-ix-models
-      uses: actions/checkout@v4
-      with:
-        lfs: true
-        fetch-depth: ${{ env.depth }}
+      uses: nschloe/action-cached-lfs-checkout@v1
 
     - uses: actions/setup-python@v5
       with:

MANIFEST.in

@@ -1,10 +1,13 @@
+# Data for testing
 prune message_ix_models/data/test/advance
+prune message_ix_models/data/test/gea
 prune message_ix_models/data/test/iea
-prune message_ix_models/data/test/MESSAGEix-GLOBIOM_1.1_R11_no-policy_baseline
+prune message_ix_models/data/test/shape
+prune message_ix_models/data/test/snapshot-*
 prune message_ix_models/data/test/ssp
-prune message_ix_models/data/water/*
 # Larger package data
 # - Not distributed on PyPI.
 # - Should be fetched with Pooch from GitHub.
 exclude message_ix_models/data/ssp/*.gz
+prune message_ix_models/data/water/*
 exclude message_ix_models/data/water/*.tar.xz

README.rst

@@ -1,21 +1,6 @@
 Tools for MESSAGEix-GLOBIOM models
 **********************************
-
-.. image:: https://img.shields.io/pypi/v/message-ix-models.svg
-   :alt: PyPI version
-   :target: https://pypi.python.org/pypi/message-ix-models/
-
-.. image:: https://readthedocs.com/projects/iiasa-energy-program-message-ix-models/badge/?version=latest
-   :alt: Documentation status
-   :target: https://docs.messageix.org/projects/models/en/latest/?badge=latest
-
-.. image:: https://github.com/iiasa/message-ix-models/actions/workflows/pytest.yaml/badge.svg
-   :alt: Test status
-   :target: https://github.com/iiasa/message-ix-models/actions/workflows/pytest.yaml
-
-.. image:: https://codecov.io/gh/iiasa/message-ix-models/branch/main/graph/badge.svg
-   :alt: Test coverage
-   :target: https://codecov.io/gh/iiasa/message-ix-models
+|pypi| |rtd| |gha| |codecov|
 
 ``message_ix_models`` provides tools for research using the MESSAGEix-GLOBIOM family of models.
 These models are built in the `MESSAGEix framework <https://docs.messageix.org>`_ and on the `ix modeling platform (ixmp) <https://docs.messageix.org/ixmp/>`_.
@@ -28,3 +13,19 @@ License
 Copyright © 2020–2024 IIASA Energy, Climate, and Environment (ECE) program
 
 Licensed under the Apache License, version 2.0.
+
+.. |pypi| image:: https://img.shields.io/pypi/v/message-ix-models.svg
+   :alt: PyPI version
+   :target: https://pypi.python.org/pypi/message-ix-models/
+
+.. |rtd| image:: https://readthedocs.com/projects/iiasa-energy-program-message-ix-models/badge/?version=latest
+   :alt: Documentation status
+   :target: https://docs.messageix.org/projects/models/en/latest/?badge=latest
+
+.. |gha| image:: https://github.com/iiasa/message-ix-models/actions/workflows/pytest.yaml/badge.svg
+   :alt: Test status
+   :target: https://github.com/iiasa/message-ix-models/actions/workflows/pytest.yaml
+
+.. |codecov| image:: https://codecov.io/gh/iiasa/message-ix-models/branch/main/graph/badge.svg
+   :alt: Test coverage
+   :target: https://codecov.io/gh/iiasa/message-ix-models

doc/api/data-sources.rst

@@ -1,6 +1,16 @@
 Tools for specific data sources
 *******************************
 
+.. _tools-gfei:
+
+Global Fuel Economy Initiative (GFEI) (:mod:`.tools.gfei`)
+==========================================================
+
+.. currentmodule:: message_ix_models.tools.gfei
+
+.. automodule:: message_ix_models.tools.gfei
+   :members:
+
 .. _tools-iea:
 
 International Energy Agency (IEA) (:mod:`.tools.iea`)
@@ -20,6 +30,26 @@ Documentation for all module contents:
 
    iea
 
+Energy efficiency indicators (:mod:`.tools.iea.eei`)
+----------------------------------------------------
+
+See :class:`.IEA_EEI`.
+This data is produced by the IEA and retrieved from the Energy Efficiency Indicators database.
+It is proprietary.
+
+The data:
+
+- Has the geographic resolution of individual countries, and scope including 41 countries:
+
+ - 24 IEA member countries for which data covering most end-uses area available: Australia, Austria, Belgium, Canada, Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Italy, Japan, Korea, Luxembourg, the Netherlands, New Zealand, Poland, Portugal, Slovak Republic, Spain, Switzerland, the United Kingdom and the United States.
+ - Others including Brazil, Chile, Lithuania, Morocco, Armenia, Azerbaijan, Belarus, Georgia, Kazakhstan, Kyrgyzstan, Republic of Moldova, Ukraine, Uzbekistan.
+
+- Includes measures/variables for energy consumption, efficiency, carbon emissions, and others for four conceptual sectors: Residential, Services, Industry and Transport.
+- The **December 2020 edition** covers the time periods 2000–2018 with annual resolution.
+
+.. note:: Currently, :mod:`.iea.eei` mainly retrieves and processes data useful for MESSAGEix-Transport.
+   To retrieve other end-use sectoral data, the code can be extended.
+
 .. _tools-iea-web:
 
 (Extended) World Energy Balances (:mod:`.tools.iea.web`)

doc/api/disutility.rst

@@ -14,15 +14,15 @@ Method & usage
 Use this code by calling :func:`add`, which takes arguments that describe the concrete usage:
 
 Consumer groups
-   This is a list of |Code| objects describing the consumer groups.
+   This is a list of :class:`.Code` objects describing the consumer groups.
    The list must be 1-dimensional, but can be composed (as in :mod:`message_data.model.transport`) from multiple dimensions.
 
 Technologies
-   This is a list of |Code| objects describing the technologies for which the consumers in the different groups experience disutility.
+   This is a list of :class:`.Code` objects describing the technologies for which the consumers in the different groups experience disutility.
    Each object must be have 'input' and 'output' annotations (:attr:`~.Code.annotations`); each of these is a :class:`dict` with the keys 'commodity', 'input', and 'unit', describing the source or sink for the technology.
 
 Template
-   This is also a |Code| object, similar to those in ``technologies``; see below.
+   This is also a :class:`.Code` object, similar to those in ``technologies``; see below.
 
 The code creates a source technology for the “disutility” commodity.
 The code does *not* perform the following step(s) needed to completely parametrize the formulation:

doc/api/report/index.rst

@@ -37,7 +37,7 @@ In short, for instance:
 
 The basic **design pattern** of :mod:`message_ix_models.report` is:
 
-- :func:`~.report.prepare_reporter` populates a new :class:`~.message_ix.Reporter` for a given |Scenario| with many keys to report all quantities of interest in a MESSAGEix-GLOBIOM–family model.
+- :func:`~.report.prepare_reporter` populates a new :class:`~.message_ix.Reporter` for a given :class:`.Scenario` with many keys to report all quantities of interest in a MESSAGEix-GLOBIOM–family model.
 - This function relies on *callbacks* defined in multiple submodules to add keys and tasks for general or tailored reporting calculations and actions.
   Additional modules **should** define callback functions and register them with :func:`~report.register` when they are to be used.
   For example:
@@ -145,7 +145,7 @@ Operators
 
      - :mod:`message_ix.report.operator`
      - :mod:`ixmp.report.operator`
-     - :mod:`genno.computations`
+     - :mod:`genno.operator`
 
    - Other submodules:
 

doc/cli.rst

@@ -172,7 +172,7 @@ Further information about the top-level options:
    These options direct it to work with a different Platform.
 
 :program:`--model MODEL --scenario SCENARIO` or :program:`--url`
-    Many commands use an *existing* |Scenario| as a starting point, and begin by cloning that Scenario to a new (model name, scenario name).
+    Many commands use an *existing* :class:`.Scenario` as a starting point, and begin by cloning that Scenario to a new (model name, scenario name).
     For any such command, these top-level options define the starting point/initial Scenario to clone/‘baseline’.
 
     In contrast, see :program:`--output-model`, below.

doc/conf.py

@@ -22,14 +22,16 @@
 # Add any Sphinx extension module names here, as strings. They can be extensions coming
 # with Sphinx (named 'sphinx.ext.*') or your custom ones.
 extensions = [
-    # "ixmp.util.sphinx_linkcode_github",
     "sphinx.ext.autodoc",
     "sphinx.ext.autosummary",
     "sphinx.ext.extlinks",
     "sphinx.ext.intersphinx",
     "sphinx.ext.napoleon",
     "sphinx.ext.todo",
     "sphinx.ext.viewcode",
+    # Others
+    "genno.compat.sphinx.rewrite_refs",
+    # "ixmp.util.sphinx_linkcode_github",
     "sphinxcontrib.bibtex",
 ]
 
@@ -43,12 +45,8 @@
 
 nitpicky = True
 nitpick_ignore_regex = {
-    # These occur because there is no .. py:module:: directive for the *top-level*
-    # module or package in the respective documentation and inventories.
-    # TODO Remove once the respective docs are fixed
-    ("py:mod", "ixmp"),
-    ("py:mod", "message_ix"),
-    ("py:mod", "message_data"),
+    # Legacy reporting docstrings are not formatted
+    ("py:.*", r"boolean \(default|str \(default|None\)|False\)"),
     # iam-units has no Sphinx docs
     ("py:.*", "iam_units.*"),
     # These are a consequence of autosummary-class.rst
@@ -59,12 +57,6 @@
 .. role:: py(code)
    :language: python
 
-.. |Annotation| replace:: :class:`~sdmx.model.common.Annotation`
-.. |Code| replace:: :class:`~sdmx.model.common.Code`
-.. |Codelist| replace:: :class:`~sdmx.model.common.Codelist`
-.. |Platform| replace:: :class:`~ixmp.Platform`
-.. |Scenario| replace:: :class:`~message_ix.Scenario`
-
 .. |n| replace:: :math:`n`
 .. |y| replace:: :math:`y`
 .. |y0| replace:: :math:`y_0`
@@ -101,6 +93,34 @@ def setup(app: "sphinx.application.Sphinx") -> None:
 # builtin themes.
 html_theme = "sphinx_rtd_theme"
 
+# -- Options for genno.compat.sphinx.rewrite_refs --------------------------------------
+
+# When base classes in upstream (genno, ixmp) packages are inherited in message_ix,
+# Sphinx will not properly resolve relative references within docstrings of methods of
+# the former. Some of these aliases are to allow Sphinx to locate the correct targets.
+reference_aliases = {
+    # genno
+    "Computer": "genno.Computer",
+    "KeyLike": ":data:`genno.core.key.KeyLike`",
+    r"(genno\.|)Key(?=Seq|[^\w]|$)": "genno.core.key.Key",
+    r"(genno\.|)Quantity": "genno.core.attrseries.AttrSeries",
+    # ixmp
+    "Platform": "ixmp.Platform",
+    "TimeSeries": "ixmp.TimeSeries",
+    # message_ix
+    r"Scenario(?=[^\w]|$)": "message_ix.Scenario",
+    "Reporter": "message_ix.report.Reporter",
+    "make_df": "message_ix.util.make_df",
+    # sdmx
+    "Code": "sdmx.model.common.Code",
+    #
+    # Many projects (including Sphinx itself!) do not have a py:module target in for the
+    # top-level module in objects.inv. Resolve these using :doc:`index` or similar for
+    # each project.
+    "pint$": ":std:doc:`pint <pint:index>`",
+    "plotnine$": ":class:`plotnine.ggplot`",
+}
+
 # -- Options for sphinx.ext.autosummary ------------------------------------------------
 
 autosummary_generate = True
@@ -145,6 +165,7 @@ def local_inv(name: str, *parts: str) -> Optional[str]:
     "pint": ("https://pint.readthedocs.io/en/stable/", None),
     "platformdirs": ("https://platformdirs.readthedocs.io/en/latest", None),
     "pooch": ("https://www.fatiando.org/pooch/latest/", None),
+    "pyam": ("https://pyam-iamc.readthedocs.io/en/stable", None),
     "pytest": ("https://docs.pytest.org/en/stable/", None),
     "python": ("https://docs.python.org/3/", None),
     "sdmx": ("https://sdmx1.readthedocs.io/en/stable/", None),
@@ -158,7 +179,8 @@ def local_inv(name: str, *parts: str) -> Optional[str]:
 
 napoleon_preprocess_types = True
 napoleon_type_aliases = {
-    "Code": ":class:`~sdmx.model.common.Code`",
+    "iterable": ":class:`~collections.abc.Iterable`",
+    "sequence": ":class:`~collections.abc.Sequence`",
     "Path": ":class:`~pathlib.Path`",
     "PathLike": ":class:`os.PathLike`",
 }

doc/data.rst

@@ -201,7 +201,7 @@ Fetch directly from a remote source
 This corresponds to section (1) above.
 Preferably, do this via :mod:`message_ix_models.util.pooch`:
 
-- Extend :mod:`.pooch.SOURCE` to store the Internet location, file name(s), and hash(es) of the file(s).
+- Extend :data:`.pooch.SOURCE` to store the Internet location, file name(s), and hash(es) of the file(s).
 - Call :func:`.pooch.fetch` to retrieve the file and cache it locally.
 - Write code in :mod:`message_ix_models` that processes the data into a common format, for instance by subclassing :class:`.ExoDataSource`.
 

doc/develop.rst

@@ -45,3 +45,45 @@ Code that will only work with certain structures…
 
 Code **may** also check a :class:`.Context` instance and automatically adapt data from certain structures to others, e.g. by interpolating data for certain periods or areas.
 To help with validation, code that does this **should** log on the :data:`logging.INFO` level to advertise these steps.
+
+.. _policy-upstream-versions:
+
+Upstream version policy
+=======================
+
+:mod:`message_ix_models` is developed to be compatible with the following versions of its upstream dependencies.
+
+:mod:`ixmp` and :mod:`message_ix`
+
+   The most recent 4 minor versions, or all minor versions released in the past two (2) years—whichever is greater.
+
+   For example, as of 2024-04-08:
+
+   - The most recent release of :mod:`ixmp` and :mod:`message_ix` are versions 3.8.0 of each project.
+     These are supported by :mod:`message_ix_models`.
+   - The previous 3 minor versions are 3.7.0, 3.6.0, and 3.5.0.
+     All were released since 2022-04-08.
+     All are supported by :mod:`message_ix_models.`
+   - :mod:`ixmp` and :mod:`message_ix` versions 3.4.0 were released 2022-01-24.
+     These this is the fifth-most-recent minor version *and* was released more than 2 years before 2024-04-08, so it is not supported.
+
+Python
+   All currently-maintained versions of Python.
+
+   The Python website displays a list of these versions (`1 <https://www.python.org/downloads/>`__, `2 <https://devguide.python.org/versions/#versions>`__).
+
+   For example, as of 2024-04-08:
+
+   - Python 3.13 is in "prerelease" or "feature" development status, and is *not* supported by :mod:`message_ix_models`.
+   - Python 3.12 through 3.8 are in "bugfix" or "security" maintenance status, and are supported by :mod:`message_ix_models`.
+   - Python 3.7 and earlier are in "end-of-life" status, and are not supported by the Python community or by :mod:`message_ix_models`.
+
+- Support for older versions of dependencies **may** be dropped as early as the first :mod:`message_ix_models` version released after changes in upstream versions.
+
+  - Conversely, some parts of :mod:`message_ix_models` **may** continue to be compatible with older upstream versions, but this compatibility is not tested and may break at any time.
+  - Users **should** upgrade their dependencies and other code to newer versions; we recommend the latest.
+- Some newer code is marked with a :func:`.minimum_version` decorator.
+
+  - This indicates that the marked code relies on features only available in certain upstream versions (of one of the packages mentioned above, or another package), newer than those listed in `pyproject.toml <https://github.com/iiasa/message-ix-models/blob/main/pyproject.toml>`__.
+  - These minima **must** be mentioned in the :mod:`message_ix_models` documentation.
+  - Users wishing to use this marked code **must** use compatible versions of those packages.

doc/index.rst

@@ -49,6 +49,7 @@ Commonly used classes may be imported directly from :mod:`message_ix_models`.
 - :doc:`api/disutility`
 - :doc:`api/report/index`
 - :doc:`api/tools`
+- :doc:`api/data-sources`
 - :doc:`api/util`
 - :doc:`api/testing`
 - :doc:`api/workflow`
@@ -73,10 +74,18 @@ Commonly used classes may be imported directly from :mod:`message_ix_models`.
 
 .. toctree::
    :maxdepth: 2
-   :caption: Variants and projects
+   :caption: Model variants
 
    water/index
-   api/project
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Research projects
+
+   project/advance
+   project/gea
+   project/shape
+   project/ssp
 
 .. toctree::
    :maxdepth: 2

doc/project/advance.rst

@@ -0,0 +1,17 @@
+.. currentmodule:: message_ix_models.project.advance
+
+ADVANCE (:mod:`.project.advance`)
+*********************************
+
+.. automodule:: message_ix_models.project.advance
+
+.. automodule:: message_ix_models.project.advance.data
+   :members:
+
+   Although free of charge, the ADVANCE data can not be downloaded automatically.
+   This source requires that users first submit personal information to register before being able to retrieve the data.
+   :mod:`message_ix_models` does not circumvent this requirement.
+   Thus:
+
+   - A copy of the data are stored in :mod:`message_data`.
+   - :mod:`message_ix_models` contains only a ‘fuzzed’ version of the data (same structure, random values) for testing purposes.