Skip to content

Commit

Permalink
Merge pull request #331 from iiasa/v3.0-prep
Browse files Browse the repository at this point in the history
Prepare for v3.0.0
  • Loading branch information
khaeru authored Jun 7, 2020
2 parents 328caa9 + 38c6ca8 commit 4fdce08
Show file tree
Hide file tree
Showing 8 changed files with 152 additions and 116 deletions.
45 changes: 36 additions & 9 deletions RELEASE_NOTES.rst
Original file line number Diff line number Diff line change
@@ -1,10 +1,42 @@
Next release
============
v3.0.0 (2020-06-05)
===================

ixmp v3.0.0 coincides with message_ix v3.0.0.

Migration notes
---------------

**Updated dependencies.** The minimum version of JPype is 0.7.5.
Excel input/output (I/O)
The file format used by :meth:`.Scenario.to_excel` and :meth:`.read_excel` is now fully specified; see :doc:`file-io`.

ixmp writes and reads items with more elements than the ~10⁶ row maximum of the Excel data format, by splitting these across multiple sheets.

The I/O code now explicitly checks for situations where the index *sets* and *names* for an item are ambiguous; see :ref:`this example <excel-ambiguous-dims>` for how to initialize and read these items.

Updated dependencies
The minimum versions of the following dependencies are increased:

- JPype1 0.7.5
- pandas 1.0
- dask 2.14 (for reporting)

Deprecations and deprecation policy
The following items, marked as deprecated in ixmp 2.0, are removed (:pull:`254`):

- :file:`$HOME/.local/ixmp/` as a configuration location.
Configuration files are now placed in the standard :file:`$HOME/.local/share/ixmp/`.
- positional and ``dbtype=`` arguments to :class:`.Platform`/:class:`.JDBCBackend`.
- ``first_model_year=``, ``keep_sol=``, and ``scen=`` arguments to :meth:`~.Scenario.clone`.
Use `shift_first_model_year`, `keep_solution`, and `scenario`, respectively.
- ``rixmp.legacy``, an earlier version of :ref:`the R interface <rixmp>` that did not use reticulate.

Newly deprecated is:

- `cache` keyword argument to :class:`.Scenario`.
Caching is controlled at the :class:`.Platform`/Backend level, using the same keyword argument.

Starting with ixmp v3.0, arguments and other features marked as deprecated will follow a standard deprecation policy: they will be removed no sooner than the second major release following the release in which they are marked deprecated.
For instance, a feature marked deprecated in ixmp version "10.5" would be retained in ixmp versions "11.x", and removed only in version "12.0" or later.


All changes
Expand All @@ -27,16 +59,11 @@ All changes
- :pull:`270`: Include all tests in the ixmp package.
- :pull:`212`: Add :meth:`Model.initialize` API to help populate new Scenarios according to a model scheme.
- :pull:`267`: Apply units to reported quantities.
- :pull:`254`: Remove deprecated items:

- ~/.local/ixmp as a configuration location.
- positional and ``dbtype=`` arguments to :class:`.Platform`/:class:`.JDBCBackend`.
- ``first_model_year=``, ``keep_sol=``, and ``scen=`` arguments to :meth:`~.Scenario.clone`.
- ``rixmp.legacy``, an earlier version of :ref:`the R interface <rixmp>` that did not use reticulate.
- :pull:`261`: Increase minimum pandas version to 1.0; adjust for `API changes and deprecations <https://pandas.pydata.org/pandas-docs/version/1.0.0/whatsnew/v1.0.0.html#backwards-incompatible-api-changes>`_.
- :pull:`243`: Add :meth:`.export_timeseries_data` to write data for multiple scenarios to CSV.
- :pull:`264`: Implement methods to get and create new subannual timeslices.


v2.0.0 (2020-01-14)
===================

Expand Down
5 changes: 5 additions & 0 deletions doc/source/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,11 @@ Scenario
GAMS code, so cannot be modified by a Scenario. See :meth:`init_equ`,
:meth:`equ_list`, and :meth:`equ`.

.. deprecated:: 3.0
The `cache` keyword argument to :class:`Scenario` has no effect and raises a warning.
Use `cache` as one of the `backend_args` to :class:`Platform` to disable/enable caching for storage backends that support it.
Use :meth:`load_scenario_data` to load all data in the Scenario into an in-memory cache.

.. autosummary::
add_par
add_set
Expand Down
2 changes: 2 additions & 0 deletions doc/source/file-io.rst
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,8 @@ Reading variables and equations

Thus, while :meth:`.to_excel` will write files containing variable and equation data, :meth:`.read_excel` can not add these to a Scenario, and only emits log messages indicating that they are ignored.

.. _excel-ambiguous-dims:

Multiple dimensions indexed by the same set
:meth:`.read_excel` provides the `init_items` argument to create new sets and parameters when reading a file.
However, the file format does not capture information needed to reconstruct the original data in all cases.
Expand Down
206 changes: 103 additions & 103 deletions doc/source/install.rst
Original file line number Diff line number Diff line change
@@ -1,22 +1,21 @@
Installation
============
************

Most users will have the |ixmp| installed automatically when `installing MESSAGEix`_.
A desktop or laptop computer is sufficient for most purposes using :mod:`ixmp`.
Most users will have :mod:`ixmp` installed automatically when `installing MESSAGEix`_.

The sections below cover other use cases:

- Installing *ixmp* to be used alone (i.e., with models or frameworks other than
|MESSAGEix|):
- Installing *ixmp* to be used alone (i.e., with models or frameworks other than |MESSAGEix|):

- see the section `Install GAMS`_,
- see the sections `Install system dependencies`_,
- then `Install ixmp via Anaconda`_.

- Installing *ixmp* from source, for development purposes: see
`Install ixmp from source`_.
- Installing *ixmp* from source, for development purposes: see `Install ixmp from source`_.

- Installing the R API to *ixmp*:

- Start with `Install GAMS`_.
- Start with `Install system dependencies`_.
- Then install *ixmp* either via Anaconda, or from source.
- Finally, see `Install rixmp`_.

Expand All @@ -25,171 +24,172 @@ The sections below cover other use cases:
.. contents::
:local:

Install system dependencies
===========================

Technical requirements
----------------------
GAMS
----

A high-quality desktop computer or laptop is sufficient for most purposes
using the |ixmp|.
:mod:`ixmp` requires `GAMS`_.

1. Download GAMS for your operating system; either the `latest version`_ or `version 29`_ (see note below).

Install GAMS
------------
2. Run the installer.

*ixmp* requires `GAMS`_.

1. Download the latest version of `GAMS`_ for your operating system; run the
installer.

2. Add GAMS to the ``PATH`` environment variable:
3. Ensure that the ``PATH`` environment variable on your system includes the path to the GAMS program:

- on Windows, in the GAMS installer…

- Check the box labeled “Use advanced installation mode.”
- Check the box labeled “Add GAMS directory to PATH environment variable”
on the Advanced Options page.
- on macOS or Linux, add the following line to your ``.bash_profile`` (Mac) or ``.bashrc`` (Linux)::
- Check the box labeled “Add GAMS directory to PATH environment variable” on the Advanced Options page.

- on other platforms (macOS or Linux), add the following line to a file such as :file:`~/.bash_profile` (macOS), :file:`~/.bashrc`, or :file:`~/.profile`::

export PATH=$PATH:/path/to/gams-directory-with-gams-binary

export PATH=$PATH:/path/to/gams-directory-with-gams-binary
.. note::
:mod:`message_ix` requires GAMS version 24.8; :mod:`ixmp` has no minimum requirement *per se*.
The latest version is recommended.

GAMS is proprietary software and requires a license to solve optimization problems.
To run both the :mod:`ixmp` and :mod:`message_ix` tutorials and test suites, a “free demonstration” license is required; the free license is suitable for these small models.
Versions of GAMS up to `version 29`_ include such a license with the installer; since version 30, the free demo license is no longer included, but may be requested via the GAMS website.

.. note::
For using `GAMS`_ to solve numerical optimisation problems, you need to
install the latest version of GAMS (in particular 24.8 or higher). If you
only have a license for an older version, install both the older and the
latest version of GAMS.
If you only have a license for an older version of GAMS, install both the older and the latest versions.


Graphviz
--------

:meth:`ixmp.reporting.Reporter.visualize` uses `Graphviz`_, a program for graph visualization.
Installing ixmp causes the python :mod:`graphviz` package to be installed.
If you want to use :meth:`.visualize` or run the test suite, the Graphviz program itself must also be installed; otherwise it is **optional**.

If you `Install ixmp via Anaconda`_, Graphviz is installed automatically via `its conda-forge package`_.
For other methods of installation, see the `Graphviz download page`_ for downloads and instructions for your system.

Install *ixmp* via Anaconda
---------------------------

After installing GAMS, we recommend that new users install Anaconda, and then
use it to install *ixmp*. Advanced users may choose to install *ixmp* from
source code (next section).
Install ``ixmp`` via Anaconda
=============================

3. Install Python via `Anaconda`_. We recommend the latest version, i.e.,
Python 3.6+.
After installing GAMS, we recommend that new users install Anaconda, and then use it to install :mod:`ixmp`.
Advanced users may choose to install :mod:`ixmp` from source code (next section).

4. Open a command prompt. We recommend Windows users use the “Anaconda Prompt”
to avoid permissions issues when installing and using *ixmp*. This program
is available in the Windows Start menu after installing Anaconda.
4. Install Python via `Anaconda`_.
We recommend the latest version; currently Python 3.8.

5. Install the ``ixmp`` package::
5. Open a command prompt.
We recommend Windows users use the “Anaconda Prompt” to avoid permissions issues when installing and using :mod:`ixmp`.
This program is available in the Windows Start menu after installing Anaconda.

6. Install the ``ixmp`` package::

$ conda install -c conda-forge ixmp


Install *ixmp* from source
--------------------------
Install ``ixmp`` from source
============================

3. (Optional) If you intend to contribute changes to *ixmp*, first register
a Github account, and fork the `ixmp repository <https://github.com/iiasa/ixmp>`_. This will create a new repository ``<user>/ixmp``.
4. (Optional) If you intend to contribute changes to *ixmp*, first register a Github account, and fork the `ixmp repository <https://github.com/iiasa/ixmp>`_.
This will create a new repository ``<user>/ixmp``.

4. Clone either the main repository, or your fork; using the `Github Desktop`_
client, or the command line::
5. Clone either the main repository, or your fork; using the `Github Desktop`_ client, or the command line::

$ git clone [email protected]:iiasa/ixmp.git

# or:
$ git clone [email protected]:USER/ixmp.git

5. Open a command prompt in the ``ixmp`` directory and type::
6. Open a command prompt in the :file:`ixmp/` directory that is created, and type::

$ pip install --editable .
$ pip install --editable .[docs,tests,tutorial]

The ``--editable`` flag ensures that changes to the source code are picked up every time ``import ixmp`` is used in Python code.
The ``[docs,tests,tutorial]`` extra dependencies ensure additional dependencies are installed.

6. (Optional) Run the built-in test suite to check that *ixmp* functions
correctly on your system::
7. (Optional) Run the built-in test suite to check that :mod:`ixmp` functions correctly on your system::

$ pip install --editable .[tests]
$ py.test
$ pytest


Install ``rixmp``
-----------------
Install development tools
-------------------------

See also the :ref:`rixmp documentation <rixmp>`.
Developers making changes to the :mod:`ixmp` source **may** need one or more of the following tools.
Users developing models using existing functionality **should not** need these tools.

1. `Install R <https://www.r-project.org>`_.
Git
Use one of:

.. warning::
Ensure the the R version installed is either 32- *or* 64-bit (and >= 3.5.0), consistently with GAMS and Java.
Having both 32- and 64-bit versions of R, or mixed 32- and 64-bit versions of different packages, can cause errors.
- https://git-scm.com/downloads
- https://desktop.github.com
- https://www.gitkraken.com

2. Enter the directory ``rixmp/`` and use R to build and install the package and its dependencies, including reticulate_::
In addition, set up an account at https://github.com, and familiarize yourself with forking and cloning repositories, as well as pulling, committing and pushing changes.

$ cd rixmp
$ Rscript -e "install.packages(c('knitr', 'reticulate'), repos='http://cran.rstudio.com/')"
$ R CMD build .
Java Development Kit (JDK)
- Install the Java Development Kit (JDK) for Java SE version 8 from https://www.oracle.com/technetwork/java/javase/downloads/index.html

3. Check that there is only one ``*tar.gz`` in the folder::
.. note:: At this point, ixmp is not compatible with JAVA SE 9.

$ R CMD INSTALL rixmp_*
- Follow the `JDK website instructions <https://docs.oracle.com/cd/E19182-01/820-7851/inst_cli_jdk_javahome_t/>`_ to set the ``JAVA_HOME`` environment variable; if ``JAVA_HOME`` does not exist, add it as a new system variable.

4. (Optional) Run the built-in test suite to check that *ixmp* and *rixmp* functions, as in *Install ixmp from source 6.* (installing
the R ``devtools`` package might be a pre-requisite). In the ``ixmp`` directory type::
- Update your ``PATH`` environment variable to point to the JRE binaries and server installation (e.g., :file:`C:\\Program Files\\Java\\jdk[YOUR JDK VERSION]\\jre\\bin\\`, :file:`C:\\Program Files\\Java\\jdk[YOUR JDK VERSION]\\jre\\bin\\server`).

$ py.test --test-r
.. warning:: Do not overwrite the existing ``PATH`` environment variable, but add to the list of existing paths.

5. (Optional) For working with Jupyter notebooks using R, install the `IR kernel <https://irkernel.github.io>`_.

6. (Optional) Install `Rtools <https://cran.r-project.org/bin/windows/Rtools/>`_ and add the path to the environment variables.

.. _reticulate: https://rstudio.github.io/reticulate/
Install ``rixmp``
=================

See also the :ref:`rixmp documentation <rixmp>`.

Install development tools
-------------------------
1. `Install R <https://www.r-project.org>`_.

Developers making changes to the *ixmp* source may need one or more of the following tools.
Users developing models using existing *ixmp* functionality **should not** need these tools.
.. warning::
Ensure the the R version installed is either 32- *or* 64-bit (and >= 3.5.0), consistently with GAMS and Java.
Having both 32- and 64-bit versions of R, or mixed 32- and 64-bit versions of different packages, can cause errors.

- **Java Development Kit (JDK).**
2. Enter the directory ``rixmp/`` and use R to build and install the package and its dependencies, including reticulate_::

- Install the Java Development Kit (JDK) for Java SE version 8 from
https://www.oracle.com/technetwork/java/javase/downloads/index.html
$ cd rixmp
$ Rscript -e "install.packages(c('knitr', 'reticulate'), repos='http://cran.rstudio.com/')"
$ R CMD build .

.. note:: At this point, ixmp is not compatible with JAVA SE 9.
3. Check that there is only one ``*tar.gz`` in the folder, then run::

- Follow the `JDK website instructions
<https://docs.oracle.com/cd/E19182-01/820-7851/inst_cli_jdk_javahome_t/>`_
to set the ``JAVA_HOME`` environment variable; if ``JAVA_HOME`` does not
exist, add as new system variable.
$ R CMD INSTALL rixmp_*

- Update your `PATH` environment variable to point to the JRE binaries and
server installation (e.g., ``C:\Program Files\Java\jdk[YOUR JDK
VERSION]\jre\bin\``, ``C:\Program Files\Java\jdk[YOUR JDK
VERSION]\jre\bin\server``).
4. (Optional) Run the built-in test suite to check that *ixmp* and *rixmp* functions, as in *Install ixmp from source 6.* (installing
the R ``devtools`` package might be a pre-requisite). In the ``ixmp`` directory type::

.. warning:: Do not overwrite the existing `PATH` environment variable, but
add to the list of existing paths.
$ pytest -m rixmp

- **Git.** Use one of:
5. (Optional) To use rixmp in Jupyter notebooks, install the `IR kernel <https://irkernel.github.io>`_.

- https://git-scm.com/downloads
- https://desktop.github.com
- https://www.gitkraken.com
6. (Optional) Install `Rtools <https://cran.r-project.org/bin/windows/Rtools/>`_ and add the path to the environment variables.

In addition, set up an account at https://github.com, and familiarize
yourself with forking and cloning repositories, as well as pulling,
committing and pushing changes.
.. _reticulate: https://rstudio.github.io/reticulate/


Troubleshooting
---------------
===============

Run ``ixmp show-versions`` on the command line to check that you have all dependencies installed, or when reporting issues.

For Anaconda users experiencing problems during installation of ixmp,
Anaconda might not have been added to the PATH system variable properly.
So, if ``install.bat`` fails, check if::
For Anaconda users experiencing problems during installation of ixmp, check that the following paths are part of the ``PATH`` environment variable, and add them if missing::

C:\[YOUR ANACONDA LOCATION]\Anaconda3;
C:\[YOUR ANACONDA LOCATION]\Anaconda3\Scripts;
C:\[YOUR ANACONDA LOCATION]\Anaconda3\Library\bin;

are all part of the PATH system variable. If they are not there, add them.

Run `ixmp show-versions` on the command line to check that you have all dependencies installed, or when reporting issues.

.. _`installing MESSAGEix`: https://message.iiasa.ac.at/en/latest/getting_started.html
.. _`Anaconda`: https://www.continuum.io/downloads
.. _`GAMS`: http://www.gams.com
.. _`latest version`: https://www.gams.com/download/
.. _`version 29`: https://www.gams.com/29/
.. _`its conda-forge package`: https://anaconda.org/conda-forge/graphviz
.. _`Graphviz download page`: https://www.graphviz.org/download/
.. _`Github Desktop`: https://desktop.github.com
1 change: 0 additions & 1 deletion ixmp/model/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,6 @@ def initialize_items(cls, scenario, items):
for key, values in item_info.items():
values = values or []
existing = getattr(scenario, key)(name)
print(name, key, values, existing)
if existing != values:
# The existing index sets or names do not match
log.warning(
Expand Down
2 changes: 2 additions & 0 deletions ixmp/tests/reporting/test_exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ def test_computationerror(caplog):


def test_computationerror_ipython(test_data_path, tmp_path, tmp_env):
# NB this requires nbformat >= 5.0, because the output kind "evalue" was
# different pre-5.0
fname = test_data_path / 'reporting-exceptions.ipynb'
nb, _ = run_notebook(fname, tmp_path, tmp_env, allow_errors=True)

Expand Down
Loading

0 comments on commit 4fdce08

Please sign in to comment.