Merge pull request #201 from iiasa/fix/materials-W23

Fix discovered issues and improve MESSAGEix-Materials

MANIFEST.in

@@ -10,4 +10,6 @@ prune message_ix_models/data/test/ssp
 # - Should be fetched with Pooch from GitHub.
 exclude message_ix_models/data/ssp/*.gz
 prune message_ix_models/data/water/*
+prune message_ix_models/data/material/*
 exclude message_ix_models/data/water/*.tar.xz
+exclude message_ix_models/data/material/*.tar.gz

doc/material/index.rst

@@ -4,8 +4,7 @@ MESSAGEix-Materials
 Description
 ===========
 
-This module adds material stock and flow accounting in MESSAGEix-GLOBIOM. The implementation currently includes four key energy/emission-intensive
-material industries: Iron&Steel, Aluminum, Cement, and Chemicals.
+This module adds material stock and flow accounting in MESSAGEix-GLOBIOM. The implementation currently includes four key energy/emission-intensive material industries: Iron&Steel, Aluminum, Cement, and Chemicals.
 
 .. contents::
    :local:
@@ -22,9 +21,9 @@ Code reference
 Data preparation
 ----------------
 
-These scripts are used to prepare and read the data into the model. They can be turned on and off individually under `DATA_FUNCTIONS` in `__init__.py`.
-For example, the buildings script (`data_buildings.py`) is only used when the buildings model outputs are given explicitly without linking the
-CHILLED/STURM model through a soft link.
+These scripts are used to prepare and read the data into the model.
+They can be turned on and off individually under `DATA_FUNCTIONS` in `__init__.py`.
+For example, the buildings script (`data_buildings.py`) is only used when the buildings model outputs are given explicitly without linking the CHILLED/STURM model through a soft link.
 
 .. automodule:: message_ix_models.model.material.data_aluminum
   :members:
@@ -56,43 +55,48 @@ CHILLED/STURM model through a soft link.
 Build and Solve the model from CLI
 ==================================
 
-Note: To include material stocks from power sector message_ix should be the following version from source:
+Note: To include material stocks from power sector message_ix should be the following
+version from source:
 https://github.com/iiasa/message_ix/tree/material_stock
 
-Use ``mix-models materials-ix build`` to add the material implementation on top of existing standard global (R12) scenarios, also giving the base scenario and indicating the relevant data location, e.g.::
+Use ``mix-models materials-ix {SSP} build`` to add the material implementation on top of existing standard global (R12) scenarios, also giving the base scenario and indicating the relevant data location, e.g.::
 
-    $ mix-models \
-      --url="ixmp://ixmp_dev/MESSAGEix-GLOBIOM 1.1-R12/baseline_DEFAULT#21" \
-      --local-data "./data" material-ix build --tag test
+    mix-models \
+        --url="ixmp://ixmp_dev/MESSAGEix-GLOBIOM 1.1-R12/baseline_DEFAULT#21" \
+        --local-data "./data" material-ix SSP2 build --tag test
 
 The output scenario name will be baseline_DEFAULT_test. An additional tag `--tag` can be used to add an additional suffix to the new scenario name.
-The mode option `--mode` has two different inputs 'by_url' (by default) or 'by_copy'. The first one uses the provided url to add the materials implementation on top of the scenario from the url.
-This is the default option. The latter is used to create a 2 degree mitigation scenario with materials by copying carbon prices to the scenario that is specified by `--scenario_name`.
+The mode option `--mode` has two different inputs 'by_url' (by default) or 'by_copy'.
+The first one uses the provided url to add the materials implementation on top of the scenario from the url.
+This is the default option. The latter is used to create a 2 degree mitigation scenario with materials by copying carbon prices to the scenario that is specified by `--scenario_name`::
 
-    $ mix-models material-ix build --tag test --mode by_copy --scenario_name baseline_DEFAULT_test.
+    mix-models --url="ixmp://ixmp_dev/MESSAGEix-Materials/scenario_name" material-ix \
+     build --tag test --mode by_copy
 
 This command line only builds the scenario but does not solve it. To solve the scenario, use ``mix-models materials-ix solve``, giving the scenario name, e.g.::
 
-    $ mix-models material-ix solve --scenario_name baseline_DEFAULT_test --add_calibration False --add_macro False
+    mix-models --url="ixmp://ixmp_dev/MESSAGEix-Materials/scenario_name" material-ix \
+     SSP2 solve --add_calibration False --add_macro False
 
 The solve command has the `--add_calibration` option to add MACRO calibration to a baseline scenario. `--add_macro` option solves the scenario with MACRO.
-Both options are False by defualt. To first calibrate the scenario and then solve that scenario with MACRO both options should be set to True.
+Both options are False by default.To first calibrate the scenario and then solve that scenario with MACRO both options should be set to True.
 
 Reporting
 =========
 
 The reporting generates specific variables related to materials, mainly Production and Final Energy.
-The resulting reporting file is generated under message_ix_models\\data\\material\\reporting_output
-with the name “New_Reporting_MESSAGEix-Materials_scenario_name.xlsx”. More detailed
-variables related to the whole energy system and emissions are not inlcuded in
-this reporting.
+The resulting reporting file is generated under message_ix_models\\data\\material\\reporting_output with the name “New_Reporting_MESSAGEix-Materials_scenario_name.xlsx”.
+More detailed variables related to the whole energy system and emissions are not included in this reporting.
 
-Reporting is executed by the following command:
+Reporting is executed by the following command::
 
-$ mix-models --url="ixmp://ixmp_dev/MESSAGEix-Materials/scenario_name" --local-data "./data" material-ix report
+    mix-models --url="ixmp://ixmp_dev/MESSAGEix-Materials/scenario_name" \
+        --local-data "./data" material-ix SSP2 report
 
-To remove any existing timeseries in the scenario the following command can be used:
-$ mix-models --url="ixmp://ixmp_dev/MESSAGEix-Materials/scenario_name" material-ix report --remove_ts True
+To remove any existing timeseries in the scenario the following command can be used::
+
+    mix-models --url="ixmp://ixmp_dev/MESSAGEix-Materials/scenario_name" material-ix \
+        SSP2 report --remove_ts True
 
 Data, metadata, and configuration
 =================================
@@ -130,7 +134,7 @@ The code relies on the following input files, stored in :file:`data/material/`:
   Final energy values to calibrate base year values for industries
 
 :file:`residual_industry_2019.xlsx`
-  Final energy values to calculate the residual industry demand.
+  Final energy values to calculate the residual industry demand
 
 :file:`nh3_fertilizer_demand.xlsx`
   Nitrogen fertilizer demand
@@ -139,7 +143,7 @@ The code relies on the following input files, stored in :file:`data/material/`:
   Techno-economic parameters for NH3 production technologies
 
 :file:`cost_conv_nh3.xlsx`
-  Cost convergance parameters for NH3 produciton technologies
+  Regional cost convergence settings for NH3 production technologies over time
 
 :file:`methanol demand.xlsx`
   Methanol demand
@@ -151,13 +155,13 @@ The code relies on the following input files, stored in :file:`data/material/`:
   Techno-economic parameters for methanol production technologies
 
 :file:`petrochemicals_techno_economic.xls`
-  Techno-economic parameters for refinery and high value chemicals production technologies
+  Techno-economic parameters for refinery and high-value chemicals production technologies
 
 :file:`steam_cracking_hist_act.csv`
-  Steam cracker historical activities
+  Steam cracker historical activities in R12 regions
 
 :file:`steam_cracking_hist_new_cap.csv`
-  Steam cracker historical capacities
+  Steam cracker historical capacities in R12 regions
 
 :file:`NTNU_LCA_coefficients.xlsx`
    Material intensity (and other) coefficients for power plants based on lifecycle assessment (LCA) data from the THEMIS database, compiled in the `ADVANCE project <http://www.fp7-advance.eu/?q=content/environmental-impacts-module>`_.
@@ -172,20 +176,50 @@ The code relies on the following input files, stored in :file:`data/material/`:
    Commodity mapping (for materials) of global 11-regional MESSAGEix-GLOBIOM model to commodities of THEMIS LCA dataset.
 
 :file:`SSP_UE_dyn_input.xlsx`
-   Calibration of end-use energy demands
+   Calibration file for industry end-use energy demands
+
+:file:`SSP_UE_dyn_input_all.xlsx`
+   Calibration file for end-use energy demands
+
+:file:`iea_mappings/all_technologies.csv`
+   Mapping of MESSAGEix-GLOBIOM technologies to IEA EWEB flows and products
+
+:file:`iea_mappings/chemicals.csv`
+   Mapping of MESSAGEix-GLOBIOM industry technologies to IEA EWEB products of chemical sector flows
+
+:file:`iea_mappings/industry.csv`
+   Mapping of MESSAGEix-GLOBIOM industry technologies to IEA EWEB products of industry sector flows not covered by MESSAGEix-Materials
+
+:file:`ammonia/demand_NH3.yaml`
+   Ammonia demand in each R12 region in year 2020
+
+:file:`other/mer_to_ppp_default.csv`
+   Default conversion factors for GDP MER to PPP used in MESSAGEix-GLOBIOM SSP2 scenarios.
+   Used to create demand projections for steel/cement/aluminum/chemicals if GDP_PPP data is not in scenario
+
+:file:`buildings/LED_LED_report_IAMC_sensitivity_R12.csv`
+   Data from MESSAGEix-Buidings LED scenarios used to get steel/cement/aluminum demands from buildings
+
+:file:`buildings/report_IRP_SSP2_BL_comm_R12.csv`
+   Commerical sector data from MESSAGEix-Buidings used to get steel/cement/aluminum demands from buildings
+
+:file:`buildings/report_IRP_SSP2_BL_resid_R12.csv`
+   Residential sector data from MESSAGEix-Buidings used to get steel/cement/aluminum demands from buildings
+
+:file:`/methanol/results_material_SHAPE_comm.csv`
+   Commercial sector data from MESSAGEix-Buidings SHAPE scenario used to get wood demands from buildings to estimate resin demands
+
+:file:`/methanol/results_material_SHAPE_resid.csv`
+   Residential sector data from MESSAGEix-Buidings SHAPE scenario used to get wood demands from buildings to estimate resin demands
+
+:file:`/methanol/results_material_SSP2_comm.csv`
+   Commercial sector data from MESSAGEix-Buidings SSP2 scenario used to get wood demands from buildings to estimate resin demands
+
+:file:`/methanol/results_material_SSP2_resid.csv`
+   Residential sector data from MESSAGEix-Buidings SSP2 scenario used to get wood demands from buildings to estimate resin demands
 
 :file:`material/set.yaml`
 ----------------------------
 
-.. literalinclude:: ../../../data/material/set.yaml
+.. literalinclude:: ../../message_ix_models/data/material/set.yaml
    :language: yaml
-
-R code and dependencies
-=======================
-
-:file:`init_modularized.R`
-This code performs regression analysis to generate the steel, cement and aluminum material demands based on historical GDP per capita data.
-The regression function is later used in python data code of the relevant material to predict the future demand based on GDP and population
-projections. This code is linked to Python workflow via the Python package `rpy2`. Depending on the local R installation(s), the environment
-variables `R_HOME` and `R_USER` may need to be set for the installation to work (see `stackoverflow <https://stackoverflow.com/questions/12698877/how-to-setup-environment-variable-r-user-to-use-rpy2-in-python>`_).
-Additional dependencies include the R packages `dplyr`, `tidyr`, `readxl` and `imputeTS` that need to be installed in the R environment.

doc/whatsnew.rst

@@ -3,7 +3,7 @@ What's new
 
 Next release
 ============
-
+- Update :doc:`/material/index` (:pull:`201`).
 - Add :doc:`/material/index` (:pull:`188`, :pull:`189`).
 - Reduce log verbosity of :func:`.apply_spec` (:pull:`202`).
 

message_ix_models/cli.py

@@ -163,7 +163,7 @@ def _log_threads(k: int, n: int):
     "message_ix_models.model.water.cli",
     "message_ix_models.project.ssp",
     "message_ix_models.report.cli",
-    "message_ix_models.model.material",
+    "message_ix_models.model.material.cli",
     "message_ix_models.testing.cli",
     "message_ix_models.util.pooch",
 ]

message_ix_models/data/material/ammonia/demand_NH3.yaml

@@ -1,3 +1,17 @@
+# Ammonia Technology Roadmap IEA. 2019 Global NH3 production = 182 Mt.
+# 70% is used for nitrogen fertilizer production. Rest is 54.7 Mt.
+# 12 Mt is missing from nitrogen fertilizer part. Possibly due to low demand
+# coming from GLOBIOM updates. Also add this to the residual demand. (66.7 Mt)
+# Approximate regional shares are from Future of Petrochemicals
+# Methodological Annex page 7. Total production for regions:
+# Asia Pacific (RCPA, CHN, SAS, PAS, PAO) = 90 Mt
+# Eurasia (FSU) = 20 Mt, Middle East (MEA) = 15, Africa (AFR) = 5
+# Europe (WEU, EEU) = 25 Mt, Central&South America (LAM) = 5
+# North America (NAM) = 20 Mt.
+# Regional shares are derived. They are based on production values not demand.
+# Some assumptions made for the regions that are not explicitly covered in IEA.
+# (CHN produces the 30% of the ammonia globally and India 10%.)
+
 - R12_AFR:
     year: 2020
     value: 2.5000