Add technoeconomic cost projections and related data

[measrainsey]

This PR adds a module that projects regionally-differentiated investment costs and fixed O&M costs for technologies.

The module projects costs in a specified reference region for all technologies by applying a decay/reduction rate. Then, costs for all other regions are calculated through one of the following methods:

1. Constant ratio: all regions' base year relative ratio to the reference region is applied through the projection period, so all regions follow the same reduction rate as the reference region
2. Convergence year: all regions' costs converge to the reference region by a specified year
3. GDP per capita-adjusted: projected GDP per capita from SSP scenarios are used to adjust the regional cost ratios into the future, using on the assumption that technology costs follow GDP per capita

Note that all costs calculations mentioned above are for investment costs. Fixed O&M costs are assumed to be a specified fraction of investment costs.

The main function that projects costs is the create_cost_projections() function, which takes the following inputs:

• Node (R11, R12, or R20)
• Reference region (by default NAM for each node)
• Base year (by default, 2021 when possible, if not fall back to use 2020)
• SSP scenario version (updated/review or previous)
• Scenario (SSP1, SSP2, SSP3, SSP4, SSP5, LED)
• Projection method (learning, GDP adjusted, or convergence with splines)
• Convergence year
• Rate of fixed cost change over time
• Output format (MESSAGE, IAMC)

The create_cost_projections() outputs a class object with two dataframes (one for investment costs, one for fixed O&M costs).

How to review

Documentation and usability (@adrivinca)

• Documentation has been added, under General purpose modeling tools > Technoeconomic investment and fixed O&M costs projection. The following link leads straight to the section in the latest docs test build: https://iiasa-energy-program-message-ix--99.com.readthedocs.build/projects/models/en/99/api/tools.html#technoeconomic-investment-and-fixed-o-m-costs-projection-tools-costs
• The demo.py script in the module gives some example of how to use the main function that creates the costs projections.
• Ensure that a colleague who has not used this tool before (a) understand and (b) use it, based on above materials provided.
  • Reviewer: read through, ask questions (as comments on specific lines, or in this Slack thread).
  • Author (@measrainsey): make edits to docs sufficient to preclude such questions.

Technical, implementation (@khaeru)

• Improve usage of the Config class.
• Other comments on the code.

PR checklist

• Continuous integration checks all ✅
• Add or expand tests; coverage checks both ✅
• Add, expand, or update documentation.
• Update doc/whatsnew.

[codecov]

Codecov Report

Attention: Patch coverage is 93.47368% with 31 lines in your changes missing coverage. Please review.

Project coverage is 76.8%. Comparing base (7dc5385) to head (3a0d7ee).
Report is 3820 commits behind head on main.

Files with missing lines	Patch %	Lines
message_ix_models/tools/costs/demo.py	0.0%	14 Missing ⚠️
message_ix_models/tools/costs/projections.py	84.7%	14 Missing ⚠️
..._ix_models/tools/costs/regional_differentiation.py	97.9%	2 Missing ⚠️
message_ix_models/tools/costs/config.py	97.2%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main     #99     +/-   ##
=======================================
+ Coverage   75.5%   76.8%   +1.2%     
=======================================
  Files        101     112     +11     
  Lines       6675    7157    +482     
=======================================
+ Hits        5046    5497    +451     
- Misses      1629    1660     +31     

Files with missing lines	Coverage Δ
message_ix_models/tests/tools/costs/test_decay.py	100.0% <100.0%> (ø)
message_ix_models/tests/tools/costs/test_gdp.py	100.0% <100.0%> (ø)
...ge_ix_models/tests/tools/costs/test_projections.py	100.0% <100.0%> (ø)
...tests/tools/costs/test_regional_differentiation.py	100.0% <100.0%> (ø)
message_ix_models/tools/costs/__init__.py	100.0% <100.0%> (ø)
message_ix_models/tools/costs/decay.py	100.0% <100.0%> (ø)
message_ix_models/tools/costs/gdp.py	100.0% <100.0%> (ø)
message_ix_models/tools/exo_data.py	98.3% <100.0%> (+<0.1%)	⬆️
message_ix_models/util/scenarioinfo.py	100.0% <100.0%> (ø)
message_ix_models/tools/costs/config.py	97.2% <97.2%> (ø)
... and 3 more

[khaeru]

This is a great start for a first message-ix-models PR!

I leave some comments about general pandas usage. I think with these applied the code will slim down considerably, making the intended behaviour more clear. That can then be targeted in a next review.

[volker-krey]

It would be nice to include needed package dependencies into message-ix-models pip install command. I had to manually install scikit-learn manually for the code to run.

[volker-krey]

In IAMC data template output, the Model and Unit columns are missing and should be added.

[volker-krey]

Time steps (currently 10-year time steps) and set of regions for which to produce projections should become easily configurable. Also, region names used in GDP/cap specification, IEA WEO mapping, etc. then need to become unambiguous. I would suggest to use RXX_NAME (e.g., R11_NAM, R12_CHN) for global model regions.

[measrainsey]

I think we should discuss whether or not it's worthwhile to implement the GDP adjustment in the cost projections.
The way the GDP is used in the spreadsheet is as the following:

1. For each region-year, the ratio of GDP is calculated relative to NAM.
2. A linear regression is applied using 2020/2021 regionally differentiated cost ratios vs regionally differentiated GDP ratios.
3. The linear regression coefficients and annual GDP ratios are then used to predict new (yearly) regionally differentiated cost ratios.

It seems the methodology was set in place to use these new regionally differentiated cost ratios with the NAM cost projections (from the learning rates) to get regional costs. However, there is an if-then flag in the spreadsheet that states if a certain γ (gamma) coefficient is 0, then don't use the GDP adjustment but instead just make each region's cost equal to NAM's cost for that year. And the γ always seems to be zero, so the GDP adjustment is technically never applied in the spreadsheet.

I tried a version of the cost module where this GDP is applied, and it leads to costs not converging at all. See the below example for coal_ppl in the SSP2 scenario.

For reference, here's what the GDP ratios look like under SSP1, SSP2, and SSP3:

[measrainsey]

Following a discussion with @volker-krey, here is what it would look like to keep the learning rates scenario constant (in this case, at SSP2) with varying SSP GDP scenarios. Showing coal_ppl, gas_cc, and solar_pv_ppl as examples:

[measrainsey]

I've updated the code to allow for the user to specify one of three methods of cost projection:

• learning rates (varying by SSP scenario)
• GDP (varying by SSP scenario)
• cost convergence at year specified, with an applied spline regression (technically this does not vary by SSP scenario, moreso by year of convergence)

Here is a visualization of the different methods of projections across scenarios and regions for gas_cc:

And here is another way of visualizing the same information, this time comparing the 3 methods more clearly:

[measrainsey]

An update on the current state of the PR

The module derives cost projections using the same three general methods as mentioned previously:

• Learning
• Convergence
• GDP-adjusted

The methdology for the GDP-adjusted cost projections have been modified to prevent large jumps in cost projections.

I have hacked together some quick input data for a low-energy demand (LED) scenario, but these values need considerate validation and verification.

The create_cost_projections() has also been updated to take the following as inputs:

• Node (R11, R12, or R20)
• Reference region (by default NAM for each node)
• Base year (by default, 2021 when possible, if not fall back to use 2020)
• SSP scenario version (updated/review or previous)
• Scenario (SSP1, SSP2, SSP3, SSP4, SSP5, LED)
• Projection method (learning, GDP adjusted, or convergence with splines)
• Convergence year
• Rate of fixed cost change over time
• Output format (MESSAGE, IAMC)

The create_cost_projections() outputs two dataframes (one for investment costs, one for fixed O&M costs).

The demo.py script has been updated to show examples of how to utilize the tool to obtain cost projections.