Flexible setup of constraints for cooling technologies

[adrivinca]

The PR changes some hard code to a flexible adjustment of constraints on cooling technologies, to improve flexibility

The cooling technologies are added as add-on to power plants (parent technologies). in some scenarios, however, the power plant are subject to constraints e.g. for phase-out, and the cooling tecnhologies growth_activity_lo need to be adjusted accordingly to avoid infeasibilities. Before this was done manually, now I added a function relax_growth_constraint that should do it automatically for each specific scenario.
I also added a test function to test the behavior of the function.

How to review

Check the code and test are in line with the standards.
Eventually run the code on a scenario at choice and see it the model solves mix-models --url=ixmp://ixmp_dev/%model%/%scenario% water-ix --regions=R12 cooling , maybe @measrainsey

PR checklist

• Continuous integration checks all ✅
• Add or expand tests; coverage checks both ✅
• Update doc/whatsnew.

[codecov]

Codecov Report

Attention: Patch coverage is 88.52459% with 7 lines in your changes missing coverage. Please review.

Project coverage is 75.5%. Comparing base (b15268e) to head (eb7aa18).
Report is 19 commits behind head on main.

Files with missing lines	Patch %	Lines
...essage_ix_models/model/water/data/water_for_ppl.py	88.3%	5 Missing ⚠️
message_ix_models/model/water/cli.py	0.0%	1 Missing ⚠️
message_ix_models/model/water/data/water_supply.py	0.0%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #242     +/-   ##
=======================================
- Coverage   76.4%   75.5%   -0.9%     
=======================================
  Files        203     203             
  Lines      15503   15546     +43     
=======================================
- Hits       11855   11748    -107     
- Misses      3648    3798    +150     

Files with missing lines	Coverage Δ
message_ix_models/model/water/report.py	7.9% <ø> (ø)
...odels/tests/model/water/data/test_water_for_ppl.py	100.0% <100.0%> (ø)
message_ix_models/model/water/cli.py	33.0% <0.0%> (-0.4%)	⬇️
message_ix_models/model/water/data/water_supply.py	77.2% <0.0%> (+0.8%)	⬆️
...essage_ix_models/model/water/data/water_for_ppl.py	94.0% <88.3%> (-0.7%)	⬇️

... and 7 files with indirect coverage changes

[adrivinca]

Thanks Frido and Meas.

I am not sure why this error appears, but I also had it on my machine, probably to do with some pandas update?

FAILED message_ix_models/tests/model/water/data/test_water_for_ppl.py::test_cool_tec[no_climate] - IndexError: index 0 is out of bounds for axis 0 with size 0
FAILED message_ix_models/tests/model/water/data/test_water_for_ppl.py::test_cool_tec[6p0] - IndexError: index 0 is out of bounds for axis 0 with size 0```

[glatterf42]

All in all, good work as far as the coding goes :)
However, I'd like to see several small improvements and please check why the tests are failing on this PR and ensure they don't do that.

[glatterf42]

I am not sure why this error appears, but I also had it on my machine, probably to do with some pandas update?

FAILED message_ix_models/tests/model/water/data/test_water_for_ppl.py::test_cool_tec[no_climate] - IndexError: index 0 is out of bounds for axis 0 with size 0
FAILED message_ix_models/tests/model/water/data/test_water_for_ppl.py::test_cool_tec[6p0] - IndexError: index 0 is out of bounds for axis 0 with size 0```

To figure this out, you can recreate a virtual environment locally that has the same pandas and numpy and Python versions installed as one of the failing workflows. Then, try to run the tests in there and use a debugger or add auxiliary output print() to study the variables as you go.

What the error message says is that years[years > x] is a numpy array of size 0, so there's not a single element inside. Thus, when you try to access the [0]th element, numpy complains. That could be because x is too large or maybe years doesn't contain a single value to begin with.

[glatterf42]

I'm sorry, I tried to rebase this branch on main to keep it up to date (since some of @measrainsey's branches were merged earlier today), but this seems to have broken the tests again.
Thanks for fixing them before! If we fix them now and add this PR to doc/whatsnew (and @measrainsey's review comes in), I think we're close to getting this merged :)

[glatterf42]

The errors were caused by updating sdmx1 to 2.19.0, let's see as to why exactly this happens...
Thanks to @khaeru for fixing this issue with the v2.19.1 release :)

[khaeru]

@adrivinca given the name of this branch (wat_SSP_upd2) I assume this is related to #234, so I have added it in the list there. If that's not the case, please clarify.

[adrivinca]

thanks, so the sdmx1 issue is solves, right?
@khaeru you are right, there will be many other, I will remember to add them to #234

I am quite in a tight timeline, so I'm happy with the current changes. I might add a couple of things before merge

[glatterf42]

I tried to rebase this branch on itself to reword the commit messages, but that accidentally picked up several commits not belonging here, not entirely sure why. So I'll not use that route; instead, we should squash and merge this PR, providing a fitting commit message.

For the future: @adrivinca, we try to have all commit messages adhere to the same form: using present tense to describe why changes are made more than what changed (which should be apparent from the code) in a short sentence. So e.g. "Fix breaking test" rather than "Fixing breaking test".

I also added this PR to doc/whatsnew, so I think we can merge it now, if you like. And I would appreciate not adding many things/anything of significance to a PR after it has been reviewed and approved unless the review asked for specific things.

[adrivinca]

Thanks for helping out. just wanted to say there is still something I am adding. And realized some of the last edits are not relaxing all the constraints I wanted. So let's not merge yet, thanks

[khaeru]

Understood—please notify again when you think it's ready for a re-review.

[adrivinca]

Hi, I had to do some changes to my same updates.
in particular I realized that the growth_activity_lo constraints was causing many infeasibility issues because many parent technologies are forced to shut down in different ways (with bound activities, or lack of input data", ect). trying to accommodate all these exceptions lead to a small and not useful dataframe. That's why I released that constraint and imposed an upper bound on all cooling technologies.
in other words, now cooling techs can be faded out quickly, but the model still can only switch to others (e.g. from once through to air cooling) withing 5% growth constraints, so that transitions should remain smooth.

SSP updated model seem to solve with unscaled infeasibilities, or normally with scaind = 1. But this is a know issue of the water module that we will try to tackle in another PR.

I recycled the relax_growth_constraint I did for growth_activity_lo, now for growth_activity_up, and adjusted the test accordingly.

One only thing that is not really good is in set.yaml line 958

    # - [solar_th_ppl, cooling__solar_th_ppl]
    - [csp_sm1_ppl, cooling__csp_sm1_ppl]
    - [csp_sm3_ppl, cooling__csp_sm3_ppl]

in previous scenarios (e.g. ENGAGE) the solar technology was called solar_th_ppl, now in the SSP updates it is different.
there is data and parameter to make both nomenclature work, but if I run, in the current case a SSP updated scenario, the code breaks when reading the yaml file with this error
ValueError: The index set 'technology' does not have an element 'solar_th_ppl'!

on the other hand, older models might break because they do not have the new technology names. So, I am unsure on how to deal with this.

[glatterf42]

Thanks for the update :)
Is solar_th_ppl the only name that changed from ENGAGE to the SSP Update scenarios? If not, we can look at how this was done before and do the same here.

I think the deciding question is: do you want your code to work with the old scenarios, too, or just with the SSP scenarios?
If it's supposed to work with all scenarios and nothing of the sort has been done before, then we should come up with a way to change the names we're looking for based on the scenario name. This could e.g. be a mapping of the sort:

{
    "ENGAGE": "solar_th_ppl",
    "SSP Update": "some new name",
}

However, this looks a lot like hardcoding, which isn't great. And I can't quite believe that nothing of the sort had to be handled before, so there should be an existing solution.
If it's only supposed to work with one scenario (which could be a valid starting point even if you intend the first option eventually), it's fine to hardcode the technology name, but then your docstrings of these functions (and maybe even the top level of your module) should include warnings that they only work for the specific scenarios [x, y, z].

[adrivinca]

thanks for your input!
actually I was looking at ENGAGE scenario of the latest generations (ENGAGE_SSP2_v4.1.8.3.1_T4.5v2_r3.1) and they also have the newest solar techs. solar_th_ppl is defines (while in SSP2 Updated) they have been stripped off, but is is not used.

So I would rather keep the new set of technologies, and happy to write a warning saying that the scenario version should be more recent than ENGAGE_SSP2_v4.1.8.3.1_T4.5v2_r3.1 for instance. Where would it be best to write this warning?

[measrainsey]

thanks @adrivinca for your hard work to implement these changes!

on my end, i can confirm that running mix-models --url=ixmp://ixmp_dev/%model%/%scenario% water-ix --regions=R12 cooling does indeed successfully solve. so i will mark my review (which is limited to this specific part) as approved

[khaeru]

I think the appropriate place for warnings is the documentation: https://github.com/iiasa/message-ix-models/blob/main/doc/water/index.rst?plain=1 , near the top.

You could add for instance:

.. note:: The current version of :mod:`.model.water` is configured to work with scenarios that have technology set elements including ``csp_sm1_ppl`` and ``csp_sm3_ppl``.
   Previous versions (in :mod:`message_ix_models` v2024.8.6 and earlier) were configured to work with scenarios including the technology ``solar_th_ppl``.
   See further discussion at :pull:`242`.

Some other thoughts:

• Another approach in documentation is to give a list of exact base scenario URLs with which the code is known to work. See https://docs.messageix.org/projects/models/en/latest/transport/index.html#base-scenarios
• It is indeed possible to inspect the base scenario and add the appropriate configuration based on its contents. This means that the values are not fixed in set.yaml, but rather added to the data structure after this file is read. We do that in .model.transport. But I think it is likely out of scope here.
• As we can see, the presence of "ENGAGE" in a scenario name does not tell us anything useful, because very different code has been used to prepare such scenarios, and we don't have a clear record of what code was used for which (model name, scenario name, version). That's why my suggested text above says directly what specific contents of scenarios, rather than just alluding to "ENGAGE scenarios" which is a poorly-defined and disparate set.