EU21 runs alter pm_cesdata — but shouldn't

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

Runs with 12 regions leave pm_cesdata alone

$ dumpgdx /p/projects/remind/modeltests/remind/output/SSP2EU-NPi-AMT_2024-01-26_22.09.54/input.gdx pm_cesdata | md5sum
3c6cf8feab921b70c4288cf126797c41  -
$ dumpgdx /p/projects/remind/modeltests/remind/output/SSP2EU-NPi-AMT_2024-01-26_22.09.54/fulldata.gdx pm_cesdata | md5sum
3c6cf8feab921b70c4288cf126797c41  -

while 21-region runs do not

$ dumpgdx /p/projects/remind/modeltests/remind/output/SSP2EU-EU21-NPi-AMT_2024-02-02_22.09.35/input.gdx pm_cesdata | md5sum
f602677e3ebf67cd46c43762f195f79d  -
$ dumpgdx /p/projects/remind/modeltests/remind/output/SSP2EU-EU21-NPi-AMT_2024-02-02_22.09.35/fulldata.gdx pm_cesdata | md5sum
ad8e33cdccb933ec2565bb54933c0943  -

Differences are all over the place

$ diff -y --suppress-common-lines <( dumpgdx /p/projects/remind/modeltests/remind/output/SSP2EU-EU21-NPi-AMT_2024-02-02_22.09.35/input.gdx pm_cesdata ) <( dumpgdx /p/projects/remind/modeltests/remind/output/SSP2EU-EU21-NPi-AMT_2024-02-02_22.09.35/fulldata.gdx pm_cesdata ) | less
2005  OAS  fegab                   price            0.1332609 |	2005  OAS  fegab                   price            0.1333000
2005  OAS  fegab                   eff              0.3666335 |	2005  OAS  fegab                   eff              0.3658576
2005  OAS  fegab                   xi               0.3634717 |	2005  OAS  fegab                   xi               0.3643495
2005  OAS  fehob                   price            0.5180366 |	2005  OAS  fehob                   price            0.5198472
2005  OAS  fehob                   eff              2.3173678 |	2005  OAS  fehob                   eff              2.3194079
2005  OAS  fehob                   xi               0.2235452 |	2005  OAS  fehob                   xi               0.2241293
2005  OAS  fesob                   price            0.3653387 |	2005  OAS  fesob                   price            0.3654902
2005  OAS  fesob                   eff              0.5389722 |	2005  OAS  fesob                   eff              0.5394467
2005  OAS  fesob                   xi               0.6778432 |	2005  OAS  fesob                   xi               0.6775279
2005  OAS  feelcb                  price            0.7850647 |	2005  OAS  feelcb                  price            0.7857541

Any ideas @Renato-Rodrigues?

[Renato-Rodrigues]

I have no idea.
I searched the REMIND code for anywhere outside the calibration module that could be reassigning the pm_cesdata parameter, but I couldn't find any GAMS statement that could be doing that.

Could you give a try setting another sequential run, but now using the /p/projects/remind/modeltests/remind/output/SSP2EU-EU21-NPi-AMT_2024-02-02_22.09.35/fulldata.gdx as input.gdx just to make sure that whatever is changing the pm_cesdata values is not part of the calibration module realisation?
If pm_cesdata keeps unchanged in this additional run, we need to check the calibration module instead.

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

I set up /p/tmp/pehl/Remind/output/SSP2EU-EU21-NPi_2024-02-09_08.41.03/, but I am unsure what you are getting at. What does it matter where SSP2EU-EU21-NPi-AMT_2024-02-02_22.09.35 got its input.gdx from for the fact that pm_cesdata is different in its fulldata.gdx? The calibration realisation is not involved in that run …

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

But should pm_cesdata come out the same from this run as it did from the AMT, we will know that the modifications are somehow consistent.

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

Well, if I put the same pm_cesdata into an SSP2EU-EU21-NPi run as come out of an SSP2EU-EU21-NPi run, then they come out the same. So, consistent alterations.

$ dumpgdx /p/tmp/pehl/Remind/output/SSP2EU-EU21-NPi_2024-02-09_08.41.03/input.gdx pm_cesdata | md5sum
ad8e33cdccb933ec2565bb54933c0943  -
$ dumpgdx /p/tmp/pehl/Remind/output/SSP2U21-NPi_2024-02-09_08.41.03/fulldata.gdx pm_cesdata | md5sum
ad8e33cdccb933ec2565bb54933c0943  -

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

Might be an artefact of the "incomplete" calibration. But the input.gdx differs from the .inc file (the very thing we want to avoid through #1545):

$ diff -yw --suppress-common-lines <( tar -Oxf /p/projects/remind/inputdata/CESparametersAndGDX/CESparametersAndGDX_9affecd97b6d34c294eca6571885263e1bb1de90.tgz indu_subsectors
-buil_simple-tran_edge_esm-POP_pop_SSP2EU-GDP_gdp_SSP2EU-En_gdp_SSP2EU-Kap_debt_limit-Reg_2b1450bc.inc | sed -n 's/pm_cesdata("\(.*\)","\(.*\)","\(.*\)","\(.*\)") = \(.*\);/\1 \2 \3 \4 \5/p' | sort | column -t ) <( dumpgdx /p/projects/remind/modeltests/remind/output/SSP2EU-
EU21-NPi-AMT_2024-02-02_22.09.35/input.gdx pm_cesdata | grep '^[0-9]' | sort ) | head
2005  CAZ  enb                     eff              11.528022 |	2005  CAZ  enb                     eff              11.532002
2005  CAZ  enb                     quantity         0.0484244 |	2005  CAZ  enb                     quantity         0.0484114
2005  CAZ  enb                     xi               0.0867451 |	2005  CAZ  enb                     xi               0.0867152
2005  CAZ  en_cement               eff              1.0584695 |	2005  CAZ  en_cement               eff              1.0585486
2005  CAZ  en_cement_non_electric  eff              1.6356880 |	2005  CAZ  en_cement_non_electric  eff              1.6352188
2005  CAZ  en_cement_non_electric  quantity         0.0137235 |	2005  CAZ  en_cement_non_electric  quantity         0.0137264
2005  CAZ  en_cement_non_electric  xi               0.6113635 |	2005  CAZ  en_cement_non_electric  xi               0.6115389
2005  CAZ  en_cement               quantity         0.0224475 |	2005  CAZ  en_cement               quantity         0.0224458
2005  CAZ  en_cement               xi               0.9447602 |	2005  CAZ  en_cement               xi               0.9446897
2005  CAZ  en_chemicals            eff              1.0093978 |	2005  CAZ  en_chemicals            eff              1.0093996

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

For the last SSP2EU-EU21-NPi run with the complete calibration before that actually finished, everything checks out:

$ md5sum <( dumpgdx SSP2EU-EU21-NPi-AMT_2023-12-22_22.08.54/input.gdx ) <( dumpgdx SSP2EU-EU21-NPi-AMT_2023-12-22_22.08.54/fulldata.gdx )
c6f95f73801ed0177817d82e5024fde0  /dev/fd/63
c6f95f73801ed0177817d82e5024fde0  /dev/fd/62

$ diff -w <( tar -Oxf /p/projects/remind/inputdata/CESparametersAndGDX/CESparametersAndGDX_6fd80d4748d4715e9f71da31d9d5494ccf2a3cbe.tgz indu_subsectors-buil_simple-tran_edge_esm-POP_pop_SSP2EU-GDP_gdp_SSP2EU-En_gd
p_SSP2EU-Kap_debt_limit-Reg_2b1450bc.inc | sed -n 's/pm_cesdata("\(.*\)","\(.*\)","\(.*\)","\(.*\)") = \(.*\);/\1 \2 \3 \4 \5/p' | sort | column -t ) <( dumpgdx /p/projects/remind/modeltests/remind/output/SSP2EU-EU21-NPi-AMT_2023-12-22_22.08.54/input.gdx pm_cesdata | grep '
^[0-9]' | sort )

[Renato-Rodrigues]

Thk you for all the tests.
I will give a try to look if I can find some reason behind this, but I am less worried because after we run a NPi scenario things apparently become consistent afterwards.

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

Nothing to look for, this was an artefact of the calibration.