Problem adding Annual Technology Emissions to config file

[FraGard]

I got csv result files for a test models using otoole and would like now to pick some of the results and convert them to the iamc format. I wrote a config file for that, attached (I just turned it to a txt file cause I was not allowed to upload yaml). It worked well for production by technology annual and for total capacity annual but I cannot make it work for the emissions. I get a long error, the end of which is:

'Level values must be unique: [nan, nan] on level 2'
config.txt
AnnualTechnologyEmission.csv

I upload also the csv result file of the emissions, for reference

[willu47]

Hi @FraGard - actually the full text of the long-error would be useful to see. Please could you paste it here in a "code block"?

[FraGard]

Thanks, sure. Note that I installed the package in my base environment. Perhaps not very clever.

(base) francescogardumi@Francescos-MBP iamc % osemosys2iamc osemosys_inputs osemosys_outputs config.yaml iamc_outputs.xlsx
Traceback (most recent call last):
  File "/Users/francescogardumi/anaconda3/bin/osemosys2iamc", line 8, in <module>
    sys.exit(entry_point())
  File "/Users/francescogardumi/anaconda3/lib/python3.10/site-packages/osemosys2iamc/resultify.py", line 404, in entry_point
    all_data = main(config, inputs_path, results_path)
  File "/Users/francescogardumi/anaconda3/lib/python3.10/site-packages/osemosys2iamc/resultify.py", line 359, in main
    data.index = data.index.set_levels(data.index.levels[2].map(iso_mapping), level=2)
  File "/Users/francescogardumi/anaconda3/lib/python3.10/site-packages/pandas/util/_decorators.py", line 331, in wrapper
    return func(*args, **kwargs)
  File "/Users/francescogardumi/anaconda3/lib/python3.10/site-packages/pandas/core/indexes/multi.py", line 943, in set_levels
    idx._set_levels(
  File "/Users/francescogardumi/anaconda3/lib/python3.10/site-packages/pandas/core/indexes/multi.py", line 826, in _set_levels
    new_codes = self._verify_integrity(levels=new_levels)
  File "/Users/francescogardumi/anaconda3/lib/python3.10/site-packages/pandas/core/indexes/multi.py", line 418, in _verify_integrity
    raise ValueError(
ValueError: Level values must be unique: [nan, nan] on level 2

[FraGard]

I tried this, copied-pasted from OSeMBE_ECEMF/config.yaml:

- iamc_variable: 'Emissions|CO2|Energy|Supply|Electricity'
  emission: [CO2]
  unit: ktCO2/yr
  osemosys_param: AnnualTechnologyEmission

But I got the error:

(base) francescogardumi@Francescos-MBP iamc % osemosys2iamc osemosys_inputs osemosys_outputs config.yaml iamc_outputs
Traceback (most recent call last):
  File "/Users/francescogardumi/anaconda3/bin/osemosys2iamc", line 8, in <module>
    sys.exit(entry_point())
  File "/Users/francescogardumi/anaconda3/lib/python3.10/site-packages/osemosys2iamc/resultify.py", line 404, in entry_point
    all_data = main(config, inputs_path, results_path)
  File "/Users/francescogardumi/anaconda3/lib/python3.10/site-packages/osemosys2iamc/resultify.py", line 318, in main
    data = extract_results(results, technologies)
UnboundLocalError: local variable 'technologies' referenced before assignment

[willu47]

I think this might be caused by not having the openentrance package installed in editable mode.

Did you follow the installation instructions in the readme to install the openentrance package like so:

pip install -e git+https://github.com/openENTRANCE/openentrance.git@main#egg=openentrance
pip install git+https://github.com/osemosys/osemosys2iamc@main#egg=osemosys2iamc

[FraGard]

Changing the order of the two command lines above did not solve the problem. I uninstalled both packages, then reinstalled them in the order above, then tried what I had tried in my comment above and got the same error. To be noted that the config for Production By Activity Annual and Total Capacity Annual works fine and produces the expected results (despite the region being RE1).

[willu47]

Hi @FraGard - I think I've spotted the issue. You need to use the key emissions not emission in your config file. A stupid detail, but critical as we have a rather fragile hard-coded search for this key word...

[FraGard]

Thanks @willu47 , I have changed it to emissions as follows

- iamc_variable: 'Emissions|CO2|Energy|Supply|Electricity'
  emissions: [CO2]
  unit: ktCO2/yr
  osemosys_param: AnnualTechnologyEmission

And the error goes back to

(base) francescogardumi@Francescos-MBP iamc % osemosys2iamc osemosys_inputs osemosys_outputs config.yaml iamc_outputs
Traceback (most recent call last):
  File "/Users/francescogardumi/anaconda3/bin/osemosys2iamc", line 8, in <module>
    sys.exit(entry_point())
  File "/Users/francescogardumi/anaconda3/lib/python3.10/site-packages/osemosys2iamc/resultify.py", line 404, in entry_point
    all_data = main(config, inputs_path, results_path)
  File "/Users/francescogardumi/anaconda3/lib/python3.10/site-packages/osemosys2iamc/resultify.py", line 359, in main
    data.index = data.index.set_levels(data.index.levels[2].map(iso_mapping), level=2)
  File "/Users/francescogardumi/anaconda3/lib/python3.10/site-packages/pandas/util/_decorators.py", line 331, in wrapper
    return func(*args, **kwargs)
  File "/Users/francescogardumi/anaconda3/lib/python3.10/site-packages/pandas/core/indexes/multi.py", line 943, in set_levels
    idx._set_levels(
  File "/Users/francescogardumi/anaconda3/lib/python3.10/site-packages/pandas/core/indexes/multi.py", line 826, in _set_levels
    new_codes = self._verify_integrity(levels=new_levels)
  File "/Users/francescogardumi/anaconda3/lib/python3.10/site-packages/pandas/core/indexes/multi.py", line 418, in _verify_integrity
    raise ValueError(
ValueError: Level values must be unique: [nan, nan, nan] on level 2

[FraGard]

This is how it is written in the OSeMBE config:

- iamc_variable: 'Emissions|CO2|Energy|Supply|Electricity'
  emission: [CO2]
  unit: ktCO2/yr
  osemosys_param: AnnualTechnologyEmission

That's where I had taken from

[willu47]

Yep, the OSeMBE config is also incorrect.