add NREL/ATB electricity costs for the US

[finozzifa]

Goals

This pull request contains the changes performed by @danielelerede-oet and myself (@finozzifa) as agreed with @martacki and Johannes (could not find his github alias).

Proposed final goal: this pull request is the first (intermediate) step of a set of changes that aim at granting the possibility to the model users to use country-specific cost assumptions.

Goal of this pull request: This work in particular proposes an intermediate step and creates a sub-folder outputs/US, where the outputs/costs_yyyy.csv files are copied and updated with NREL/ATB data.

Input and output schemas

NREL/ATB input values and schema

The NREL/ATB electricity data source is available here.

We require the cost assumptions for the years 2020, 2025, 2030, 2035, 2040, 2045 and 2050. The cost assumptions for 2020 are obtained from atb_e_2022 dataset, whereas those for the other years from the atb_e_2024 dataset. The schema of these files is unfortunately slightly different. Namely:

Schema of atb_e_2022

 #   Column                 Non-Null Count   Dtype  
---  ------                 --------------   -----  
 0   atb_year               286998 non-null  int64  
 1   core_metric_key        286812 non-null  object 
 2   core_metric_parameter  286998 non-null  object 
 3   core_metric_case       286998 non-null  object 
 4   crpyears               286998 non-null  object 
 5   technology             286998 non-null  object 
 6   technology_alias       285510 non-null  object 
 7   techdetail             286998 non-null  object 
 8   display_name           285510 non-null  object 
 9   default                285510 non-null  float64
 10  scenario               286998 non-null  object 
 11  core_metric_variable   286998 non-null  object 
 12  units                  286998 non-null  object 
 13  value                  286998 non-null  float64

Schema of atb_e_2024

 #   Column                 Non-Null Count   Dtype  
---  ------                 --------------   -----  
 0   atb_year               572232 non-null  int64  
 1   core_metric_key        572232 non-null  object 
 2   core_metric_parameter  572232 non-null  object 
 3   core_metric_case       572232 non-null  object 
 4   tax_credit_case        212388 non-null  object 
 5   crpyears               572232 non-null  object 
 6   technology             572232 non-null  object 
 7   technology_alias       572232 non-null  object 
 8   techdetail             572232 non-null  object 
 9   techdetail2            506772 non-null  object 
 10  resourcedetail         488328 non-null  object 
 11  display_name           572232 non-null  object 
 12  default                572232 non-null  int64  
 13  scale                  506772 non-null  object 
 14  maturity               506772 non-null  object 
 15  scenario               572232 non-null  object 
 16  core_metric_variable   572232 non-null  int64  
 17  units                  572232 non-null  object 
 18  value                  572232 non-null  float64

We consider a subset of such columns. This can be configured with the configuration from the config.yaml from config["nrel_atb"]["nrel_atb_columns_to_keep"]. For this pull request, the columns taken are

["atb_year", "core_metric_parameter", "core_metric_case", "core_metric_variable", "technology", "technology_alias", "display_name", "scenario", "units", "value", "tax_credit_case"]

where in particular

• atb_year equals the year in the file name. For example atb_year = 2022 for atb_e_2022 or atb_year = 2024 for atb_e_2024
• core_metric_variable equals the year for which the cost assumption is made
• core_metric_parameter has various values. We consider

Column value	Unit
CAPEX	$/kW
CF
Fixed O&M	$/KW-yr
Variable O&M	$/MWh
Fuel	$/MWh
Additional OCC	$/KW
WACC Real

• scenario equals Moderate, Conservative, Advanced
• tax_credit_case equals ITC, None, PTC + ITC, PTC

outputs/US/costs_yyyy.csv values and schema

NREL/ATB column	outputs/US/costs_yyyy.csv column	Notes
display_name	technology
core_metric_parameter	parameter
units	unit
	source	taken from config["nrel_atb"]["nrel_atb_source_link"]
	further description	left blank
atb_year	currency_year
scenario	scenario
core_metric_case	financial_case
tax_credit_case	tax_credit_case

Changes

Changes in the workflow

The workflow has been updated as follows:

1. Step 1: rule compile_cost_assumptions generates outputs/costs_yyyy.csv files
2. Step 2: rule compile_cost_assumptions_nrel takes the outputs/costs_yyyy.csv files, reads-in the nrel/atb inputs, processes them and outputs a dedicated set of costs for the US in outputs/US/costs_yyyy.csv

The "high level" description of what compile_cost_assumptions_nrel.py does

The "high level" description of what the script does is:

• it loops through the years 2020, 2025, 2030, 2035, 2040, 2045, 2050
• for each year, it reads the corresponding outputs/cost_yyyy.csv file
• for each year, it reads the corresponding atb input file, filters the rows corresponding to the given year and extracts the necessary columns
• for each year, it normalizes the Fixed O&M by Additional OCC (for retrofits technologies) or CAPEX (for any other technology) and changes its unit from $/KW-yr to %-yr
• for each year, it performs the following technology renaming. NREL/ATB contains many more technologies than those listed in the table below. Those technologies are included anyway in the final outputs/US/costs_year.csv files but not renamed.

display_name	PyPSA technology name
Coal-new -> 2nd Gen Tech	coal
Coal-new	coal
NG F-Frame CT	CCGT
NG Combustion Turbine (F-Frame)	CCGT
Hydropower - NPD 1	hydro
Hydropower - NSD 1	ror
Pumped Storage Hydropower - National Class 1	PHS
Nuclear - Large	nuclear
Nuclear - AP100	nuclear
Geothermal - Hydro / Flash	geothermal
Land-Based Wind - Class 1	onwind
Land-Based Wind - Class 1 - Technology 1	onwind
Offshore Wind - Class 1	offwind
Utility PV - Class 1	solar-utility
Commercial PV - Class 1	solar-rooftop
Utility-Scale Battery Storage - 6Hr	battery storage
Biopower - Dedicated	biomass
CSP - Class 2	csp-tower

• for each year, it updates the cost values in the corresponding outputs/US/costs_year.csv file so that, the script
  • appends NREL/ATB technology entries not present in the original file outputs/costs_year.csv
  • leaves untouched technology entries not present in the NREL/ATB dataset
  • updates the technology entries present in both the NREL/ATB dataset and outputs/costs_year.csv, with NREL/ATB values

Other noteworthy changes

environment.yaml

We choose to take the the input datasets atb_e_2022 and atb_e_2024 in parquet format. This is because the corresponding csv files have a size which is significantly larger. This choice brings about the following addition to the environment.yaml file

  - pyarrow
  - fastparquet

unit tests

We added a test folder to include unit tests for the functions included in scripts/compile_cost_assumptions_nrel.py

inputs/manual_input.csv

New technologies have been added to manual_inputs.csv.

Checklist

• Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
• Data source for new technologies is clearly stated.
• Newly introduced dependencies are added to environment.yaml (if applicable).
• A note for the release notes doc/release_notes.rst of the upcoming release is included.
• I consent to the release of this PR's code under the GPLv3 license.

[danielelerede-oet]

Thanks @finozzifa for the detailed description. I would just add something about manual_inputs.csv. The file now contains technical information about the new technologies added from the NREL ATB dataset (i.e. mainly efficiencies, lifetimes and capture rates) and all the parameters required for the characterization of hydrogen production technologies (e.g. biomass gasification, different kinds of electrolyzers, etc.) as from the JRC-EU-TIMES Model dataset.