Pandas KeyError in run_generate_validation_comparisons_site.py

[krosenfeld-IDM]

Continuing to log errors. After all simulations and analyzers on COMPS run (see #63 and #61). I get another pandas related error in the plot creation stage:

INI File Used: /home/krosenfeld/playpen/malaria-model_validation/idmtools.ini
plotting with subset = all.
Folder /home/krosenfeld/playpen/malaria-model_validation/report/_plots was created
Traceback (most recent call last):
  File "/home/krosenfeld/playpen/malaria-model_validation/env/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3653, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 147, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 176, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Incidence'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/krosenfeld/playpen/malaria-model_validation/create_plots/run_generate_validation_comparisons_site.py", line 95, in <module>
    run(subset=args.subset)
  File "/home/krosenfeld/playpen/malaria-model_validation/create_plots/run_generate_validation_comparisons_site.py", line 49, in run
    generate_age_incidence_outputs(coord_csv, simulation_output_filepath, base_reference_filepath, plot_output_filepath,
  File "/home/krosenfeld/playpen/malaria-model_validation/create_plots/helpers_coordinate_each_relationship.py", line 44, in generate_age_incidence_outputs
    combined_df = prepare_inc_df(coord_csv, simulation_output_filepath, base_reference_filepath,
  File "/home/krosenfeld/playpen/malaria-model_validation/create_plots/helpers_reformat_sim_ref_dfs.py", line 302, in prepare_inc_df
    sim_df['Incidence'] = sim_df['Incidence'] * sim_df['p_detect_case']
  File "/home/krosenfeld/playpen/malaria-model_validation/env/lib/python3.9/site-packages/pandas/core/frame.py", line 3761, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/krosenfeld/playpen/malaria-model_validation/env/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3655, in get_loc
    raise KeyError(key) from err
KeyError: 'Incidence'
[Tue Jul 25 10:18:20 2023]
Error in rule plot:
    jobid: 0
    input: /home/krosenfeld/playpen/malaria-model_validation/simulations/COMPS_ID/Suite, /home/krosenfeld/playpen/malaria-model_validation/simulations/COMPS_ID/eradication_found, /home/krosenfeld/playpen/malaria-model_validation/simulations/COMPS_ID/sif.id, COMPS_ID/chonyi_1999_download, COMPS_ID/ngerenya_1999_download, COMPS_ID/dielmo_1990_download, COMPS_ID/ndiop_1993_download, COMPS_ID/ebolakounou_1997_download, COMPS_ID/koundou_1997_download, COMPS_ID/dongubougou_1999_download, COMPS_ID/sotuba_1999_download, COMPS_ID/matsari_1970_download, COMPS_ID/rafin_marke_1970_download, COMPS_ID/sugungum_1970_download, COMPS_ID/navrongo_2000_download, COMPS_ID/laye_2007_download, COMPS_ID/dapelogo_2007_download
    output: /home/krosenfeld/playpen/malaria-model_validation/simulations/COMPS_ID/plot_completed

RuleException:
CalledProcessError in file /home/krosenfeld/playpen/malaria-model_validation/simulations/snakefile, line 167:
Command 'set -euo pipefail;  python3 /home/krosenfeld/playpen/malaria-model_validation/create_plots/run_generate_validation_comparisons_site.py --subset all' returned non-zero exit status 1.
  File "/home/krosenfeld/playpen/malaria-model_validation/simulations/snakefile", line 167, in __rule_plot
  File "/home/krosenfeld/playpen/malaria-model_validation/env/lib/python3.9/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message

[krosenfeld-IDM]

I found the error. In AnnualSummaryReportAnalyzer.py the reduce step in the groupby was going across both columns and should have been done just along the individual columns. Closed by #62 .