Bug while reconciling with the Top Down approach

[D3NE3N]

Hey,

First of all, thank you very much for your great work on this project and providing the package. It contributes a lot to the domain of hierarchical reconciliation.

However, we've encountered a bug when applying the Top-Down approach. In certain cases, the Top-Down approach performs correctly, while in others, it does not. To be more specific, it produces zeros at the highest hierarchy levels, despite the actual forecasts being positive at all these highest levels as well as at all other levels below.

Because of this issue, I have prepared a reproducible example.
Archiv.zip

import pandas as pd
import pickle

from hierarchicalforecast.core import HierarchicalReconciliation
from hierarchicalforecast.methods import TopDown

# read data - the files are attached

Y_hat_df = pd.read_pickle("Y_hat_df.pkl")
Y_df = pd.read_pickle("Y_df.pkl")
S = pd.read_pickle("S.pkl")
with open("tags.pkl", 'rb') as pkl_file:

    tags = pickle.load(pkl_file)

# reconcile

reconcilers = [
            TopDown(method="forecast_proportions"),
        ]

hrec = HierarchicalReconciliation(reconcilers=reconcilers)

Y_rec_df = hrec.reconcile(
            Y_hat_df=Y_hat_df,
            Y_df=Y_df,
            S=S,
            tags=tags,
        )

# highlight bug

pred_tiger_before_reconciliation = Y_rec_df[Y_rec_df.index == 'Tiger']['model'].sum()
pred_tiger_after_reconciliation = Y_rec_df[Y_rec_df.index == 'Tiger']['model/TopDown_method-forecast_proportions'].sum()

pred_hedgehog_before_reconciliation = Y_rec_df[Y_rec_df.index == 'Hedgehog']['model'].sum()
pred_hedgehog_after_reconciliation = Y_rec_df[Y_rec_df.index == 'Hedgehog']['model/TopDown_method-forecast_proportions'].sum()

pred_lion_before_reconciliation = Y_rec_df[Y_rec_df.index == 'Lion']['model'].sum()
pred_lion_after_reconciliation = Y_rec_df[Y_rec_df.index == 'Lion']['model/TopDown_method-forecast_proportions'].sum()

print("Tiger: ", str(pred_tiger_before_reconciliation) + " vs. " + str(pred_tiger_after_reconciliation))
print("Hedgehog: ", str(pred_hedgehog_before_reconciliation) + " vs. " + str(pred_hedgehog_after_reconciliation))
print("Lion: ", str(pred_lion_before_reconciliation) + " vs. " + str(pred_lion_after_reconciliation))

Output

Tiger: 117566.20513857048 vs. 117566.2
Hedgehog: 40888.12916468866 vs. 0.0
Lion: 574437.3661258915 vs. 0.0

For the top-level 'Tiger,' the top-down approach works. However, for the top levels 'Hedgehog' and 'Lion,' as mentioned above, it generates incorrect zero values. These zero values are also passed along the whole hierarchy.

I'm excited about an upcoming exchange.

[jmoralez]

Hey @D3NE3N, thanks for the report. Would it be possible for you to provide the datasets as CSV and the commands to generate the aggregation?

[D3NE3N]

Following your request, I have provided the data as CSV and the necessary commands to generate the aggregation.

It is important to note that the data includes some noise due to anonymization purposes which is why the values provided in this example differ slightly from those above. However, the bug remains identical.

data corresponds to the time series data, while hierarchy_levels contains the respective hierarchy information. Y_hat_d represents the associated forecasts. See
Archiv.zip
for the attached CSVs.

Here is the code for a complete reproduction

Import packages

import pandas as pd

from hierarchicalforecast.core import HierarchicalReconciliation
from hierarchicalforecast.methods import TopDown

Load data

data = pd.read_csv("data.csv") # file is attached
hierarchy_levels= [['level1'], ['level1', 'level2'], ['level1', 'level2', 'level3'], ['level1', 'level2', 'level3', 'level4']]

Y_hat_df = pd.read_csv("Y_hat_df.csv", index_col=0) # file is attached

Generate aggregation

Y_df, S, tags = aggregate(df=data, spec=hierarchy_levels)

The following code is identical to the one above

reconcilers = [
            TopDown(method="forecast_proportions"),
        ]

hrec = HierarchicalReconciliation(reconcilers=reconcilers)

Y_rec_df = hrec.reconcile(
            Y_hat_df=Y_hat_df,
            Y_df=Y_df,
            S=S,
            tags=tags,
        )

# highlight bug

pred_tiger_before_reconciliation = Y_rec_df[Y_rec_df.index == 'Tiger']['model'].sum()
pred_tiger_after_reconciliation = Y_rec_df[Y_rec_df.index == 'Tiger']['model/TopDown_method-forecast_proportions'].sum()

pred_hedgehog_before_reconciliation = Y_rec_df[Y_rec_df.index == 'Hedgehog']['model'].sum()
pred_hedgehog_after_reconciliation = Y_rec_df[Y_rec_df.index == 'Hedgehog']['model/TopDown_method-forecast_proportions'].sum()

pred_lion_before_reconciliation = Y_rec_df[Y_rec_df.index == 'Lion']['model'].sum()
pred_lion_after_reconciliation = Y_rec_df[Y_rec_df.index == 'Lion']['model/TopDown_method-forecast_proportions'].sum()

print("Tiger: ", str(pred_tiger_before_reconciliation) + " vs. " + str(pred_tiger_after_reconciliation))
print("Hedgehog: ", str(pred_hedgehog_before_reconciliation) + " vs. " + str(pred_hedgehog_after_reconciliation))
print("Lion: ", str(pred_lion_before_reconciliation) + " vs. " + str(pred_lion_after_reconciliation))

Output

Tiger: 125950.50688862089 vs. 0.0

Hedgehog: 25298.394988848817 vs. 0.0

Lion: 352189.52839933697 vs. 352189.5

[jmoralez]

Thanks for that. Looking at the code

hierarchicalforecast/hierarchicalforecast/methods.py

Line 412 in 2296c25

idx_top = int(S.sum(axis=1).argmax())

it seems that there should be only one top hierarchy (which in this case is choosing Lion).

I believe you need to create another level that has the same value for every row and introduce that as the first level in the hierarchy.

[D3NE3N]

Thank you very much for the very helpful answer!

Due to the missing cohort at the very top level, which has to be created artificially, the error described above arose with Top Down. Maybe it makes sense to specify a warning/error if the S matrix is not defined in a proper way for reconciliation?

Can a simple statement be made as to whether this (false S matrix) also directly affects other reconciliation strategies such as MinTrace or MiddleOut? The background to the question is that those two strategies (with the above S matrix) have been used in the past and may have generated incorrect forecasts.

[jmoralez]

Maybe it makes sense to specify a warning/error if the S matrix is not defined in a proper way for reconciliation?

Yes, I believe we should raise an error in that case, since the results will be wrong.

Can a simple statement be made as to whether this (false S matrix) also directly affects other reconciliation strategies such as MinTrace or MiddleOut?

I believe it's always expected to have a single top hierarchy, as can be seen in the diagram here, so I recommend doing that for all methods.