-
Notifications
You must be signed in to change notification settings - Fork 72
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Very large memory usage of ERM reconciler with method = 'reg' or 'reg_bu' #163
Comments
Thanks for the comment @dclaz, and sorry for the inconvenience. Currently, the lasso ERM is creating matrices using Kronecker products which are very inefficient in memory and computation. We plan to work on the efficiency of both |
Thanks for your rapid reply @kdgutier. Sadly I don't think I can afford a multi-petabyte machine in this instance. |
Yep sorry, An efficient Meanwhile Thanks for reporting the memory issue. |
That would be amazing, thank you so much. |
Hello, is there some extra step being taken with the ERM regularization methods (
reg
andreg_bu
) that require a very large memory overhead? Relative to ERM withmethod=closed
?I've got a large-scale time series forecast task and there is a hierarchical nature that I hope can be used to improve both the aggregate and granular forecasts.
The most granular series are sales office / material combinations, which can be nested under either a sales office or material
The unique counts of each of these are given below:
So in total, there are
11449+3213+28=14690
series that are to be forecasted. The S matrix looks like this:I've produced test forecasts and have been able to successfully compare the BottomUp and ERM(Closed) reconcilers with the base forecasts, but unfortunately the bottom-up reconciler cannot improve the granular sales office X material forecasts and the ERM-Closed method results in very poor MSE at the sales office X material level (but vastly improves the others).
I was hoping that utilizing regularisation with the ERM reconciler with
method='reg'
ormethod='reg_bu'
would be helpful here, but I run into a memory issue:My understanding is that these methods use a LASSO to obtain the optimal weights. I'm not sure about what the reconciler is doing in the background, but I do believe there should be very memory-efficient implementations of the LASSO that could be leveraged here.
For very wide S matrices, I think that:
$$\mathbf{P}^{*} = \text{argmin}{\mathbf{P}} ||\mathbf{Y}-\mathbf{S} \mathbf{P} \hat{Y} ||^{2}{2} + \lambda ||\mathbf{P}-\mathbf{P}{\text{BU}}||{1}
$$
Should be much easier to compute than
Additionally, I am surprised that the closed-form solution for ERM works at all and only takes ~13 seconds, given it has to invert a 14690x14690 matrix in this scenario.
The text was updated successfully, but these errors were encountered: