Make CatBoost-based estimators loss a variable parameters

[VitalyPetrov]

Description

From my experience i have found that RMSE is not the best loss function when it comes to time series forecasting. One can achieve higher values of metrics (MAPE, SMAPE) utilizing another losses (e.g. MAE, Quantile) as RMSE tends to focus on anomalies rather then normal / ordinary values. It would be great if you can introduce that function at your package

Thoughts

No response

Related

underlying model initialization

[d-a-bunin]

Hi! Currently you can set a custom loss just by passing a parameter loss_function into the model. It isn't documented because it goes into kwargs. We are making this behavior more clear in issue: #445.

[VitalyPetrov]

Sure you can but this approach seems to be a little bit unobvious for the end user -- one need to look into the source code in order to find what params are tunable and how your parameters match with Catboost-ones.

But I agree that in your case the correspondence is one-to-one for the most parameters

[d-a-bunin]

We recently merged the PR (#454) with the description about the kwargs. Now user could look at documentation, find out that there is kwargs and go the the documentation of catboost to check the name of parameter for the loss.

If this discussion gets many upvotes we could think about extracting parameter loss_function from kwargs into a separate constructor parameter.

[d-a-bunin]

one need to look into the source code in order to find what params are tunable

Why should the user look up the source code if there is an API documentation? Are there any problems with it that makes it inconvenient?

As for me, I frequently look up the source code of the other libraries that I use. It often makes you better understand what exactly is going on inside.

[d-a-bunin]

There is probably an another dimension of this discussion. It could be nice to add this parameter into params_to_tune for example to make it tunable by default. However, it probably doesn't require making it a separate parameter (kwargs should be fine).

However, in order to do this we should define a list of loss_function values that are reasonable to try during default tuning.