Potential Issue re: Spline models and Bambi's interpret module #796
Comments
|
I imagine there is quite a bit of work in adapting the interpret function to account for the static spline bases but i'm just wondering if there is something obvious i'm missing? |
|
Hey @NathanielF thanks for raising the issue. This should work. What version of Bambi are you using? A couple months ago we added enhancements of how the interpret package parses hsgp and basis spline terms in the Bambi model. It may be that you need to install from source to get these latest changes. Nonetheless, I will attempt to reproduce your example and get back with you. |
|
Thanks @GStechschulte I'm using a 0.13: I'd be keen to discover if i can use Bambi's native interpret function. For the moment i've hacked to a routine to extract the components: which gives me this plot:
Work in Progress here: https://nathanielf.github.io/posts/post-with-code/GAMs_and_GPs/gams_and_gps.html Diving into both splines, hierarchical spline modelling and HSGP approximations using Bambi. Would be great if i could "show-off" the full functionality. |
|
@NathanielF @GStechschulte I'm working on this, I think the issue is where looking up some variable in the data frame when that should not be the case. |
|
@GStechschulte I see in the code @NathanielF shared he's setting @NathanielF are you trying to use |
|
Ah ok. That maybe it. I will have another look! |
|
It's still broken in the latest release and the main branch, I'm about to push a branch with a quick fix and open an issue explaining the problem. Thanks @NathanielF for raising this, it allowed us to find a bug :) |
|
Nice! Happy to help |
|
Is this something close to what you're looking for? fig, ax = bmb.interpret.plot_predictions(
spline_model, spline_idata, "Edu"
)
fig, ax = bmb.interpret.plot_predictions(
spline_model, spline_idata, {"Edu": np.linspace(0.7, 1, num=100)}
)
|
|
@NathanielF this is the branch https://github.com/tomicapretto/bambi/tree/spline_interpret and this is a notebook with the code https://github.com/tomicapretto/bambi/blob/spline_interpret/spline_prediction.ipynb (I'll delete it before merging into Bambi's main) The main point is that in |
|
The plot predictions code works for me too. But just out of curiosity i tried the plot slopes without the I've change these to two covariates and i'm now getting a different error: UpdateIt works if we convert object to a list But gives a kind of nonsensical plot 
|
|
Ok, starting to make progress. I think i just wasn't using the API correctly. If i pass in a dictionary for the wrt variable in the slopes calculation than i start to see variation across the income variable: 
I guess i'm just not entirely sure what plot slopes will do with a spline based model? |
|
Thanks a lot @tomicapretto and @NathanielF for your comments.
That's correct. For all of the interpret functions, we are typically interested in how the response variable changes given some values for the covariates; generally speaking.
Yeah, it seems
Plot slopes isn't doing anything per say with the model. What it is doing is saying, "okay, cool. You have defined and fit a model on some data. Now, lets generate some "new data", feed that data to the model, and see what the predictions are". I think Richard McElreath uses the term "seeing through the eyes of the model". Under the hood, slopes is assembling a pairwise grid of values that allows you to compare the predictions of the response variable made with respect to Regarding your two comments
Not sure why this is required in your example. When I run our documentation examples for slopes where a
Same as above. You shouldn't have to pass a dict. I am looking into the last two comments. :) |
|
@NathanielF regarding your comment:
I don't have this problem. Cherry-picking @tomicapretto hot fix commit I can do fig, ax = bmb.interpret.plot_slopes(
spline_model,
spline_idata,
wrt={"Edu": [.8, .9]} ,
conditional={
"Income": np.linspace(0.1, 1, 10), # No need to convert array to list
"Health": np.linspace(0.1, 1, 3) # No need to convert array to list
}
)
I also tested out a couple other parameter data type variations (array, list, and float) fig, ax = bmb.interpret.plot_predictions(
spline_model,
spline_idata,
{
"Edu": np.linspace(0.7, 1, num=100),
"Income": [0.1, 0.3, 0.5, 0.7, 0.9],
"Health": 0.893
},
subplot_kwargs={"main": "Edu", "group": 'Income', "panel": "Income"},
fig_kwargs={"figsize": (12, 4)},
legend=False
)
plt.tight_layout()
and everything is working as expected. It is interesting to see how the predictions for |
|
Very good. Not sure about the array list issue, but I was also able to get plot predictions working to show similar effects of incomes. The response just blows up at extreme realisations - which threw me. Thanks for your work on this! Unrelated, but I wondered if you had thoughts on the hierarchical spline model I've implemented in that blog post? Or how something similar could be handled in bambi? Especially the predicting on new years/groups step. |
|
@NathanielF anytime! I hope to read your blog in depth over the coming days 👍🏼 |
Haven't looked into it in depth, but I'm very curious about it so I will be reading it and I'll provide feedback here if you want |
|
That would be amazing @tomicapretto . I think all the mechanics are there in the blog post. I need to finesse the writing a bit. But fyi the basic theme I want to interrogate is that unpooled and pooled spline models seem brittle for extrapolation, but a hierarchical model seems to offer a way to "abstract" over the individual loss curves and pull out something like the generic shape of the loss curves to predict on new years.... I'm thinking this is a good way to balance the overfit tendency of spline models....? |
|
Just FYI @tomicapretto and @GStechschulte I've tightened the writing here and showed how to do sampling for "new groups" over my hierarchical spline model for the insurance curves. Additionally i've shown how adding hierarchies to a multiple regression spline modelling can improve the extrapolation to new groups 
|
|
@NathanielF thanks for the update! I'm reading through it right now (it's been in my mind, but I couldn't find time time) |
|
No worries. Appreciate any feedback you might have. |
I'm possibly doing something a bit naive, but i just wondered if there was interpretation functions available for spline based models like you get with mgcv in R.
This breaks with the following error:
The last command breaks with the following error:
The text was updated successfully, but these errors were encountered: