A more thorough explanation of overfitting, need for balance, etc

[malcolmbarrett]

We're not satisfied with the way we usually address these questions

[malcolmbarrett]

Here's where I sit with this today:

1. Overfitting is not exactly the same problem as in prediction because we're not taking this model and making predictions with it on different data. It remains true that the right causal model needn't predict particularly well to be unbiased but
2. Overfitting is model misspecification (https://statmodeling.stat.columbia.edu/2017/07/15/what-is-overfitting-exactly/) and so from that perspective overfit models will have residual confounding and thus biased estimates
3. Overfitting can exacerbate positivity violations, causing bias (I don't have a good example of this but it was inspired by some of Sherri's work here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5824732/)

[malcolmbarrett]

Bias-variance tradeoff and overfitting of PS model? https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5756087/

My instinct is that overfitting lowers the statistical bias at the expense of bias from the true effect, and that in a frequentist since, it increases variance by virtue of the bias from the true effect, leaving poor nominal coverage of the CIs

Include this either with the callout box or somewhere else on its own.

[malcolmbarrett]

callout box in ps model fitting

[malcolmbarrett]

what about when you have the right model but can't model it, e.g., too many confounders vs sample size. (lasso, dim reduction)