Version 2.0 Breaking Changes

[palday]

I've noticed that some packages rely on TableRegressionModel to support GLM: jmboehm/RegressionTables.jl#128, yufongpeng/AnovaBase.jl#52 and yufongpeng/AnovaGLM.jl#6. Even if they adapt to support the new approach, we'd better bump version to 2.0 to avoid any breakage. That can also be the occasion to drop some long-deprecated API. We should check whether we would like to make any other breaking changes. (A few other packages use TableRegressionModel for their own models, it would be good that they also stop using it but there's no hurry.)

Originally posted by @nalimilan in #339 (comment)

Here's a quick list of potentially issues that we might want to try to address as part of a push towards 2.0. Several are relatively straightforward, some could potentially be solved via more extensive documentation, and some will require Decisions to be made (e.g. all the stuff with weights).

• Get rid of TableRegressionModel by storing formula directly in model #339
• remove deprecations #575
• all the issues related to handling of rank deficiency / multicollinearity because this involves potentially changing defaults
  • Failed to "solve" logistic regression that R can  #449
  • Incorrect linear regression results #426
  • predict gives incorrect CIs when collinear columns have been dropped #413
  • PosDefException with simple linear regression when x/y ranges are small #375
  • Validation, CI, Residual variance, compare with lm() in R #280
• Expose a way for the user to choose between QR/Cholesky with the formula interface? #574
• Defaulting to the slightly slower but more stable QR method? #573
• More helpful errors #483
• Taking weighting seriously #487
• Change wts keyword argument name to weights #350
• nobs() should be number of obs; wobs() should be current nobs #259
• wts option doesn't automatically subset when a DataFrame has missing values #255
• Support for a categoricalArray as the dependent variable for GLM models  #240
• Set minimum Julia version to 1.6 #514
• Remove all branches in runtests.jl associated with unsupported versions of Julia #568
• Make internal fieldnames more transparent or at least add some comments to the struct definitions #572

There are several other issues I would like to see addressed sooner rather than later, but all are technically nonbreaking, at least under ColPrac guidelines, e.g., changes to the show methods, as raised in

• ftest and -0.0 #461
• add custom summary to types in GLM.jl #469

[mousum-github]

Right now, we are working on GLM with QR decomposition in two steps

1. LM with QR
2. GLM with QR
   and target is to complete by this calendar year.

Hope this will solve some issues related to the PosDefException as mentioned above.

[mousum-github]

I would like to have Multiple dependent variables, and Quasi Likelihood in GLM 2.0

[nalimilan]

Nice to hear you're working on QR! I think we can wait until you finish that before tagging 2.0. OTC, multiple dependent variables and quasi-likelihood do not change current behavior so they can be added later (and we have to discuss whether they should live in this package or in a separate one).

[nalimilan]

I don't think we should do anything about #259. Anyway #487 will change nobs to return an integer, as now the presence of weights is part of the type so there's no type instability. People can use size(modelmatrix(m), 1) to find out the number of rows in the matrix if they need that information.

#483, #255 and #240 would be good to have, but not breaking AFAICT.

[jariji]

I hope 2.0 fixes #496 and throws an error on missing values to protect users from making analytical errors by accident. lm(...; skipmissing=true) seems fine to me.

[andreasnoack]

I've split the items to show the status of the issue/PR for each of the changes and also converted the items not already tracked in individual issues to new issues. I've also added them to milestones. Most of them the the 2.0 milestone but some of the issues mentioned in comments as out of scope, I have put on the "out of scope for 2.0" milestone to show that they are deliberately considered out of scope. I hope these steps will make it possible for us to release 2.0 relatively soon. It's overdue.