Investigate alternate meta-learners

[marcdotson]

Using LOO for model stacking produces improvement in terms of LOO only for the conjoint ensemble. What about alternate meta-learners? Use the meta-learner branch.

[marcdotson]

Updated comparison of results for simulated data with both ANA and respondent quality estimated with a heterogenous, 1000-member ensemble:

Model	LOO	Hit Rate	Hit Prob
HMNL	-2691	0.453	0.391
Ensemble (LOO Weights)	-2652	0.464	0.366
Ensemble (Equal Weights)	-2714	0.444	0.361
Ensemble (MNL Weights w/Predicted Probs)	-2715	0.444	0.361
Ensemble (Simple Count Weights)	-2711	0.444	0.361
Ensemble (Simple Probability Weights)	-2710	0.444	0.361

Updated comparison of results for real data where we account for both ANA and respondent quality with a heterogeneous, 1000-member ensemble:

Model	LOO	Hit Rate	Hit Prob
HMNL	-2756	0.403	0.348
Ensemble (LOO Weights)	-2771	0.417	0.296
Ensemble (Equal Weights)	-2870	0.414	0.291
Ensemble (MNL Weights w/Predicted Probs)	-2872	0.380	0.287
Ensemble (Simple Count Weights)	-2867	0.380	0.287
Ensemble (Simple Probability Weights)	-2834	0.387	0.289

And to check for any issues with how we are currently implementing the respondent quality pathology, here are updated results for simulated data with ANA only with a heterogenous, 1000-member ensemble:

Model	LOO	Hit Rate	Hit Prob
HMNL	-2787	0.411	0.368
Ensemble (LOO Weights)	-2723	0.410	0.347
Ensemble (Equal Weights)	-2789	0.414	0.349
Ensemble (MNL Weights w/Predicted Probs)	-2772	0.414	0.350
Ensemble (Simple Count Weights)	-2790	0.419	0.349
Ensemble (Simple Probability Weights)	-2785	0.414	0.349

And here are results for real data where we account for ANA only with a heterogeneous, 1000-member ensemble:

Model	LOO	Hit Rate	Hit Prob
HMNL	-2756	0.367	0.336
Ensemble (MNL Weights w/Predicted Probs)	-2923	0.380	0.286
Ensemble (Simple Count Weights)	-2925	0.380	0.286
Ensemble (Simple Probability Weights)	-2890	0.387	0.288

[marcdotson]

FWIW, @RogerOverNOut I've re-run the model with just ANA for both simulated and real data. You said it might be easier to work with that when trying your alternative meta-learner. I had to make some much clearer names. See the shared folder.

[RogerOverNOut]

Thanks Marc

…

On Fri, Jun 4, 2021, 2:31 AM Marc Dotson ***@***.***> wrote: FWIW, @RogerOverNOut <https://github.com/RogerOverNOut> I've re-run the model with just ANA for both simulated and real data. You said it might be easier to work with that when trying your alternative meta-learner. I had to make some much clearer names. See the shared folder. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#68 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHEP55B2FNQKO7T5VHS2JL3TRBXKLANCNFSM45N7IGWA> .

[marcdotson]

@jeff-dotson @RogerOverNOut I made my first pass using an MNL as a meta-learner. The results are in the above tables under "Ensemble (Logit Weights)."

I haven't done this before, so I wanted to describe what I'm doing. Please let me know if there's a red flag you see:

• I'm using the first half of the test data for validation, the second half of the test data as test data.
• I created a new function predictive_fit_stacking() based on Roger's predictive_fit_ensemble() to produce choice predictions for each of the ensemble members.
• The validation choice data is the Y and the predicted choices are the X for the meta-learner. Since we only have a single predicted choice for each choice scenario for each ensemble member, I just duplicate the predicted choice for each of the choice alternatives in each choice scenario. I realize in writing this I should be using the probabilities instead of the choices themselves -- I'll try that next.
• The meta-learner is an aggregate Bayesian MNL which you can see in mnl.stan.
• I average the betas associated with each of the ensemble members across draws and then normalize so we have weights.

[marcdotson]

MNL using probabilities instead of the choices has been added to the tables above. I've also added weights using simple counts of the hits as well as a sum of probabilities for the hits. Note that weights produced using simple counts of the hits is the same as weights from an MNL meta-leaner using probabilities.

[marcdotson]

Using {logitr} for the meta-learner would help boost speed.