Demonstrate parameter and constraint recovery

[marcdotson]

Before returning to more meta-learner alternatives, Jeff remembered an idea we've discussed previously, running the model using the actual constraint matrices used to produce the simulated data via simulation-experiments. This potentially allows us to see the best possible performance for any ensemble. It might help us figure out any problems with how we're implementing pathologies or with the estimation itself.

No Pathologies
Let's start with no pathologies. Without constraints, this is purely parameter recovery:

Model	LOO	Hit Rate	Hit Prob
HMNL	-2405	0.561	0.479
Ensemble (Simple Count Weights)	-2154	0.567	0.479

It appears that the HMNL and the ensemble are able to recover parameters. This is a good sign that things are working as intended.

ANA Only
ANA only with a homogeneous pathology:

Model	LOO	Hit Rate	Hit Prob
HMNL	-3014	0.444	0.366
Ensemble (Simple Count Weights)	-3003	0.456	0.374

This seems to be our initial problem. We aren't recovering parameters as well and the ensemble, even though it knows the actual constraints, is only doing as well as the HMNL. The fact that the HMNL is doing just as well suggests that there simply isn't really any benefit to using the ensemble. In other words, the pathology isn't present (enough) or the HMNL is able to effectively account for ANA on its own as a homogeneous pathology.

ANA only with a heterogeneous pathology:

Model	LOO	Hit Rate	Hit Prob
HMNL	-2575	0.522	0.419
Ensemble (Simple Count Weights)	-2736	0.453	0.372

Screening Only
Screening only with a homogeneous pathology:

Model	LOO	Hit Rate	Hit Prob
HMNL	-1311	0.742	0.683
Ensemble (Simple Count Weights)	-1159	0.714	0.578

Note the scale: the HMNL is way off compared the ensembles (and it is parameters 1 and 3 that are being screened on homogeneously). In other words, screening is a pathology that the HMNL can't account for as well as ANA. That said, the ensemble fit statistics don't perform any better. This takes us back to how we actually compute the ensemble hit rates and hit probabilities.

Screening only with a heterogeneous pathology:

Model	LOO	Hit Rate	Hit Prob
HMNL	-1733	0.528	0.492
Ensemble (Simple Count Weights)	-39532	0.500	0.487

Respondent Quality Only
Respondent quality only with a homogeneous pathology:

Model	LOO	Hit Rate	Hit Prob
HMNL	-2238	0.586	0.495
Ensemble (Simple Count Weights)	-1912	0.561	0.492

Respondent quality isn't tied to specific attribute levels, so at the very least it's good to see that each of the parameter estimates struggles.

Respondent quality only with a heterogeneous pathology:

Model	LOO	Hit Rate	Hit Prob
HMNL	-2238	0.586	0.495
Ensemble (Simple Count Weights)	-1886	0.575	0.497

ANA and Screening
ANA and screening with homogeneous pathologies:

Model	LOO	Hit Rate	Hit Prob
HMNL	-713	0.856	0.827
Ensemble (Simple Count Weights)	-736	0.797	0.757

ANA and screening with heterogeneous pathologies:

Model	LOO	Hit Rate	Hit Prob
HMNL	-1726	0.500	0.476
Ensemble (Simple Count Weights)	-49042	0.489	0.458

Screening and Respondent Quality
Screening and respondent quality with homogeneous pathologies:

Model	LOO	Hit Rate	Hit Prob
HMNL	-1216	0.694	0.671
Ensemble (Simple Count Weights)	-1109	0.558	0.562

Screening and respondent quality with heterogeneous pathologies:

Model	LOO	Hit Rate	Hit Prob
HMNL	-1654	0.486	0.467
Ensemble (Simple Count Weights)	-36118	0.461	0.446

ANA and Respondent Quality
ANA and respondent quality with homogeneous pathologies:

Model	LOO	Hit Rate	Hit Prob
HMNL	-2617	0.467	0.400
Ensemble (Simple Count Weights)	-2354	0.472	0.402

ANA and respondent quality with heterogeneous pathologies:

Model	LOO	Hit Rate	Hit Prob
HMNL	-2895	0.444	0.369
Ensemble (Simple Count Weights)	-2896	0.475	0.352

ANA, Screening, and Respondent Quality
ANA, screening, and respondent quality with homogeneous pathologies:

Model	LOO	Hit Rate	Hit Prob
HMNL	-1039	0.819	0.794
Ensemble (Simple Count Weights)	-14565	0.764	0.703

ANA, screening, and respondent quality with heterogeneous pathologies:

Model	LOO	Hit Rate	Hit Prob
HMNL	-1973	0.581	0.512
Ensemble (Simple Count Weights)	-49459	0.542	0.465

[marcdotson]

FWIW, this is what I had put together previously, @jeff-dotson. However, all of these rely on the use of an ensemble rather than a single model that knows the simulated pathologies exactly.

[marcdotson]

A single model to establish the upper bounds confirms what's demonstrated above (at least for homogeneous pathologies; not sure how to get our present setup to manage heterogeneous pathologies with a single model), @jeff-dotson. We should review the big three: simulate_data.R, clever_randomization.R, and hmnl_ensemble.stan when we talk about our approach to how the ensemble is managing pathologies.

Model    Pathologies  Heterogeneous LOO   `Hit Rate` `Hit Prob`
HMNL     None         No            NA         0.578      0.483
Ensemble None         No            NA         0.574      0.482
HMNL     ANA          No            NA         0.614      0.532
Ensemble ANA          No            NA         0.604      0.527
HMNL     Screen       No            NA         0.739      0.685
Ensemble Screen       No            NA         0.742      0.609
HMNL     ANA & Screen No            NA         0.865      0.832
Ensemble ANA & Screen No            NA         0.717      0.682

[marcdotson]

The following updated results uses averaged respondent-level Betas rather than the mean of the heterogeneity distribution Gammas when predicting out-of-sample hit rates and hit probability. This is a fairer comparison between the constrained-parameters method we're employing for the conjoint ensemble and the standard HMNL since we are imposing the constraints on the Betas rather than the Gammas.

If we were only concerned with homogeneous pathologies, we could constrain the Gammas directly. However, the bite starts with including heterogeneous pathologies, as demonstrated below. To alleviate any concerns about this choice, we could include predictions based on both the Betas and the Gammas. This might further demonstrate the limits of flexibility for the standard HMNL, especially for predicting out-of-sample.

   Model                Pathologies  Heterogeneous LOO   `Hit Rate` `Hit Prob`
   <chr>                <chr>        <chr>         <lgl>      <dbl>      <dbl>
 1 HMNL                 None         No            NA         0.582      0.484
 2 Ensemble Upper Bound None         No            NA         0.578      0.482
 3 HMNL                 ANA          No            NA         0.45       0.384
 4 Ensemble Upper Bound ANA          No            NA         0.457      0.383
 5 HMNL                 Screen       No            NA         0.875      0.862
 6 Ensemble Upper Bound Screen       No            NA         0.867      0.854
 7 HMNL                 ANA & Screen No            NA         0.911      0.892
 8 Ensemble Upper Bound ANA & Screen No            NA         0.91       0.881
 9 HMNL                 None         Yes           NA         0.582      0.484
10 Ensemble Upper Bound None         Yes           NA         0.578      0.482
11 HMNL                 ANA          Yes           NA         0.41       0.369
12 Ensemble Upper Bound ANA          Yes           NA         0.421      0.372
13 HMNL                 Screen       Yes           NA         0.546      0.525
14 Ensemble Upper Bound Screen       Yes           NA         0.556      0.552
15 HMNL                 ANA & Screen Yes           NA         0.594      0.551
16 Ensemble Upper Bound ANA & Screen Yes           NA         0.569      0.566

The previous results also appeared to show that the models were increasingly accurate as the underlying pathologies became more complicated. Fortunately these latest results appear to avoid what may have been a coding issue or just an artifact of data simulation.

[marcdotson]

Here I try to replicate the conjoint ensemble proof-of-concept with the constraints operating post-hoc in the generated quantities block.

   Model                Pathologies  Heterogeneous LOO   `Hit Rate` `Hit Prob`
   <chr>                <chr>        <chr>         <lgl>      <dbl>      <dbl>
 1 HMNL                 None         No            NA         0.582      0.484
 2 Ensemble Upper Bound None         No            NA         0.578      0.482
 3 HMNL                 ANA          No            NA         0.45       0.384
 4 Ensemble Upper Bound ANA          No            NA         0.45       0.383
 5 HMNL                 Screen       No            NA         0.875      0.862
 6 Ensemble Upper Bound Screen       No            NA         0.865      0.851
 7 HMNL                 ANA & Screen No            NA         0.911      0.892
 8 Ensemble Upper Bound ANA & Screen No            NA         0.903      0.883
 9 HMNL                 None         Yes           NA         0.582      0.484
10 Ensemble Upper Bound None         Yes           NA         0.578      0.482
11 HMNL                 ANA          Yes           NA         0.41       0.369
12 Ensemble Upper Bound ANA          Yes           NA         0.41       0.368
13 HMNL                 Screen       Yes           NA         0.546      0.525
14 Ensemble Upper Bound Screen       Yes           NA         0.551      0.548
15 HMNL                 ANA & Screen Yes           NA         0.594      0.551
16 Ensemble Upper Bound ANA & Screen Yes           NA         0.569      0.565

[jeff-dotson]

Very interesting. Thanks for the update. From: Marc Dotson ***@***.***> Date: Tuesday, October 31, 2023 at 5:42 PM To: marcdotson/conjoint-ensembles ***@***.***> Cc: ***@***.*** # ***@***.***>, Mention ***@***.***> Subject: Re: [marcdotson/conjoint-ensembles] Demonstrate parameter and constraint recovery (#70) The following updated results uses averaged respondent-level Betas rather than the mean of the heterogeneity distribution Gammas when predicting out-of-sample hit rates and hit probability. This is a fairer comparison between the constrained-parameters method we're employing for the conjoint ensemble and the standard HMNL since we are imposing the constraints on the Betas rather than the Gammas. If we were only concerned with homogeneous pathologies, we could constrain the Gammas directly. However, the bite starts with including heterogeneous pathologies, as demonstrated below. To alleviate any concerns about this choice, we could include predictions based on both the Betas and the Gammas. This might further demonstrate the limits of flexibility for the standard HMNL, especially for predicting out-of-sample. Model Pathologies Heterogeneous LOO `Hit Rate` `Hit Prob` <chr> <chr> <chr> <lgl> <dbl> <dbl> 1 HMNL None No NA 0.582 0.484 2 Ensemble Upper Bound None No NA 0.578 0.482 3 HMNL ANA No NA 0.45 0.384 4 Ensemble Upper Bound ANA No NA 0.457 0.383 5 HMNL Screen No NA 0.875 0.862 6 Ensemble Upper Bound Screen No NA 0.867 0.854 7 HMNL ANA & Screen No NA 0.911 0.892 8 Ensemble Upper Bound ANA & Screen No NA 0.91 0.881 9 HMNL None Yes NA 0.582 0.484 10 Ensemble Upper Bound None Yes NA 0.578 0.482 11 HMNL ANA Yes NA 0.41 0.369 12 Ensemble Upper Bound ANA Yes NA 0.421 0.372 13 HMNL Screen Yes NA 0.546 0.525 14 Ensemble Upper Bound Screen Yes NA 0.556 0.552 15 HMNL ANA & Screen Yes NA 0.594 0.551 16 Ensemble Upper Bound ANA & Screen Yes NA 0.569 0.566 The previous results also appeared to show that the models were increasingly accurate as the underlying pathologies became more complicated. Fortunately these latest results appear to avoid what may have been a coding issue or just an artifact of data simulation. — Reply to this email directly, view it on GitHub<#70 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABW2EBVANZ7M4EBVM3RQ4QLYCGEHPAVCNFSM47XQGGFKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZYHAYTQMRSGUZA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[marcdotson]

@ethanbudge here are the file descriptions for the original conjoint ensemble archive:

• noana2.csv Output for no ANA, no screening.
• rate.ana2.csv Output for ANA and screening conditioned on ensemble size.
• screening.ana.csv Output for ANA and screening.
• Rate Chart.xlsx Chart produced for no ANA, no screening conditioned on ensemble size.
• design.R Generates a fractional factorial design by sampling.
• predEnsemble.R Use same randomization strategy to make a post-hoc ensemble prediction.
• predFit.R Computes predictive fit for test data.
• rhierMNL.r Estimates an HMNL.
• run.ensemble.R Control file for running ensemble.
• RunLoop.R Control file for looping over running the ensemble.
• simData.all.R Simulates multinomial logit data.
• simData.R Simulates multinomial logit data (deprecated).

I've gone through the code. It's very clean. A few things I found, and I'm interested to see what you find:

• The probability of simulating data with ANA or screening is the same (0.75) and the probability of inducing ANA or screening in the ensemble prediction is the same (0.50).
• This initial algorithm did not allow screening on the attribute that had ANA as well as the final attribute's levels, which is either price or a hard-coded specification of the outside option.
• It looks like the model was fit using the entire dataset instead of the training data split.
• The model was fit using Metropolis-within-Gibbs rather than Hamiltonian Monte Carlo.
• The population mean of beta was generated by averaging betas across individuals, with no reference to the population covariance.

[marcdotson]

@jeff-dotson I can't replicate the previous results.

[marcdotson]

I tried to replicate using information leakage we saw in the initial results. Still not there.

[jeff-dotson]

Interesting. Let’s discuss. Jeff From: Marc Dotson ***@***.***> Date: Tuesday, December 19, 2023 at 1:27 PM To: marcdotson/conjoint-ensembles ***@***.***> Cc: ***@***.*** # ***@***.***>, Mention ***@***.***> Subject: Re: [marcdotson/conjoint-ensembles] Demonstrate parameter and constraint recovery (#70) I tried to replicate using information leakage we saw in the initial results. Still not there. image.png (view on web)<https://github.com/marcdotson/conjoint-ensembles/assets/29615257/87f66858-6727-4840-b6ee-fa182e6406b9> — Reply to this email directly, view it on GitHub<#70 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABW2EBXST7MF7Q6N6LU4BQTYKH2BRAVCNFSM47XQGGFKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBWGM2DENZWGQ2A>. You are receiving this because you were mentioned.Message ID: ***@***.***>