Effects of, and optimal settings for, mismatch ratios

[hyanwong]

In the tsdate preprint, we show a few plots of what happens when you adjust the mismatch ratio values when running the new tsinfer version. We conclude that for human-like datasets, mismatch ratios of 1 in both match samples and match ancestors passes gives a reasonable result. However, this may not always be the optimal value: it will be dependent on the underlying error model, and may differ between organisms. This discussion is aimed at collecting extra information about these mismatch parameters. FYI, here's the most relevant plot in the current preprint, based on 1500 samples and 1000-genomes-project-like error:

Which is created by running test_mismatch_rates.py

[hyanwong]

There are a few extra metrics that we could plot:

1. "Generalised" RF distances, such as Martin Smith's MIC (Mutual Clustering Information metric).
2. The accuracy of dating mutations when the inferred TS is then dated using tsdate
3. The edge+mutation count when mutations are re-layed on the trees using parsimony (either with or without specifying the ancestral state)
4. The proportion of sites at which the focal ancestor is actually present in the tree

I've tried all of these on the standard OOA simulated dataset above with the following results:

1. MIC distance - seems to be basically the same as the RF metric but takes longer to calculate. Here's a example with only 150 samples, same 1000G error

2. Accuracy of further tsdate inference. This is pretty indirect, but very applicable to our case. In the 1500 sample case with 1000G error, the plots suggest that we should have a higher match samples ratio and a lower match ancestors ratio (here a *higher* correlation coefficient is better, so the optimal values are mismatch ratio = 1e-3 for match ancestors and 1e3 for match samples, which does make some sense.

3. Relaying using parsimony - if we do this, the number of mutations required when there is a high match ancestors mismatch ratio, seems relatively static, so that we can crank up the ancestors mismatch ratio which drops the number of edges but has little effect on the number of mutations so that by the measure of edges+mutations, a high match-ancestors mismatch ratio is beneficial (which seems odd)

4. Surprisingly often, at a focal site, we don't actually include the ancestor on which we expected the focal mutation to occur. We can measure this by looking at the proportion of sites that have the "focal" ancestor in the tree at the site position. Here's how that looks on the parameter map (lower mismatch ratios give more focal ancestors, as might be expected, but there's a jump around mismatch ratios of 10e-3)

[jeromekelleher]

(Spotted a typo: need a p in "prootion" for fig title)

[hyanwong]

Thanks @jeromekelleher - corrected now. I don't think we want to add this particular plot to the tsdate paper though, do we? It's just to show in this discussion for further discussion

[hyanwong]

I now think that measuring the quality of imputation is the best way to go for this. See #652