Proposal to remove mention of bootstrap (and other) thresholds from docs

[roblanf]

Our docs occasionally mention thresholds for interpreting things like bootstraps and concordance factors. I want to suggest that we remove these, because I think they are both incorrect and misleading. For roughly the following reasons:

• Bootstrap: a measure of sampling variance. Will always go up as the dataset gets more information. Completely unrelated to whether a branch is 'true' or not, since there are endless examples of incorrect branches with 100% support. Good paper here: https://academic.oup.com/mbe/article/29/2/457/1024815

• aLRT: same problem as for bootstrap, except that this is just a test of the null hypothesis that a branch length is zero

• concordance factors: a measure of biological variation. Not a measure of support, there is no threshold that matters here - this is an observation about a dataset that shouldn't change monotonically as the amount of information increases. We wrote about this here: https://ecoevorxiv.org/repository/view/6484/

What I think is true

All of these measures are useful and informative, for all sorts of great reasons.

What I think is false

There exist thresholds that can be meaningfully converted into binary outcomes for any of these measures (e.g. bootstrap > threshold means x, but bootstrap < threshold means y).

What I think we should do (and discuss here)

I think we should expunge all mention of thresholds from the docs, and instead explain why they are not sensible, with references to good discussions in the literature. I think currently our docs do a disservice to the field by promoting the notion that thresholds exist and are meaningful.

An apology

For being strident. I really like our docs though, and I think this is an important point for discussion!

[hahnlab-user]

Totally agree, though I at least can imagine cases where bootstrap thresholds make sense. For instance, use thresholds on gene trees to collapse weakly supported branches. But a threshold never makes sense for concordance factors.

[roblanf]

I guess my point is to remove mention of thresholds like "a value of [statistic] above [threshold] means that the branch is [true/reliable/good/]"

My point is the value is just a value, and any threshold is dependent on the dataset and the question (like your example of collapsing gene tree branches).

So, maybe better than removing all mention of thresholds is to put something in that's more nuanced along these lines?

[hahnlab-user]

I think it's okay that bootstraps (or aLRTs) are dependent on the size of the dataset in question--that doesn't make them necessarily bad, it just makes them like any sort of confidence interval/ standard error. Maybe just understanding that CFs don't have this property is the nuanced view?

[bqminh]

This document (http://www.iqtree.org/doc/Frequently-Asked-Questions#how-do-i-interpret-ultrafast-bootstrap-ufboot-support-values) was written in the context of single gene phylogenies. In fact the UFBoot 2013 paper only analysed gene trees and these numbers reflected these analysis results. I think they are still useful. But the document didn't mention about that. I will add a few sentences to clarify it, and that these shouldn't be generalised to phylogenomic data (in fact the whole bootstrap thing shouldn't be applied to concatenation tree analysis).

[roblanf]

I'm not sure I agree with that. I think bootstraps are a good measure of sampling variance, whatever the alignment size. It's just that I think it's better to avoid mentions of thresholds along the lines of "bootstraps > x are good"

[roblanf]

right on queue, a new approach to calculating a bootstrap if you have zero sampling variance (i.e. you've sampled the whole genome, as in a lot of virus work):

https://www.biorxiv.org/content/10.1101/2024.06.22.600199v1

Looks like an interesting approach...