About topic reduction "auto"

[HerrmannM]

Hi all,

I was wondering on what exactly the clustering for the automatic topic reduction does operate (looking at the embeddings case, not the c-tf-idf case)

1. Topic embeddings are created by taking the average of all document embedding.
2. Embeddings are normalised
3. HDBScan is applied with some predefined params, giving the new clusters

HDBScan is using an euclidean distance and not a cosine similarity, but the embeddings are generally made to be compared with cosine similarity. In 'normal' use, UMAP projects embeddings in euclidean space, and then HDBScan is used.

• Did I miss something ? E.g. are the topic embeddings computed by averaging what comes out of umap?
• If not, how could we evaluate the impact of using the euclidean distance rather than the cosine similarity?

Thank you!

[MaartenGr]

Applying L2 normalization to embeddings and then computing the Euclidean distance between them will result in a measure that is closely related to cosine similarity.

Did I miss something ? E.g. are the topic embeddings computed by averaging what comes out of umap?

The topic embeddings are generally created by taking the average of the input embeddings, so not the reduced embeddings that come out of UMAP.

If not, how could we evaluate the impact of using the euclidean distance rather than the cosine similarity?
See above.

[HerrmannM]

Thank you!

For anyone else wondering how the distances are related: https://stats.stackexchange.com/questions/146221/is-cosine-similarity-identical-to-l2-normalized-euclidean-distance (so both should lead to the same results when use in a 'relative' manner, e.g. ranking).