Discrepancies in embedding outputs for single vs. multiple lines

[raulod]

I have noticed that the generated embeddings differ slightly when embedding a single line versus multiple lines. I am curious to understand why this happens.

single line

./llama-embedding -m paraphrase-MiniLM-L6-23M-v2-F16.gguf -p "line 1"

embedding 0: -0.063674  0.047630 -0.038471 -0.020687 -0.058054  0.008873  0.066752  0.006669  0.052501 -0.041890  0.022065 -0.019708  0.009104 -0.056419

Added one more line

./llama-embedding -m paraphrase-MiniLM-L6-23M-v2-F16.gguf -p "line 1\nline 2"

embedding 0: -0.063660  0.047625 -0.038473 -0.020686 -0.058066  0.008888  0.066752  0.006660  0.052509 -0.041868  0.022060 -0.019701  0.009107 -0.056428 -0.032168 -0.024372 
embedding 1: -0.048970  0.078183 -0.054204 -0.004430 -0.077422  0.023646  0.043569  0.014006  0.047194 -0.061538  0.037463  0.011851  0.012090 -0.090341 -0.017245 -0.037014 

As you can see, the values for embedding 0 in both cases are slightly different. This pattern is consistent and can be reproduced.

[ggerganov]

This is expected and is a side effect of the unified KV cache currently used in llama.cpp. In the second case, both inputs are placed in the same batch and the computation currently is not invariant of the batch size due to some floating point max/sum operations in the attention (#3014 (comment)). Same effect is observed when processing the same causal prompt with different batch sizes.

[raulod]

Is cache relevant for embedding?

I think there is more to this than just the cache. I tried processing both sentences sequentially instead of batching them. Each sentences was decoded separately using the same ctx. The cache is getting cleared in batch_decode function so cache should not impact.

//` clear previous kv_cache values (irrelevant for embeddings)
    llama_kv_cache_clear(ctx);

In this case too, the results are slightly different.

[ggerganov]

I tried processing both sentences sequentially instead of batching them.

I'm not sure what you mean - "line 1" and "line 2" is normal to produce different embeddings. I'm saying that "line 1" will also produce different embeddings if it is processed alone or with something else in the same batch.