How to generate embeddings using Microsoft.ML.OnnxRuntimeGenAI with phi-3 c#

[pavinan]

How to generate embeddings using Microsoft.ML.OnnxRuntimeGenAI with phi-3 c#

[Ben-Epstein]

I was wondering the same!

[MaxAkbar]

Based on this it does not support embedding: Phi3Family.md. That said I have used Semantic Kernel having a local BERT model. This link is in c# but Semantic Kernel supports python and Java as well.

[rhenry74]

any way to get embeddings would be cool

[stephnux]

Is there an api for this? This can be usefull for RAG

[rhenry74]

I have had some success using onnx to get embeddings using WhereIsAI/UAE-Large-V1
on hugging face <- note the onnx subfolder

Maybe this onnx model could be run thru the genai tool and then utilized to get embeddings via onnxruntime-genai.
i think this is the tooling to use

If you see a file called genai_config.json come out you have a good chance of getting it to work... i think :-)

[kunal-vaishnavi]

Can you clarify which specific information you are looking for?

1. Weights and biases in self.model.embed_tokens
2. inputs_embeds
3. Last hidden state before language modeling head
4. Weights and biases in self.lm_head

[stephnux]

The idea is to create BertOnnxTextEmbeddingGenerationService.cs for Semantic Kernel / Kernel memory but for Phi3 onnx model.

Also see LLamaEmbedder.cs from LLamaSharp.

I'm not exactly sure but I think both 1 and 3 will be helpful.

According to this python code it seems like the Last Hidden State OnnxEmbeddingModels

[pavinan]

It seems my misunderstanding, gen ai models are different from embeddings model. Phi3 is gen ai model. In demos I noticed Microsoft is using "smart components" local embedding model

[elephantpanda]

Personally I would like to get the output vectors for a given token position. So I can examine all the probabilities and make my own choice of which token to pick or even display the probabilites. So I can design a custom function in order to have more control over which token it pick. I would like to get the probability vectors for then entire output.

In the vision model, I would also like to be able to examine the input generated after feeding in the images and modify it if I so please.

[kunal-vaishnavi]

The embeddings (a.k.a. last hidden states) are now accessible via ONNX Runtime GenAI. You can create the ONNX model to output the hidden states using ONNX Runtime GenAI's model builder with --extra_options include_hidden_states=true.