Combining RAG with function calling

[adrian-watson-pts]

Is it possible to combine data retrieval with Azure AI Search and chat-completion function calling?

When attempting it with a generalized message (which won't call functions) I get a response from AI Search successfully.
However when trying it where a function call will be made, the function does execute but the GetChatMessageContentAsync ends up throwing an exception.

Code example

Working function calling without combining with RAG:

// ChatHistory chatHistory <= created elsewhere
var openAIPromptExecutionSettings = new OpenAIPromptExecutionSettings
{
    ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions
};

var chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();

await chatCompletionService.GetChatMessageContentAsync(chatHistory, executionSettings: openAIPromptExecutionSettings, kernel: kernel);

Combining with RAG:

// ChatHistory chatHistory <= created elsewhere
var chatExtensionConfiguration = new AzureSearchChatExtensionConfiguration
{
    Authentication = new OnYourDataApiKeyAuthenticationOptions("<redacted>"),
    IndexName = "<redacted>",
    SearchEndpoint = new Uri("<redacted>")
};

var chatExtensionsOptions = new AzureChatExtensionsOptions { Extensions = { chatExtensionConfiguration } };
var openAIPromptExecutionSettings = new OpenAIPromptExecutionSettings
{
    AzureChatExtensionsOptions = chatExtensionsOptions,
    ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions
};

var chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();

await chatCompletionService.GetChatMessageContentAsync(chatHistory, executionSettings: openAIPromptExecutionSettings, kernel: kernel);

Error output

[13:39:56 INF] Function <redacted> succeeded.
[13:39:56 VRB] Function result: <redacted>
[13:39:56 INF] Function completed. Duration: 0.0087746s
[13:41:14 ERR] HTTP POST /api/v1/chat responded 500 in 80853.5833 ms
Microsoft.SemanticKernel.HttpOperationException: Service request failed.
Status: 400 (Bad Request)

Content:
{"error": {"requestid": "65b0cb1e-438f-4dbe-956f-ac4bc575afd2", "code": 400, "message": "Invalid chat message detected: message content must be string"}}

Headers:
skip-error-remapping: REDACTED
x-envoy-upstream-service-time: REDACTED
apim-request-id: REDACTED
x-ms-client-request-id: d8a44898-e39f-48b2-bfb7-cc9837a72da0
Strict-Transport-Security: REDACTED
X-Content-Type-Options: REDACTED
x-ms-region: REDACTED
x-ratelimit-remaining-requests: REDACTED
x-ratelimit-remaining-tokens: REDACTED
Date: Thu, 25 Jul 2024 03:39:56 GMT
Content-Length: 153
Content-Type: application/json; charset=UTF-8

 ---> Azure.RequestFailedException: Service request failed.
Status: 400 (Bad Request)

Content:
{"error": {"requestid": "65b0cb1e-438f-4dbe-956f-ac4bc575afd2", "code": 400, "message": "Invalid chat message detected: message content must be string"}}

Headers:
skip-error-remapping: REDACTED
x-envoy-upstream-service-time: REDACTED
apim-request-id: REDACTED
x-ms-client-request-id: d8a44898-e39f-48b2-bfb7-cc9837a72da0
Strict-Transport-Security: REDACTED
X-Content-Type-Options: REDACTED
x-ms-region: REDACTED
x-ratelimit-remaining-requests: REDACTED
x-ratelimit-remaining-tokens: REDACTED
Date: Thu, 25 Jul 2024 03:39:56 GMT
Content-Length: 153
Content-Type: application/json; charset=UTF-8

   at Azure.Core.HttpPipelineExtensions.ProcessMessageAsync(HttpPipeline pipeline, HttpMessage message, RequestContext requestContext, CancellationToken cancellationToken)
   at Azure.AI.OpenAI.OpenAIClient.GetChatCompletionsAsync(ChatCompletionsOptions chatCompletionsOptions, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.RunRequestAsync[T](Func`1 request)
   --- End of inner exception stack trace ---
   at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.RunRequestAsync[T](Func`1 request)
   at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.GetChatMessageContentsAsync(ChatHistory chat, PromptExecutionSettings executionSettings, Kernel kernel, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.ChatCompletion.ChatCompletionServiceExtensions.GetChatMessageContentAsync(IChatCompletionService chatCompletionService, ChatHistory chatHistory, PromptExecutionSettings executionSettings, Kernel kernel, CancellationToken cancellationToken)
<Now into application call stack>

[dmytrostruk]

@adrian-watson-pts It looks like it's not supported by Azure .NET SDK, which we are using at the moment:
https://github.com/Azure/azure-sdk-for-net/tree/49b0ace48d35cb4b2ace93b517036b660fc12ca4/sdk/openai/Azure.AI.OpenAI#use-your-own-data-with-azure-openai

NOTE: The concurrent use of Chat Functions and Azure Chat Extensions on a single request is not yet supported. Supplying both will result in the Chat Functions information being ignored and the operation behaving as if only the Azure Chat Extensions were provided. To address this limitation, consider separating the evaluation of Chat Functions and Azure Chat Extensions across multiple requests in your solution design.

[hk4git]

As you said "consider separating the evaluation of Chat Functions and Azure Chat Extensions across multiple requests in your solution design", does this mean both are not available on single request ? What to do If I am relying on data to come/search from either AzureAISearch (Azure Chat Extensions) or from custom native function (Plugin) ? Anywork around to make both work in a single request. Please provide example as well if possible.
Also is there any future plan to make them (Azure Chat Extensions & Plugins) work togather?

[adrian-watson-pts]

The solution to this is to make two calls, first to the knowledge/data retrieval service (Azure AI Search in my case) and then add its response (as ChatHistory) into the chat-completion call to Semantic Kernel.
That pattern is similar to how LangChain works for RAG: https://python.langchain.com/v0.2/docs/tutorials/rag/

[HuskyDanny]

I am also thinking about combining RAG with function calling. The problem I want to resolve is I want the llm to decide do I need to retrieve data or not. For example, saying hello at first didn't need to retrieve the data.

I think the other way to do it is by agent to hook up with the retrieval plugin? Didn't try that yet, but if the functional calling is not working, I think this might be the only way to go?

[dmytrostruk]

I am also thinking about combining RAG with function calling. The problem I want to resolve is I want the llm to decide do I need to retrieve data or not. For example, saying hello at first didn't need to retrieve the data.

@HuskyDanny This case can be handled by function calling functionality. If you have a plugin that is responsible for data retrieval, with automatic function calling this plugin will be executed only when it's necessary. Otherwise, it will be ignored. As you mentioned, for message hello this plugin won't be called.

[HuskyDanny]

Thanks for the reply @dmytrostruk , I am confused among these concepts : 1. ReAct based Agent 2.Functional call based Agent 3. Planner in SK. If we were gonna implement agent or auto agents in our application, which one is recommended? Our scenario is for a nl2sql task.

[gitbatuhan]

I'm on the same confusion. @HuskyDanny , did you figured out? I think i will continue with Functional call based agent and enabling auto function calling.

[sabbadino]

I am exploring the best to combine rag and function calling .. I am currently trying to implement it using the new agent functionality in sk :

1. one agent declares it takes care of replying to some documentation (rag)
2. the second agent is setup with some function calls
3. the group chat aggregator is given instructions to to decide what agent to use

I Will post some code sample in a GitHub repo when I have something working properly

[sabbadino]

And one of the plugin has the responsibility for the rag part , i guess .. still know that open ai docs recommend an optimal maximum num of plugins (function calls) of 10 .. and for sure no more than 20 .. so at a certain point u will have the LLM perform not optimally if you put too much decision into a single LLM "setup" .. at that point u will need another LLM setup " on top" dedicated to route user prompt to "more specific" LLM setup ..

It's quite obvious, but all those python unreadable samples make this concept obscure and magic talks coming from Gandalf like ppl :)

[sabbadino]

Unfortunately the code is in python, but this article gives a good example of the scenario . But I might of course change my mind in the future :)

https://shine-snowman-27f.notion.site/2024-08-16_Introducing-Agent-based-RAG-9b7141ae1cd7-3e370c7fc7664b40a0c2ef93d3b57b01

Still I still personally do not believe into this multi agent calls for a single user prompt ..

[vavjeeva]

You are absolutely correct and going with multiple agents is the right way to implement it. Since i am doing as POC, i start with simple and keep improving it later. Also, semantic kernel is still in maturing state so there going to be lot of breaking changes in future anyway.

[sabbadino]

Yes , it will take still a while before I take sk to prod . Still I am monitoring it a lot now they have agents :) ..
But I do not imply that I think that ppl writing sk is doing a poor job .. still with so many moving parts i still prefer to stay on "bare metal" (that is azure open ai SDK , azure ai search SDK , ms translate SDK ,voice to speech and viceversa SDK , etc ) :)

[vavjeeva]

here is the repository for combining RAG and function calling using ollama. everything local.

https://github.com/vavjeeva/SKOllamaLocalRAGSearchWithFunctionCalling

https://jeevasubburaj.com/2024/09/07/building_a_smart_private_chatbot_with_local_content_search_and_function_calling_with_apis_using_semantic_kernel_and_ollama/

[AlexSchlecht]

Is it the same like this:

#8488