Right approach to make sense of JSON data

[rajyraman]

I am pretty new to SK and LLMs in general. The use case I want to try is craft a prompt to search through a large JSON data that comes from API response. Initially I thought I can send it in the prompt itself as context, but then learnt that there are going to be too many tokens in that case and also it might exceed the token limit. So I am now exploring one of these two options:

1. Use function calling and do the retrieval and making sense of JSON data (from API) in the function itself.
2. Use Semantic Memory and load the JSON into VolatileMemoryStore. Then craft the prompt to search this data.

What is the better approach to follow, or is there any other different approach?

[dmytrostruk]

Hi @rajyraman !
I'm wondering, if you need to perform a search through a large JSON data that comes from API response, do you need to send a prompt at all? I think you can convert your JSON to embeddings and then perform a vector search with your query to find specific data in this JSON? In this case, there is no need in function calling, and using VolatileMemoryStore or other memory connector to retrieve related data should be enough. Let us know if this could cover your case. Thank you!

[dmytrostruk]

I really like the experience that I see in https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Agents/OpenAIAssistant_FileManipulation.cs. Because I don't have to deal with any of the embedding, retrieval bit. Just feed the file and you can query with prompt. The is what I see as ideal experience. But seems like it is not possible with Azure OAI.

@rajyraman If you are interested in Assistant API and file manipulation, it's possible to use it with Azure OpenAI. The example you shared above use constructor for communication with OpenAI, but it also has constructor overload which accepts endpoint parameter, which you can use to pass Azure OpenAI Endpoint URL. You could try this approach and let us know if it covers your scenario. Thanks!

[rajyraman]

@dmytrostruk - It does not work with the Azure Open AI endpoint.

This is exactly as per https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0026-file-service.md#context-and-problem-statement which says "Azure Open AI does not currently support the OpenAI file service API."

Please clarify which constructor accepts in AOI endpoint URL.

[dmytrostruk]

Please clarify which constructor accepts in AOI endpoint URL.

Sorry, I was talking about OpenAIFileService, which is used in this example. In example, the service is initialized with OpenAI key, but there is another constructor that accepts Azure OpenAI settings:

semantic-kernel/dotnet/src/Connectors/Connectors.OpenAI/Files/OpenAIFileService.cs

Lines 42 to 53 in 3b0d086

	/// <summary>
	/// Create an instance of the Azure OpenAI chat completion connector
	/// </summary>
	/// <param name="endpoint">Azure Endpoint URL</param>
	/// <param name="apiKey">Azure OpenAI API Key</param>
	/// <param name="organization">OpenAI Organization Id (usually optional)</param>
	/// <param name="version">The API version to target.</param>
	/// <param name="httpClient">Custom <see cref="HttpClient"/> for HTTP requests.</param>
	/// <param name="loggerFactory">The <see cref="ILoggerFactory"/> to use for logging. If null, no logging will be performed.</param>
	public OpenAIFileService(
	Uri endpoint,
	string apiKey,

This is exactly as per https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0026-file-service.md#context-and-problem-statement which says "Azure Open AI does not currently support the OpenAI file service API."

@crickman Could you please provide an information about ADR? Is it up-to-date?

[rajyraman]

@dmytrostruk - According to this https://github.com/microsoft/semantic-kernel/blob/feature_agents_openai/dotnet/docs/OPENAI-CONNECTOR-MIGRATION.md#4-openai-file-service-deprecation lot of things have still not been merged into main yet. Once #7126 is merged I can probably try it or is there any other way?

[dmytrostruk]

According to this https://github.com/microsoft/semantic-kernel/blob/feature_agents_openai/dotnet/docs/OPENAI-CONNECTOR-MIGRATION.md#4-openai-file-service-deprecation lot of things have still not been merged into main yet. Once #7126 is merged I can probably try it or is there any other way?

@rajyraman This part of work is a migration to new OpenAI SDK. After it's merged, it's possible that code syntax or namespaces will change a little bit, but in terms of functionality everything should work pretty much the same. If you are interested in Assistants API, I think it's possible to achieve your scenario with current functionality in main branch using Azure OpenAI. @crickman will provide more information if needed.

[Cobra86]

I'm using Azure AI Search with SK. It's working perfectly. The only issue is that Azure AI Search/SK only supports one index.

[crickman]

@rajyraman - Our implementation of OpenAIFileService does appear to struggle with Azure endpoint for certain interactions; although, it was designed to support targeting both OpenAI and Azure Open AI services.

The current version of SK is based on V1 SDK's and we have a major update just around the corner where we've migrated to the V2 SDK's. This update impacts the memory connectors and the assistant-agents. As part of this, we've made obsolete the OpenAIFileService in favor of direclty using the FileClient from the V2 SDK. This sample has been updated to include this change in a feature-branch and is expected to be merged to main in the very near future!!!

In the meantime, I've re-written this sample to use the V1 AssistantsClient and confirmed functionality against our team's Azure deployments.

// Copyright (c) Microsoft. All rights reserved.
using System.Text;
using Azure;
using Azure.AI.OpenAI.Assistants;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Agents;
using Microsoft.SemanticKernel.Agents.OpenAI;
using Microsoft.SemanticKernel.ChatCompletion;
using Resources;

namespace Agents;

/// <summary>
/// Demonstrate using code-interpreter to manipulate and generate csv files with <see cref="OpenAIAssistantAgent"/> .
/// </summary>
public class OpenAIAssistant_FileManipulation(ITestOutputHelper output) : BaseTest(output)
{
    /// <summary>
    /// Target OpenAI services.
    /// </summary>
    protected override bool ForceOpenAI => false;

    [Fact]
    public async Task AnalyzeCSVFileUsingOpenAIAssistantAgentAsync()
    {
        AssistantsClient client = new(new Uri(TestConfiguration.AzureOpenAI.Endpoint), new AzureKeyCredential(TestConfiguration.AzureOpenAI.ApiKey));
        await using Stream stream = EmbeddedResource.ReadStream("sales.csv")!;
        OpenAIFile fileInfo = await client.UploadFileAsync(stream, OpenAIFilePurpose.Assistants);

        Console.WriteLine(this.ApiKey);

        // Define the agent
        OpenAIAssistantAgent agent =
            await OpenAIAssistantAgent.CreateAsync(
                kernel: new(),
                config: new(this.ApiKey, this.Endpoint),
                new()
                {
                    EnableCodeInterpreter = true, // Enable code-interpreter
                    ModelId = this.Model,
                    FileIds = [fileInfo.Id] // Associate uploaded file
                });

        // Create a chat for agent interaction.
        AgentGroupChat chat = new();

        // Respond to user input
        try
        {
            await InvokeAgentAsync("Which segment had the most sales?");
            await InvokeAgentAsync("List the top 5 countries that generated the most profit.");
            await InvokeAgentAsync("Create a tab delimited file report of profit by each country per month.");
        }
        finally
        {
            await agent.DeleteAsync();
            await client.DeleteFileAsync(fileInfo.Id);
        }

        // Local function to invoke agent and display the conversation messages.
        async Task InvokeAgentAsync(string input)
        {
            chat.AddChatMessage(new ChatMessageContent(AuthorRole.User, input));

            Console.WriteLine($"# {AuthorRole.User}: '{input}'");

            await foreach (ChatMessageContent content in chat.InvokeAsync(agent))
            {
                Console.WriteLine($"# {content.Role} - {content.AuthorName ?? "*"}: '{content.Content}'");

                foreach (AnnotationContent annotation in content.Items.OfType<AnnotationContent>())
                {
                    Console.WriteLine($"\n* '{annotation.Quote}' => {annotation.FileId}");
                    BinaryData fileContent = await client.GetFileContentAsync(annotation.FileId!);
                    Console.WriteLine(Encoding.Default.GetString(fileContent.ToArray()));
                }
            }
        }
    }
}

[rajyraman]

@crickman - Is there a specific version I should use? I am getting this exception.

[rajyraman]

@crickman - Seems like I can use the JSON file to do search using Code Interpreter. Now I need to understand why the code is not working.