Skip to content

Commit

Permalink
Rename Content Storage to Document Storage (#499)
Browse files Browse the repository at this point in the history
Breaking changes:

* Renamed IContentStorage to IDocumentStorage
* Renamed namespaces
* Renamed exceptions
* Renamed DI methods
* Renamed AzureBlobs nuget package
* Configuration schema changes, deprecated ContentStorageType => use
DocumentStorageType
* Update bicep/arm templates, rename env var matching config settings

Other changes:

* Bump version to 0.60
* Improve README docs
* Fix Postgres docs
* Refactor memory db comparison tests
  • Loading branch information
dluc authored May 18, 2024
1 parent aa92ec4 commit 431a362
Show file tree
Hide file tree
Showing 81 changed files with 583 additions and 621 deletions.
2 changes: 1 addition & 1 deletion Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<Project>
<PropertyGroup>
<!-- Central version prefix - applies to all nuget packages. -->
<Version>0.51.0</Version>
<Version>0.60.0</Version>

<!-- C# lang version, https://learn.microsoft.com/dotnet/csharp/whats-new -->
<LangVersion>12</LangVersion>
Expand Down
239 changes: 117 additions & 122 deletions README.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ Here's comparison table:
* [Quickstart: test KM in few minutes](quickstart)
* [**Memory service**, web clients and plugins](service)
* [**Memory API**, memory ingestion and information retrieval](functions)
* [KM **Extensions**: vector DBs, AI models, Data formats, Orchestration, Content storage](extensions)
* [KM **Extensions**: vector DBs, AI models, Data formats, Orchestration, Document storage](extensions)
* [Embedding **serverless** memory in .NET apps](serverless)
* [**Security**, service and users](security)
* [**How-to guides**, customizing KM and examples](how-to)
Expand Down
Binary file added docs/infra-async.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/infra-sync.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
77 changes: 14 additions & 63 deletions docs/packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,77 +7,28 @@ layout: default
---
# .NET packages

* **Microsoft.KernelMemory.WebClient:** The web client library, can be used to call
a running instance of the Memory web service. .NET Standard 2.0 compatible.
* **Microsoft.KernelMemory.WebClient:** .NET web client to call a running instance of Kernel Memory web service.

[![Nuget package](https://img.shields.io/nuget/vpre/Microsoft.KernelMemory.WebClient)](https://www.nuget.org/packages/Microsoft.KernelMemory.WebClient/)
[![Example code](https://img.shields.io/badge/example-code-blue)](examples/001-dotnet-WebClient)
[![Example code](https://img.shields.io/badge/example-code-blue)](https://github.com/microsoft/kernel-memory/tree/main/examples/001-dotnet-WebClient)

* **Microsoft.KernelMemory.SemanticKernelPlugin:** a Memory plugin for Semantic Kernel,
replacing the original Semantic Memory available in SK. .NET Standard 2.0 compatible.

[![Nuget package](https://img.shields.io/nuget/vpre/Microsoft.KernelMemory.SemanticKernelPlugin)](https://www.nuget.org/packages/Microsoft.KernelMemory.SemanticKernelPlugin/)
[![Example code](https://img.shields.io/badge/example-code-blue)](examples/011-dotnet-using-MemoryPlugin)

* **Microsoft.KernelMemory.Abstractions:** The internal interfaces and models
shared by all packages, used to extend KM to support third party services.
.NET Standard 2.0 compatible.

[![Nuget package](https://img.shields.io/nuget/v/Microsoft.KernelMemory.Abstractions)](https://www.nuget.org/packages/Microsoft.KernelMemory.Abstractions/)

* **Microsoft.KernelMemory.MemoryDb.AzureAISearch:** Memory storage using
**[Azure AI Search](extensions/AzureAISearch)**.

[![Nuget package](https://img.shields.io/nuget/v/Microsoft.KernelMemory.MemoryDb.AzureAISearch)](https://www.nuget.org/packages/Microsoft.KernelMemory.MemoryDb.AzureAISearch/)

* **Microsoft.KernelMemory.MemoryDb.Postgres:** Memory storage using
**[PostgreSQL](extensions/Postgres)**.

[![Nuget package](https://img.shields.io/nuget/v/Microsoft.KernelMemory.MemoryDb.Postgres)](https://www.nuget.org/packages/Microsoft.KernelMemory.MemoryDb.Postgres/)

* **Microsoft.KernelMemory.MemoryDb.Qdrant:** Memory storage using
**[Qdrant](extensions/Qdrant)**.

[![Nuget package](https://img.shields.io/nuget/v/Microsoft.KernelMemory.MemoryDb.Qdrant)](https://www.nuget.org/packages/Microsoft.KernelMemory.MemoryDb.Qdrant/)

* **Microsoft.KernelMemory.AI.AzureOpenAI:** Integration with **[Azure OpenAI](extensions/OpenAI)** LLMs.

[![Nuget package](https://img.shields.io/nuget/v/Microsoft.KernelMemory.AI.AzureOpenAI)](https://www.nuget.org/packages/Microsoft.KernelMemory.AI.AzureOpenAI/)
* **Microsoft.KernelMemory.Core:** Kernel Memory core library including all extensions, can be used to build custom
pipelines and handlers, contains
also the serverless client to use memory in a synchronous way without the web service.

* **Microsoft.KernelMemory.AI.LlamaSharp:** Integration with **[LLama](extensions/LlamaSharp)** LLMs.

[![Nuget package](https://img.shields.io/nuget/v/Microsoft.KernelMemory.AI.LlamaSharp)](https://www.nuget.org/packages/Microsoft.KernelMemory.AI.LlamaSharp/)

* **Microsoft.KernelMemory.AI.OpenAI:** Integration with **[OpenAI](extensions/OpenAI)** LLMs.

[![Nuget package](https://img.shields.io/nuget/v/Microsoft.KernelMemory.AI.OpenAI)](https://www.nuget.org/packages/Microsoft.KernelMemory.AI.OpenAI/)

* **Microsoft.KernelMemory.DataFormats.AzureAIDocIntel:** Integration with
[Azure AI Document Intelligence](extensions/AzureAIDocIntel).

[![Nuget package](https://img.shields.io/nuget/v/Microsoft.KernelMemory.DataFormats.AzureAIDocIntel)](https://www.nuget.org/packages/Microsoft.KernelMemory.DataFormats.AzureAIDocIntel/)

* **Microsoft.KernelMemory.Orchestration.AzureQueues:** Ingestion and synthetic memory
pipelines via [Azure Queue Storage](extensions/AzureQueues).

[![Nuget package](https://img.shields.io/nuget/v/Microsoft.KernelMemory.Orchestration.AzureQueues)](https://www.nuget.org/packages/Microsoft.KernelMemory.Orchestration.AzureQueues/)

* **Microsoft.KernelMemory.Orchestration.RabbitMQ:** Ingestion and synthetic memory
pipelines via [RabbitMQ](extensions/RabbitMQ).

[![Nuget package](https://img.shields.io/nuget/v/Microsoft.KernelMemory.Orchestration.RabbitMQ)](https://www.nuget.org/packages/Microsoft.KernelMemory.Orchestration.RabbitMQ/)
[![Nuget package](https://img.shields.io/nuget/vpre/Microsoft.KernelMemory.Core)](https://www.nuget.org/packages/Microsoft.KernelMemory.Core/)
[![Example code](https://img.shields.io/badge/example-code-blue)](https://github.com/microsoft/kernel-memory/tree/main/examples/002-dotnet-Serverless)

* **Microsoft.KernelMemory.ContentStorage.AzureBlobs:** Used to store content on
[Azure Storage Blobs](extensions/AzureBlobs).
* **Microsoft.KernelMemory.Service.AspNetCore:** an extension to load Kernel Memory into your ASP.NET apps.

[![Nuget package](https://img.shields.io/nuget/v/Microsoft.KernelMemory.ContentStorage.AzureBlobs)](https://www.nuget.org/packages/Microsoft.KernelMemory.ContentStorage.AzureBlobs/)
[![Nuget package](https://img.shields.io/nuget/vpre/Microsoft.KernelMemory.Service.AspNetCore)](https://www.nuget.org/packages/Microsoft.KernelMemory.Service.AspNetCore/)
[![Example code](https://img.shields.io/badge/example-code-blue)](https://github.com/microsoft/kernel-memory/tree/main/examples/204-dotnet-ASP.NET-MVC-integration)

* **Microsoft.KernelMemory.Core:** The core library, can be used to build custom
pipelines and handlers, and contains a serverless client to use memory in a
synchronous way, without the web service. .NET 6+.
* **Microsoft.KernelMemory.SemanticKernelPlugin:** a Memory plugin for Semantic Kernel,
replacing the original Semantic Memory available in SK.

[![Nuget package](https://img.shields.io/nuget/vpre/Microsoft.KernelMemory.Core)](https://www.nuget.org/packages/Microsoft.KernelMemory.Core/)
[![Example code](https://img.shields.io/badge/example-code-blue)](examples/002-dotnet-Serverless)
[![Nuget package](https://img.shields.io/nuget/vpre/Microsoft.KernelMemory.SemanticKernelPlugin)](https://www.nuget.org/packages/Microsoft.KernelMemory.SemanticKernelPlugin/)
[![Example code](https://img.shields.io/badge/example-code-blue)](https://github.com/microsoft/kernel-memory/tree/main/examples/003-dotnet-SemanticKernel-plugin)

### Packages for Python, Java and other languages

Expand Down
2 changes: 1 addition & 1 deletion examples/001-dotnet-WebClient/file9-settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
// "Microsoft.KernelMemory.Handlers.TextPartitioningHandler": "Information",
// "Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.Handlers.SaveEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.ContentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.DocumentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.Pipeline.Queue.AzureQueues": "Information",
"Microsoft.AspNetCore": "Warning"
}
Expand Down
2 changes: 1 addition & 1 deletion examples/002-dotnet-Serverless/appsettings.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
// "Microsoft.KernelMemory.Handlers.TextPartitioningHandler": "Information",
// "Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.Handlers.SaveEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.ContentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.DocumentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.Pipeline.Queue.AzureQueues": "Information",
"Microsoft.AspNetCore": "Warning"
}
Expand Down
2 changes: 1 addition & 1 deletion examples/002-dotnet-Serverless/file9-settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
// "Microsoft.KernelMemory.Handlers.TextPartitioningHandler": "Information",
// "Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.Handlers.SaveEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.ContentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.DocumentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.Pipeline.Queue.AzureQueues": "Information",
"Microsoft.AspNetCore": "Warning"
}
Expand Down
2 changes: 1 addition & 1 deletion examples/101-dotnet-custom-Prompts/appsettings.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
// "Microsoft.KernelMemory.Handlers.TextPartitioningHandler": "Information",
// "Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.Handlers.SaveEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.ContentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.DocumentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.Pipeline.Queue.AzureQueues": "Information",
"Microsoft.AspNetCore": "Warning"
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
// "Microsoft.KernelMemory.Handlers.TextPartitioningHandler": "Information",
// "Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.Handlers.SaveEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.ContentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.DocumentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.Pipeline.Queue.AzureQueues": "Information",
"Microsoft.AspNetCore": "Warning"
}
Expand Down
2 changes: 1 addition & 1 deletion examples/104-dotnet-custom-LLM/appsettings.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
// "Microsoft.KernelMemory.Handlers.TextPartitioningHandler": "Information",
// "Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.Handlers.SaveEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.ContentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.DocumentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.Pipeline.Queue.AzureQueues": "Information",
"Microsoft.AspNetCore": "Warning"
}
Expand Down
4 changes: 2 additions & 2 deletions examples/105-dotnet-serverless-llamasharp/appsettings.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
// "Microsoft.KernelMemory.Handlers.TextPartitioningHandler": "Information",
// "Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.Handlers.SaveEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.ContentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.DocumentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.Pipeline.Queue.AzureQueues": "Information",
"Microsoft.AspNetCore": "Warning"
}
Expand All @@ -33,7 +33,7 @@
"Deployment": "",
// The max number of tokens supported by model deployed
// See https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models
"MaxTokenTotal": 8191,
"MaxTokenTotal": 8191
},
"OpenAI": {
// Name of the model used to generate text (text completion or chat completion)
Expand Down
2 changes: 1 addition & 1 deletion examples/106-dotnet-retrieve-synthetics/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ await memory.ImportDocumentAsync(new Document("doc1")
== file4-KM-Readme.pdf summary ==
Kernel Memory is an AI service designed for efficient indexing of datasets, supporting features like Retrieval Augmented Generation, synthetic memory, and custom semantic memory processing. It integrates with platforms like Semantic Kernel, Microsoft Copilot, and ChatGPT, and is available as a GPT Plugin, web clients, a .NET library, and a Docker container. It allows natural language querying and provides answers with citations from indexed data.
Semantic Memory, part of the Semantic Kernel project, is a library for C#, Python, and Java that supports vector search and wraps database calls. Kernel Memory builds upon this, offering additional features like text extraction from various file formats, secure data frameworks, and a .NET codebase for ease of use across languages and platforms.
Kernel Memory supports a wide range of data formats and backends, including Microsoft Office files, PDFs, web pages, images with OCR, and JSON files. It integrates with various AI and vector storage services and offers content storage and orchestration options.
Kernel Memory supports a wide range of data formats and backends, including Microsoft Office files, PDFs, web pages, images with OCR, and JSON files. It integrates with various AI and vector storage services and offers document storage and orchestration options.
Kernel Memory can be used in serverless mode, embedded in applications, or as a service for scalable document ingestion and information retrieval. It supports custom ingestion pipelines, data lineage, and citations for verifying answer accuracy.
The service provides a web API with OpenAPI documentation for easy access and testing. It also includes a Docker image for quick deployment and a web client for file import and querying. Custom memory ingestion pipelines can be defined with .NET handlers, and the service offers a range of .NET packages for integration with different services and platforms. Python and Java packages are also planned, with contributions for other languages welcomed.
Expand Down
2 changes: 1 addition & 1 deletion examples/106-dotnet-retrieve-synthetics/appsettings.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
// "Microsoft.KernelMemory.Handlers.TextPartitioningHandler": "Information",
// "Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.Handlers.SaveEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.ContentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.DocumentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.Pipeline.Queue.AzureQueues": "Information",
"Microsoft.AspNetCore": "Warning"
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
// "Microsoft.KernelMemory.Handlers.TextPartitioningHandler": "Information",
// "Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.Handlers.SaveEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.ContentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.DocumentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.Pipeline.Queue.AzureQueues": "Information",
"Microsoft.AspNetCore": "Warning"
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
// "Microsoft.KernelMemory.Handlers.TextPartitioningHandler": "Information",
// "Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.Handlers.SaveEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.ContentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.DocumentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.Pipeline.Queue.AzureQueues": "Information",
"Microsoft.AspNetCore": "Warning"
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
// "Microsoft.KernelMemory.Handlers.TextPartitioningHandler": "Information",
// "Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.Handlers.SaveEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.ContentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.DocumentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.Pipeline.Queue.AzureQueues": "Information",
"Microsoft.AspNetCore": "Warning"
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

using Microsoft.KernelMemory;
using Microsoft.KernelMemory.Configuration;
using Microsoft.KernelMemory.ContentStorage.DevTools;
using Microsoft.KernelMemory.DocumentStorage.DevTools;
using Microsoft.KernelMemory.FileSystem.DevTools;
using Microsoft.KernelMemory.MemoryStorage.DevTools;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
// "Microsoft.KernelMemory.Handlers.TextPartitioningHandler": "Information",
// "Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.Handlers.SaveEmbeddingsHandler": "Information",
// "Microsoft.KernelMemory.ContentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.DocumentStorage.AzureBlobs": "Information",
// "Microsoft.KernelMemory.Pipeline.Queue.AzureQueues": "Information",
"Microsoft.AspNetCore": "Warning"
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
namespace Microsoft.Discord.TestApplication;

/// <summary>
/// KM pipeline handler fetching discord data files from content storage
/// KM pipeline handler fetching discord data files from document storage
/// and storing messages in Postgres.
/// </summary>
public sealed class DiscordMessageHandler : IPipelineStepHandler, IDisposable, IAsyncDisposable
Expand Down
2 changes: 1 addition & 1 deletion examples/301-discord-test-application/Program.cs
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// Copyright (c) Microsoft. All rights reserved.

using Microsoft.KernelMemory;
using Microsoft.KernelMemory.ContentStorage.DevTools;
using Microsoft.KernelMemory.DocumentStorage.DevTools;
using Microsoft.KernelMemory.Sources.DiscordBot;

namespace Microsoft.Discord.TestApplication;
Expand Down
2 changes: 1 addition & 1 deletion examples/301-discord-test-application/appsettings.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"DiscordToken": "",
// Index where to store files, e.g. disk folder, Azure blobs folder, etc.
"Index": "discord",
// File name used when uploading a message to content storage.
// File name used when uploading a message to document storage.
"FileName": "discord-msg.json",
// Handlers processing the incoming Discord events
"Steps": [
Expand Down
4 changes: 2 additions & 2 deletions extensions/AzureAISearch/AzureAISearch/AzureAISearchMemory.cs
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@
using Azure.Search.Documents.Models;
using Microsoft.Extensions.Logging;
using Microsoft.KernelMemory.AI;
using Microsoft.KernelMemory.ContentStorage;
using Microsoft.KernelMemory.Diagnostics;
using Microsoft.KernelMemory.DocumentStorage;
using Microsoft.KernelMemory.MemoryStorage;

namespace Microsoft.KernelMemory.MemoryDb.AzureAISearch;
Expand Down Expand Up @@ -92,7 +92,7 @@ public AzureAISearchMemory(

default:
this._log.LogCritical("Azure AI Search authentication type '{0}' undefined or not supported", config.Auth);
throw new ContentStorageException($"Azure AI Search authentication type '{config.Auth}' undefined or not supported");
throw new DocumentStorageException($"Azure AI Search authentication type '{config.Auth}' undefined or not supported");
}
}

Expand Down
Loading

0 comments on commit 431a362

Please sign in to comment.