Skip to content

Commit

Permalink
Fix dotnet publish // LLamaSharp conflicts (#500)
Browse files Browse the repository at this point in the history
## Motivation and Context (Why the change? What's the scenario?)

LLamaSharp doesn't allow to include multiple backends at once in a
packaged .NET app, requiring to build once for each backend (CPU,
CUDA11, CUDA12, etc). The current code fails to "publish" due to
concurring backends.

See also #498

## High level description (Approach, Design)

Solve the "publish" issue by removing all LLamaSharp backends, leaving
it to the end user/developer to choose one.

The need to build multiple times, once per LLamaSharp backend, persists,
due to how LLamaSharp is designed. (if anyone knows of a workaround
please let us know!)
  • Loading branch information
dluc authored May 18, 2024
1 parent 431a362 commit 8cc78e6
Show file tree
Hide file tree
Showing 5 changed files with 42 additions and 2 deletions.
2 changes: 2 additions & 0 deletions Directory.Packages.props
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,9 @@
<PackageVersion Include="HtmlAgilityPack" Version="1.11.61" />
<PackageVersion Include="LLamaSharp" Version="0.12.0" />
<PackageVersion Include="LLamaSharp.Backend.Cpu" Version="0.12.0" />
<PackageVersion Include="LLamaSharp.Backend.Cuda11" Version="0.12.0" />
<PackageVersion Include="LLamaSharp.Backend.Cuda12" Version="0.12.0" />
<PackageVersion Include="LLamaSharp.Backend.OpenCL" Version="0.12.0" />
<PackageVersion Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.22.0" />
<PackageVersion Include="Microsoft.Bcl.AsyncInterfaces" Version="8.0.0" />
<PackageVersion Include="Microsoft.Data.SqlClient" Version="5.2.0" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,14 @@
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>

<ItemGroup>
<!-- To run the example choose one of these backends -->
<PackageReference Include="LLamaSharp.Backend.Cpu" />
<!-- <PackageReference Include="LLamaSharp.Backend.Cuda11" /> -->
<!-- <PackageReference Include="LLamaSharp.Backend.Cuda12" /> -->
<!-- <PackageReference Include="LLamaSharp.Backend.OpenCL" /> -->
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\..\extensions\AzureOpenAI\AzureOpenAI.csproj" />
<ProjectReference Include="..\..\extensions\OpenAI\OpenAI.csproj" />
Expand Down
13 changes: 13 additions & 0 deletions examples/105-dotnet-serverless-llamasharp/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,19 @@
using Microsoft.KernelMemory;
using Microsoft.KernelMemory.AI.OpenAI;

/* PLEASE NOTE:
*
* The example is pre-configured to use a CPU backend.
*
* If your device provides CUDA or other backends, you can configure the example by editing the project file (*.csproj)
* and editing the LLamaSharp PackageReference, removing the CPU backend and choosing a different one.
*
* Kernel Memory web service is configured to not have ANY pre-configured LLamaSharp backend, e.g. KM Docker
* image does not work with LLamaSharp without manually mounting a LLamaSharp backend assembly into the
* Docker image at runtime. Same for custom .NET apps, which need to choose a LLamaSharp backend manually
* installing one of the available LLamaSharp.Backend packages.
*/

var llamaConfig = new LlamaSharpConfig();
var azureOpenAIEmbeddingConfig = new AzureOpenAIConfig();

Expand Down
12 changes: 10 additions & 2 deletions extensions/LlamaSharp/LlamaSharp/LlamaSharp.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,16 @@

<ItemGroup>
<PackageReference Include="LLamaSharp" />
<PackageReference Include="LLamaSharp.Backend.Cpu" />
<PackageReference Include="LLamaSharp.Backend.Cuda12" />

<!-- Kernel Memory web service is configured to not have ANY pre-configured LLamaSharp backend, e.g. KM Docker
image does not work with LLamaSharp without manually mounting a LLamaSharp backend assembly into the
Docker image at runtime. Same for custom .NET apps, which need to choose a LLamaSharp backend manually
installing one of the available LLamaSharp.Backend packages. -->

<!-- <PackageReference Include="LLamaSharp.Backend.Cpu" /> -->
<!-- <PackageReference Include="LLamaSharp.Backend.Cuda11" /> -->
<!-- <PackageReference Include="LLamaSharp.Backend.Cuda12" /> -->

<PackageReference Include="System.Linq.Async" />
</ItemGroup>

Expand Down
9 changes: 9 additions & 0 deletions service/Service/Service.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,15 @@
<PackageReference Include="Swashbuckle.AspNetCore" />
</ItemGroup>

<ItemGroup>
<!-- If you plan on using LLamaSharp, you will need to enable one of these backends,
which depend on the hardware hosting Kernel Memory code. -->

<!-- <PackageReference Include="LLamaSharp.Backend.Cpu" /> -->
<!-- <PackageReference Include="LLamaSharp.Backend.Cuda11" /> -->
<!-- <PackageReference Include="LLamaSharp.Backend.Cuda12" /> -->
</ItemGroup>

<!--
This dependency is needed only to generate appsetting.development.json via
Expand Down

0 comments on commit 8cc78e6

Please sign in to comment.