From 46c423defceacbd4571e69d817a45b47135d9a20 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 26 Oct 2023 19:07:35 +0200 Subject: [PATCH 1/7] docs: Add LocalAI --- mlops-engines.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/mlops-engines.md b/mlops-engines.md index f032171..dd12216 100644 --- a/mlops-engines.md +++ b/mlops-engines.md @@ -29,6 +29,7 @@ Inference Engine | Open-Source | GPU optimisations | Ease of use [](#vllm) | 🟒 Yes | Continuous Batching, Tensor Parallelism, Paged Attention | 🟒 Easy [](#bentoml) | 🟒 Yes | None | 🟒 Easy [](#modular) | πŸ”΄ No | N/A | 🟑 Moderate +[](#localai) | 🟒 Yes | 🟒 Yes | 🟒 Easy ``` {{ table_feedback }} @@ -127,6 +128,18 @@ Cons: This is not an exhaustive list of MLOps engines by any means. There are many other tools and frameworks developer use to deploy their ML models. There is ongoing development in both the open-source and private sectors to improve the performance of LLMs. It's up to the community to test out different services to see which one works best for their use case. +## LocalAI + +[LocalAI](https://localai.io) is the free, Open Source OpenAI alternative. LocalAI act as a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs with various backend such as llama.cpp or vllm, generate images, generate audio, transcribe audio, self-hosted, on-prem with consumer grade hardware, supporting multiple model families. + +Pros: +- wide range of models supported +- supports for LocalAI functions (OpenAI functions, but self-hosted) +- easy to integrate + +Cons: +- binary version is harder to run and compile locally +- high learning curve due to high degree of customization ## Challenges in Open Source From 2bf01fb381734cf1490b86eeb11fcc5b86e1b200 Mon Sep 17 00:00:00 2001 From: Casper da Costa-Luis Date: Fri, 27 Oct 2023 13:29:37 +0100 Subject: [PATCH 2/7] misc copyedits & cross-references --- desktop-apps.md | 2 +- mlops-engines.md | 8 +++++--- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/desktop-apps.md b/desktop-apps.md index 35a8087..57cc607 100644 --- a/desktop-apps.md +++ b/desktop-apps.md @@ -208,7 +208,7 @@ koboldcpp Julius Model Configuration [local.ai]: https://www.localai.app -The [local.ai] App from https://github.com/louisgv/local.ai ([not to be confused](https://github.com/louisgv/local.ai/discussions/71) with [LocalAI](https://localai.io) from https://github.com/mudler/LocalAI) is a simple application for loading LLMs after you manually download a `ggml` model from online. +The [local.ai] App from https://github.com/louisgv/local.ai ([not to be confused](https://github.com/louisgv/local.ai/discussions/71) with [](mlops-engines.md#localai) from https://github.com/mudler/LocalAI) is a simple application for loading LLMs after you manually download a `ggml` model from online. ### UI and Chat diff --git a/mlops-engines.md b/mlops-engines.md index dd12216..14d9e61 100644 --- a/mlops-engines.md +++ b/mlops-engines.md @@ -130,16 +130,18 @@ This is not an exhaustive list of MLOps engines by any means. There are many oth ## LocalAI -[LocalAI](https://localai.io) is the free, Open Source OpenAI alternative. LocalAI act as a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs with various backend such as llama.cpp or vllm, generate images, generate audio, transcribe audio, self-hosted, on-prem with consumer grade hardware, supporting multiple model families. +[LocalAI](https://localai.io) from https://github.com/go-skynet/LocalAI ([not to be confused](https://github.com/louisgv/local.ai/discussions/71) with [](desktop-apps.md#localai) from https://github.com/louisgv/local.ai) is the free, Open Source alternative to OpenAI. LocalAI act as a drop-in replacement REST API that's compatible with OpenAI API specifications for local inferencing. It can run LLMs (with various backend such as https://github.com/ggerganov/llama.cpp or [](#vllm)), generate images, generate audio, transcribe audio, and can be self-hosted (on-prem) with consumer-grade hardware. Pros: + - wide range of models supported -- supports for LocalAI functions (OpenAI functions, but self-hosted) +- support for "LocalAI functions" (self-hosted verison of OpenAI functions) - easy to integrate Cons: + - binary version is harder to run and compile locally -- high learning curve due to high degree of customization +- high learning curve due to high degree of customisation ## Challenges in Open Source From a36a06debf88f5aea8fc8b0b14dbe29168ab0b9c Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 28 Oct 2023 15:28:54 +0200 Subject: [PATCH 3/7] Update mlops-engines.md Co-authored-by: Casper da Costa-Luis --- mlops-engines.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mlops-engines.md b/mlops-engines.md index 14d9e61..649d360 100644 --- a/mlops-engines.md +++ b/mlops-engines.md @@ -135,7 +135,7 @@ This is not an exhaustive list of MLOps engines by any means. There are many oth Pros: - wide range of models supported -- support for "LocalAI functions" (self-hosted verison of OpenAI functions) +- support for [functions](https://localai.io/features/openai-functions) (self-hosted [OpenAI functions](https://platform.openai.com/docs/guides/gpt/function-calling)) - easy to integrate Cons: From 278fbe73ae2197f8467641b73cce79ef722b168e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 28 Oct 2023 15:29:04 +0200 Subject: [PATCH 4/7] Update mlops-engines.md Co-authored-by: Casper da Costa-Luis --- mlops-engines.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mlops-engines.md b/mlops-engines.md index 649d360..1c39db2 100644 --- a/mlops-engines.md +++ b/mlops-engines.md @@ -134,7 +134,7 @@ This is not an exhaustive list of MLOps engines by any means. There are many oth Pros: -- wide range of models supported +- [wide range of models supported](https://localai.io/model-compatibility) - support for [functions](https://localai.io/features/openai-functions) (self-hosted [OpenAI functions](https://platform.openai.com/docs/guides/gpt/function-calling)) - easy to integrate From 4f5920ba5e5ccb09df6f70c4f6f4746e8d6e2961 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 29 Oct 2023 13:52:00 +0000 Subject: [PATCH 5/7] fixups --- mlops-engines.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mlops-engines.md b/mlops-engines.md index 1c39db2..c3c4316 100644 --- a/mlops-engines.md +++ b/mlops-engines.md @@ -130,17 +130,17 @@ This is not an exhaustive list of MLOps engines by any means. There are many oth ## LocalAI -[LocalAI](https://localai.io) from https://github.com/go-skynet/LocalAI ([not to be confused](https://github.com/louisgv/local.ai/discussions/71) with [](desktop-apps.md#localai) from https://github.com/louisgv/local.ai) is the free, Open Source alternative to OpenAI. LocalAI act as a drop-in replacement REST API that's compatible with OpenAI API specifications for local inferencing. It can run LLMs (with various backend such as https://github.com/ggerganov/llama.cpp or [](#vllm)), generate images, generate audio, transcribe audio, and can be self-hosted (on-prem) with consumer-grade hardware. +[LocalAI](https://localai.io) from https://github.com/mudler/LocalAI ([not to be confused](https://github.com/louisgv/local.ai/discussions/71) with [](desktop-apps.md#localai) from https://github.com/louisgv/local.ai) is the free, Open Source alternative to OpenAI. LocalAI act as a drop-in replacement REST API that's compatible with OpenAI API specifications for local inferencing. It can run LLMs (with various backend such as https://github.com/ggerganov/llama.cpp or [](#vllm)), generate images, generate audio, transcribe audio, and can be self-hosted (on-prem) with consumer-grade hardware. Pros: - [wide range of models supported](https://localai.io/model-compatibility) - support for [functions](https://localai.io/features/openai-functions) (self-hosted [OpenAI functions](https://platform.openai.com/docs/guides/gpt/function-calling)) -- easy to integrate +- [easy to integrate](https://localai.io/integrations/) Cons: -- binary version is harder to run and compile locally +- binary version is harder to run and compile locally. [For example, see issue 1196](https://github.com/mudler/LocalAI/issues/1196). - high learning curve due to high degree of customisation ## Challenges in Open Source From 13f4cd7d039c389085b472b5f280a1daf7cf053d Mon Sep 17 00:00:00 2001 From: Casper da Costa-Luis Date: Sun, 5 Nov 2023 15:11:40 +0000 Subject: [PATCH 6/7] slight copyedits --- mlops-engines.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mlops-engines.md b/mlops-engines.md index c3c4316..ee90937 100644 --- a/mlops-engines.md +++ b/mlops-engines.md @@ -136,11 +136,11 @@ Pros: - [wide range of models supported](https://localai.io/model-compatibility) - support for [functions](https://localai.io/features/openai-functions) (self-hosted [OpenAI functions](https://platform.openai.com/docs/guides/gpt/function-calling)) -- [easy to integrate](https://localai.io/integrations/) +- [easy to integrate](https://localai.io/integrations) Cons: -- binary version is harder to run and compile locally. [For example, see issue 1196](https://github.com/mudler/LocalAI/issues/1196). +- binary version is harder to run and compile locally. https://github.com/mudler/LocalAI/issues/1196. - high learning curve due to high degree of customisation ## Challenges in Open Source From 9295d860e2f991abc9d6c55f33ee947ca56e5fe6 Mon Sep 17 00:00:00 2001 From: Casper da Costa-Luis Date: Sun, 5 Nov 2023 23:06:10 +0000 Subject: [PATCH 7/7] update links, drop langchain embedding models --- model-formats.md | 2 +- sdk.md | 3 +-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/model-formats.md b/model-formats.md index 4c62556..6d15189 100644 --- a/model-formats.md +++ b/model-formats.md @@ -280,7 +280,7 @@ Some [clients & libraries supporting `GGUF`](https://huggingface.co/TheBloke/Lla - [LM Studio](https://lmstudio.ai) -- an easy-to-use and powerful local GUI with GPU acceleration on both Windows (NVidia and AMD), and macOS ```{seealso} -For more info on `GGUF`, see https://github.com/ggerganov/llama.cpp/pull/2398 and its [spec](https://github.com/philpax/ggml/blob/gguf-spec/docs/gguf.md). +For more info on `GGUF`, see https://github.com/ggerganov/llama.cpp/pull/2398 and its [spec](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md). ``` ### Limitations diff --git a/sdk.md b/sdk.md index 7a34a48..f029d9e 100644 --- a/sdk.md +++ b/sdk.md @@ -46,11 +46,10 @@ The list of vector stores that LangChain supports can be found [here](https://ap ### Models -This is the heart of most LLM models where the core functionality resides. There are broadly 3 different [models](https://docs.langchain.com/docs/components/models) that LLMs provide. They are Language, Chat, and Embedding model. +This is the heart of most LLMs, where the core functionality resides. There are broadly [2 different types of models](https://python.langchain.com/docs/modules/model_io/models) which LangChain integrates with: - **Language**: Inputs & outputs are `string`s - **Chat**: Run on top of a Language model. Inputs are a list of chat messages, and output is a chat message -- **Embedding**: Inputs is a `string` and outputs are a list of `float`s (vector) ### Tools