update docs for remote ai worker

livepeer · Oct 28, 2024 · 94d75ef · 94d75ef
1 parent 975c7c4
commit 94d75ef
Show file tree

Hide file tree

Showing 3 changed files with 128 additions and 0 deletions.
diff --git a/ai/orchestrators/ai-worker.mdx b/ai/orchestrators/ai-worker.mdx
@@ -0,0 +1,124 @@
+---
+title: Split or Combined Orchestrator with AI Worker
+---
+
+## AI Worker
+The Orchestrator uses an **AI Worker** to manage the runner containers that do the inference.
+The **AI Worker** can be a separate process run on compute machines separate from the Orchestrator or run as one process with the Orchestrator.
+Split or combined must be selected at startup of the Orchestrator and cannot be run together (combined Orchestrator cannot also have separate AI Workers).
+
+The Orchestrator and AI Worker both use the `aiModels.json` configuration (see [Configuring AI Models](models-config)).
+For split Orchestrator and **AI Worker**:
+> - Orchestrator only uses the fields needed to set pricing.
+> - **AI Worker** only uses the fields needed to manage the runner containers
+
+
+## Split Orchestrator and AI Worker
+
+Orchestrator split from the AI Worker will manage AI Workers connected and allocate work based on capacity for each AI Worker connected.
+Capacity set by the AI Worker is one-to-one `runner container count per pipeline/model_id = capacity per pipeline/model_id` and
+is updated when AI Workers connect and disconnect. 
+<Info>
+    Note that if running external runners, the external runner containers would connect to the AI Worker not the Orchestrator.
+</Info>
+
+## Combined Orchestrator and AI Worker
+The **AI Worker** is setup at Orchestrator launch and capacity is set the same where `runner container count per pipeline/model_id = capacity per pipeline/model_id` but only considers the GPUs available to the Orchestrator on the machine or externally connected.
+
+## Example configurations
+
+<Info>
+    See [Start Your AI Orchestrator](start-orchestrator) for full launch command and see [Configuring AI Models](models-config) for extended information on setting up model configurations.
+</Info>
+
+<Accordion title="Split Orchestrator and AI Worker">
+    ```bash
+    #Orchestrator
+    docker run \
+        --name livepeer_ai_orchestrator \
+        ...
+        -orchestrator \
+        ...
+        -aiModels /root/.lpData/aiModels.json \
+
+    #AI Worker
+    docker run \
+        -name livepeer_ai_worker \
+        -aiWorker \
+        -orchAddr [service addr of orch] \
+        -orchSecert very-big-secret \
+        -nvidia "all" \
+        -v 6 \
+        -aiModels /root/.lpData/aiModels.json \
+        -aiModelsDir /root/.lpData/models \
+        -testTranscoder=false\
+        -aiRunnerImage livepeer/ai-runner:latest #OPTIONAL
+
+    ```
+    <Info>
+      Pricing used in this example is subject to change and should be set competitively based on market research and costs to provide the compute.
+    </Info>
+    `aiModels.json`
+    ```json
+
+    #Orchestrator
+    [
+      {
+        "pipeline": "text-to-image",
+        "model_id": "SG161222/RealVisXL_V4.0_Lightning",
+        "price_per_unit": 4768371,
+        "pixels_per_unit": 1,
+      }
+    ]
+
+    #AI Worker
+    [
+      {
+        "pipeline": "text-to-image",
+        "model_id": "SG161222/RealVisXL_V4.0_Lightning",
+        "warm": true,
+        "optimization_flags": {
+            "SFAST": true
+        }
+      }
+    ]
+    ```
+</Accordion>
+<Accordion title="Combined Orchestrator and AI Worker">
+    ```bash
+    #Orchestrator
+    docker run \
+        --name livepeer_ai_orchestrator \
+        ...
+        -orchestrator \
+        -transcoder \
+        -nvidia "all" \
+        -v 6 \
+        ...
+        -aiWorker
+        -aiModels /root/.lpData/aiModels.json \
+        -aiModelsDir /root/.lpData/models \
+        -aiRunnerImage livepeer/ai-runner:latest #OPTIONAL
+    ```
+    <Info>
+      Pricing used in this example is subject to change and should be set competitively based on market research and costs to provide the compute.
+    </Info>
+    `aiModels.json`
+    ```json
+
+    #Orchestrator
+    [
+      {
+        "pipeline": "text-to-image",
+        "model_id": "SG161222/RealVisXL_V4.0_Lightning",
+        "price_per_unit": 4768371,
+        "pixels_per_unit": 1,
+        "warm": true,
+        "optimization_flags": {
+            "SFAST": true
+        }
+      }
+    ]
+    ```
+</Accordion>
+
diff --git a/ai/orchestrators/models-config.mdx b/ai/orchestrators/models-config.mdx
@@ -10,6 +10,9 @@ Orchestrators specify supported AI models in an `aiModels.json` file, typically
 located in the `~/.lpData` directory. Below is an example configuration showing
 currently **recommended** models and their respective prices.
 
+<Info>
+  Pricing used in this example is subject to change and should be set competitively based on market research and costs to provide the compute.
+</Info>
 ```json
 [
   {

diff --git a/mint.json b/mint.json
@@ -547,6 +547,7 @@
           "iconType": "solid",
           "pages": [
             "ai/orchestrators/get-started",
+            "ai/orchestrators/ai-worker",
             "ai/orchestrators/models-config",
             "ai/orchestrators/models-download",
             "ai/orchestrators/start-orchestrator",