Extract Python components into their own packages

The following packages were introduced: * "indexify-python-utils" with common functionality for components written in Python. Not published to PyPi. It currently includes: ** Logging configuration for services written in Python (used by Function Executor and Executor). ** HTTP client utils (used by Python SDK and Executor). ** Common build and testing scripts for all other Python packages. * "indexify-python-sdk" for Python SDK library. Not published to PyPi. * "indexify-function-executor" for Function Executor. Not published to PyPi. * "indexify-executor" for Executor. Not published to PyPi. * "indexify" (depends on all the 4 packages above ^ and includes the CLI). Published to PyPi. This is the same "all in one" package we were using right now. It should be compatible with all existing customer UX. The idea is that "indexify" package combines all Open Source Indexify components in one package and customers can use the package to develop their workflows and to run local or distributed Indexify clusters using the CLI and also to build their function images. Also added README.md for all the packages and their tests to clarify what functionality belongs where and how the packages relate to each other. Versioning: "indexify-python-sdk" package has the current latest version 0.2.48. "indexify" package has the same current version 0.2.48. All other packages have version 0.1.0 as it's not as important to keep their versions the same as the current 0.2.48. The versioning policy from now on is the following: * Versions of all the packages are independent except "indexify" package. * When a package version gets incremented then "indexify" package version gets incremented too. ** This ensures that all important enough updates in any Indexify component results in user visible version change. Also update github actions to: * Work with "indexify" package folder instead of python-sdk package folder. * Run all the linter checks and tests in all the packages. * Use poetry version 2.0.0 strictly to prevent random regressions in CI. And update the lock files with never versions of the dependencies from toml files. Testing: In all the packages ran: ``` make check make test ```
tensorlakeai · Jan 7, 2025 · 29c3246 · 29c3246
1 parent 0ea9024
commit 29c3246
Show file tree

Hide file tree

Showing 133 changed files with 9,542 additions and 917 deletions.
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -27,17 +27,17 @@ Consider providing screenshots, logs, code or other visual aids to help the revi
 
 ## Contribution Checklist
 
-- [ ] If the python-sdk was changed, please run `make fmt` in `python-sdk/`.
+- [ ] If a Python package was changed, please run `make fmt` in the package directory.
 - [ ] If the server was changed, please run `make fmt` in `server/`.
 - [ ] Make sure all PR Checks are passing.
 <!--
-You can run the tests manually:
-
 Notes:
 
-- Tests can be run manually: start the server and executor, `cd python-sdk`,
-  run `make test`.
-- To test if changes to the server are backward compatible with the latest
-  release, label the PR with `ci_compat_test`. This might report failures
-  unrelated to your change if previous incompatible changes were pushed without
-  being released yet -->
+Tests of a Python package can be run manually. Start a Server and an Executor then
+run `make test` in the Python package directory.
+
+To test if changes to the server are backward compatible with the latest
+release, label the PR with `ci_compat_test`. This might report failures
+unrelated to your change if previous incompatible changes were pushed without
+being released yet
+-->
diff --git a/.github/workflows/publish_indexify_pypi.yaml b/.github/workflows/publish_indexify_pypi.yaml
@@ -21,7 +21,7 @@ permissions:
 
 defaults:
   run:
-    working-directory: ./python-sdk
+    working-directory: ./indexify
 
 jobs:
   extract-version:
@@ -38,28 +38,10 @@ jobs:
         run: pip install toml
       - id: version_extraction
         run: echo "version=$(python -c 'import toml; print(toml.load("pyproject.toml")["tool"]["poetry"]["version"])')" >> $GITHUB_OUTPUT
-
-  # create-release:
-  #   name: Create GitHub Release
-  #   runs-on: ubuntu-latest
-  #   needs:
-  #     - extract-version
-  #   steps:
-  #     - name: Create GitHub Release
-  #       id: create_release
-  #       uses: actions/create-release@v1
-  #       with:
-  #         tag_name: "v${{ needs.extract-version.outputs.version }}"
-  #         prerelease: ${{ github.event.inputs.prerelease }}
-  #         body: ${{ github.event.inputs.release_message }}
-  #       env:
-  #         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
   publish-indexify-to-pypi:
-    name: Publish Indexify Python Client
+    name: Publish Indexify package
     runs-on: ubuntu-latest
-    # needs:
-    #   - create-release
     environment:
       name: pypi
       url: https://pypi.org/p/indexify
@@ -71,10 +53,10 @@ jobs:
         with:
           python-version: "3.9"
       - name: Install Poetry
-        run: pipx install poetry
+        run: pipx install --force 'poetry==2.0.0'
       - name: Build python-sdk
         run: make build
-      - name: Publish Indexify Client to PyPI
+      - name: Publish Indexify to PyPI
         uses: pypa/gh-action-pypi-publish@release/v1
         with:
           packages-dir: python-sdk/dist/

diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml
@@ -10,11 +10,16 @@ on:
       - 'main'
     paths:
       - 'server/**'
+      - 'executor/**'
+      - 'function_executor/**'
+      - 'indexify/**'
+      - 'python_utils/**'
       - 'python-sdk/**'
       - '.github/workflows/tests.yaml'
 
 env:
   CARGO_TERM_COLOR: always
+  INDEXIFY_URL: http://localhost:8900
 
 jobs:
   lint_server:
@@ -44,7 +49,6 @@ jobs:
         with:
           cache-directories: |
             server/target
-
       - name: Build indexify-server
         run: |
           cd server
@@ -63,52 +67,39 @@ jobs:
           name: indexify-server
           path: server/target/debug/indexify-server
 
-  lint_python_sdk:
-    name: Lint Indexify Python SDK
+  lint_python_packages:
+    name: Lint Python packages
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
       - name: Install poetry
-        run: pipx install poetry
+        run: pipx install --force 'poetry==2.0.0'
       - name: Setup Python
         uses: actions/setup-python@v5
         with:
           python-version: 3.11
           cache: 'poetry'
-      - name: Build python-sdk
-        run: |
-          cd python-sdk
-          make build
-      - name: Lint python-sdk
-        run: |
-          cd python-sdk
-          make check
+      - name: Build and lint executor
+        run: cd executor && make build && make check
+      - name: Build and lint function_executor
+        run: cd function_executor && make build && make check
+      - name: Build and lint indexify
+        run: cd indexify && make build && make check
+      - name: Build and lint python_utils
+        run: cd python_utils && make build && make check
+      - name: Build and lint python-sdk
+        run: cd python-sdk && make build && make check
 
   acceptance_tests:
     name: Run Acceptance Tests
-    needs: [build_server, lint_python_sdk]
+    needs: [build_server, lint_python_packages]
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
       - name: Download indexify-server
         uses: actions/download-artifact@v4
         with:
           name: indexify-server
-      - name: Install poetry
-        run: pipx install poetry
-      - name: Setup Python
-        uses: actions/setup-python@v5
-        with:
-          python-version: 3.11
-          cache: 'poetry'
-      - name: Build python-sdk
-        run: |
-          cd python-sdk
-          make build
-      - name: Lint python-sdk
-        run: |
-          cd python-sdk
-          make check
       - name: Start Background Indexify Server
         uses: JarvusInnovations/background-action@v1
         with:
@@ -125,11 +116,21 @@ jobs:
           log-output: true
           # always logging the output to debug test failures.
           log-output-if: true
+
+      - name: Install poetry
+        run: pipx install --force 'poetry==2.0.0'
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.11
+          cache: 'poetry'
+      - name: Build indexify
+        run: cd indexify && make build
       - name: Start Background Indexify Executor
         uses: JarvusInnovations/background-action@v1
         with:
           run: |
-            cd python-sdk
+            cd indexify
             poetry run indexify-cli executor &
             echo $! > /tmp/indexify-executor.pid &
 
@@ -160,11 +161,16 @@ jobs:
                 sleep 5
             fi
           done
-      - name: Run All Tests
-        run: |
-          cd python-sdk
-          export INDEXIFY_URL=http://localhost:8900
-          make test
+      - name: Run executor tests
+        run: cd executor && make test
+      - name: Run function_executor tests
+        run: cd function_executor && make test
+      - name: Run indexify tests
+        run: cd indexify && make test
+      - name: Run python_utils tests
+        run: cd python_utils && make test
+      - name: Run python-sdk tests
+        run: cd python-sdk && make test
       - name: Terminate processes
         # We want to test clean termination of processes.
         run: |
@@ -189,7 +195,7 @@ jobs:
   last_release_acceptance_tests:
     name: 'Last Release Acceptance Tests (trigger with label: ci_compat_test)'
     if: contains(github.event.pull_request.labels.*.name, 'ci_compat_test')
-    needs: [build_server, lint_python_sdk]
+    needs: [build_server, lint_python_packages]
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
@@ -202,17 +208,6 @@ jobs:
         uses: actions/download-artifact@v4
         with:
           name: indexify-server
-      - name: Install poetry
-        run: pipx install poetry
-      - name: Setup Python
-        uses: actions/setup-python@v5
-        with:
-          python-version: 3.11
-          cache: 'poetry'
-      - name: Build python-sdk
-        run: |
-          cd python-sdk
-          make build
       - name: Start Background Indexify Server
         uses: JarvusInnovations/background-action@v1
         with:
@@ -229,11 +224,20 @@ jobs:
           log-output: true
           # always logging the output to debug test failures.
           log-output-if: true
+      - name: Install poetry
+        run: pipx install --force 'poetry==2.0.0'
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.11
+          cache: 'poetry'
+      - name: Build indexify
+        run: cd indexify && make build
       - name: Start Background Indexify Executor
         uses: JarvusInnovations/background-action@v1
         with:
           run: |
-            cd python-sdk
+            cd indexify
             poetry run indexify-cli executor &
             echo $! > /tmp/indexify-executor.pid &
 
@@ -264,11 +268,16 @@ jobs:
                 sleep 5
             fi
           done
-      - name: Run All Tests
-        run: |
-          cd python-sdk
-          export INDEXIFY_URL=http://localhost:8900
-          make test
+      - name: Run executor tests
+        run: cd executor && make test
+      - name: Run function_executor tests
+        run: cd function_executor && make test
+      - name: Run indexify tests
+        run: cd indexify && make test
+      - name: Run python_utils tests
+        run: cd python_utils && make test
+      - name: Run python-sdk tests
+        run: cd python-sdk && make test
       - name: Terminate processes
         # We want to test clean termination of processes.
         run: |

diff --git a/.vscode/settings.json b/.vscode/settings.json
@@ -0,0 +1,9 @@
+{
+    "python.analysis.extraPaths": [
+        "./python-sdk/src",
+        "./function_executor/src",
+        "./executor/src",
+        "./indexify/src",
+        "./python_utils/src"
+    ]
+}
diff --git a/executor/Makefile b/executor/Makefile
@@ -0,0 +1 @@
+include ../python_utils/common.mk
diff --git a/executor/README.md b/executor/README.md
@@ -0,0 +1,29 @@
+## Overview
+
+Executor registers at Indexify Server and continiously pulls tasks assigned to it from the Indexify Server
+and executes them. While registering it shares its capabilities like available hardware with the Indexify
+Server and periodically updates the Server about its current state. Executor spins up Function Executors
+to run customer functions. Executor should never link with Indexify Python-SDK. It should not know anything
+about programming languages and runtime environments used by Indexify Functions. Function Executor is
+responsible for this.
+
+## Deployment
+
+An Executor can run in a Virtual Machine, container or a in bare metal host. Each vm/container/bare metal
+host in an Indexify cluster deployment runs a single Executor daemon process.
+Open Source users manage the Executors fleet themself e.g. using Kubernetes, any other cluster
+orchestrator or even manually.
+
+An Executor can be configured to only run particular functions. This allows to route certain Indexify
+functions to only particular subsets of Executors to e.g. implement a load balancing strategy.
+
+This package doesn't provide an executable entry point that runs an Executor. This is intentional
+as Executor has many configurable subcomponents. `indexify` package provides a cli with `executor`
+command that runs Executor with functionality available in Open Source offering.
+
+## Threat model
+
+A VM/container/bare metal host where an Executor is running is fully trusted. This works well for single
+tenant deployments where customer functions' code is fully trusted. If this is not the case then Function
+Executors that run customer functions need to get isolated from Executor using e.g. Virtual Machines.
+This functionality is not included into the Open Source offering.
diff --git a/python-sdk/config.yaml → executor/config.yaml b/python-sdk/config.yaml → executor/config.yaml