Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extract Python components into their own packages #1147

Merged
merged 1 commit into from
Jan 7, 2025
Merged

Conversation

eabatalov
Copy link
Contributor

@eabatalov eabatalov commented Jan 6, 2025

The following packages were introduced:

  • "indexify-python-utils" with common functionality for components written in Python. Not published to PyPi. It currently includes:
    ** Logging configuration for services written in Python (used by Function Executor and Executor).
    ** HTTP client utils (used by Python SDK and Executor).
    ** Common build and testing scripts for all Python packages.
  • "indexify-python-sdk" for Python SDK library. Not published to PyPi.
  • "indexify-function-executor" for Function Executor. Not published to PyPi.
  • "indexify-executor" for Executor. Not published to PyPi.
  • "indexify" (depends on all the 4 packages above ^ and includes the CLI). Published to PyPi. This is the same "all in one" package we were using right now. It should be compatible with all existing customer UX. The idea is that "indexify" package combines all Open Source Indexify components in one package and customers can use the package to develop their workflows and to run local or distributed Indexify clusters using the CLI and also to build their function images.

Also added README.md for all the packages and their tests to clarify what functionality belongs where and how the packages relate to each other.

Versioning:

"indexify-python-sdk" package has the current latest version 0.2.48. "indexify" package has the same current version 0.2.48. All other packages have version 0.1.0 as it's not as important to keep their versions the same as the current 0.2.48.

The versioning policy from now on is the following:

  • Versions of all the packages are independent except "indexify" package.
  • When a package version gets incremented then "indexify" package version gets incremented too. ** This ensures that all important enough updates in any Indexify component results in user visible version change.

Also update github actions to:

  • Work with "indexify" package folder instead of python-sdk package folder.
  • Run all the linter checks and tests in all the packages.
  • Use poetry version 2.0.0 strictly to prevent random regressions in CI.

And update the lock files with never versions of the dependencies from toml files.

Testing:

make check
make test

Contribution Checklist

  • If the python-sdk was changed, please run make fmt in python-sdk/.
  • [n/a] If the server was changed, please run make fmt in server/.
  • Make sure all PR Checks are passing.

@eabatalov eabatalov force-pushed the eugene-sdk-split branch 24 times, most recently from a08b2ef to ac594f9 Compare January 7, 2025 12:00
The following packages were introduced:

* "indexify-python-utils" with common functionality for components written in Python.
  Not published to PyPi. It currently includes:
  ** Logging configuration for services written in Python (used by Function Executor and Executor).
  ** HTTP client utils (used by Python SDK and Executor).
  ** Common build and testing scripts for all other Python packages.
* "indexify-python-sdk" for Python SDK library. Not published to PyPi.
* "indexify-function-executor" for Function Executor. Not published to PyPi.
* "indexify-executor" for Executor. Not published to PyPi.
* "indexify" (depends on all the 4 packages above ^ and includes the CLI).
  Published to PyPi. This is the same "all in one" package we were using right now.
  It should be compatible with all existing customer UX.
  The idea is that "indexify" package combines all Open Source Indexify components
  in one package and customers can use the package to develop their workflows and to
  run local or distributed Indexify clusters using the CLI and also to build their
  function images.

Also added README.md for all the packages and their tests to clarify what functionality
belongs where and how the packages relate to each other.

Versioning:

"indexify-python-sdk" package has the current latest version 0.2.48.
"indexify" package has the same current version 0.2.48.
All other packages have version 0.1.0 as it's not as important to keep their versions
the same as the current 0.2.48.

The versioning policy from now on is the following:
* Versions of all the packages are independent except "indexify" package.
* When a package version gets incremented then "indexify" package version
  gets incremented too.
  ** This ensures that all important enough updates in any Indexify component
     results in user visible version change.

Also update github actions to:
* Work with "indexify" package folder instead of
python-sdk package folder.
* Run all the linter checks and tests in all the packages.
* Use poetry version 2.0.0 strictly to prevent random regressions in CI.

And update the lock files with never versions of the dependencies from toml files.

Testing:

In all the packages ran:

```
make check
make test
```
@eabatalov eabatalov marked this pull request as ready for review January 7, 2025 12:30
@eabatalov eabatalov merged commit 4546bbd into main Jan 7, 2025
8 checks passed
@eabatalov eabatalov deleted the eugene-sdk-split branch January 7, 2025 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant