Compile automatically pythons package in one shared library and remove the source code when we build a distribution.

Introduction

• Cython can compile a module but not a package ( a collection of modules )
• Sometime, you want to publish an optimized version of your python package/module.
• Sometime, you don't want to publish the source code. You can use PyArmor, or Pyminifier but if you want to compile the API, it's not a good approach.

Using cython

Cython is complex for the normal python developer.

• To compile the code, you must use a .pyx file. It's break your simple usage : edit, test, edit, test. Now, you must use : edit, package, test, edit, package, test.
• Cython propose to create a shared library for each module. If in you package, you have ten python file, you will have ten shared library.
• The wheel package merge the python source code and the shared library. It's big and not confidential.

We can do better.

Based of the idea presented here, this component can help you to continue to use a classical python source code, and generate a single optimized module when you create the wheel.

And, it's VERY SIMPLE to use !

Usage

For each package where you want to compile and/or hide the source code, add a file __compile__.py with nothing. With that, you can continue to use your classical python file (or add some .pyx files). For the developer point of vue, you continue to use the interpreted python code.

Then, to be compatible with PEP-0517,

• in the pyproject.toml

[build-system]
requires = ["setuptools>=42", "wheel", "cythonpackage[build]"]
build-backend = "setuptools.build_meta"

• in the setup.py:

from setuptools import setup, find_packages

setup(
    setup_requires=['cythonpackage[build]'],
    cythonpackage=True,
    ...
    packages=find_packages(),
)

• in the setup.cfg, add other projects information:

[metadata]
name = my_compiled_project
...

Now, you can build your compiled wheel.

python3 -m pip install --upgrade build
python -m build

You can check inside the dist/*.whl to see if the source code in __compile__ package are removed and replaced with a shared library. All others .py were pre-compiled.

In another virtualenv and directory, try to install this wheel.

$ mkdir -p tmp
$ cd tmp
$ virtualenv test
$ source test/bin/activate
$ pip install ../dist/*.whl
$ python ...  # use your package

We use another virtualenv and another directory to remove the confusion between your local python source code and the compiled version. Sometime, you use the interpreted version if the source code are accessible with the PYTHONPATH.

You can check the running context with:

if cython.compiled:
    print("Use compiled version")
else:
    print("Use slow interpreted version")

Use can try the sample present here: https://github.com/pprados/test-cythonpackage Try to rename the pyproject.toml or setup.py to test different kind of builds.

Multiple architecture

The wheel file generated is specific for an architecture and a Python version. The name describe that : test_cythonpackage-0.0.0-cp38-cp38-manylinux_2_31_x86_64.whl To be compatible with multiple combinaison of python version or architecture, you must build wheels for all architecture. Try cibuildwheel to generate a version of each Operating System and python versions.

$ pip install cibuildwheel
$ python3 -m cibuildwheel --output-dir dist --platform linux

To create a classical version of your package, without compilation and with source code, use:

$ CYTHONPACKAGE=false python3 -m setup.py bdist_wheel

Use a CI action like github action. You can generate something like:

test_cythonpackage-0.0.0-cp37-cp37m-macosx_10_15_x86_64.whl
test_cythonpackage-0.0.0-cp37-cp37m-manylinux_2_12_i686.manylinux_2_5_i686.manylinux1_i686.manylinux2010_i686.whl
test_cythonpackage-0.0.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux_2_5_x86_64.manylinux1_x86_64.manylinux2010_x86_64.whl
test_cythonpackage-0.0.0-cp37-cp37m-musllinux_1_1_i686.whl
test_cythonpackage-0.0.0-cp37-cp37m-musllinux_1_1_x86_64.whl
test_cythonpackage-0.0.0-cp37-cp37m-win32.whl
test_cythonpackage-0.0.0-cp37-cp37m-win_amd64.whl
test_cythonpackage-0.0.0-cp38-cp38-macosx_10_15_x86_64.whl
test_cythonpackage-0.0.0-cp38-cp38-manylinux_2_12_i686.manylinux_2_5_i686.manylinux1_i686.manylinux2010_i686.whl
test_cythonpackage-0.0.0-cp38-cp38-manylinux_2_12_x86_64.manylinux_2_5_x86_64.manylinux1_x86_64.manylinux2010_x86_64.whl
test_cythonpackage-0.0.0-cp38-cp38-musllinux_1_1_i686.whl
test_cythonpackage-0.0.0-cp38-cp38-musllinux_1_1_x86_64.whl
test_cythonpackage-0.0.0-cp38-cp38-win32.whl
test_cythonpackage-0.0.0-cp38-cp38-win_amd64.whl
test_cythonpackage-0.0.0-cp39-cp39-macosx_10_15_x86_64.whl
test_cythonpackage-0.0.0-cp39-cp39-manylinux_2_12_i686.manylinux_2_5_i686.manylinux1_i686.manylinux2010_i686.whl
test_cythonpackage-0.0.0-cp39-cp39-manylinux_2_12_x86_64.manylinux_2_5_x86_64.manylinux1_x86_64.manylinux2010_x86_64.whl
test_cythonpackage-0.0.0-cp39-cp39-musllinux_1_1_i686.whl
test_cythonpackage-0.0.0-cp39-cp39-musllinux_1_1_x86_64.whl
test_cythonpackage-0.0.0-cp39-cp39-win32.whl
test_cythonpackage-0.0.0-cp39-cp39-win_amd64.whl
test_cythonpackage-0.0.0-pp37-pypy37_pp73-macosx_10_15_x86_64.whl
test_cythonpackage-0.0.0-pp37-pypy37_pp73-manylinux_2_12_i686.manylinux_2_5_i686.manylinux1_i686.manylinux2010_i686.whl
test_cythonpackage-0.0.0-pp37-pypy37_pp73-manylinux_2_12_x86_64.manylinux_2_5_x86_64.manylinux1_x86_64.manylinux2010_x86_64.whl

and add a fallback with a classical version with source code.

test_cythonpackage-0.0.0-py3-none-any.whl

The PIP-0425 explain how the best package are selected.

Using standard PEP-517

Don't forget to add a file __compile__.py with nothing in package to compile.

With classical setuptools,

• in pyproject.toml

[build-system]
requires = ["setuptools>=42", "wheel"]
build-backend = "setuptools.build_meta"

• in setup.py

from setuptools import setup, find_packages

setup(
    setup_requires=['cythonpackage[build]'],
    cythonpackage=True,
    packages=find_packages(),
)

• in the setup.cfg, add other projects information:

[metadata]
name = my_compiled_project
...
and
```shell
$ pip wheel --use-pep517 .

Using Poetry

Poetry propose a new approach to build a wheel, compatible with PEP 517. At this time, the last version (1.1.*) is not compatible with cython and it's not possible to add a plugin with enough features, but you can use it in a more complex approach:

Don't forget to add a file __compile__.py with nothing in package to compiled.

Create a pyproject.toml file with something like this:

[build-system]
build-backend = "poetry.core.masonry.api"
requires = ["cythonpackage[poetry]"]

[tool.poetry]
name = "test"
version = "0.0.0"
description = "Test"
authors = ["Me <[email protected]>"]
packages = [
    { include = "foo" },
    { include = "foo2" },
    { include = "foo3" },
]
exclude = ["**/[!__]*.py"] # Remove source code
build = 'poetry_build.py'

[tool.poetry.dependencies]
python = "^3.7"
cythonpackage = "*"

[build-system]
requires = ["poetry-core>=1.2.0a2"]
build-backend = "poetry.core.masonry.api"

and a file ./poetry_build.py:

import cythonpackage
def build(setup_kw):
    cythonpackage.build_cythonpackage(setup_kw)

Then, you can build the package with:

python -m build -w

or

pip wheel -w dist .

Using setup.py only (obsolete)

Don't forget to add a file __compile__.py with nothing in package to compiled.

In the setup.py, add setup_requires and cythonpackage=True:

from setuptools import setup, find_packages

setup(
    name="myname",
    version="v0.0.0",
    setup_requires=['cythonpackage[build]'],
    install_requires=['cythonpackage'],
    cythonpackage=True,
    packages=find_packages(),
)

Now, you can build your compiled wheel.

python setup.py bdist_wheel

Using PBR (obsolete)

PBR is a library for managing setuptools packaging needs in a consistent manner.

Don't forget to add a file __compile__.py with nothing in package to compiled.

You can use use PBR with CythonPackage in setup.cfg:

[metadata]
name = my_compiled_project
setup_requires=cythonpackage,pbr
cythonpackage=True
pbr=True
...
[options]
install_requires =
    cythonpackage

or setup.py

from setuptools import setup

setup(
    setup_requires=['pbr','cythonpackage[build]'],
    pbr=True,
    cythonpackage=True,
)

Now, you can build your compiled wheel.

python setup.py bdist_wheel

Sample

The project test-cythonpackage propose a tiny exemple to use CythonPackage, and generate all binary version, with a GitHub Action.

Advanced usage

To make this magic, we manipulate some special parameters at different levels. You can remove some manipulation with a dictionary in cythonpackage parameter.

from setuptools import setup, find_packages

setup(
    setup_requires=['pbr','cythonpackage'],
    pbr=True,
    cythonpackage={
        "inject_ext_modules": True,
        "inject_init": True,
        "remove_source": True,
        "compile_py": True,
        "optimize": 1,
        "exclude": []  # List of glob
    },
)

and you can de-activate CythonPackage with the environment variable CYTHONPACKAGE=false

CYTHONPACKAGE=false python setup.py bdist_wheel

to generate a classical version, without compilation and with python source code.

inject_ext_modules

Detect all packages with __compile__.py, and generate a list of Extension to generate the compiled version of the module __compile__ with all the source code. But with this, it's important to add an extension in sys.meta_path to use only one shared library. It's done at the begining in all __init__.py files.

If you set this parameter to False, you must write yourself the ext_modules parameter

setup(...
      ext_modules=cythonize(
        [
            Extension(
                    name=f"foo.__compile__",
                    sources=['foo/*.py']
                )
        ],
        build_dir="build/cythonpackage",
        compiler_directives={'language_level': 3}
      )
)

inject_init

By default, during the build process, all the __init__.py file for the package with __compile__.py are patched on the fly, to inject two line:

import cythonpackage
cythonpackage.init(__name__)

If this manipulation break something, set this parameter to False and add yourself these two lines at THE BEGINNING of the file, before others import.

remove_source

Because the shared library is enough to use the package, the source code is not necessary. And, sometime, it's possible to have a confusion between the compiled version and the interpreted version. To be sure to use the compiled version, we remove all the source code of the compiled files.

If you set this parameter to False, the source code were inside the wheel.

compile_py / optimize

The objectif of this kind of build, it to optimize the usage of the package. Normally, it's possible to compile the source code to python-byte-code with python setup.py build_py --compile. But the bdist_wheel can not receive the --compile parameter.

With CythonPackage, by default, the compile is set to True, and the optimize is set to 1 to remove the assertions.

You can change these parameters.

exclude

Sometime, a specific python source code can not be compiled with Cython. You can exclude this file with a list of glob pattern.

setup(
    setup_requires=['pbr','cythonpackage'],
    pbr=True,
    cythonpackage={
        "exclude": ["foo/*special.py"]  # List of glob
    },
    install_requires = ["cythonpackage"]
)