From 4ca0341a8b7bd5a2d6250d16d83d7de002433408 Mon Sep 17 00:00:00 2001 From: Mathieu Lamiot Date: Tue, 3 Sep 2024 18:35:51 +0200 Subject: [PATCH 1/2] add python standards: repo, tests, style --- technical_standards/index.md | 3 +- technical_standards/python/index.md | 7 ++ technical_standards/python/repository.md | 51 +++++++++ technical_standards/python/style.md | 131 +++++++++++++++++++++++ technical_standards/python/test.md | 70 ++++++++++++ 5 files changed, 261 insertions(+), 1 deletion(-) create mode 100644 technical_standards/python/index.md create mode 100644 technical_standards/python/repository.md create mode 100644 technical_standards/python/style.md create mode 100644 technical_standards/python/test.md diff --git a/technical_standards/index.md b/technical_standards/index.md index 8ec6b08..472297f 100644 --- a/technical_standards/index.md +++ b/technical_standards/index.md @@ -1 +1,2 @@ -- [PHP](php/index.md) \ No newline at end of file +- [PHP](php/index.md) +- [Python](python/index.md) \ No newline at end of file diff --git a/technical_standards/python/index.md b/technical_standards/python/index.md new file mode 100644 index 0000000..63e01cd --- /dev/null +++ b/technical_standards/python/index.md @@ -0,0 +1,7 @@ +# Python standards +- [Repository](repository.md) +- [Code style](style.md) +- [Tests](test.md) + +## Django standards +- [] \ No newline at end of file diff --git a/technical_standards/python/repository.md b/technical_standards/python/repository.md new file mode 100644 index 0000000..d546fa4 --- /dev/null +++ b/technical_standards/python/repository.md @@ -0,0 +1,51 @@ +--- +notion_page: https://www.notion.so/wpmedia/Python-Repository-preparation-3e292f3b1ffc4ee3adfdf1aceefa3fe2?pvs=4 +title: Python - Repository preparation +--- + +# Python - Repository preparation + +## Requirements + +Dependencies for a codebase usually differ based on the environment the code is running: linters and test tools are required for development and CI, but not for production for instance. Therefore, we recommend using separate requirement files to ease dependency management and keep the footprint of the app small. + +In the app repository, we use a `requirements/` directory, containing the following files: +- `base.txt`: this file lists the dependencies needed in all contexts; +- `ci.txt`: this file lists the dependencies needed in a CI context, on top of `base.txt`. It usually begins with `-r base.txt`. +- `dev.txt`: this file lists the dependencies needed in a development environment. It usually begins with `-r ci.txt`. +- `prod.txt`: this file lists the dependencies only needed in production and deployed environments. It usually begins with `-r base.txt`. + +Note that those files are related to each others as some requirements include other ones. To achieve this, we use the following syntax as first line of a file, so that it includes all dependencies from the referenced file: + +``` +`-r base.txt` +``` + +To automatically generate those files, you can use: + +``` +mkdir requirements +touch requirements/{base.txt,ci.txt,dev.txt,prod.txt} +``` + +## Configuration file + +As we use various tools to enhance the developer experience, it is easier to gather all configuration within a single `.toml` file. +In the project’s root directory, create a `pyproject.toml` file. This will include all the configuration for third party packages. + +```bash +touch pyproject.toml +``` + +## Setting up a local virtual environment + +As developers often switch from one repository to another, and that the environment usually differ (different dependencies, versions, etc.), it is safer to use a virtual environment per project. This will allow you to install dependencies for this repository only, and maybe use different ones on another repository without interferences. +To create and run a virtual environment, you can use the following commands in the repository: +```bash +# create the virtual env +python3 -m venv .venv +# activate the virtualenv +. .venv/bin/activate +``` + +## Setting up a Docker container \ No newline at end of file diff --git a/technical_standards/python/style.md b/technical_standards/python/style.md new file mode 100644 index 0000000..3facdd1 --- /dev/null +++ b/technical_standards/python/style.md @@ -0,0 +1,131 @@ +--- +notion_page: https://www.notion.so/wpmedia/Python-Code-Style-81237a443b514d8dafb033efe7cdb4f1?pvs=4 +title: Python - Code Style +--- + +# Python - Code Style + +This document summarizes our standard code styling tools for Python codebases. + +Our code style guidelines include linting and formatting. All the following should automatically be checked as part of the CI of all repositories. + +## Linting + +### pylint + +For linting, we use `pylint`. + +#### Install and configure + +To use it, the package must be added to `requirements/ci.txt`. Note that`pylint-django` must also be added for Django projects. To do so, add the following lines to `requirements/ci.txt` and replace the targeted version with the latest one available. + +``` +pylint== +pylint-django== +``` + +Add the following lines `pyproject.toml`. Note that some lines mentioning *django* are only needed for Django projects. + +```toml +[tool.pylint.MAIN] +load-plugins = ["pylint_django"] +ignore = [ + "migrations", + "tests", + "test", + "venv", + ".venv", + "setup.py", + "manage.py", +] +django-settings-module = "repository_name.settings.ci" + +``` + +#### Run + +Now you can run `pylint` and resolve any linting issues you will encounter ; either in a virtual environment, in your docker image or directly in the CI: + +``` +pylint --recursive=y . +``` + +## Formatting + +For code formatting, we use two tools: +- [`black`](https://black.readthedocs.io/en/stable/): This tool is a code formatter. While strongly opinionated, its capacity to automatically format code and to promote small diff codebase changes makes it very useful to work on large projects. +- [`isort`](https://pycqa.github.io/isort/): This tool automatically sorts imports at the beginning of Python files, making dependency management easier and more readable. + +### black + +#### Install and configure + +To use `black`, the package must be installed by adding the following line to `requirements/ci.txt`: + +``` +black== +``` + +Paste the following lines to `pyproject.toml` to configure the tool: + +```toml +[tool.black] +exclude = ''' +/( + \.git + | \.hg + | \.mypy_cache + | \.tox + | \.venv + | venv + | \.local + | \.pytest_cache + | _build + | buck-out + | build + | dist + | migrations +)/ +''' +``` + +#### Run + +Now you can run `black .` and resolve any linting issues you will encounter ; either in a virtual environment or in your docker image. + +To use `black` in a CI, you don't want the format to be enforced but just to report not compliant lines so that you can fix them. To do so, you can use the check option: + +``` +black --check . +``` + +### isort + +#### Install and configure + +To use `isort`, the package must be installed by adding the following line to `requirements/ci.txt`: + +``` +isort== +``` + +Paste the following lines to `pyproject.toml` to configure the tool: + +```toml +[tool.isort] +py_version = 311 +multi_line_output = 3 +include_trailing_comma = true +skip_glob = ["venv/*", "**/migrations/**"] +profile = "black" +``` + +#### Run + +Now you can run `isort .` and resolve any linting issues you will encounter ; either in a virtual environment or in your docker image. + +To use `isort` in a CI, you don't want the format to be enforced but just to report not compliant lines so that you can fix them. To do so, you can use the check option: + +``` +isort --check . +``` \ No newline at end of file diff --git a/technical_standards/python/test.md b/technical_standards/python/test.md new file mode 100644 index 0000000..5c7e164 --- /dev/null +++ b/technical_standards/python/test.md @@ -0,0 +1,70 @@ +--- +notion_page: https://www.notion.so/wpmedia/Python-Tests-b6d5253be84142ee87a2fd120c192ca9?pvs=4 +title: Python - Tests +--- + +# Python - Tests + +For built-in unit and integration tests of our Python projects, we use [`pytest`](https://docs.pytest.org/en/stable/). +Running `pytest` and ensuring all tests are passing must be part of the CI of all repositories. + +## Setting up pytest. + +To start using `pytest`, some packages must be added to `requirements/ci.txt`: `pytest` and `pytest-mock`. Note that`pytest-django` must also be added for Django projects. To do so, add the following lines to `requirements/ci.txt` and replace the targeted version with the latest one available. + +``` +pytest== +pytest-django== +pytest-mock== +``` + +Make sure that in `requirements/dev.txt` the first line is + +``` +-r ci.txt +``` + +For the new packages to be added to your docker image, build the image again with + +``` +docker-compose build +``` + +In your `pyproject.toml` file, add the following lines. + +```toml +[tool.pytest.ini_options] +minversion = "6.0" +DJANGO_SETTINGS_MODULE = "repository_name.settings.ci" +addopts = "-x --strict-markers --ds=repository_name.settings.ci" +python_files = ["test*.py", "test_*.py"] + +``` + +Creates a `tests.py` file and add the following code. + +```python + +def test_dummy(client): + result = 1+1 + assert result == 2 +``` + +Now enter the container and run the tests. + +```bash +docker-compose exec app bash +pytest +``` + +You should see one passing test. + +## Reporting test coverage + +`pytest` can report test coverage after running. This is useful to understand which lines of code are covered with tests. We use this feature as part of our [automated CI](../../ways_of_working/processes/reviews.md). + +To generate an XML coverage report, use the following options when running `pytest`: + +``` +pytest --cov=. --cov-report=xml +``` \ No newline at end of file From 807a81110c970eb3b1a9aafc1329c357c857e3a1 Mon Sep 17 00:00:00 2001 From: Mathieu Lamiot Date: Wed, 4 Sep 2024 10:21:54 +0200 Subject: [PATCH 2/2] adds docker container instructions for Python --- technical_standards/python/index.md | 5 +- technical_standards/python/repository.md | 180 ++++++++++++++++++++++- 2 files changed, 180 insertions(+), 5 deletions(-) diff --git a/technical_standards/python/index.md b/technical_standards/python/index.md index 63e01cd..2f0a873 100644 --- a/technical_standards/python/index.md +++ b/technical_standards/python/index.md @@ -1,7 +1,4 @@ # Python standards - [Repository](repository.md) - [Code style](style.md) -- [Tests](test.md) - -## Django standards -- [] \ No newline at end of file +- [Tests](test.md) \ No newline at end of file diff --git a/technical_standards/python/repository.md b/technical_standards/python/repository.md index d546fa4..b2ced05 100644 --- a/technical_standards/python/repository.md +++ b/technical_standards/python/repository.md @@ -41,6 +41,7 @@ touch pyproject.toml As developers often switch from one repository to another, and that the environment usually differ (different dependencies, versions, etc.), it is safer to use a virtual environment per project. This will allow you to install dependencies for this repository only, and maybe use different ones on another repository without interferences. To create and run a virtual environment, you can use the following commands in the repository: + ```bash # create the virtual env python3 -m venv .venv @@ -48,4 +49,181 @@ python3 -m venv .venv . .venv/bin/activate ``` -## Setting up a Docker container \ No newline at end of file +## Setting up Docker containers + +Being able to run our apps locally is critical to ensure efficient development and debug processes. Ensuring that what happens locally is as close as possible to production is also very important to avoid wasting time on "local only" issues. To achieve this, all our Python projects must be ready to run locally within docker containers. +If you are not familiar with Docker, check out their [Getting started guide](https://docs.docker.com/get-started/). + +Docker containers are also used on our live servers as we deploy them within kubernetes pods. + +To prepare the app to run on Docker, a few files must be added and configured. The following chapter presents the baseline configuration we use at WP Media. Note that this can then evolve based on the needs of the project. + +### Development container + +In the project root, create a *docker* directory, and inside that create an *app* directory. + +```bash +touch docker-compose.yaml +mkdir -p docker/app/ +touch docker/app/dev.Dockerfile +``` + + + +In the `dev.Dockerfile`, paste the following code. + +```docker +FROM python:3.11-bullseye + +RUN set -xe; + +ENV PYTHONDONTWRITEBYTECODE 1 +ENV PYTHONUNBUFFERED 1 + +# The following configuration is only needed for Django apps. +ENV DJANGO_ALLOW_ASYNC_UNSAFE="true" +ENV DJANGO_SETTINGS_MODULE=ticket_ai.settings.dev + +# Create a user named app, to run as that instead of root. +RUN groupadd -r app && useradd --create-home --no-log-init -u 1000 -r -g app app && mkdir /app && chown app:app /app +WORKDIR /app + +RUN mkdir /app/requirements/ +ADD ./requirements/* /app/requirements/ +RUN pip install --no-cache-dir --upgrade pip && pip install --no-cache-dir -r requirements/dev.txt +ADD . /app + +USER app + +EXPOSE 8000/tcp +CMD python3 /app/manage.py migrate --noinput && python3 /app/manage.py runserver 0.0.0.0:8000 +``` + +In the *./docker-compose.yaml* paste the following code. + +```yaml +services: + app: + build: + context: . + dockerfile: docker/app/dev.Dockerfile + ports: + - "8000:8000" + volumes: + - .:/app + +``` + +Run docker-compose and visit http://127.0.0.1:8000. You should see the json response hello world. + +```bash +# to build the image. +docker-compose build + +# to start the app again. +docker-compose up + +``` + +At this point we already have the app running in the container. + +### Production containers + +The previous section describes the local setup. For production needs, we use a different Dockerfile. It typically defines an image with smaller footprint as we don't need debug tools in production, uses an uwsgi production server, and it also points to the production specific requirements. + +```bash +touch docker/app/Dockerfile +``` + +```docker +FROM python:3.11-slim + +RUN set -xe; + +ENV PYTHONDONTWRITEBYTECODE 1 +ENV PYTHONUNBUFFERED 1 + +# The following configuration is only needed for Django apps. +ENV DJANGO_ALLOW_ASYNC_UNSAFE="true" +ENV DJANGO_SETTINGS_MODULE=repository_name.settings.prod + +# Create a user named app, to run as that instead of root. +RUN groupadd -r app && useradd --create-home --no-log-init -u 1000 -r -g app app && mkdir /app && chown app:app /app +WORKDIR /app + +RUN mkdir /app/requirements/ + +COPY ./docker/app/entrypoint.sh /usr/local/bin/ +ADD ./requirements/* /app/requirements/ +RUN pip install --no-cache-dir --upgrade pip && pip install --no-cache-dir -r requirements/prod.txt +ADD . /app + +USER app + +# Add .local/bin to PATH (this is where python bins are put when installed through pip) +ENV PATH="${PATH}:/home/app/.local/bin" + +EXPOSE 8000/tcp +ENTRYPOINT ["tini", "--", "entrypoint.sh"] +CMD ["tini", "--", "uwsgi", "--ini", "uwsgi.ini"] +``` + +In the production configuration, the app is started through a script `entrypoint.sh`. You must create it in `docker/app/` and use the following content: + +``` +#!/bin/sh +set -e + +exec "$@" +``` + +Note that further instructions can be added to this file to be performed when starting the app. For instance, for Django apps, we usually add `python manage.py migrate` before `exec "$@"`. + +#### uwsgi + +For production, we rely on [uWSGI](https://uwsgi-docs.readthedocs.io/en/latest/) as server suite. + +uWSGI must be added to `requirements/prod.txt`: + +``` +`-r base.txt` + +uWSGI== +``` + +Then, a `uwsgi.ini` configuration file must be added at the root of the repository: + +``` +touch uwsgi.ini +``` + +The following file content is a baseline we recommend for our Python apps. Note that this can evolve based on the needs of the app, and that some values there must be configured as environment variables as part of the kubernetes deployment. + +``` +[uwsgi] +strict = true +uid = app +gid = app +env = DJANGO_SETTINGS_MODULE=$(DJANGO_SETTINGS_MODULE) +env = LANG=en_US.UTF-8 +chdir = /app +module = repository_name.wsgi:application +pidfile = /tmp/repository_name-master.pid +master = True +vacuum = True +max-requests = $(UWSGI_MAX_REQUESTS) +processes = $(UWSGI_PROCESSES) +harakiri = $(UWSGI_HARAKIRI) +lazy-apps = true +enable-threads = true +http-socket = :8000 +http-enable-proxy-protocol = 1 +log-x-forwarded-for = true +ignore-sigpipe = true +ignore-write-errors = true +disable-write-exception = true +``` + +The following articles are good references about the use of `uwsgi`, and provide insights about the above configuration: +- https://www.bloomberg.com/company/stories/configuring-uwsgi-production-deployment/ +- https://blog.ionelmc.ro/2022/03/14/how-to-run-uwsgi/ \ No newline at end of file