Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructure la configuration CircleCI #1650

Merged
merged 30 commits into from
Sep 1, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
3e8c175
Sépare le job test-python du build
HAEKADI Aug 18, 2021
12eb901
Sépare le job test-yaml du build
HAEKADI Aug 18, 2021
6cb6661
Regroupe les jobs de linting
HAEKADI Aug 18, 2021
199e071
Ajoute des dépendances au job check_version_and_changelog
HAEKADI Aug 18, 2021
256e197
Corrige typos
HAEKADI Aug 18, 2021
dae2772
Ajoute explication du job test-python
HAEKADI Aug 18, 2021
1f7b8ca
Unify job names
HAEKADI Aug 20, 2021
95b1c16
Sépare le build de l'install
HAEKADI Aug 20, 2021
51b7bb3
Ajoute la dépendance install au build
HAEKADI Aug 20, 2021
510f8b3
Sauvegarde le cache du build
HAEKADI Aug 20, 2021
ab3137d
Nomme le job build
HAEKADI Aug 20, 2021
b6942b9
Ajoute la dépendance test_api au check_version
HAEKADI Aug 20, 2021
6a502af
Améliore la formulation
HAEKADI Aug 20, 2021
f5dfc4d
Sauvegarde le cache après le build
HAEKADI Aug 24, 2021
a5e4b26
Supprime la redondance de l'utilisation de OpenFisca Core
HAEKADI Aug 24, 2021
5c1f067
Supprime le split des tests Python
HAEKADI Aug 24, 2021
2cb7b28
Restructure la config
HAEKADI Aug 24, 2021
f039ea3
Supprime les redondances des dépendances dans deploy
HAEKADI Aug 24, 2021
781d9e1
Change l'ordre des déclarations
HAEKADI Aug 24, 2021
c937ebd
Teste sans le cache du build
HAEKADI Aug 25, 2021
2f60c8c
Teste avec le cache du build
HAEKADI Aug 25, 2021
8c0e1d0
Change la dépendance des tests Python
HAEKADI Aug 25, 2021
54acb66
Ajoute revision cache aux tests Python
HAEKADI Aug 25, 2021
16d6925
Supprime le job install
HAEKADI Aug 25, 2021
623fdbc
Ajoute le job install
HAEKADI Aug 25, 2021
206e9f0
Ajoute au job test_python
HAEKADI Aug 25, 2021
c35aaed
Ajoute deuxième cache pour le build
HAEKADI Aug 25, 2021
346cd07
Change le cache utilisé dans les jobs
HAEKADI Aug 25, 2021
79f5a52
Metà jour le CHANGELOG
HAEKADI Sep 1, 2021
511f6bd
Met à jour le setup
HAEKADI Sep 1, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
124 changes: 85 additions & 39 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
# CircleCI 2.0 configuration file. See <https://circleci.com/docs/2.0/language-python/>.
version: 2
jobs:
build:
parallelism: 2 # Speed up the tests, while letting two containers available for other builds.
install:
docker:
- image: python:3.7

Expand All @@ -21,66 +20,107 @@ jobs:

- run:
name: Install dependencies
command: |
make build
# pip install --editable git+https://github.com/openfisca/openfisca-core.git@BRANCH_NAME#egg=OpenFisca-Core # use a specific branch of OpenFisca-Core
command: make install

# - run:
# name: Use a specific branch of OpenFisca-Core for testing purposes
# command: pip install --editable git+https://github.com/openfisca/openfisca-core.git@BRANCH_NAME#egg=OpenFisca-Core

- save_cache:
key: v1-py3-deps-{{ .Branch }}-{{ checksum "setup.py" }}
paths:
- /tmp/venv/openfisca_france

lint_files:
docker:
- image: python:3.7

steps:
- checkout

- restore_cache:
key: v1-py3-deps-{{ .Branch }}-{{ checksum "setup.py" }}

- run:
name: Activate virtualenv
command: echo "source /tmp/venv/openfisca_france/bin/activate" >> $BASH_ENV

- run: make check-syntax-errors

- run: make check-style

- run:
name: Lint Python files
command: .circleci/lint-changed-python-files.sh

- run:
name: Lint YAML tests
command: .circleci/lint-changed-yaml-tests.sh

build:
docker:
- image: python:3.7

steps:
- checkout

- restore_cache:
key: v1-py3-deps-{{ .Branch }}-{{ checksum "setup.py" }}

- run:
name: Activate virtualenv
command: echo "source /tmp/venv/openfisca_france/bin/activate" >> $BASH_ENV

- run:
name: Build package
command: make build

- save_cache:
key: v2-py3-deps-{{ .Branch }}-{{ checksum "setup.py" }}
paths:
- /tmp/venv/openfisca_france

- save_cache:
key: v1-py3-build-{{ .Revision }}
paths:
- dist

- run:
name: Run tests
command: |
pytest `circleci tests glob "tests/**/[^_]*.py" | circleci tests split`
openfisca test `circleci tests glob "tests/**/*.{yaml,yml}" | circleci tests split`

lint_yaml_files:
test_python: # Python tests are separated from YAML tests, because the latter take longer. This allows Python tests to use any other available container and yield it, once finished, to another job instead of relying only on the two containers assigned to the YAML tests.
docker:
- image: python:3.7

steps:
- checkout

- restore_cache:
key: v1-py3-deps-{{ .Branch }}-{{ checksum "setup.py" }}
key: v2-py3-deps-{{ .Branch }}-{{ checksum "setup.py" }}

- run:
name: Activate virtualenv
command: |
echo "source /tmp/venv/openfisca_france/bin/activate" >> $BASH_ENV
command: echo "source /tmp/venv/openfisca_france/bin/activate" >> $BASH_ENV

- run:
name: Lint files
command: |
.circleci/lint-changed-yaml-tests.sh
name: Run Python tests
command: pytest `circleci tests glob "tests/**/[^_]*.py"`

lint_python_files:
test_yaml:
parallelism: 2 # Speed up the yaml tests, while letting two containers available for other jobs.
docker:
- image: python:3.7

steps:
- checkout

- restore_cache:
key: v1-py3-deps-{{ .Branch }}-{{ checksum "setup.py" }}
key: v2-py3-deps-{{ .Branch }}-{{ checksum "setup.py" }}

- run:
name: Activate virtualenv
command: |
echo "source /tmp/venv/openfisca_france/bin/activate" >> $BASH_ENV
command: echo "source /tmp/venv/openfisca_france/bin/activate" >> $BASH_ENV

- run:
name: Lint files
command: |
make check-syntax-errors check-style
.circleci/lint-changed-python-files.sh
name: Run YAML tests
command: openfisca test `circleci tests glob "tests/**/*.{yaml,yml}" | circleci tests split`

test_api:
docker:
Expand All @@ -90,7 +130,7 @@ jobs:
- checkout

- restore_cache:
key: v1-py3-deps-{{ .Branch }}-{{ checksum "setup.py" }}
key: v2-py3-deps-{{ .Branch }}-{{ checksum "setup.py" }}

- run:
name: Activate virtualenv
Expand All @@ -99,8 +139,7 @@ jobs:

- run:
name: Test the Web API
command: |
.circleci/test-api.sh
command: .circleci/test-api.sh

check_version_and_changelog:
docker:
Expand All @@ -126,7 +165,7 @@ jobs:
- checkout

- restore_cache:
key: v1-py3-deps-{{ .Branch }}-{{ checksum "setup.py" }}
key: v2-py3-deps-{{ .Branch }}-{{ checksum "setup.py" }}

- restore_cache:
key: v1-py3-build-{{ .Revision }}
Expand All @@ -153,24 +192,31 @@ workflows:
version: 2
build_and_deploy:
jobs:
- check_version_and_changelog
- build
- test_api:
- install
- lint_files:
requires:
- install
- build:
requires:
- install
- test_python:
requires:
- build
- lint_yaml_files:
- test_yaml:
requires:
- build
- lint_python_files:
- test_api:
requires:
- build
- check_version_and_changelog:
requires:
- lint_files
- test_python
- test_yaml
- test_api
- deploy:
requires:
- check_version_and_changelog
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this downwards to align dependencies order with reading order.

I would be at ease with minimising the number of dependencies, knowing they are transitive: if check_version_and_changelog passes and it requires all the other jobs, we maybe don't need to list all of them explicitly. Do whatever feels safer to you though 🙂

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Je trouve aussi que c'est redondant. Je les avais gardées pour des raisons de lisibilité du fichier de config. Mais vu que tu penses aussi qu'il y a une redondance, je les ai supprimées.

Copy link
Contributor Author

@HAEKADI HAEKADI Aug 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Donc pour le deploy, je ne garde que le check_version_and_changelog 🤔

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

J'ai l'impression… ça fait un peu peur, non ? 😅

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Un peu 😬 Mais le DAG montre clairement les dépendances, donc ça ira je pense 😅

- build
- test_api
- lint_yaml_files
- lint_python_files
filters:
branches:
only: master
4 changes: 2 additions & 2 deletions .circleci/lint-changed-python-files.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ last_tagged_commit=`git describe --tags --abbrev=0 --first-parent` # --first-pa

if ! changes=$(git diff-index --name-only --diff-filter=ACMR --exit-code $last_tagged_commit -- "*.py")
then
echo "Hello boss, I'm linting the following changed files:"
echo "Linting the following Python files:"
echo $changes
flake8 $changes
else echo "Could't find changed files, come visit again!"
else echo "No changed Python files to lint"
fi
4 changes: 2 additions & 2 deletions .circleci/lint-changed-yaml-tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ last_tagged_commit=`git describe --tags --abbrev=0 --first-parent` # --first-pa

if ! changes=$(git diff-index --name-only --diff-filter=ACMR --exit-code $last_tagged_commit -- "tests/*.yaml")
then
echo "Hello boss, I'm linting the following changed files:"
echo "Linting the following changed YAML tests:"
echo $changes
yamllint $changes
else echo "Could't find changed files, come visit again!"
else echo "No changed YAML tests to lint"
fi
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
# Changelog

### 72.1.2 [#1650](https://github.com/openfisca/openfisca-france/pull/1650)

* Amélioration technique.
* Zones impactées : configuration de l'intégration continue.
* Détails :
- Sépare les jobs `test_python` et `test_yaml` du `build`.
- Sépare le job `install` du job `build`.
- Regroupe les deux jobs `lint_python_files` et `lint_yaml_files` dans un seul job.
- Rend le job `check-version-and-changelog` dépendant de tous les autres jobs.

## 72.1.1 [#1657](https://github.com/openfisca/openfisca-france/pull/1657)

* Changement mineur.
Expand Down
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

setup(
name = "OpenFisca-France",
version = "72.1.1",
version = "72.1.2",
author = "OpenFisca Team",
author_email = "[email protected]",
classifiers = [
Expand Down