Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor liboqs CI and update Ubuntu images #1909

Merged
merged 41 commits into from
Sep 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
493c570
Add reusable workflow for basic checks
SWilson4 Aug 26, 2024
bfd75aa
Add PR tests workflow that calls the basic checks
SWilson4 Aug 26, 2024
1116783
Run in Focal container (revert later)
SWilson4 Aug 26, 2024
b71d770
Fix missing arg in copy_from_upstream
SWilson4 Aug 27, 2024
524ac16
Fix working directory
SWilson4 Aug 27, 2024
e032685
Debug environment var
SWilson4 Aug 27, 2024
d472ed2
Run PR tests on push for testing purposes
SWilson4 Aug 27, 2024
d37f82f
Run android, ios, windows, and zephyr tests on PR
SWilson4 Aug 27, 2024
b74957c
Add workflow_dispatch triggers for individual PR tests
SWilson4 Aug 27, 2024
958a4c2
Add macos tests to PR runs
SWilson4 Aug 27, 2024
d6379de
Run scorecard in PR tests
SWilson4 Aug 27, 2024
572eba4
Finish refactor and add documentation
SWilson4 Aug 27, 2024
bcbb508
Test out full tests trigger [full tests]
SWilson4 Aug 27, 2024
c2f84c2
Add @SWilson4 as a CODEOWNER for .github/workflows [skip ci]
SWilson4 Aug 27, 2024
93bc112
Empty commit to re-trigger CI for demonstration purposes
SWilson4 Aug 27, 2024
a9ad9ac
Store LIBJADE_ALG_LIST in github repo variable
SWilson4 Aug 28, 2024
912be88
Descriptive headers in CI.md
SWilson4 Aug 28, 2024
fe8d28f
Expand in CONTRIBUTING.md [full tests]
SWilson4 Aug 28, 2024
b44b969
Fix variable quoting [full tests]
SWilson4 Aug 28, 2024
bd158a3
Use concurrency groups to automatically cancel workflows on double-push
SWilson4 Aug 28, 2024
f6ef87f
Test concurrency group feature [full tests]
SWilson4 Aug 28, 2024
8a0db55
Use latest CI image for basic checks
SWilson4 Aug 28, 2024
c898c3a
Add safe directory with absolute path
SWilson4 Aug 29, 2024
c59823d
Fix syntax in update_docs_from_yaml.py
SWilson4 Aug 29, 2024
5ed008b
Run basic checks with strict warnings
SWilson4 Aug 29, 2024
505649d
Move all Focal jobs to Noble
SWilson4 Aug 30, 2024
da1076c
Update image in extended tests and formatting script
SWilson4 Aug 30, 2024
e34813b
Update PLATFORMS.md [full tests]
SWilson4 Aug 30, 2024
d8d511e
Update old compiler version
SWilson4 Aug 30, 2024
9fc90d1
Fix scan-build error
SWilson4 Aug 30, 2024
d353e4c
Add --status-bugs flag to scan-build cmake [full tests]
SWilson4 Aug 30, 2024
5dce311
Don't test Dilithium/ML-DSA for leaks on clang [full tests]
SWilson4 Aug 30, 2024
e1d6195
Test all triggers [full test] [extended test] [trigger downstream]
SWilson4 Aug 30, 2024
2628e77
Inherit secrets from caller workflow [full tests] [extended tests] [t…
SWilson4 Aug 30, 2024
7ba9900
Update suppression files for Falcon and McEliece [extended tests]
SWilson4 Sep 3, 2024
d06d640
Enable all McEliece variants in weekly tests [extended tests]
SWilson4 Sep 3, 2024
3f2a056
Update permissions for workflows that call scorecard
SWilson4 Sep 4, 2024
b029ac2
Update boringssl to GitHub Actions
SWilson4 Sep 4, 2024
3e6b034
REVERT: test out main branch workflow
SWilson4 Sep 4, 2024
9ea3515
Fix fail-fast logic for downstream trigger
SWilson4 Sep 4, 2024
a8d4721
Revert "REVERT: test out main branch workflow"
SWilson4 Sep 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners

* @dstebila
/.circleci @baentsch
/.github/workflows @SWilson4
/scripts/copy_from_upstream @baentsch @bhess @alexrow
/src/common @dstebila
/src/kem/bike @crockeea
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/android.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: android build
permissions:
contents: read

on: [ push, pull_request ]
on: [workflow_call, workflow_dispatch]

jobs:

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/apple.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: apple build
permissions:
contents: read

on: [ push, pull_request ]
on: [workflow_call, workflow_dispatch]

jobs:

Expand Down
75 changes: 75 additions & 0 deletions .github/workflows/basic.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
name: Basic checks

permissions:
contents: read

on: [workflow_call, workflow_dispatch]

jobs:

stylecheck:
name: Check code formatting
runs-on: ubuntu-latest
container: openquantumsafe/ci-ubuntu-latest:latest
steps:
- name: Checkout code
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # pin@v4
- name: Ensure code conventions are upheld
run: python3 -m pytest --verbose tests/test_code_conventions.py
- name: Check that doxygen can parse the documentation
run: mkdir build && ./scripts/run_doxygen.sh $(which doxygen) ./docs/.Doxyfile ./build
- name: Validate CBOM
run: scripts/validate_cbom.sh

upstreamcheck:
name: Check upstream code is properly integrated
runs-on: ubuntu-latest
container: openquantumsafe/ci-ubuntu-latest:latest
steps:
- name: Checkout code
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # pin@v4
- name: Configure
run: |
git config --global user.name "ciuser" && \
git config --global user.email "[email protected]" && \
git config --global --add safe.directory "$PWD" && \
echo "LIBOQS_DIR=$PWD" >> "$GITHUB_ENV"
- name: Verify copy_from_upstream state after copy
working-directory: 'scripts/copy_from_upstream'
run: |
python3 copy_from_upstream.py copy && \
! git status | grep -i modified
- name: Verify copy_from_upstream state after libjade
working-directory: 'scripts/copy_from_upstream'
run: |
python3 copy_from_upstream.py libjade && \
! git status | grep -i modified

buildcheck:
name: Check that code passes a basic build
needs: [ stylecheck, upstreamcheck ]
runs-on: ubuntu-latest
container: openquantumsafe/ci-ubuntu-latest:latest
env:
KEM_NAME: ml_kem_768
SIG_NAME: ml_dsa_65
steps:
- name: Checkout code
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # pin@v4
- name: Configure
run: |
mkdir build && \
cd build && \
cmake -GNinja -DOQS_STRICT_WARNINGS=ON \
-GNinja \
-DOQS_MINIMAL_BUILD="KEM_$KEM_NAME;SIG_$SIG_NAME" \
--warn-uninitialized .. > config.log 2>&1 && \
cat config.log && \
cmake -LA -N .. && \
! (grep -i "uninitialized variable" config.log)
- name: Build code
run: ninja
working-directory: build
- name: Build documentation
run: ninja gen_docs
working-directory: build
111 changes: 31 additions & 80 deletions .github/workflows/commit-to-main.yml
Original file line number Diff line number Diff line change
@@ -1,88 +1,39 @@
name: Trigger basic downstream CI
name: Main branch tests

permissions:
contents: read

on:
push:
branches: [ "main" ]
branches: ['main']

jobs:
trigger-downstream-ci:
strategy:
fail-fast: false
runs-on: ubuntu-latest
steps:
# TODO: missing projects?
- name: Trigger OQS-OpenSSL CI
run: |
curl --silent \
--write-out "\n%{response_code}\n" \
--user ${{ secrets.BUILD_TRIGGER_TOKEN }}: \
--request POST \
--header "Content-Type: application/json" \
--data '{ "branch": "OQS-OpenSSL_1_1_1-stable", "parameters": { "run_downstream_tests": true } }' \
https://circleci.com/api/v2/project/gh/open-quantum-safe/openssl/pipeline | tee curl_out \
&& grep -q "201" curl_out
- name: Trigger OQS-BoringSSL CI
run: |
curl --silent \
--write-out "\n%{response_code}\n" \
--user ${{ secrets.BUILD_TRIGGER_TOKEN }}: \
--request POST \
--header "Content-Type: application/json" \
--data '{ "branch": "master", "parameters": { "run_downstream_tests": true } }' \
https://circleci.com/api/v2/project/gh/open-quantum-safe/boringssl/pipeline | tee curl_out \
&& grep -q "201" curl_out
- name: Trigger OQS-OpenSSH CI
run: |
curl --silent \
--write-out "\n%{response_code}\n" \
--request POST \
--header "Accept: application/vnd.github+json" \
--header "Authorization: Bearer ${{ secrets.OQSBOT_GITHUB_ACTIONS }}" \
--header "X-GitHub-Api-Version: 2022-11-28" \
--data '{"ref":"OQS-v9"}' \
https://api.github.com/repos/open-quantum-safe/openssh/actions/workflows/ubuntu.yaml/dispatches | tee curl_out \
&& grep -q "204" curl_out
- name: Trigger oqs-provider CI
run: |
curl --silent \
--write-out "\n%{response_code}\n" \
--user ${{ secrets.BUILD_TRIGGER_TOKEN }}: \
--request POST \
--header "Content-Type: application/json" \
--data '{ "branch": "main" }' \
https://circleci.com/api/v2/project/gh/open-quantum-safe/oqs-provider/pipeline | tee curl_out \
&& grep -q "201" curl_out
- name: Trigger liboqs-dotnet CI
run: |
curl --silent \
--write-out "\n%{response_code}\n" \
--user ${{ secrets.BUILD_TRIGGER_TOKEN }}: \
--request POST \
--header "Content-Type: application/json" \
--data '{ "branch": "master" }' \
https://circleci.com/api/v2/project/gh/open-quantum-safe/liboqs-dotnet/pipeline | tee curl_out \
&& grep -q "201" curl_out
- name: Trigger liboqs-java CI
run: |
curl --silent \
--write-out "\n%{response_code}\n" \
--user ${{ secrets.BUILD_TRIGGER_TOKEN }}: \
--request POST \
--header "Content-Type: application/json" \
--data '{ "branch": "master" }' \
https://circleci.com/api/v2/project/gh/open-quantum-safe/liboqs-java/pipeline | tee curl_out \
&& grep -q "201" curl_out
- name: Trigger liboqs-python CI
run: |
curl --silent \
--write-out "\n%{response_code}\n" \
--request POST \
--header "Accept: application/vnd.github+json" \
--header "Authorization: Bearer ${{ secrets.OQSBOT_GITHUB_ACTIONS }}" \
--header "X-GitHub-Api-Version: 2022-11-28" \
--data '{"event_type":"liboqs-upstream-trigger"}' \
https://api.github.com/repos/open-quantum-safe/liboqs-python/dispatches | tee curl_out \
&& grep -q "204" curl_out

platform-tests:
uses: ./.github/workflows/platforms.yml

scorecard:
uses: ./.github/workflows/scorecard.yml
secrets: inherit
# complete list of permissions keys as per
# https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/controlling-permissions-for-github_token#defining-access-for-the-github_token-permissions
# accessed September 4, 2024
permissions:
actions: read
attestations: read
checks: read
contents: read
deployments: read
id-token: write
issues: read
discussions: read
packages: read
pages: read
pull-requests: read
repository-projects: read
security-events: write
statuses: read

basic-downstream:
uses: ./.github/workflows/downstream-basic.yml
secrets: inherit
83 changes: 83 additions & 0 deletions .github/workflows/downstream-basic.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
name: Trigger basic downstream CI

permissions:
contents: read

on: [workflow_call, workflow_dispatch]

jobs:

trigger-downstream-ci:
runs-on: ubuntu-latest
steps:
- name: Trigger OQS-BoringSSL CI
if: ${{ !cancelled() }} # run all steps independent of failures
run: |
curl --silent \
--write-out "\n%{response_code}\n" \
--request POST \
--header "Accept: application/vnd.github+json" \
--header "Authorization: Bearer ${{ secrets.OQSBOT_GITHUB_ACTIONS }}" \
--header "X-GitHub-Api-Version: 2022-11-28" \
--data '{"event_type":"liboqs-upstream-trigger"}' \
https://api.github.com/repos/open-quantum-safe/boringssl/dispatches | tee curl_out \
&& grep -q "204" curl_out
- name: Trigger OQS-OpenSSH CI
if: ${{ !cancelled() }} # run all steps independent of failures
run: |
curl --silent \
--write-out "\n%{response_code}\n" \
--request POST \
--header "Accept: application/vnd.github+json" \
--header "Authorization: Bearer ${{ secrets.OQSBOT_GITHUB_ACTIONS }}" \
--header "X-GitHub-Api-Version: 2022-11-28" \
--data '{"ref":"OQS-v9"}' \
https://api.github.com/repos/open-quantum-safe/openssh/actions/workflows/ubuntu.yaml/dispatches | tee curl_out \
&& grep -q "204" curl_out
- name: Trigger oqs-provider CI
if: ${{ !cancelled() }} # run all steps independent of failures
run: |
curl --silent \
--write-out "\n%{response_code}\n" \
--user ${{ secrets.BUILD_TRIGGER_TOKEN }}: \
--request POST \
--header "Content-Type: application/json" \
--data '{ "branch": "main" }' \
https://circleci.com/api/v2/project/gh/open-quantum-safe/oqs-provider/pipeline | tee curl_out \
&& grep -q "201" curl_out
- name: Trigger liboqs-cpp CI
if: ${{ !cancelled() }} # run all steps independent of failures
run: |
curl --silent \
--write-out "\n%{response_code}\n" \
--request POST \
--header "Accept: application/vnd.github+json" \
--header "Authorization: Bearer ${{ secrets.OQSBOT_GITHUB_ACTIONS }}" \
--header "X-GitHub-Api-Version: 2022-11-28" \
--data '{"event_type":"liboqs-upstream-trigger"}' \
https://api.github.com/repos/open-quantum-safe/liboqs-cpp/dispatches | tee curl_out \
&& grep -q "204" curl_out
- name: Trigger liboqs-go CI
if: ${{ !cancelled() }} # run all steps independent of failures
run: |
curl --silent \
--write-out "\n%{response_code}\n" \
--request POST \
--header "Accept: application/vnd.github+json" \
--header "Authorization: Bearer ${{ secrets.OQSBOT_GITHUB_ACTIONS }}" \
--header "X-GitHub-Api-Version: 2022-11-28" \
--data '{"event_type":"liboqs-upstream-trigger"}' \
https://api.github.com/repos/open-quantum-safe/liboqs-go/dispatches | tee curl_out \
&& grep -q "204" curl_out
- name: Trigger liboqs-python CI
if: ${{ !cancelled() }} # run all steps independent of failures
run: |
curl --silent \
--write-out "\n%{response_code}\n" \
--request POST \
--header "Accept: application/vnd.github+json" \
--header "Authorization: Bearer ${{ secrets.OQSBOT_GITHUB_ACTIONS }}" \
--header "X-GitHub-Api-Version: 2022-11-28" \
--data '{"event_type":"liboqs-upstream-trigger"}' \
https://api.github.com/repos/open-quantum-safe/liboqs-python/dispatches | tee curl_out \
&& grep -q "204" curl_out
Original file line number Diff line number Diff line change
@@ -1,23 +1,17 @@
name: Release tests
name: Downstream release tests

permissions:
contents: read

on: [workflow_call, workflow_dispatch]

# Trigger oqs-provider release tests.
# Runs whenever a release is published, or when a commit message ends with "[trigger downstream]"
# When triggered by a release, the liboqs release tag and the provider "<release tag>-tracker" branch are used.
# When triggered by a commit message, the triggering liboqs branch and the provider "<liboqs branch>-tracker" branch are used.
# When triggered by a release (see release.yml), the liboqs release tag and the provider "<release tag>-tracker" branch are used.
# When triggered by a commit message (see filter.yml), the triggering liboqs branch and the provider "<liboqs branch>-tracker" branch are used.
# If the tracker branch does not exist, the downstream pipeline should detect it and run on the main branch instead.

on:
push:
workflow_dispatch:
release:
types: [ published ]

jobs:
oqs-provider-release-test:
if: github.event_name == 'release' || contains( github.event.head_commit.message, '[trigger downstream]' )
runs-on: ubuntu-latest
steps:
- name: Checkout release tests script
Expand Down
Loading
Loading