Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Github Actions build step for multiplatform builds of python C extensions #3981

Closed
wants to merge 55 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
c5e834b
First iteration on GA actions
vitorguidi May 9, 2024
8f7aefc
Dumping gcp steps
vitorguidi May 9, 2024
fd2c45e
CDing into proper file to build deps
vitorguidi May 9, 2024
510e536
Debugging why artifact upload failed
vitorguidi May 9, 2024
bb7106e
Trying to echo into github actions console
vitorguidi May 9, 2024
fd16355
Adding id to build_wheel step so I can reference its outputs
vitorguidi May 9, 2024
9de692b
Adding dummy step to print output variables
vitorguidi May 9, 2024
5611603
Adding $ to GITHUB_OUTPUT to properly write wheel file location
vitorguidi May 9, 2024
19b818f
Trying to get absolute path from find command
vitorguidi May 9, 2024
8ee0394
Tidying workflow up to actually produce the build artifact and upload it
vitorguidi May 9, 2024
d056d59
Building grpcio and grpciotools
vitorguidi May 9, 2024
3c8172e
First experimentation with cibuildwheel for grpcio 1.63.0
vitorguidi May 10, 2024
e358dd7
Fixing conditional run on platform
vitorguidi May 10, 2024
7a30d9c
Fixing absence of if field in unix fetch step
vitorguidi May 10, 2024
1836d91
Fixing tar syntax on unix
vitorguidi May 10, 2024
3cc33e9
Attempting to fix bad path
vitorguidi May 10, 2024
c964651
Fixing typo
vitorguidi May 10, 2024
4519c81
Printing paths for CI debugging
vitorguidi May 10, 2024
85f3cf8
Avoiding short circuit in mkdir
vitorguidi May 10, 2024
bf3f105
Disable fail fast in matrix job so I can at least debug linux
vitorguidi May 10, 2024
887d942
First attempt at splitting builds per platform/arch/python version
vitorguidi May 10, 2024
7981004
Fixed bad reference to python version
vitorguidi May 10, 2024
1873748
Attempting to matrix map
vitorguidi May 10, 2024
3b337d5
Fixing bad references
vitorguidi May 10, 2024
f125d60
Attempting to install from tar.gz to fix windows
vitorguidi May 10, 2024
21c9cb3
Running workflow only for windows, removing cd command from install w…
vitorguidi May 10, 2024
913a1dc
Fixing typo on targz file to be built
vitorguidi May 10, 2024
2bde073
Fixing windows architecture
vitorguidi May 10, 2024
f719d4f
Hardcoding all pypi source files in matrix
vitorguidi May 13, 2024
3b8c556
Removing mac os arm - we do not support it in clusterfuzz
vitorguidi May 13, 2024
cf81787
Skipping musl linux builds
vitorguidi May 13, 2024
838e439
Fetching filename and uri from matrix instead of hardcoding in step
vitorguidi May 13, 2024
da003ca
Fixing typo in CIBW_SKIP
vitorguidi May 13, 2024
c8c4f7d
Attempting to fix invalid GA yaml
vitorguidi May 13, 2024
ca0847a
Trying to fix yet again
vitorguidi May 13, 2024
451baf6
Following the error trail in GA logs
vitorguidi May 13, 2024
aed7cee
Temporarily disabling tests for cheaper iterations
vitorguidi May 13, 2024
e1471ed
Limiting to 256 jobs
vitorguidi May 13, 2024
23159d1
Dropping macos14
vitorguidi May 13, 2024
1b9466c
Ignoring mac until github replies, testing other deps to see who breaks
vitorguidi May 13, 2024
c5d43bf
Running for only relevant libs
vitorguidi May 13, 2024
390188b
Resuming installation of cibuildwheel
vitorguidi May 13, 2024
dc10ab3
Giving a better name to build artifacts
vitorguidi May 13, 2024
4d686a2
Attempting to build proto 4 to see if windows still breaks
vitorguidi May 14, 2024
941d77a
Fixing bad cibuildwheel install step
vitorguidi May 14, 2024
0e389bc
Trying to compile cryptography
vitorguidi May 15, 2024
81bed70
apt must be run as sudo
vitorguidi May 15, 2024
6f57891
Omitting windows setup
vitorguidi May 15, 2024
93be9c2
Attempting to build cryptography with rust
vitorguidi May 15, 2024
8dea175
Installing pre reqs
vitorguidi May 15, 2024
ca9b823
Adding rust toolchain setup step
vitorguidi May 15, 2024
294be81
Upgrading pip prior to rustup
vitorguidi May 15, 2024
ed51539
Attemptign to fix broken pip upgrade command
vitorguidi May 15, 2024
4777f40
Attempting to build deps with gaps
vitorguidi May 21, 2024
2c7ba03
Building gap deps only for python3.11
vitorguidi May 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions .github/workflows/build_python_deps_linux.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
name: Build python deps on linux
on:
push:
branches:
- feature/ga-python-builds

jobs:
build_wheels:
name: ${{ matrix.libraries[1] }}-${{ matrix.wheel_combinations[0] }}-${{ matrix.wheel_combinations[1] }}-py${{ matrix.python_versions[0] }}
runs-on: ${{ matrix.wheel_combinations[0] }}
strategy:
matrix:
libraries: [

["https://files.pythonhosted.org/packages/8c/34/7dafc9052bd9b2b41c5a8912aeeca01e179d16de17e9c275633d4b807330/grpcio-1.38.0.tar.gz", "grpcio-1.38.0.tar.gz"],
["https://files.pythonhosted.org/packages/c4/b8/3512f0e93e0db23a71d82485ba256071ebef99b227351f0f5540f744af41/psutil-5.7.0.tar.gz", "psutil-5.7.0.tar.gz"],
["https://files.pythonhosted.org/packages/00/3a/7a53f54006933efd80dd908cd5b63d6c8b5006d60d1a6563dd90eb866de6/grpcio-tools-1.38.0.tar.gz", "grpcio-tools-1.38.0.tar.gz"],
["https://files.pythonhosted.org/packages/bf/10/ff66fea6d1788c458663a84d88787bae15d45daa16f6b3ef33322a51fc7e/MarkupSafe-2.0.1.tar.gz", "MarkupSafe-2.0.1.tar.gz"],

]
python_versions: [
#["3.7","cp37-*"],
# ["3.8","cp38-*"],
# ["3.9","cp39-*"],
# ["3.10","cp310-*"],
["3.11","cp311-*"],
# ["3.12","cp312-*"],
]
wheel_combinations: [
["ubuntu-latest", "x86_64"],
["windows-2019", "AMD64"],
["macos13", "x86_64"],
# ["macos14", "x86_64"],
]
fail-fast: false

steps:
# Used to host cibuildwheel
- uses: actions/setup-python@v5
with:
python_version: ${{ matrix.python_versions[0] }}

- name: Install python deps
run: |
python -m pip install --upgrade pip
python -m pip install cibuildwheel==2.17.0

- uses: actions-rust-lang/setup-rust-toolchain@v1
if: matrix.wheel_combinations[0] == 'ubuntu-latest'

- name: Setup linux
if: matrix.wheel_combinations[0] == 'ubuntu-latest'
run: |
sudo apt-get install -y libffi-dev

- name: Download and Untar File - Windows
if: matrix.wheel_combinations[0] == 'windows-latest'
run: |
Invoke-WebRequest -Uri ${{ matrix.libraries[0] }} -OutFile ${{ matrix.libraries[1] }}

- name: Download and Untar File - Unix
if: matrix.wheel_combinations[0] != 'windows-latest'
run: |
wget -O ${{ matrix.libraries[1] }} ${{ matrix.libraries[0] }}

- name: Build wheels
run: |
python -m cibuildwheel ${{ matrix.libraries[1] }} --output-dir wheelhouse
# to supply options, put them in 'env', like:
env:
CIBW_ARCHS: ${{ matrix.wheel_combinations[1] }}
CIBW_BUILD: ${{ matrix.python_versions[1] }}
CIBW_SKIP: '*-musllinux_*'


- uses: actions/upload-artifact@v4
with:
name: ${{ matrix.libraries[1] }}-${{ matrix.wheel_combinations[0] }}-${{ matrix.wheel_combinations[1] }}-py${{ matrix.python_versions[0] }}
path: ./wheelhouse/*.whl

60 changes: 30 additions & 30 deletions .github/workflows/tests.yaml
Original file line number Diff line number Diff line change
@@ -1,37 +1,37 @@
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# # Copyright 2020 Google LLC
# #
# # Licensed under the Apache License, Version 2.0 (the "License");
# # you may not use this file except in compliance with the License.
# # You may obtain a copy of the License at
# #
# # http://www.apache.org/licenses/LICENSE-2.0
# #
# # Unless required by applicable law or agreed to in writing, software
# # distributed under the License is distributed on an "AS IS" BASIS,
# # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# # See the License for the specific language governing permissions and
# # limitations under the License.

name: Run basic tests
on: [pull_request]
# name: Run basic tests
# on: [pull_request]

permissions: read-all
# permissions: read-all

jobs:
build:
runs-on: ubuntu-20.04
# jobs:
# build:
# runs-on: ubuntu-20.04

steps:
- uses: actions/checkout@v3
- run: | # Needed for git diff to work.
git fetch origin master --depth 1
git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/master
# steps:
# - uses: actions/checkout@v3
# - run: | # Needed for git diff to work.
# git fetch origin master --depth 1
# git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/master

- name: Setup python environment
uses: actions/setup-python@b55428b1882923874294fa556849718a1d7f2ca5
with:
python-version: 3.7
# - name: Setup python environment
# uses: actions/setup-python@b55428b1882923874294fa556849718a1d7f2ca5
# with:
# python-version: 3.7

- name: Run basic tests
run: ./local/tests/ci_tests.bash
# - name: Run basic tests
# run: ./local/tests/ci_tests.bash

Loading