Skip to content

Commit

Permalink
Introduce dependency test suite
Browse files Browse the repository at this point in the history
The plan is to build out this test suite to test against the AWS
CLI's dependencies to help facilitate dependency upgrades.

To start, this test suite contains the following new test cases
to better monitor the overall dependency closure of the awscli
package:

* Assert expected packages in runtime closure. This will alert
  us if a dependency introduces a new transitive depenency
  to the AWS CLI closure.

* Assert expected unbounded dependencies in runtime closure.
  Specifically these are dependencies that do not have a version
  ceiling. This will alert us if a new unbounded dependency is
  introduced into the AWS CLI runtime dependency closure.

* Assert expected packages used in building the awscli package
  and any of its runtime dependencies. Specifically, this
  targets the use case of a user installing the awscli
  (and all of its dependencies) from sdist as it will capture
  which build packages are pulled into the isolated virtual
  environment when building the package.

* Assert expected unbounded dependencies in building the awscli
  package and any of its runtime dependencies. This helps monitor
  out of all of the build packages used, which ones may have no
  version ceiling when installed into an isolated virutal environment

See additional implementation notes below:

* Why it is a separate test directory
  * Do not block builds
* Note how to figure it out for runtime dependencies

* Note how to figure out for build dependencies
  • Loading branch information
kyleknap committed Jan 29, 2024
1 parent a0d2a4f commit c0ec804
Show file tree
Hide file tree
Showing 6 changed files with 506 additions and 8 deletions.
27 changes: 27 additions & 0 deletions .github/workflows/run-dep-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: Run dependency tests

on:
push:
pull_request:
branches-ignore: [ master ]

jobs:
build:

runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
os: [ubuntu-latest, macOS-latest, windows-latest]

steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: python scripts/ci/install
- name: Run tests
run: python scripts/ci/run-dep-tests
33 changes: 25 additions & 8 deletions requirements-dev-lock.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@
atomicwrites==1.4.1 \
--hash=sha256:81b2c9071a49367a7f770170e5eec8cb66567cfbbc8c73d20ce5ca4a8d71cf11
# via -r requirements-dev.txt
build==1.0.3 \
--hash=sha256:538aab1b64f9828977f84bc63ae570b060a8ed1be419e7870b8b4fc5e6ea553b \
--hash=sha256:589bf99a67df7c9cf07ec0ac0e5e2ea5d4b37ac63301c4986d1acb126aa83f8f
# via -r requirements-dev.txt
colorama==0.4.4 \
--hash=sha256:5941b2b48a20143d2267e95b1c2a7603ce057ee39fd88e7329b0c292aa16869b \
--hash=sha256:9f47eda37229f68eee03b24b9748937c7dc3868f906e8ba69fbcbdd3bc5dc3e2
Expand Down Expand Up @@ -79,22 +83,29 @@ exceptiongroup==1.1.3 \
--hash=sha256:097acd85d473d75af5bb98e41b61ff7fe35efe6675e4f9370ec6ec5126d160e9 \
--hash=sha256:343280667a4585d195ca1cf9cef84a4e178c4b6cf2274caef9859782b567d5e3
# via pytest
importlib-metadata==7.0.1 \
--hash=sha256:4805911c3a4ec7c3966410053e9ec6a1fecd629117df5adee56dfc9432a1081e \
--hash=sha256:f238736bb06590ae52ac1fab06a3a9ef1d8dce2b7a35b5ab329371d6c8f5d2cc
# via build
iniconfig==1.1.1 \
--hash=sha256:011e24c64b7f47f6ebd835bb12a743f2fbe9a26d4cecaa7f53bc4f35ee9da8b3 \
--hash=sha256:bc3af051d7d14b2ee5ef9969666def0cd1a000e121eaea580d4a313df4b37f32
# via pytest
packaging==21.3 \
--hash=sha256:dd47c42927d89ab911e606518907cc2d3a1f38bbd026385970643f9c5b8ecfeb \
--hash=sha256:ef103e05f519cdc783ae24ea4e2e0f508a9c99b2d4969652eed6a2e1ea5bd522
# via pytest
packaging==23.2 \
--hash=sha256:048fb0e9405036518eaaf48a55953c750c11e1a1b68e0dd1a9d62ed0c092cfc5 \
--hash=sha256:8c491190033a9af7e1d931d0b5dacc2ef47509b34dd0de67ed209b5203fc88c7
# via
# -r requirements-dev.txt
# build
# pytest
pluggy==1.0.0 \
--hash=sha256:4224373bacce55f955a878bf9cfa763c1e360858e330072059e10bad68531159 \
--hash=sha256:74134bbf457f031a36d68416e1509f34bd5ccc019f0bcc952c7b909d06b37bd3
# via pytest
pyparsing==3.0.9 \
--hash=sha256:2b020ecf7d21b687f219b71ecad3631f644a47f01403fa1d1036b0c6416d70fb \
--hash=sha256:5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc
# via packaging
pyproject-hooks==1.0.0 \
--hash=sha256:283c11acd6b928d2f6a7c73fa0d01cb2bdc5f07c57a2eeb6e83d5e56b97976f8 \
--hash=sha256:f271b298b97f5955d53fb12b72c1fb1948c22c1a6b70b315c54cedaca0264ef5
# via build
pytest==7.4.0 \
--hash=sha256:78bf16451a2eb8c7a2ea98e32dc119fd2aa758f1d5d66dbf0a59d69a3969df32 \
--hash=sha256:b4bf8c45bd59934ed84001ad51e11b4ee40d40a1229d2c79f9c592b0a3f6bd8a
Expand All @@ -109,9 +120,15 @@ tomli==2.0.1 \
--hash=sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc \
--hash=sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f
# via
# build
# coverage
# pyproject-hooks
# pytest
wheel==0.38.1 \
--hash=sha256:7a95f9a8dc0924ef318bd55b616112c70903192f524d120acc614f59547a9e1f \
--hash=sha256:ea041edf63f4ccba53ad6e035427997b3bb10ee88a4cd014ae82aeb9eea77bb9
# via -r requirements-dev.txt
zipp==3.17.0 \
--hash=sha256:0e923e726174922dce09c53c59ad483ff7bbb8e572e00c7f7c46b88556409f31 \
--hash=sha256:84e64a1c28cf7e91ed2078bb8cc8c259cb19b76942096c8d7b84947690cabaf0
# via importlib-metadata
4 changes: 4 additions & 0 deletions requirements-dev.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,7 @@ pytest==7.4.0
pytest-cov==4.1.0
atomicwrites>=1.0 # Windows requirement
colorama>0.3.0 # Windows requirement

# Dependency test specific deps
packaging==23.2
build==1.0.3
35 changes: 35 additions & 0 deletions scripts/ci/run-dep-tests
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#!/usr/bin/env python
# Don't run tests from the root repo dir.
# We want to ensure we're importing from the installed
# binary package not from the CWD.

import os
import sys
from contextlib import contextmanager
from subprocess import check_call

_dname = os.path.dirname

REPO_ROOT = _dname(_dname(_dname(os.path.abspath(__file__))))


@contextmanager
def cd(path):
"""Change directory while inside context manager."""
cwd = os.getcwd()
try:
os.chdir(path)
yield
finally:
os.chdir(cwd)


def run(command):
env = os.environ.copy()
env['TESTS_REMOVE_REPO_ROOT_FROM_PATH'] = 'true'
return check_call(command, shell=True, env=env)


if __name__ == "__main__":
with cd(os.path.join(REPO_ROOT, "tests")):
run(f"{sys.executable} {REPO_ROOT}/scripts/ci/run-tests dependencies")
12 changes: 12 additions & 0 deletions tests/dependencies/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
Loading

0 comments on commit c0ec804

Please sign in to comment.