Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(DockerImage): allow mounting existing or volume copy volumes #32832

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

cgatt
Copy link

@cgatt cgatt commented Jan 10, 2025

Issue # (if applicable)

Closes #32805
Closes #20601

Reason for this change

This change is to better support image bundling in atypical (e.g. CI, DinD) environments by allowing volumes to be mounted using the more portable "Volume Copy" method, or to mount an existing volume. This supports use cases such as caching dependencies in CI environments to reduce asset bundling times.

Description of changes

The primary change made was the addition of two new DockerVolume varients (VolumeCopyDockerVolume and ExistingDockerVolume), facilitated by the new DockerVolumeHelper private class. This replaces the old asset-staging private classes AssetBundlingBindMount and AssetBundlingVolumeCopy, reproducing all the behaviours in a more flexible manner.
The decision to move these utility functions to private/bundling.ts was to better encapsulate the docker related behaviours and abstract them away from the asset management.

This currently uses a discriminated union type to clearly show which properties are required for each volume type. This passes linting, but if it's an issue it can be implemented just as an extension to the DockerVolume interface. However, I'm unsure how to best provide a clear developer experience with this approach.

How It Works

The interfaces for DockerRunOptions.volumes and the related BundlingOptions.volumes have been changed to accept VolumeCopyDockerVolume and ExistingDockerVolume as well as the existing DockerVolume type. The BundlingDockerImage.run() function has been modified slightly to instantiate a DockerVolumeHelper, which accepts the docker run options and uses them to prepare the volumes passed in, creating the volume command strings that run then uses.

If any if the passed volumes are VOLUME_COPY volumes, a copyContainer is spun up with each VOLUME_COPY volume atttached anonymously. The input copy is then performed using the same approach from AssetBundlingVolumeCopy. As these volumes are mounted to the primary container via --volumes-from, them being anonymous doesnt matter and makes creation and deletion of the helper more atomic, reducing the possibility of orphaned volumes or containers. Error handling has been added in multiple places to further prevent this happening in the case of a build failure.

BundlingDockerImage.run() makes use of the volume commands from DockerVolumeHelper to complete its image run, then runs DockerVolumeHelper.cleanup() to copy any outputs from the volume copy volumes, and delete the container (using the -v option to remove attach anonymous volumes).

Describe any new or updated permissions being added

N/A - All changes are at bundling time only

Description of how you validated changes

Unit tests were added to cover the new expected volume behaviours, and existing tests updated to match any changes to argument order. Tests were made to cover the new helper class and new error handling cleanup behaviour. No integration tests were made or modified as this is a local/bundling change,

Checklist


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@cgatt cgatt requested a review from a team as a code owner January 10, 2025 08:43
@github-actions github-actions bot added beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK bug This issue is a bug. effort/medium Medium work item – several days of effort p2 labels Jan 10, 2025
@aws-cdk-automation aws-cdk-automation requested a review from a team January 10, 2025 08:44
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

@cgatt
Copy link
Author

cgatt commented Jan 10, 2025

Exemption Request (Integration Test): Unsure what sort of integration test would be beneficial here, as it's a synth time change.

Clarification Request (Readme): Happy to add a writeup, but the current DockerImage volume and mounting behaviour has no docs, so I wasnt sure where it would be appropriate to put it.

@aws-cdk-automation aws-cdk-automation added pr/reviewer-clarification-requested The contributor has requested clarification on feedback, a failing build, or a failing PR Linter run pr-linter/exemption-requested The contributor has requested an exemption to the PR Linter feedback. labels Jan 10, 2025
@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: 06dda4d
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@aws-cdk-automation aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Jan 13, 2025
@aws-cdk-automation
Copy link
Collaborator

The pull request linter fails with the following errors:

❌ Features must contain a change to a README file.
❌ Features must contain a change to an integration test file and the resulting snapshot.

PRs must pass status checks before we can provide a meaningful review.

If you would like to request an exemption from the status checks or clarification on feedback, please leave a comment on this PR containing Exemption Request and/or Clarification Request.

✅ A exemption request has been requested. Please wait for a maintainer's review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK bug This issue is a bug. effort/medium Medium work item – several days of effort p2 pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. pr/reviewer-clarification-requested The contributor has requested clarification on feedback, a failing build, or a failing PR Linter run pr-linter/exemption-requested The contributor has requested an exemption to the PR Linter feedback.
Projects
None yet
3 participants