feat(cli): support CloudFormation simplified resource import #29087

tmokmss · 2024-02-13T07:21:28Z

Issue # (if applicable)

Reason for this change

This feature allows to automatically import exsting resources with the same physical name, such as S3 bucket, DDB table, etc, during a CFn deployment.

Because resource import is a vital feature for CDK users e.g. to refactor a construct tree, cdk migrate, etc, it would benefit many potential users if cdk natively support it.

Description of changes

This PR adds a CLI option --import-exsting-resources: boolean to cdk deploy command and pass it to createChangeSet API call.

Description of how you validated changes

Added a cli integ test.

Checklist

My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

maybe revert later

Co-authored-by: Luca Pizzini <[email protected]>

tmokmss · 2024-04-10T02:50:09Z

@SankyRed Thanks! I guess we also need to add pr-linter/exempt-integ-test label to pass the linter?

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

comcalvi

we probably want this feature on diff changesets as well... @TheRealAmazonKendra what do you think?

comcalvi · 2024-04-23T22:33:13Z

packages/aws-cdk/lib/api/deploy-stack.ts

      ResourcesToImport: this.options.resourcesToImport,
      Description: `CDK Changeset for execution ${this.uuid}`,
      ClientToken: `create${this.uuid}`,
+      ImportExistingResources: importExistingResources,


what happens if they specify both ResourcesToImport and ImportExistingResources?

I guess it will not work, but the spec says nothing about such restriction, so I would like to keep it as-is.

Actually users cannot set both ImportExistingResources and ResourcesToImport from the CDK CLI, because the former is only used in cdk deploy, and cdk deploy does not have any command argument to directly set ResourcesToImport.

The spec isn't really the source of truth we all wish it was so it's not uncommon we enforce logical things the spec does not. However in this specific instance I believe it is correct not to enforce these arguments be exclusive because there is a difference between the resources supported by importExistingResources and ResourcesToImport. Auto-import is a subset of resources supported by ResourcesToImport so it is entirely possible to have both. It is entirely possible then someone could build a command which fails, but I believe this won't really be a common occurence and this is fine to leave as is

rachit-shah · 2024-08-29T18:54:01Z

@comcalvi @tmokmss @TheRealAmazonKendra
We use CDK deployment and face resource conflicts often. We'd like to avoid calling CFN APIs directly since we are using CDK app.

What exactly is this PR blocked on? It has been months since the PR was raised. If it's adding this feature in a different cdk operation (cdk diff), can that be a separate PR/issue?

Pull request has been modified.

tmokmss · 2024-09-17T04:21:13Z

@comcalvi Hi, I resolved the merge conflict. Can you take a look at this once again?

HBobertz · 2024-09-19T20:44:23Z

packages/aws-cdk/lib/api/deploy-stack.ts

      ResourcesToImport: this.options.resourcesToImport,
      Description: `CDK Changeset for execution ${this.uuid}`,
      ClientToken: `create${this.uuid}`,
+      ImportExistingResources: importExistingResources,


The spec isn't really the source of truth we all wish it was so it's not uncommon we enforce logical things the spec does not. However in this specific instance I believe it is correct not to enforce these arguments be exclusive because there is a difference between the resources supported by importExistingResources and ResourcesToImport. Auto-import is a subset of resources supported by ResourcesToImport so it is entirely possible to have both. It is entirely possible then someone could build a command which fails, but I believe this won't really be a common occurence and this is fine to leave as is

HBobertz · 2024-09-19T20:52:51Z

packages/aws-cdk/README.md

+This allows you to automatically import the resources in your template that already
+exist in your AWS account during CloudFormation deployments.
+With this feature, you can reduce the manual effort of import operations and avoid
+deployment failures because of naming conflicts.


Suggested change

This allows you to automatically import the resources in your template that already

exist in your AWS account during CloudFormation deployments.

With this feature, you can reduce the manual effort of import operations and avoid

deployment failures because of naming conflicts.

Automatically import resources in your CDK application which represent

unmanaged resources in your account.

Reduces the manual effort of import operations and avoids

deployment failures due to naming conflicts with unmanaged resources in your account

adding some active voice to be more similar to how our other command readme's are written.

packages/aws-cdk/README.md

packages/aws-cdk/lib/api/deploy-stack.ts

packages/aws-cdk/README.md

comcalvi · 2024-09-19T21:38:30Z

packages/aws-cdk/test/api/deploy-stack.test.ts

@@ -880,6 +880,41 @@ describe('disable rollback', () => {

 });

+describe('import-existing-resources', () => {


Can we get a CLI integ test that creates a single resource and then imports it with this new flag?

@comcalvi
Thanks, but I don't fully agree with implementing such test on CDK side. Because:

Such test would more or less complicate the test code, not noly the success case, but also ensuring there is no orphaned resource when test failed halfway through. This possibly increases maintenance cost of the cli integ test code.

It is not CDK's responsibility to ensure that a resource is successfully imported when the flag is set. It is CFn's responsibility to guarantee the functionality. CDK should just make sure the flag is set properly (as tested in the current code.)

What do you think?

comcalvi · 2024-09-19T21:39:11Z

Existing Resource Import Methods

CDK has 3 ways to import stuff into a CDK stack:

From CFN:
- CfnInclude
From AWS, but not in CFN:
- cdk import
From either:
- cdk migrate

CfnInclude requires you to create a CDK Stack with the same name as the CFN Stack, and imports everything from that CFN stack into CDK. This doesn't rely on any special changeset operation, it's just a regular stack update.

cdk import lets you specify...I think at least one resource ID to import into your stack. This resource is external to CDK / CFN. This executes an IMPORT type changeset. The key is that this type of changeset can only contain import operations; if you do anything else, it will fail.

cdk migrate also uses an import type changeset to import resources, but it also allows you to perform other operations; this is because cdk migrate first performs the import deployment, and then performs a second deployment containing any additional operations.

The New Resource Import Method

CFN launched a feature that allows a changeset to absorb new resources into the stack. This is specified as a flag on the changeset and it effectively turns any "Resource already exists" errors into resource import operations. This means that if your stack attempts to create a resource with the same physical ID as another resource in the same environment, it will import it instead.

It's unclear if this flag works with IMPORT type changesets. Those changesets should not create anything, but autoimport likely needs a CREATE to import the resource...maybe IMPORT changesets recognize that?

That doesn't affect this PR though, because this, conceptually, is a standard deploy option. It's saying "perform a regular deployment, but instead of "create this resource and throw if it doesn't exist", it's "try to create this resource, and import it if it already exists"

Bottom Line

We should add this as a flag to cdk deploy. --import-existing-resources is a good flag name.

Co-authored-by: Calvin Combs <[email protected]>

Pull request has been modified.

Co-authored-by: Hogan Bobertz <[email protected]> Co-authored-by: Calvin Combs <[email protected]>

tmokmss · 2024-09-20T01:03:00Z

@HBobertz @comcalvi

Thanks! I addressed all the comments from you :)

It's unclear if this flag works with IMPORT type changesets. Those changesets should not create anything, but autoimport likely needs a CREATE to import the resource...maybe IMPORT changesets recognize that?

As far as I tested, CFn decides whether a resource is created or imported when we create a changeset. If a resource will be imported, the changeset type for the resource will become IMPORT, and if not, it wil become CREATE.

aws-cdk-automation · 2024-09-20T01:29:02Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: ced4fb1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

tmokmss and others added 23 commits November 19, 2023 23:58

feat(cli): support ImportExstingResources

0f8965f

update aws-sdk, but I'm not sure if it's allowed

e42494e

maybe revert later

Update cli.integtest.ts

ca9cd25

fix integ

e803106

also update aws-sdk

a2fa8b4

Update deployments.ts

d66b937

possible refactor to remove dead code

7e04e54

revert refactor; might do this in another PR

30c1e50

Merge branch 'main' into allow_import_resources

86f6ebc

Merge branch 'main' into allow_import_resources

9229d92

Update yarn.lock

3a2a739

Update README.md

2a6aae5

Update cli.integtest.ts

07fa306

Update README.md

526daed

Update deploy-stack.test.ts

d5f41e6

Update deploy-stack.test.ts

2abaeba

reject only when method=direct && importExistingResources==true

3cc542d

Update packages/aws-cdk/README.md

e812eaa

Co-authored-by: Luca Pizzini <[email protected]>

Update packages/aws-cdk/README.md

6e8813c

Co-authored-by: Luca Pizzini <[email protected]>

Update packages/aws-cdk/README.md

dd75942

Co-authored-by: Luca Pizzini <[email protected]>

Merge branch 'main' into allow_import_resources

8b5ea7e

Merge branch 'main' into allow_import_resources

552e0eb

Merge branch 'main' into allow_import_resources

8803a58

aws-cdk-automation requested a review from a team February 13, 2024 07:21

github-actions bot added effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p1 star-contributor [Pilot] contributed between 25-49 PRs to the CDK labels Feb 13, 2024

mergify bot mentioned this pull request Feb 13, 2024

feat(cli): support CloudFormation simplified resource import #28401

Closed

tmokmss had a problem deploying to test-pipeline February 13, 2024 07:22 — with GitHub Actions Failure

aws-cdk-automation removed pr/needs-maintainer-review This PR needs a review from a Core Team Member labels Apr 9, 2024

SankyRed added the pr-linter/exempt-integ-test The PR linter will not require integ test changes label Apr 10, 2024

aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Apr 10, 2024

comcalvi previously requested changes Apr 23, 2024

View reviewed changes

aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Apr 23, 2024

merge

c8be1f5

tmokmss requested a deployment to test-pipeline September 17, 2024 04:09 — with GitHub Actions Waiting

tmokmss requested a review from comcalvi September 17, 2024 04:18

aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Sep 17, 2024

HBobertz previously requested changes Sep 19, 2024

View reviewed changes

aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Sep 19, 2024

comcalvi previously requested changes Sep 19, 2024

View reviewed changes

Update packages/aws-cdk/README.md

8eb1549

Co-authored-by: Calvin Combs <[email protected]>

tmokmss requested a deployment to test-pipeline September 20, 2024 00:51 — with GitHub Actions Waiting

Apply suggestions from code review

293a61f

Co-authored-by: Hogan Bobertz <[email protected]> Co-authored-by: Calvin Combs <[email protected]>

tmokmss requested a deployment to test-pipeline September 20, 2024 00:53 — with GitHub Actions Waiting

Update README.md

ced4fb1

tmokmss requested a deployment to test-pipeline September 20, 2024 00:59 — with GitHub Actions Waiting

aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Sep 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cli): support CloudFormation simplified resource import #29087

feat(cli): support CloudFormation simplified resource import #29087

tmokmss commented Feb 13, 2024 •

edited

Loading

tmokmss commented Apr 10, 2024 •

edited

Loading

comcalvi left a comment

comcalvi Apr 23, 2024

tmokmss Apr 24, 2024

HBobertz Sep 19, 2024 •

edited

Loading

rachit-shah commented Aug 29, 2024

tmokmss commented Sep 17, 2024

HBobertz Sep 19, 2024 •

edited

Loading

HBobertz Sep 19, 2024

HBobertz Sep 19, 2024

comcalvi Sep 19, 2024

tmokmss Sep 20, 2024 •

edited

Loading

comcalvi commented Sep 19, 2024

tmokmss commented Sep 20, 2024

aws-cdk-automation commented Sep 20, 2024

		@@ -880,6 +880,41 @@ describe('disable rollback', () => {

		});

		describe('import-existing-resources', () => {

feat(cli): support CloudFormation simplified resource import #29087

Are you sure you want to change the base?

feat(cli): support CloudFormation simplified resource import #29087

Conversation

tmokmss commented Feb 13, 2024 • edited Loading

Issue # (if applicable)

Reason for this change

Description of changes

Description of how you validated changes

Checklist

tmokmss commented Apr 10, 2024 • edited Loading

comcalvi left a comment

Choose a reason for hiding this comment

comcalvi Apr 23, 2024

Choose a reason for hiding this comment

tmokmss Apr 24, 2024

Choose a reason for hiding this comment

HBobertz Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

rachit-shah commented Aug 29, 2024

tmokmss commented Sep 17, 2024

HBobertz Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

HBobertz Sep 19, 2024

Choose a reason for hiding this comment

HBobertz Sep 19, 2024

Choose a reason for hiding this comment

comcalvi Sep 19, 2024

Choose a reason for hiding this comment

tmokmss Sep 20, 2024 • edited Loading

Choose a reason for hiding this comment

comcalvi commented Sep 19, 2024

Existing Resource Import Methods

The New Resource Import Method

Bottom Line

tmokmss commented Sep 20, 2024

aws-cdk-automation commented Sep 20, 2024

AWS CodeBuild CI Report

tmokmss commented Feb 13, 2024 •

edited

Loading

tmokmss commented Apr 10, 2024 •

edited

Loading

HBobertz Sep 19, 2024 •

edited

Loading

HBobertz Sep 19, 2024 •

edited

Loading

tmokmss Sep 20, 2024 •

edited

Loading