feat(s3): replicating objects

[badmintoncryer]

Issue # (if applicable)

Closes #1680.

Reason for this change

AWS S3 supports configuring object replication , but the s3.Bucket construct does not support it.

Description of changes

Added replicationRules to BucketProps.

Replication configuration version

There are two versions of replication configuration. This PR uses only the V2 replication configuration to enable the specification of the Filter element and S3 Replication Time Control (S3 RTC).

To use V2 replication configuration, this PR explicitly specifies Filter.Prefix property.

        const prefix = rule.prefixFilter ?? '';
        const filter = isAndFilter ? {
          and: {
            prefix,
            tagFilters: rule.tagFilter,
          },
        } : {
          prefix,
        };

V2 replication configuration has some restriction:

• Must specify DeleteMarkerReplication

ReplicationStack | 4/7 | 9:22:08 PM | CREATE_FAILED        | AWS::S3::Bucket  | SourceBucket (SourceBucketDDD2130A) Resource handler returned message:
Delete marker replication is not supported if any Tag filter is specified. Please refer to S3 Developer Guide for more information. (Service: S3, Status Code: 400, Request ID: XXX, Extended Request ID: XXX)

• Must specify Priority

ReplicationStack | 4/7 | 9:12:08 PM | CREATE_FAILED        | AWS::S3::Bucket  | SourceBucket (SourceBucketDDD2130A) Resource handler returned message:
Priority must be specified for this version of Cross Region Replication configuration schema. Please refer to S3 Developer Guide for more information. (Service: S3, Status Code: 400, Request ID: XXX, Extended Request ID: XXX)

These restriction is not documented but there are some posts about these points.

• https://repost.aws/questions/QUiEc8wFE_Q16fX5WG-YWnrA/cloudformation-support-for-s3-replication-to-multiple-destination-buckets

To resolve these problems,I made the priority required and explicitly set the deleteMarkerReplication.

       const prefix = rule.prefixFilter ?? ''; // set empty string to use V2 replication configuration
        const filter = isAndFilter ? {
          and: {
            prefix,
            tagFilters: rule.tagFilter,
          },
        } : {
          prefix,
        };

        return {
          id: rule.id,
          priority: rule.priority,
          status: 'Enabled',
          destination: {
            bucket: rule.destination.bucket.bucketArn,
            account: rule.destination.account,
            storageClass: rule.storageClass?.toString(),
            accessControlTranslation: rule.destination.accessControlTransition ? {
              owner: 'Destination',
            } : undefined,
            encryptionConfiguration: rule.kmsKey ? {
              replicaKmsKeyId: rule.kmsKey.keyArn,
            } : undefined,
            replicationTime: rule.replicationTimeControl !== undefined ? {
              status: rule.replicationTimeControl ? 'Enabled' : 'Disabled',
              time: {
                minutes: 15,
              },
            } : undefined,
            metrics: rule.replicationTimeControlMetrics !== undefined ? {
              status: rule.replicationTimeControlMetrics ? 'Enabled' : 'Disabled',
              eventThreshold: {
                minutes: 15,
              },
            } : undefined,
          },
          filter,
          // To avoid deploy error when there are multiple replication rules with undefined deleteMarkerReplication,
          // CDK explicitly set the deleteMarkerReplication if it is undefined.
          deleteMarkerReplication: {
            status: rule.deleteMarkerReplication ? 'Enabled' : 'Disabled',
          },
          sourceSelectionCriteria,
        };

IAM permission

There is a documentation to setup IAM permissions for service role.

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "s3:GetReplicationConfiguration",
            "s3:ListBucket"
         ],
         "Resource":[
            "arn:aws:s3:::SRC-BUCKET"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:GetObjectVersionForReplication",
            "s3:GetObjectVersionAcl",
            "s3:GetObjectVersionTagging"
         ],
         "Resource":[
            "arn:aws:s3:::SRC-BUCKET/*"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:ReplicateObject",
            "s3:ReplicateDelete",
            "s3:ReplicateTags"
         ],
         "Resource":"arn:aws:s3:::DST-BUCKET/*"
      }
   ]
}

However, there are discrepancies between the automatically generated IAM policies in the management console and the IAM policies in the documentation.

Generated Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetReplicationConfiguration",
                "s3:GetObjectVersionForReplication",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionTagging",
                "s3:GetObjectRetention",
                "s3:GetObjectLegalHold"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::SRC-BUCKET",
                "arn:aws:s3:::SRC-BUCKET/*"
            ]
        },
        {
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags",
                "s3:GetObjectVersionTagging",
                "s3:ObjectOwnerOverrideToBucketOwner"
            ],
            "Effect": "Allow",
            "Condition": {
                "StringLikeIfExists": {
                    "s3:x-amz-server-side-encryption": [
                        "aws:kms",
                        "aws:kms:dsse",
                        "AES256"
                    ]
                }
            },
            "Resource": [
                "arn:aws:s3:::DST-BUCKET/*"
            ]
        },
        {
            "Action": [
                "kms:Decrypt"
            ],
            "Effect": "Allow",
            "Condition": {
                "StringLike": {
                    "kms:ViaService": "s3.ap-northeast-1.amazonaws.com",
                    "kms:EncryptionContext:aws:s3:arn": [
                        "arn:aws:s3:::SRC-BUCKET/*"
                    ]
                }
            },
            "Resource": [
                "arn:aws:kms:ap-northeast-1:123456789012:key/hogehuga"
            ]
        },
        {
            "Action": [
                "kms:Encrypt"
            ],
            "Effect": "Allow",
            "Condition": {
                "StringLike": {
                    "kms:ViaService": [
                        "s3.ap-northeast-1.amazonaws.com"
                    ],
                    "kms:EncryptionContext:aws:s3:arn": [
                        "arn:aws:s3:::DST-BUCKET*"
                    ]
                }
            },
            "Resource": [
                "arn:aws:kms:ap-northeast-1:123456789012:key/hogefuga"
            ]
        }
    ]
}

I adopted the policy from the document. I look forward to hearing your thoughts on this matter.

Description of how you validated changes

Added both unit and integ tests.

Checklist

• My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[aws-cdk-automation]

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

[shikha372]

Hi @badmintoncryer , Thank you for submitting this PR, as this feature is adding some additional pemissions for s3, we'll need to have an internal security review while we go through this PR. Will keep you posted !!

[badmintoncryer]

I considered bidirectional replication. First, I added an addReplicationRules() method to allow the destination bucket to be set lazily.

  public addReplicationRules(rules: ReplicationRule[]): void {
    this.replicationRules.push(...rules);
  }

However, it seems that the circular reference between the source and destination buckets cannot be resolved.

Do you have any suggestions on the implementation?

[shikha372]

I considered bidirectional replication. First, I added an addReplicationRules() method to allow the destination bucket to be set lazily.

  public addReplicationRules(rules: ReplicationRule[]): void {
    this.replicationRules.push(...rules);
  }

However, it seems that the circular reference between the source and destination buckets cannot be resolved.

Do you have any suggestions on the implementation?

Thanks @badmintoncryer , working on adding cross-account integ test will give it a try at my end for bi-directional.

[badmintoncryer]

Thank you very much @shikha372

[codecov]

Codecov Report

Attention: Patch coverage is 14.03509% with 49 lines in your changes missing coverage. Please review.

Project coverage is 66.79%. Comparing base (364ce86) to head (b12f672).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #30966      +/-   ##
==========================================
- Coverage   66.96%   66.79%   -0.17%     
==========================================
  Files         329      329              
  Lines       18663    18719      +56     
  Branches     3258     3273      +15     
==========================================
+ Hits        12497    12504       +7     
- Misses       5839     5887      +48     
- Partials      327      328       +1     

Flag	Coverage Δ
suite.unit	66.79% <14.03%> (-0.17%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
packages/aws-cdk	80.71% <ø> (ø)
packages/aws-cdk-lib/core	82.08% <ø> (ø)

[mergify]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[mergify]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[badmintoncryer]

@Mergifyio update

[mergify]

update

✅ Branch has been successfully updated

[mergify]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[badmintoncryer]

@mergify update

[mergify]

update

✅ Branch has been successfully updated

[aws-cdk-automation]

AWS CodeBuild CI Report

• CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
• Commit ID: 147733f
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[mergify]

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

[shikha372]

Hi @badmintoncryer , tried to merge it yesterday but I think there is unit test coverage missing for new addReplication method, I can get back to it next week

[badmintoncryer]

Thank you very much @shikha372 . Looking forward to the updates!