[Docs] add cloudformation backed guide for s3-sqs setup for filebeats (…

…elastic#40642) * add cloudformation backed guide for s3-sqs setup for filebeats Signed-off-by: Kavindu Dodanduwa <[email protected]> * Update x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc Co-authored-by: kaiyan-sheng <[email protected]> --------- Signed-off-by: Kavindu Dodanduwa <[email protected]> Co-authored-by: kaiyan-sheng <[email protected]>
niraj-elastic · Aug 29, 2024 · 372b126 · 372b126
1 parent 1f77e6e
commit 372b126
Showing 1 changed file with 120 additions and 4 deletions.
diff --git a/x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc b/x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc
@@ -573,10 +573,126 @@ sqs:GetQueueAttributes
 [float]
 === S3 and SQS setup
 
-Enable bucket notification: any new object creation in S3 bucket will also
-create a notification through SQS. Please see
-https://docs.aws.amazon.com/AmazonS3/latest/dev/ways-to-add-notification-config-to-bucket.html#step1-create-sqs-queue-for-notification[create-sqs-queue-for-notification]
-for more details.
+To configure SQS notifications for an existing S3 bucket, you can follow
+https://docs.aws.amazon.com/AmazonS3/latest/dev/ways-to-add-notification-config-to-bucket.html#step1-create-sqs-queue-for-notification[create-sqs-queue-for-notification] guide.
+
+Alternatively, you can follow steps given which utilize a CloudFormation template to create a S3 bucket connected to a SQS with object creation notifications already enabled.
+
+. First copy the CloudFormation template given below to a desired location. For example, to file `awsCloudFormation.yaml`
+
++
+[%collapsible]
+.CloudFormation template
+====
+[source,yaml]
+----
+AWSTemplateFormatVersion: '2010-09-09'
+Description: |
+  Create a S3 bucket connected to a SQS for filebeat validations
+Resources:
+  S3BucketWithSQS:
+    Type: AWS::S3::Bucket
+    Properties:
+      BucketName: !Sub ${AWS::StackName}-s3bucket
+      BucketEncryption:
+        ServerSideEncryptionConfiguration:
+          - ServerSideEncryptionByDefault:
+              SSEAlgorithm: aws:kms
+              KMSMasterKeyID: alias/aws/s3
+      PublicAccessBlockConfiguration:
+        IgnorePublicAcls: true
+        RestrictPublicBuckets: true
+      NotificationConfiguration:
+        QueueConfigurations:
+          - Event: s3:ObjectCreated:*
+            Queue: !GetAtt SQSWithS3BucketConnected.Arn
+    DependsOn:
+      - S3BucketWithSQSToSQSWithS3BucketConnectedPermission
+  S3BucketWithSQSBucketPolicy:
+    Type: AWS::S3::BucketPolicy
+    Properties:
+      Bucket: !Ref S3BucketWithSQS
+      PolicyDocument:
+        Id: RequireEncryptionInTransit
+        Version: '2012-10-17'
+        Statement:
+          - Principal: '*'
+            Action: '*'
+            Effect: Deny
+            Resource:
+              - !GetAtt S3BucketWithSQS.Arn
+              - !Sub ${S3BucketWithSQS.Arn}/*
+            Condition:
+              Bool:
+                aws:SecureTransport: 'false'
+  SQSWithS3BucketConnected:
+    Type: AWS::SQS::Queue
+    Properties:
+      MessageRetentionPeriod: 345600
+  S3BucketWithSQSToSQSWithS3BucketConnectedPermission:
+    Type: AWS::SQS::QueuePolicy
+    Properties:
+      PolicyDocument:
+        Version: '2012-10-17'
+        Statement:
+          - Effect: Allow
+            Principal:
+              Service: s3.amazonaws.com
+            Action: sqs:SendMessage
+            Resource: !GetAtt SQSWithS3BucketConnected.Arn
+            Condition:
+              ArnEquals:
+                aws:SourceArn: !Sub arn:${AWS::Partition}:s3:::${AWS::StackName}-s3bucket
+      Queues:
+        - !Ref SQSWithS3BucketConnected
+Outputs:
+  S3BucketArn:
+    Description: The ARN of the S3 bucket to insert logs
+    Value: !GetAtt S3BucketWithSQS.Arn
+  SQSUrl:
+    Description: The SQS URL to use for filebeat
+    Value: !GetAtt SQSWithS3BucketConnected.QueueUrl
+----
+====
++
+
+. Next, create a CloudFormation stack sourcing the copied.
+
++
+[source,sh]
+----
+aws cloudformation create-stack --stack-name <STACK_NAME> --template-body file://awsCloudFormation.yaml
+----
++
+
+. Then, obtain the S3 bucket ARN and SQS queue url using stack's output
+
++
+For this, you can describe the stack created above. The S3 ARN is set to `S3BucketArn` output and SQS url is set to `SQSUrl` output.
+The output will be populated once the `StackStatus` is set to `CREATE_COMPLETE`.
++
+
++
+[source,sh]
+----
+aws cloudformation describe-stacks --stack-name <STACK_NAME>
+----
++
+
+. Finally, you can configure filebeat to use SQS notifications
+
++
+[source,yaml]
+----
+filebeat.inputs:
+- type: aws-s3
+  queue_url: <URL_FROM_STACK>
+  expand_event_list_from_field: Records
+  credential_profile_name: elastic-beats
+----
++
+
+With this configuration, filebeat avoids polling and utilizes SQS notifications to extract logs from the S3 bucket.
 
 [float]
 === S3 -> SNS -> SQS setup