Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ETL-535] Set up SQS to receive S3 event notification #75

Merged
merged 5 commits into from
Sep 7, 2023
Merged

Conversation

rxu17
Copy link
Contributor

@rxu17 rxu17 commented Sep 5, 2023

Purpose: This PR contains initial work done as part of ETL-532 which led us to move forward with using SQS queue.

This PR only does the following:

  • Sets up a working SQS queue that can receive messages from S3 pre-ETL bucket
  • Updates the S3 event config lambda so that the S3 pre-ETL bucket can send S3 notifications to a SQS queue

@rxu17 rxu17 requested a review from a team as a code owner September 5, 2023 22:00
Copy link
Contributor

@philerooski philerooski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor changes requested.

Description: Arn for the S3 Event Config Lambda Function
Description: Arn for the S3 Event Config Destination

S3ToGlueDestinationType:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add an AllowedValues property to this.

@@ -0,0 +1,13 @@
template:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we want an analogous config file in config/prod?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise for the s3-event-config-lambda

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess when I originally created this PR, I wasn't sure if we were going to push this right away or leave it in dev branch to avoid messing with production. Based on our recent discussion, it seems like we should be okay to also have it in prod so will do.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. It will exist independently of everything else for now. We do modify the behavior of the event config lambda, but we aren't using it right now so it shouldn't matter.

@rxu17 rxu17 temporarily deployed to develop September 6, 2023 23:27 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 6, 2023 23:27 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 6, 2023 23:27 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 6, 2023 23:27 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 6, 2023 23:31 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 6, 2023 23:31 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 6, 2023 23:34 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 6, 2023 23:34 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 6, 2023 23:34 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 6, 2023 23:34 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 6, 2023 23:38 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 6, 2023 23:38 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 6, 2023 23:52 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 6, 2023 23:55 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 7, 2023 00:19 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 7, 2023 00:19 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 7, 2023 00:19 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 7, 2023 00:19 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 7, 2023 00:19 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 7, 2023 00:19 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 7, 2023 00:20 — with GitHub Actions Inactive
@rxu17 rxu17 temporarily deployed to develop September 7, 2023 00:20 — with GitHub Actions Inactive
@rxu17 rxu17 merged commit f729901 into main Sep 7, 2023
14 checks passed
@rxu17 rxu17 deleted the etl-535 branch September 7, 2023 00:43
Copy link
Member

@thomasyu888 thomasyu888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥 Excellent @rxu17 ! Also great review @philerooski !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants