-
Notifications
You must be signed in to change notification settings - Fork 227
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
141 changed files
with
2,647 additions
and
150 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
--- | ||
title: Tutorials | ||
meta_desc: Here is the amazing meta description. It's more than 50 characters but less than 160. | ||
aliases: | ||
- /learn | ||
--- | ||
|
||
Here is a line or two to explain where you are and what you'll find here. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes
File renamed without changes
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,129 @@ | ||
--- | ||
title: Automate AWS Aurora Serverless v2 Deployment with Pulumi | ||
meta_desc: A comprehensive guide to automate file expiration in AWS S3 using Pulumi. | ||
layout: topic | ||
estimated_time: 5 | ||
providers: | ||
- aws | ||
collections: | ||
- serverless | ||
--- | ||
|
||
## Overview | ||
|
||
In this tutorial, you will learn how to automate the deployment of AWS Aurora Serverless v2 clusters using Pulumi. AWS Aurora Serverless v2 provides a cost-effective, scalable solution for running your relational databases without managing database servers. Pulumi allows you to define, deploy, and manage your cloud infrastructure using familiar programming languages. | ||
|
||
## Prerequisites | ||
|
||
Before you begin, ensure you have the following: | ||
|
||
- An AWS account with appropriate permissions to create and manage Aurora Serverless v2 clusters. | ||
- Pulumi CLI installed on your local machine. You can download it from the [Pulumi website](https://www.pulumi.com/docs/get-started/install/). | ||
- Node.js and npm installed. You can download them from the [Node.js website](https://nodejs.org/). | ||
- AWS CLI installed and configured with your AWS credentials. You can download it from the [AWS CLI website](https://aws.amazon.com/cli/). | ||
- Pulumi ESC configured for managing secrets. Follow the setup guide on the [Pulumi ESC documentation](https://www.pulumi.com/docs/esc/). | ||
|
||
## Define the Aurora Serverless v2 Cluster | ||
|
||
1. Open the `index.ts` file in your project directory and add the following code: | ||
|
||
```typescript | ||
import * as pulumi from "@pulumi/pulumi"; | ||
import * as aws from "@pulumi/aws"; | ||
|
||
// Create an RDS subnet group | ||
const subnetGroup = new aws.rds.SubnetGroup("aurora-subnet-group", { | ||
subnetIds: ["subnet-12345678", "subnet-87654321"], // Replace with your subnet IDs | ||
tags: { | ||
Name: "aurora-subnet-group", | ||
}, | ||
}); | ||
|
||
// Create a security group for the Aurora cluster | ||
const securityGroup = new aws.ec2.SecurityGroup("aurora-security-group", { | ||
description: "Allow access to Aurora Serverless v2 cluster", | ||
ingress: [ | ||
{ | ||
protocol: "tcp", | ||
fromPort: 3306, | ||
toPort: 3306, | ||
cidrBlocks: ["0.0.0.0/0"], // Replace with your IP range | ||
}, | ||
], | ||
}); | ||
|
||
// Use Pulumi ESC to manage secrets securely | ||
const config = new pulumi.Config(); | ||
const dbPassword = config.requireSecret("dbPassword"); | ||
|
||
// Create an Aurora Serverless v2 cluster | ||
const cluster = new aws.rds.Cluster("aurora-cluster", { | ||
engine: "aurora-mysql", // or "aurora-postgresql" based on your preference | ||
engineMode: "serverless", | ||
scalingConfiguration: { | ||
autoPause: true, | ||
minCapacity: 2, | ||
maxCapacity: 64, | ||
}, | ||
masterUsername: "admin", | ||
masterPassword: dbPassword, | ||
skipFinalSnapshot: true, | ||
dbSubnetGroupName: subnetGroup.name, | ||
vpcSecurityGroupIds: [securityGroup.id], | ||
backupRetentionPeriod: 7, // Retain backups for 7 days | ||
preferredBackupWindow: "07:00-09:00", // Backup window | ||
preferredMaintenanceWindow: "sun:23:00-mon:01:30", // Maintenance window | ||
// kmsKeyId: "your-kms-key-id", // Optional: Replace with your KMS key ID for encryption | ||
}); | ||
|
||
// Output the cluster endpoint | ||
export const clusterEndpoint = cluster.endpoint; | ||
``` | ||
|
||
This Pulumi TypeScript code creates an RDS subnet group named `aurora-subnet-group`, which includes the specified subnet IDs (`subnet-12345678` and `subnet-87654321`). This subnet group allows the Aurora cluster to span across multiple Availability Zones, ensuring high availability. Additionally, the code creates a security group named `aurora-security-group`, which allows inbound traffic on port 3306 (the default port for MySQL) from any IP address (`0.0.0.0/0`). This security group ensures that only specified traffic can access the Aurora cluster. | ||
|
||
The code then creates an Aurora Serverless v2 cluster named `aurora-cluster`. The cluster is configured to use the MySQL engine (`aurora-mysql`) and the serverless engine mode. It is set up with automatic pause and scaling, with a minimum capacity of 2 ACUs and a maximum capacity of 64 ACUs. | ||
|
||
Instead of using a static password, the tutorial now uses Pulumi ESC to securely manage the database password. This ensures that sensitive information is not hard-coded into your infrastructure code, enhancing security. | ||
|
||
Finally, the code exports the endpoint of the Aurora cluster, allowing you to use it in other parts of your infrastructure or applications. This setup provides a scalable, serverless MySQL database cluster on AWS with essential networking and security configurations, along with automated backups and maintenance settings. | ||
|
||
## Step 3: Configure Secrets with Pulumi ESC | ||
|
||
1. Set the database password using Pulumi ESC: | ||
|
||
```bash | ||
pulumi config set --secret dbPassword your-secure-password | ||
``` | ||
|
||
This command securely stores the database password in Pulumi ESC, ensuring it is not exposed in your code or version control. | ||
|
||
## Step 4: Deploy the Infrastructure | ||
|
||
1. Deploy your Pulumi stack: | ||
|
||
```bash | ||
pulumi up | ||
``` | ||
|
||
Confirm the deployment and wait for the resources to be created. | ||
|
||
## Step 5: Verify the Deployment | ||
|
||
1. Log in to the [AWS Management Console](https://aws.amazon.com/console/). | ||
2. Navigate to the RDS service. | ||
3. Verify that the Aurora Serverless v2 cluster has been created and is in an available state. | ||
|
||
## Conclusion | ||
|
||
In this tutorial, you have successfully automated the deployment of an AWS Aurora Serverless v2 cluster using Pulumi. By leveraging Pulumi ESC, you securely managed sensitive information, enhancing the security of your infrastructure. You can now manage your cloud infrastructure with code, making deployments more consistent and repeatable. | ||
|
||
For more information on Pulumi and AWS Aurora Serverless v2, refer to the [Pulumi documentation](https://www.pulumi.com/docs/) and [AWS Aurora Serverless v2 documentation](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless.html). | ||
|
||
## Learn more about Pulumi | ||
|
||
Pulumi is free, [open source](https://github.com/pulumi/pulumi), and optionally pairs with the [Pulumi Cloud](/docs/pulumi-cloud/) to make managing infrastructure secure, reliable, and hassle-free. | ||
|
||
- Follow the [Getting Started](/docs/get-started/) guide to give Pulumi a try. | ||
|
||
- [Join our community on Slack](https://slack.pulumi.com/) to discuss this guide, and let us know what you think. |
120 changes: 120 additions & 0 deletions
120
content/tutorials/automating-s3-file-expiration/_index.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,120 @@ | ||
--- | ||
title: Automating AWS S3 File Expiration with Pulumi | ||
meta_desc: | | ||
A comprehensive guide to automate file expiration in AWS S3 using Pulumi. | ||
page_title: Automate AWS S3 File Expiration with Pulumi | ||
layout: topic | ||
estimated_time: 5 | ||
providers: | ||
- aws | ||
collections: | ||
- serverless | ||
--- | ||
|
||
In this guide, we'll walk through the process of automating AWS S3 file expiration using Pulumi. Lifecycle rules in AWS S3 allow you to specify actions on objects that meet certain criteria over time, such as transitioning objects to a different storage class or automatically deleting them after a specified period. By following these simple steps in this guide, you'll be able to efficiently manage the lifecycle policies for objects stored in S3 buckets, ensuring that outdated files are automatically expired and removed. | ||
|
||
With Pulumi, we can automate S3 file expiration by creating a Pulumi program that sets up these lifecycle rules. We'll use the aws.s3.BucketLifecycleConfigurationV2 resource, which allows us to define these rules programmatically. | ||
|
||
Here's a step-by-step explanation of what we'll do: | ||
|
||
Define the S3 Bucket: We'll create a new S3 bucket or use an existing one where the files are stored. | ||
Set Up Lifecycle Rules: We'll define lifecycle rules to specify how files should be managed as they age. For example, we can define a rule to delete files after 30 days. | ||
Apply the Configuration: We'll apply the lifecycle configuration to the S3 bucket using Pulumi. | ||
Now, let's write a Pulumi program in TypeScript that creates an S3 bucket with a lifecycle policy to transition objects to Glacier after 90 days. | ||
|
||
```typescript | ||
import * as pulumi from "@pulumi/pulumi"; | ||
import * as aws from "@pulumi/aws"; | ||
|
||
// Create an S3 bucket | ||
const bucket = new aws.s3.Bucket("my-automated-bucket", { | ||
// Bucket settings can be added here | ||
}); | ||
|
||
// Define a lifecycle rule to transition objects to Glacier after 90 days | ||
const bucketLifecyclePolicy = new aws.s3.BucketLifecycleConfigurationV2("my-bucket-lifecycle", { | ||
bucket: bucket.id, | ||
rules: [ | ||
{ | ||
id: "archiveToGlacier", | ||
status: "Enabled", | ||
filter: { | ||
prefix: "documents/", | ||
}, | ||
transitions: [ | ||
{ | ||
days: 90, | ||
storageClass: "GLACIER", | ||
}, | ||
], | ||
}, | ||
], | ||
}); | ||
|
||
// Define a bucket policy to enforce server-side encryption with AWS managed keys (SSE-S3) | ||
const bucketPolicy = new aws.s3.BucketPolicy("my-bucket-policy", { | ||
bucket: bucket.id, | ||
policy: bucket.id.apply(id => JSON.stringify({ | ||
Version: "2012-10-17", | ||
Statement: [ | ||
{ | ||
Sid: "EnforceSSE", | ||
Effect: "Deny", | ||
Principal: "*", | ||
Action: "s3:PutObject", | ||
Resource: `arn:aws:s3:::${id}/*`, | ||
Condition: { | ||
StringNotEquals: { | ||
"s3:x-amz-server-side-encryption": "AES256", | ||
}, | ||
}, | ||
}, | ||
], | ||
})), | ||
}); | ||
|
||
// Export the name of the bucket | ||
export const bucketName = bucket.id; | ||
``` | ||
|
||
In this program, we start by importing the AWS module from Pulumi. We then create an S3 bucket named my-automated-bucket. After that, we define a lifecycle configuration for this bucket. The lifecycle configuration includes a rule named archiveToGlacier, which transitions objects under the documents/ prefix to the Glacier storage class after 90 days. | ||
|
||
The filter property with the prefix sub-property ensures that this rule only applies to objects stored under the documents/ folder. The transitions property inside the rule controls the transition of the objects. The days sub-property specifies the number of days after object creation when the objects should be transitioned to Glacier. | ||
|
||
Finally, we export the bucket name, which can be useful if you want to reference this bucket from other parts of your Pulumi program or from other Pulumi stacks. | ||
|
||
## Verify the configuration of your S3 file expiration | ||
|
||
After deployment, you can verify the lifecycle configuration in the AWS Management Console: | ||
|
||
- Navigate to the S3 service. | ||
- Find and select your newly created bucket. | ||
- Go to the "Management" tab. | ||
- Check the "Lifecycle rules" section to see the applied rules. | ||
|
||
## Wrapping up | ||
|
||
This simple Pulumi program will ensure that any files uploaded to the documents/ folder in your S3 bucket will be automatically transitioned to Glacier after 90 days, helping you manage storage costs and keep your bucket tidy without manual intervention. | ||
|
||
## Additional use cases for S3 automation with Pulumi | ||
|
||
Automating S3 with Pulumi can extend beyond file expiration to address various other needs. Here are some additional use cases: | ||
|
||
- **Automated Data Archiving**: Set up lifecycle policies to automatically transition older data to Glacier for cost-effective long-term archiving. | ||
- **Security Compliance**: Automatically apply and enforce bucket policies to meet security and compliance requirements across all S3 buckets. | ||
- **Disaster Recovery**: Automatically replicate data across different regions to ensure high availability and disaster recovery. | ||
- **Data Processing Workflows**: Trigger Lambda functions to process data as soon as it's uploaded to S3, for use cases like image resizing, data transformation, or machine learning inference. | ||
- **Audit and Monitoring**: Continuously monitor access and changes to S3 objects and generate alerts or reports for audit purposes. | ||
|
||
By leveraging Pulumi with AWS S3, you can automate and streamline various aspects of your AWS S3 management, leading to more efficient, cost-effective, and secure cloud storage operations. | ||
|
||
For more advanced configurations, refer to the [Pulumi AWS documentation](/docs/reference/pkg/aws/s3/bucketlifecycleconfiguration/) and the [AWS S3 Lifecycle Management guide](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html). | ||
|
||
## Want to learn more about Pulumi? | ||
|
||
Pulumi is free, [open source](https://github.com/pulumi/pulumi), and optionally pairs with the [Pulumi Cloud](/docs/pulumi-cloud/) to make managing infrastructure secure, reliable, and hassle-free. | ||
|
||
- Follow the [Getting Started](/docs/get-started/) guide to give Pulumi a try. | ||
|
||
- [Join our community on Slack](https://slack.pulumi.com/) to discuss this guide, and let us know what you think. |
File renamed without changes
File renamed without changes
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes
Oops, something went wrong.