Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

powershell scripting for s3 #1

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
s3 sdk updated
kessiey committed Jun 9, 2024
commit 98c6358f961597568cb14ab6b6e5254b2e59dc2d
19 changes: 19 additions & 0 deletions s3/sdk/java/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@

# CREATE A NEW MAVEN PROJECT
```sh
mvn archetype:generate \
-DgroupId=com.mycompany.app \
-DartifactId=my-app \
-DarchetypeArtifactId=maven-archetype-quickstart \
-DarchetypeVersion=1.4 \
-DinteractiveMode=false
```

```
mvn -B archetype:generate \
-DarchetypeGroupId=software.amazon.awssdk \
-DarchetypeArtifactId=archetype-lambda -Dservice=s3 -Dregion=US_EAST_1 \
-DarchetypeVersion=2.25.69 \
-DgroupId=com.example.myapp \
-DartifactId=myapp
```
6 changes: 3 additions & 3 deletions s3/sdk/java/myapp/pom.xml
Original file line number Diff line number Diff line change
@@ -13,9 +13,9 @@
<maven.shade.plugin.version>3.2.1</maven.shade.plugin.version>
<maven.compiler.plugin.version>3.6.1</maven.compiler.plugin.version>
<exec-maven-plugin.version>1.6.0</exec-maven-plugin.version>
<aws.java.sdk.version>2.21.29</aws.java.sdk.version>
<aws.java.sdk.version>2.25.69</aws.java.sdk.version>
<aws.lambda.java.version>1.2.3</aws.lambda.java.version>
<junit5.version>5.8.1</junit5.version>
<junit5.version>5.10.1</junit5.version>
</properties>

<dependencyManagement>
@@ -48,7 +48,7 @@

<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>url-connection-client</artifactId>
<artifactId>aws-crt-client</artifactId>
</dependency>

<dependency>
4 changes: 2 additions & 2 deletions s3/sdk/java/myapp/src/main/java/com/example/myapp/App.java
Original file line number Diff line number Diff line change
@@ -2,7 +2,7 @@

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3AsyncClient;

/**
* Lambda function entry point. You can change to use other pojo type or implement
@@ -11,7 +11,7 @@
* @see <a href=https://docs.aws.amazon.com/lambda/latest/dg/java-handler.html>Lambda Java Handler</a> for more information
*/
public class App implements RequestHandler<Object, Object> {
private final S3Client s3Client;
private final S3AsyncClient s3Client;

public App() {
// Initialize the SDK client outside of the handler method so that it can be reused for subsequent invocations.
Original file line number Diff line number Diff line change
@@ -2,9 +2,9 @@
package com.example.myapp;

import software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider;
import software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient;
import software.amazon.awssdk.http.crt.AwsCrtAsyncHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3AsyncClient;

/**
* The module containing all dependencies required by the {@link App}.
@@ -14,13 +14,13 @@ public class DependencyFactory {
private DependencyFactory() {}

/**
* @return an instance of S3Client
* @return an instance of S3AsyncClient
*/
public static S3Client s3Client() {
return S3Client.builder()
public static S3AsyncClient s3Client() {
return S3AsyncClient.builder()
.credentialsProvider(EnvironmentVariableCredentialsProvider.create())
.region(Region.US_EAST_1)
.httpClientBuilder(UrlConnectionHttpClient.builder())
.httpClientBuilder(AwsCrtAsyncHttpClient.builder())
.build();
}
}
Binary file modified s3/sdk/java/myapp/target/classes/com/example/myapp/App.class
Binary file not shown.
Binary file not shown.
43 changes: 29 additions & 14 deletions s3/sdk/ruby/Gemfile.lock
Original file line number Diff line number Diff line change
@@ -2,32 +2,47 @@ GEM
remote: https://rubygems.org/
specs:
aws-eventstream (1.3.0)
aws-partitions (1.855.0)
aws-sdk-core (3.188.0)
aws-eventstream (~> 1, >= 1.0.2)
aws-partitions (1.941.0)
aws-sdk-core (3.197.0)
aws-eventstream (~> 1, >= 1.3.0)
aws-partitions (~> 1, >= 1.651.0)
aws-sigv4 (~> 1.5)
aws-sigv4 (~> 1.8)
jmespath (~> 1, >= 1.6.1)
aws-sdk-kms (1.73.0)
aws-sdk-core (~> 3, >= 3.188.0)
aws-sdk-kms (1.83.0)
aws-sdk-core (~> 3, >= 3.197.0)
aws-sigv4 (~> 1.1)
aws-sdk-s3 (1.139.0)
aws-sdk-core (~> 3, >= 3.188.0)
aws-sdk-s3 (1.152.0)
aws-sdk-core (~> 3, >= 3.197.0)
aws-sdk-kms (~> 1)
aws-sigv4 (~> 1.6)
aws-sigv4 (1.7.0)
aws-sigv4 (~> 1.8)
aws-sigv4 (1.8.0)
aws-eventstream (~> 1, >= 1.0.2)
coderay (1.1.3)
jmespath (1.6.2)
method_source (1.0.0)
nokogiri (1.15.5-x86_64-linux)
method_source (1.1.0)
nokogiri (1.16.5-aarch64-linux)
racc (~> 1.4)
nokogiri (1.16.5-arm-linux)
racc (~> 1.4)
nokogiri (1.16.5-arm64-darwin)
racc (~> 1.4)
nokogiri (1.16.5-x86-linux)
racc (~> 1.4)
nokogiri (1.16.5-x86_64-darwin)
racc (~> 1.4)
nokogiri (1.16.5-x86_64-linux)
racc (~> 1.4)
pry (0.14.2)
coderay (~> 1.1)
method_source (~> 1.0)
racc (1.7.3)
racc (1.8.0)

PLATFORMS
aarch64-linux
arm-linux
arm64-darwin
x86-linux
x86_64-darwin
x86_64-linux

DEPENDENCIES
@@ -36,4 +51,4 @@ DEPENDENCIES
pry

BUNDLED WITH
2.4.20
2.5.11
75 changes: 50 additions & 25 deletions s3/sdk/ruby/s3.rb
Original file line number Diff line number Diff line change
@@ -1,31 +1,48 @@
# Required libraries
require 'aws-sdk-s3' # AWS SDK for S3 operations
require 'pry' # Interactive shell for debugging
require 'pry' # Interactive shell for debugging
require 'securerandom' # For generating secure random strings (UUIDs)

# S3 Bucket configuration
bucket_name = ENV['BUCKET_NAME'] # Fetching bucket name from environment variables
region = 'ca-central-1' # AWS region where the bucket will be created
# Prompt user for S3 bucket name
puts "Enter the bucket name:"
bucket_name = gets.chomp # Get user input and remove trailing newline

# Prompt user for AWS region
puts "Enter the AWS region (e.g., us-west-2):"
region = gets.chomp # Get user input and remove trailing newline

# Check if the bucket name and region are provided
if bucket_name.nil? || bucket_name.empty? || region.nil? || region.empty?
puts "Bucket name and region are required."
exit 1
end

# Initialize AWS S3 client
client = Aws::S3::Client.new # Creating a new instance of the S3 client

# Create S3 bucket
resp = client.create_bucket({
bucket: bucket_name, # Name of the bucket
create_bucket_configuration: {
location_constraint: region # Region for the bucket
}
})
# binding.pry
client = Aws::S3::Client.new(region: region) # Creating a new instance of the S3 client with the specified region

begin
# Create S3 bucket
resp = client.create_bucket({
bucket: bucket_name, # Name of the bucket
create_bucket_configuration: {
location_constraint: region # Region for the bucket
}
})
puts "Bucket '#{bucket_name}' created successfully in region '#{region}'."
rescue Aws::S3::Errors::BucketAlreadyExists
puts "Bucket '#{bucket_name}' already exists."
rescue Aws::S3::Errors::ServiceError => e
puts "Failed to create bucket: #{e.message}"
exit 1
end

# Determine the number of files to create and upload
number_of_files = 1 + rand(6) # Generating a random number between 1 and 6
puts "number_of_files: #{number_of_files}"
puts "Number of files to create and upload: #{number_of_files}"

# Loop to create and upload each file
number_of_files.times.each do |i|
puts "i: #{i}"
number_of_files.times do |i|
puts "Creating and uploading file #{i+1} of #{number_of_files}..."
filename = "file_#{i}.txt" # Generating a filename for each file
output_path = "/tmp/#{filename}"# Specifying the output path for the file

@@ -34,12 +51,20 @@
f.write(SecureRandom.uuid) # Write a random UUID to the file
end

# Open and read the file in binary mode, then upload it to S3
File.open(output_path, 'rb') do |f| # 'rb' mode for reading binary data
client.put_object( # Uploading the file to S3
bucket: bucket_name, # Bucket to upload to
key: filename, # Key (filename) for the object in the bucket
body: f # File content
)
begin
# Open and read the file in binary mode, then upload it to S3
File.open(output_path, 'rb') do |f| # 'rb' mode for reading binary data
client.put_object( # Uploading the file to S3
bucket: bucket_name, # Bucket to upload to
key: filename, # Key (filename) for the object in the bucket
body: f # File content
)
end
puts "File '#{filename}' uploaded successfully."
rescue Aws::S3::Errors::ServiceError => e
puts "Failed to upload file '#{filename}': #{e.message}"
end
end
end

# Uncomment the following line if you want to start an interactive session
# binding.pry