Week 8 — Serverless Image Processing

Overview
Pre-Requisites
CDK Pipeline Creation
Create CloudFront Distribution
Avatar Upload Implementation
Issue when creating Serverless Image Process
Dynamically passing user handle to profile
Ruby Initialisation for Application
Code to dynamically get the GitPod name in Ruby
Repo for JWT Token
S3 Bucket Issue
Journal Summary

Overview

Videos for week 8

Week 8 Livestream
Serverless Image Process CDK
Serving Avatars via CloudFront
Implement Users Profile Page
Implement Migrations Backend Endpoint and Profile Form
Implement Avatar Uploading (Part 1)
Fix CORS for API Gateway
Fix CORS Final AWS Lambda Layers
Render Avatar from CloudFront

The aim of this week is to allow users to upload their own profile images via Serverless Image Process. To do so we use the CDK - Cloud Development Kit to create a CDK Pipeline.

CDK Pipelines can automatically build, test, and deploy new versions of our pipeline. CDK Pipelines are self-updating. Once we add application stages or stacks, the pipeline automatically reconfigures itself to deploy them.

We will use the CDK pipeline implemented in JavaScript that will perform the following tasks for us.

Use the sharp package to process an uploaded image and resize it to create a thumbnail
Write an AWS Lambda function
Deploy our Lambda function
Import an existing S3 bucket that contains the source image
Create an S3 bucket that will be used to process the uploaded image
Create a SNS (Simple Notification Service) to process on the PUT function and invoke our Lambda function

To invoke the lambda the following changes need to be made to the application

Implement a file upload function in the frontend
Our PostGres database needs to be updated to include a biography field
Update SQL scripts to retrieve this information when matched to the users cognito ID

Pre-Requisites

The following npm packages installed globally (aws-cdk, aws-cdk-lib, dotenv)
The following npm packages installed for our lambda (sharp, @aws-sdk/client-s3)
S3 Bucket which we will upload our images to assets with the name assets.<domainname>

CDK Pipeline Creation

Install Global Packages

Run the following command to install the required packages

npm i aws-cdk aws-cdk-lib dotenv -g

To automate the installation of these packages and our lambda package (explained in detail later) for our gitpod environment we can add a task by inserting the following section in our .gitpod.yml

  - name: cdk
    before: |
      npm install aws-cdk -g
      npm install aws-cdk-lib -g
      cd thumbing-serverless-cdk
      npm i      
      cp env.example .env

Folder for Pipeline

We will store our Pipeline in a folder called thumbing-serverless-cdk in the root of our repository.

cd /workspace/aws-bootcamp-cruddur-2023
mkdir thumbing-serverless-cdk

Initialise CDK Pipeline

Navigate to the thumbing-serverless-cdk folder and initialise it for typescript.

cdk init app --language typescript

Prepare and define CDK Pipeline environment

Created a S3 bucket named assets.tajarba.com in my AWS account. This will be used to store avatar images, banners for the website
Create the following file .env.example. This will be used by the lamba application to define the source and output buckets
Create lambda function that will be invoked by our CDK stack in aws\lambdas\process-images
Add the following code in the thumbing-serverless-cdk/libthumbing-serverless-cdk-stack.ts

Create Sample .env.example file

cd /workspace/aws-bootcamp-cruddur-2023/thumbing-serverless-cdk
touch .env.example

Sample .env.example file

.env.example

ASSETS_BUCKET_NAME="assets.tajarba.com"
THUMBING_S3_FOLDER_INPUT=""
THUMBING_S3_FOLDER_OUTPUT="avatars"
THUMBING_WEBHOOK_URL="https://api.tajarba.com/webhooks/avatar"
THUMBING_TOPIC_NAME="cruddur-assets"
THUMBING_FUNCTION_PATH="/workspace/aws-bootcamp-cruddur-2023/aws/lambdas/process-images"
UPLOADS_BUCKET_NAME="tajarba-uploaded-avatars"

S3 Bucket for images

assets.<domain_name> e.g. assets.tajarba.com

Create Lambda Function

Create the application in aws\lambdas\process-images

cd /workspace/aws-bootcamp-cruddur-2023/
mkdir -p aws/lambdas/
cd aws/lambdas/process-images
touch index.js s3-image-processing.js test.js
npm init -y
npm install sharp @aws-sdk/client-s3 --save

The code for these files is located in process-images

Optionally there is also an example.json that can be used to test the application using AWS Lambdas test function.

Bootstrap environment

Bootstrapping is the process of provisioning resources for the AWS CDK before you can deploy AWS CDK apps into an AWS environment. (An AWS environment is a combination of an AWS account and Region).

Bootstrap the application using the command. The command assumes that you have set the AWS_ACCOUNT_ID and AWS_DEFAULT_REGIONS correctly.

cdk bootstrap "aws://$AWS_ACCOUNT_ID/$AWS_DEFAULT_REGION"

Synthesise Application

Once bootstrapped you can first generate a CloudFormation template for the application using

cdk synth

Deploy Environment

Deploy the CDK using AWS CloudFormation

cdk deploy

To verify the application has been deployed successfully, run the following command.

cdk ls

Sharp Installation script

To use the sharp package within a lambda function the node_modules directory of the deployment package must include binaries for the Linux x64 platform. Once the npm package has been installed we need to run the following npm command.

cd /workspace/aws-bootcamp-cruddur-2023/thumbing-serverless-cdk
npm install
rm -rf node_modules/sharp
SHARP_IGNORE_GLOBAL_LIBVIPS=1 npm install --arch=x64 --platform=linux --libc=glibc sharp

This process has been automated in the following script. sharp install script

Sharp Documentation

Test Deployed Lambda

Run the bin/avatar/upload script that uploads a file data.jpg to the source directory that the lambda is looking at
Verify that the image has been uploaded to the destination bucket and that it has been resized to 512x512

Verify Original Image was uploaded

Check how it looks, it should be 1920x1080

Confirm that the lambda has placed the file in the s3 bucket

Check that the processed image was resized to 512x512

Create CloudFront Distribution

CloudFront is a CDN (Content Delivery Network) by AWS. We will use it to serve content from our buckets by responding to HTTPS requests. It also allows us granular control over how assets are displayed.

CloudFront Pre-Requisites

Domain name registered to the <domainname> you are using. I am using https://www.tajarba.com and have it registered with IONOS
Domain's name servers registered with Route 53
Certificate registered for <domainname> in the us-east-1 zone in addition to your local region

Certificate Creation

Go to AWS Certificate Manager (ACM)
Click Request Certificate
Select Request a public certificate
In Fully qualified domain name enter <domainname> e.g. tajarba.com
Select Add Another Name to this certificated and add *.tajarba.com
Ensure DNS validation - recommended is selected
Click Request

Cloudfront Distribution Creation

The below settings use assets.tajarba.com as its example. Everything else can be left as default

Option	Value
Origin domain	Choose Amazon S3 bucket `assets.tajarba.com`
Name	Set Automatically when you select the S3 bucket
Origin access	Select Origin access control settings (recommended)
Origin access control	`assets.tajarba.com`
Create a control setting	Select and choose the following Sign requests (recommended),Origin type=S3
Viewer protocol policy	Redirect HTTP to HTTPS
Cache policy and origin request policy (recommended)	Selected
Cache policy	CachingOptimized
Origin request policy	CORS-CustomOrigin
Response headers policy	SimpleCORS
Alternate domain name (CNAME)	`assets.tajarba.com`
Custom SSL certificate	Certificate created for `tajarba.com`

Once the CloudFront distribution has been created, we need to copy it's bucket policy. To copy this go to Origins, select the origin assets.tajarba.com and click Edit. Scroll to Bucket Policy and click Copy Policy

This policy needs to be applied to the bucket assets.tajarba.com under Permissions -> Bucket Policy

Route 53 Record Creation

Go to Route 53
Click Create hosted zone
Domain name -> tajarba.com
Type = Public hosted zone
Click Create Hosted Zone

Enable Invalidation

When uploading a new version of an image until it expires it will keep displaying the old version of the file. To stop this from happening we need to enable invalidation

In Cloudfront select the cloudfront distribution
Select Invalidations
Add the pattern /* and click Create Invalidation
It will take a minute or so for the change to take effect

DB Changes

To display Biographic information about the user we need to add a text column called BIO. Andrew implemented a migration script but I chose to just change my schema.sql and seed.sql for convenience.

Schema.sql

CREATE TABLE public.users (
  uuid UUID DEFAULT uuid_generate_v4() PRIMARY KEY,
  display_name text NOT NULL,
  handle text NOT NULL ,
  email text NOT NULL ,
  cognito_user_id text,
  bio text,
  created_at TIMESTAMP default current_timestamp NOT NULL
);

Seed.sql

INSERT INTO public.users (display_name, handle, email,bio,cognito_user_id)
VALUES
  ('Shehzad Ali', 'shehzad','shehzad@exampro.co','NothingSoFar','MOCK'),
  ('Andrew Bayko', 'bayko','bayko@exampro.co','NothingSoFar','MOCK'),
  ('Andrew Brown', 'andrewbrown','andrew@exampro.co','NothingSoFar','MOCK');

Avatar Upload Implementation

To upload avatars we will utilise a API gateway that will call a lambda function lambda-authorizer which authenticates the current user. This then calls cruddur-upload-avatar which returns a pre-signed URL. This pre-signed URL allows access to Cruddur to upload to the S3 bucket.

Pre-Requisites for Avatar Upload

Create a lambda function to authorise the currently logged in user aws/lambdas/lambda-authorizer
Create a lambda function to upload the image aws/lambdas/cruddur-upload-avatar/
Create an API gateway which invokes the lambda functions

Implement CruddurAvatarUpload

Create the skeleton structure of the application

cd /workspace/aws-bootcamp-cruddur-2023/
mkdir -p aws/lambdas/cruddur-upload-avatar/
cd aws/lambdas/cruddur-upload-avatar/
touch function.rb
bundle init

After running bundle init a Gemfile will have been created. Add the following packages to it ["aws-sdk-s3", "ox", "jwt"] by editing it as below

# frozen_string_literal: true

source "https://rubygems.org"

# gem "rails"
gem "aws-sdk-s3"
gem "ox"
gem "jwt"

Install the required packages bundle install
Update function.rb with this code function.rb
Update the Access-Control-Allow-Origin sections with the URL of the frontend application e.g. "Access-Control-Allow-Origin": "https://3000-shehzadashi-awsbootcamp-sf7toclaf7t.ws-eu96b.gitpod.io"
Verify lambda function works bundle exec ruby function.rb. This should return a pre-signed URL

Implement Lambda-Authoriser

Create the skeleton structure of the application

cd /workspace/aws-bootcamp-cruddur-2023/
mkdir -p aws/lambdas/lambda-authorizer/
cd aws/lambdas/lambda-authorizer/
touch index.js
npm init -y
npm install aws-jwt-verify --save

Update index.js with this code index.js
Download the files in this folder to a zip file as shown below. This file will be uploaded later as a lambda. Downloading these files will ensure all the required packages are available to the lambda function

Create functions in AWS Lambda

CruddurAvatarUpload

Create a Ruby Application named CruddurAvatarUpload
Upload the code from function.rb ensuring it has the correct GitPod frontend URL set in Access-Control-Allow-Origin
Set an environment variable UPLOADS_BUCKET_NAME with tajarba-uploaded-avatars the location where avatars are to be uploaded to
Edit runtime settings to have the handler set as function.handler
Modify the current permissions policy and attach a new inline policy PresignedUrlAvatarPolicy using this S3 Policy

S3 Policy using our bucket name

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::tajarba-uploaded-avatars/*"
        }
    ]
  }

CruddurApiGatewayLambdaAuthorizer

Create a Node.js Application named CruddurApiGatewayLambdaAuthorizer
Upload the zip file of the code we created earlier. If packaged and uploaded correctly it should look like this

Set the environment variables USER_POOL_ID and CLIENT_ID with your Cognito clients USER_POOL_I and AWS_COGNITO_USER_POOL_CLIENT_ID respectively

Update S3 Bucket COR Policy

Under the permissions for tajarba-uploaded-avatars edit Cross-Origin resource sharing (CORS) with this S3 CORS Policy

Create API Gateway

In API Gateway, create a HTTP API with api.<domain_name> e.g. api.tajarba.com
Creat the two routes below

Option	Value
POST	/avatars/key_upload with authoriser `CruddurJWTAuthorizer` which invokes lambda `CruddurApiGatewayLambdaAuthorizer`, and with integration `CruddurAvatarUpload`
OPTIONS	/{proxy+} without authoriser, but with integration `CruddurAvatarUpload`

Routes for API Gateway - Post

Routes for API Gateway - Options

Authorisation for API Gateway

Integration for KeyUpload

Integration for Options

Cors for API Gateway

There should be no CORS configuration

Triggers for CruddurAvatarUpload

Issue when creating Serverless Image Process

When creating the lambda, I face a few errors. Because of costs I had been working off my local machine. This installed sharp but once the lambda had been uploaded it would complain that sharp was missing. I resolved this by switching over to Gitpod

Once this issue had been resolve the lambda would still not trigger. I checked that the code was correct by testing it against sample JSON. This passed so I looked into various aspects in the cloudformation stack. I looked at the roles in cloudformation and saw the following error.

This however was a red herring and wasted time. The reason for the code not working was that there were typos in the .env file. The lambda was looking at

avatar/original

while the image was being uploaded to

avatars/original

Once this was resolved the lambda processed images successfully.

Dynamically passing user handle to profile

This makes the Profile Icon handle dynamic by removing @andrewbrown as a hardcoded URL.

In DesktopNavigation.js the following is hardcoded

profileLink = <DesktopNavigationLink 
      url="/@andrewbrown" 
      name="Profile"
      handle="profile"
      active={props.active} />

During my video grading Andrew mentioned that since the user had been already passed we should be able to access the property of it. A bit of trial and error later the following code works.

profileLink = <DesktopNavigationLink 
      url={"/@" + props.user.handle}
      name="Profile"
      handle="profile"
      active={props.active} />

This shows the @bayko profile when logged in as Bayko

Initially double clicking on this caused an error as it kept appending the username to the end of the URL repeatedly. I managed to resolve this by appending / at the beginning of the URL which I had missed out initially.

I also wrote this up in a Blog post to provide more detail.

https://shehzadashiq.hashnode.dev/aws-cloud-project-bootcamp-dynamic-user-handles

Ruby Initialisation for Application

To initialise ruby in a directory run the following command

bundle init

To install the packages in a ruby bundle run the following command

bundle install

UPLOADS_BUCKET_NAME needs to be configured as a GitPOD environment variable

gp env UPLOADS_BUCKET_NAME="tajarba-uploaded-avatars"

Test Ruby function by running

bundle exec ruby function.rb

Code to dynamically get the GitPod name in Ruby

I had written this up purely as a test.

workspace_id = ENV['GITPOD_WORKSPACE_ID']
workspace_cluster_host = ENV['GITPOD_WORKSPACE_CLUSTER_HOST']

workspace_url = "https://#{workspace_id}.#{workspace_cluster_host}"

puts "Workspace URL: #{workspace_url}"

Repo for JWT Token

Repo documenting usage of JWT Tokens https://github.com/awslabs/aws-jwt-verify

npm install aws-jwt-verify --save

S3 Bucket Issue

Created the bucket according to Andrew's directions. Items in the avatar folder were visible however items in the banners folder were not.

Trying to solve this by creating another bucket and troubleshooting this

This did not resolve the issue so I have placed the banner in the avatars directory. Not the ideal solution but I need a workaround for now.

In this instance we can see that the Avatar image is displaying fine

Banner image placed in the banner folder does not display only a 1x1 pixel is displayed

Verified that the banner.jpg exists in S3 bucket

Verified that all images exist in avatars folder

The same image uploaded to the avatars folder displays without any issue

This has had me scratching my head for days but I could not resolve it. I created another bucket and cloudfront distribution too however the cloudfront distribution could not resolve the domain name so I abandoned testing it with that approach.

I opened a ticket and one of the other bootcampers mentioned that this might be a problem with Ad-Blockers. It turns out that Kaspersky was treating the image as a banner and hiding it. To counteract this I have created a new folder called images and placed the banner in it.

Setting in Kaspersky that caused the issue

CORS Not Working

Following the videos and looking through the discord, I could get CORS working.

To get uploads working I had to disable the CruddurApiGatewayLambdaAuthoriser completely :(

I continued with week 9 and then come back to troubleshoot this issue as it had put me quite far behind in my work.

Journal Summary

Completed all the homework.

Challenges. I managed to remove the hard-coded user profile of andrewbrown and instead use dynamic profiles by changing the profileLink url to

"url={"/@" + props.user.handle}"

Issues

Working from my local environment despite following the instructions to install the sharp and without no errors the lambda would complain that sharp had not been installed. The same issue happened despite using WSL2 and a Linux VM. To resolve this I had to resort to using GitPod.
Lambda would not trigger complaining "Entity does not exist. One of the entities that you specified for the operation does not exist. The role with the name ThumbingServerlessCDKStack-ThumbLambdaServiceRole cannot be found". This however was not the issue. The fault was that I had misconfigured the source path. The lambda was looking at avatar/original while the image was being uploaded to avatars/original.
The initial issue was Kaspersky blocking all banners on the website. One of the boot campers pointed this out to me. To overcome this I had to place the banner in another folder in my s3 bucket and ensure the file was not named banner.jpg
CORS issues which were due to misconfiguration of the following

I had kept copy pasting the URL of the workspace and missed out the port number of the frontend for "Access-Control-Allow-Origin" in cruddur-upload-avatar/function.rb
I had attached an authorisation to the OPTIONS route in the API gateway. This resulted in a 401 error
In cruddur-upload-avatar/function.rb I had a put at the end of the function for debugging. Andrew explained that this would result in the pre-signed URL not being returned

Once these issues were fixed I was able to successfully upload the image based on the currently logged in users id.

Files

week8.md

Latest commit

History