S3 path deprecation in 2020: how will this affect the Serverless Image Handler?

[lithiumoxide]

Hi,

Amazon are deprecating S3 paths to buckets. Instead of the bucket name coming after the S3 domain, the bucket name will become a subdomain of the main S3 domain. (See: https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/)

I use version 3.1 of the solution and have a requirement to continue using it rather than version 4. I see that in version 3.1 the Lambda function uses the "old" path based method, as show by the environment variables TC_AWS_ENDPOINT and TC_AWS_LOADER_BUCKET.

How will these changes to S3 impact the Serverless Image Handler? Can we expect an update to the code to cater for this "new" subdomain-only method of accessing a bucket?

Conor

[nicosh]

@lithiumoxide
Out with the Old
In response to feedback on the original deprecation plan that we announced last week, we are making an important change. Here’s the executive summary:

Original Plan – Support for the path-style model ends on September 30, 2020.

Revised Plan – Support for the path-style model continues for buckets created on or before September 30, 2020. Buckets created after that date must be referenced using the virtual-hosted model.

[knihit]

Hi @lithiumoxide based on the revised plan (https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/), you can still continue using the old style S3 endpoints for v3.1. Please let us know if you have questions in this regard. If not can we close this issue.

[lithiumoxide]

Hi,

Thanks for the updates. We plan on creating new deployments of the solution after the S3 path deprecation has been carried out (we use CloudFormation to set it up).

Does anything need to be changed in the template or code to cater for the changes in S3 after September 2020? My initial thoughts are that, right now, the variables TC_AWS_ENDPOINT and TC_AWS_LOADER_BUCKET are concatenated to form a path which is then used by the Lambda function. Do these need to be altered in any way to place the bucket name before the S3 domain?

Conor

[beomseoklee]

@lithiumoxide, that would be true if you create a new stack after September 30, 2020.
May I know what makes you stay on v3.1 rather than v4?

[lithiumoxide]

Hi @beomseoklee. Our main concern is ongoing support for Thumbor-style requests. The URLs are generated dynamically by code in our CMS system, so the structure of the request URL must be known beforehand. Deviation away from Thumbor-style/version 3.1 filters can potentially end up with significant redevelopment work in our core product and in the website design for sites generated by our CMS.

[Furytron]

@lithiumoxide according to the documentation there is still limited legacy support for Thumbor style URLs. Only files in the root of the S3 bucket will be processed and files in any subfolder will cause an error.

So you would have to ensure all files are uploaded into the S3 bucket root directory and of course all have unique names, so they do not overwrite each other. That could become messy with 1000s of files.

Edit: Just to note that the docs are out of date and it does support paths in S3: #184 (comment)

[lithiumoxide]

Thanks for that @Furytron. Unfortunately, we regularly deal with websites using thousands of images, and sometimes tens of thousands. Due to the nature of our use cases, image filenames are not required to be unique, which poses yet another problem with placing images in the root. It seems that v4 is significantly incompatible with what we require, but v3.1 did exactly what we need.

[alexga12]

Quick question, is there a limit as to how many buckets can be used in the SourceBuckets parameter?

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[github-actions]

This issue was closed because it has been inactive for 7 days since being marked as closing-soon.