Support Thumbor smart cropping and available detectors

[Furytron]

Thumbor is supported as far as I know through your mapping code: https://github.com/aws-solutions/serverless-image-handler/blob/develop/source/image-handler/thumbor-mapper.ts, however there is no case for "smart".

So would it be possible to implement the Thumbor style smart cropping using the URL path /smart/ ?
Smart cropping documentation: https://thumbor.readthedocs.io/en/latest/usage.html#smart-cropping

It would also be great to allow the enabling of the smart cropping detectors through the URL as well.

Possibly using something like the below:

/smart(face_detector)/
/smart(glasses_detector)/

Thumbor detectors: https://thumbor.readthedocs.io/en/latest/detectors.html

[fisenkodv]

@Furytron thank you for your feedback. We will look into it and add it to our backlog. Meanwhile you can take a look at https://docs.aws.amazon.com/solutions/latest/serverless-image-handler/smart-cropping.html and https://docs.aws.amazon.com/solutions/latest/serverless-image-handler/content-moderation.html to see how you can apply smart cropping, please also refer to Step 2. Create and use image requests to see how to send the request.

[Furytron]

@fisenkodv Great, thanks for the docs. Unfortunately, our implementation of the Serverless Image Handler was initially designed with Thumbor style URLs, back in version 3.x. So we still have the requirement of using those URL style requests. We have not implemented the SharpJS method, which you have linked to. But good to know. Thanks!

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[Furytron]

keep open

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[Furytron]

Can you guys turn the auto-close off? It gives the impression no one cares about these issues

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[marco910]

Any updates on this? Would be great to see this feature in one of the future versions of the Serverless Image Handler.

[tjegan]

Any update on this request?

[simonkrol]

Hi @Furytron, @marco910, @tjegan

While we support Thumbor style requests, those requests are converted (in the mapping code linked), to base64 encoded requests that primarily use Sharp to perform image manipulation. I can look at expanding Thumbor-style support to include the ability to use the Amazon Rekognition smart cropping currently available for base64 encoded requests. Let me know if that would be acceptable for your use cases.

Thanks for your feedback,
Simon

[marco910]

Hi @simonkrol

We would really appreciate it if we could use smart cropping via Thumbor-styles. Because of the way our application works we're not able to use base64 encoded requests.

Maybe it would work to add /smart, /smart:faceIndex(1) and /smart:padding(40) to the Thumbor-styles? An image request could then look like this: https://img.domain.com/smart/smart:faceIndex(1)/smart:padding(40)/filters:quality(80)/image.jpg

Feel free to let me know if you need more information on that :)

[Furytron]

Thanks @simonkrol . I would echo what @marco910 has said above and the examples given.

A key part would be the ability to chain those smart croppings too, like #343. So taking @marco910 examples above, I would imagine the URLs to look like the below, note that I removed the starting "smart" part of the path, it seems redundant to me.

https://img.domain.com/smart:faceIndex(1):padding(40)/filters:quality(80):blur(5)/path/to/image.jpg

I'd also like to note that we should not require "fit-in" to be in the path for this functionality, similarly to #344.

[Furytron]

Additionally, if #521 is merged, you could end up with URLs combinations like the below, where there is chaining and no chaining, along with having one or multiple keywords: s3, smart, and filters. Just need to be careful to not break existing URLs and support this too.

https://img.domain.com/smart:faceIndex(1)/filters:quality(80)/filters:blur(5)/path/to/image.jpg
https://img.domain.com/s3:test-bucket/smart:faceIndex(1)/filters:quality(80)/filters:blur(5)/path/to/image.jpg
https://img.domain.com/s3:test-bucket/smart:faceIndex(1):padding(40)/filters:quality(80):blur(5)/path/to/image.jpg
https://img.domain.com/s3:test-bucket/smart:faceIndex(1):padding(40)/path/to/image.jpg

[simonkrol]

Hi @Furytron and @marco910,

We're working through an implementation for this now. To maintain consistency with the other filters available for Thumbor-style requests, our current plan is to implement a single filter called smart_crop, which would accept two comma delimited numbers as the value, representing the faceIndex and padding respectively. For the requests you created above, this would look something like:

https://img.domain.com/filters:smart_crop(1)/filters:quality(80)/filters:blur(5)/path/to/image.jpg
https://img.domain.com/s3:test-bucket/filters:smart_crop(1)/filters:quality(80)/filters:blur(5)/path/to/image.jpg
https://img.domain.com/s3:test-bucket/filters:smart_crop(1,40):quality(80):blur(5)/path/to/image.jpg
https://img.domain.com/s3:test-bucket/filters:smart_crop(1,40)/path/to/image.jpg

A request could be made with just a padding specified (which would default to faceIndex 0) like:

https://img.domain.com/s3:test-bucket/filters:smart_crop(,40)/path/to/image.jpg

Let me know if that would work for your use cases,
Simon

[Furytron]

Thanks @simonkrol. Using the "filters" keyword approach looks fine with me, and compliments the existing chaining solution.

So would faceIndex be 0, or 1 from the examples above? As in, smart_crop(1) would the faceIndex, and smart_crop(2) or smart_crop(3) would be an additional detectors like "glasses_detector" etc?

Will there only be a single detector available on release or multiple detectors?

[simonkrol]

Hi @Furytron,
Unfortunately because the solution doesn't actually use Thumbor, the smart cropping available would be limited to our integration with AWS Rekognition, which we've only set up to detect faces. In these requests, the x (0,1,2...etc) in filters:smart_crop(x,y) would represent which person in the image to crop around.

So on release only the detection of faces would be available. We currently have no plans to expand smart_cropping beyond faces, though that can certainly change.

Simon

[marco910]

Hi @simonkrol, good to hear that you're planing to implement the feature into the supported Thumbor filters!

https://img.domain.com/filters:smart_crop(1)/filters:quality(80)/filters:blur(5)/path/to/image.jpg
https://img.domain.com/s3:test-bucket/filters:smart_crop(1)/filters:quality(80)/filters:blur(5)/path/to/image.jpg
https://img.domain.com/s3:test-bucket/filters:smart_crop(1,40):quality(80):blur(5)/path/to/image.jpg
https://img.domain.com/s3:test-bucket/filters:smart_crop(1,40)/path/to/image.jpg

When I'm right there are not more options than the faceIndex and padding for smart crop when using JSON requests. So, the smart_crop(1,40) syntax would be fine for use.

I'm excited to use this in production soon :)