Adds Random Resized Crop #457

beresandras · 2022-05-28T23:28:06Z

This PR implements the Random Resized Crop augmentation, currently minimally.

Example augmentations of the elephant image from the keras.io guide:

Corresponding issue: #131

Details to discuss:

interface:
- currently an area_factor and aspect_ratio_factor is used, both between 0 and 1, following the API design guidelines. They are translated to new areas and aspect ratios via new_value = 1 +/- factor(). The maximal area_factor of exactly 1.0 can lead to a crop with 0% or 200% area.
- we could also use min/max area and min/max aspect ratio, similarly to the torchvision implementation. This might be easier to understand, and would enable the user to set area = 8% - 100%, aspect_ratio = 3/4 - 4/3, corresponding to the commonly used Inception crops (Section 6).
- do we even want crops to go outside the image?
crop method: currently the crop + resize is implemented with the ImageProjectiveTransformV3 operation. It could be done with tf.image.crop_and_resize() as well, but it does not have a fill_mode parameter.
batched implementation: I proposed a batched implementation originally in the issue, as it is not difficult to do, however, it does not fit well with the current BaseImageAugmentationLayer API.
log-uniform aspect ratio distribution: When having aspect ratios e.g. between 1:2 and 2:1 (0.5 and 2.0) one might want to keep the distribution centered at 1.0. To achieve that, we could use a log-uniform sampling between the two value, similarly to the torchvision implementation. This would somehow require a default log-uniform factor sampler, or a hardcoded sampling. I am not sure how this should be implemented.
aliasing: we should not allow image areas above 4 (2x-2x zoom out along both sides), as they would lead to aliasing, and the ImageProjectiveTransformV3 does not have an antialias argument, in contrast to tf.image.resize().

TODO:

tests

beresandras · 2022-05-28T23:33:53Z

Sample code to test it:

import matplotlib.pyplot as plt
import tensorflow as tf

from keras_cv.layers.preprocessing.random_resized_crop import RandomResizedCrop

SIZE = (300, 300)
elephants = tf.keras.utils.get_file(
    "african_elephant.jpg", "https://i.imgur.com/Bvro0YD.png"
)
elephants = tf.keras.utils.load_img(elephants, target_size=SIZE)
elephants = tf.keras.utils.img_to_array(elephants) / 255

augmenter = RandomResizedCrop(area_factor=0.75, aspect_ratio_factor=0.25)
elephants = augmenter(elephants).numpy()

plt.imsave("aug_image.png", elephants)

LukeWood

This looks wonderful! Great work

LukeWood · 2022-05-31T20:49:33Z

keras_cv/layers/preprocessing/random_resized_crop.py

+class RandomResizedCrop(tf.keras.__internal__.layers.BaseImageAugmentationLayer):
+    """Randomly crops an image and resizes it to its original resolution.
+    Args:
+        TODO


lets document these factors, great job!!!

LukeWood · 2022-05-31T20:50:01Z

keras_cv/layers/preprocessing/random_resized_crop.py

+    ):
+        super().__init__(seed=seed, **kwargs)
+        self.area_factor = preprocessing.parse_factor(
+            area_factor, param_name="area_factor", seed=seed


are there high and low bounds on this value? Be sure to document them, and pass them to min and max in parse_factor

keras_cv/layers/preprocessing/random_resized_crop.py

LukeWood · 2022-05-31T20:52:01Z

@beresandras let me know if you need help adding test cases! This is looking fantastic! Thanks for your swift contribution!

LukeWood · 2022-06-02T17:12:53Z

FYI @beresandras I updated the API a bit.

LukeWood · 2022-06-02T17:13:24Z

Some notes:

we need to include a crop size
we want to resize images first, then crop second.

LukeWood · 2022-06-07T01:08:16Z

Hey @beresandras are you intending to finish this PR up? If not, we can try to find someone to run it across the finish line!

No worries either way.

beresandras · 2022-06-08T09:20:48Z

Yes, I will finish it in the next few days.

beresandras · 2022-06-14T14:41:08Z

@LukeWood unfortunately it seems that I won't be able to finish the PR for another week. Please try to find someone to complete it if you can.

LukeWood · 2022-06-14T17:16:12Z

@LukeWood unfortunately it seems that I won't be able to finish the PR for another week. Please try to find someone to complete it if you can.

Thanks for letting me know. I will try to do so!

We can also pick back up next week if needed

AdityaKane2001 · 2022-06-16T12:08:31Z

@sayakpaul @LukeWood

I would like to work on this one. I'll create a PR asap.

LukeWood

Minor changes

LukeWood · 2022-06-23T02:31:44Z

keras_cv/layers/preprocessing/random_resized_crop.py

+
+
+@tf.keras.utils.register_keras_serializable(package="keras_cv")
+class RandomResizedCrop(tf.keras.__internal__.layers.BaseImageAugmentationLayer):


lets use keras_cv.layers.BaseImageAugmentationLayer going forward

LukeWood · 2022-06-23T02:33:53Z

keras_cv/layers/preprocessing/random_resized_crop.py

+        interpolation: interpolation method used in the `ImageProjectiveTransformV3` op.
+             Supported values are `"nearest"` and `"bilinear"`.
+             Defaults to `"bilinear"`.
+        aspect_ratio_factor: (Optional) A tuple of two floats, a single float or


I think aspect ratios respect the aspect ratio convention, 1.0 does not change the image. aspect_ratio=4/3 shifts aspect ratio by 4/3, etc.

As such, we should default this ratio to (1, 1) so it performs a no-op. I don't think distorting aspect ratio should be a default behavior.

Perhaps we make this argument required?

LukeWood · 2022-06-24T17:01:38Z

Thanks for the contribution! Closing in favor of #499

* Created random resized crop files * Added test from #457 * Used `tf.image.crop_and_resize` instead of `ImageProjectionTransformV3` * Minor bug * Formatted * Reformatted * Update keras_cv/layers/preprocessing/random_resized_crop.py Co-authored-by: Sayak Paul <[email protected]> * Apply suggestions from code review Co-authored-by: Sayak Paul <[email protected]> * Doc changes * Doc changes * Made requested changes * Reformatted and tested * Update keras_cv/layers/preprocessing/random_resized_crop.py Co-authored-by: Sayak Paul <[email protected]> * Made requested changes * Created random resized crop files * Added test from #457 * Used `tf.image.crop_and_resize` instead of `ImageProjectionTransformV3` * Minor bug * Formatted * Reformatted * Doc changes * Update keras_cv/layers/preprocessing/random_resized_crop.py Co-authored-by: Sayak Paul <[email protected]> * Apply suggestions from code review Co-authored-by: Sayak Paul <[email protected]> * Doc changes * Made requested changes * Reformatted and tested * Luke edits * remove merge conflict * Fix broken test cases * Made changes to rrc * Minor changes * Added checks * Added checks * Added checks for inputs * Docstring updates * Made requested changes * Minor changes * Added demo * Formatted Co-authored-by: Sayak Paul <[email protected]> Co-authored-by: Luke Wood <[email protected]>

* Created random resized crop files * Added test from keras-team#457 * Used `tf.image.crop_and_resize` instead of `ImageProjectionTransformV3` * Minor bug * Formatted * Reformatted * Update keras_cv/layers/preprocessing/random_resized_crop.py Co-authored-by: Sayak Paul <[email protected]> * Apply suggestions from code review Co-authored-by: Sayak Paul <[email protected]> * Doc changes * Doc changes * Made requested changes * Reformatted and tested * Update keras_cv/layers/preprocessing/random_resized_crop.py Co-authored-by: Sayak Paul <[email protected]> * Made requested changes * Created random resized crop files * Added test from keras-team#457 * Used `tf.image.crop_and_resize` instead of `ImageProjectionTransformV3` * Minor bug * Formatted * Reformatted * Doc changes * Update keras_cv/layers/preprocessing/random_resized_crop.py Co-authored-by: Sayak Paul <[email protected]> * Apply suggestions from code review Co-authored-by: Sayak Paul <[email protected]> * Doc changes * Made requested changes * Reformatted and tested * Luke edits * remove merge conflict * Fix broken test cases * Made changes to rrc * Minor changes * Added checks * Added checks * Added checks for inputs * Docstring updates * Made requested changes * Minor changes * Added demo * Formatted Co-authored-by: Sayak Paul <[email protected]> Co-authored-by: Luke Wood <[email protected]>

beresandras added 2 commits May 28, 2022 20:50

initial commit based on shear

8f8914a

minimal random resized crop added

22dd713

remove default none transformation

12f624b

LukeWood suggested changes May 31, 2022

View reviewed changes

Update docstring and API for RandomResizedCrop

0bb7661

reformat

f908536

AdityaKane2001 added a commit to AdityaKane2001/keras-cv that referenced this pull request Jun 16, 2022

Added test from keras-team#457

ebb52bd

AdityaKane2001 mentioned this pull request Jun 16, 2022

Random Resized Crop #499

Merged

LukeWood suggested changes Jun 23, 2022

View reviewed changes

LukeWood pushed a commit to AdityaKane2001/keras-cv that referenced this pull request Jun 24, 2022

Added test from keras-team#457

309d266

LukeWood closed this Jun 24, 2022

LukeWood mentioned this pull request Sep 16, 2022

Split RandomlyResizedCrop into two API surfaces (RandomlyZoomedCrop, RandomCropAndResize) #738

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds Random Resized Crop #457

Adds Random Resized Crop #457

beresandras commented May 28, 2022 •

edited

Loading

beresandras commented May 28, 2022

LukeWood left a comment

LukeWood May 31, 2022

LukeWood May 31, 2022

LukeWood commented May 31, 2022

LukeWood commented Jun 2, 2022

LukeWood commented Jun 2, 2022

LukeWood commented Jun 7, 2022

beresandras commented Jun 8, 2022

beresandras commented Jun 14, 2022

LukeWood commented Jun 14, 2022

AdityaKane2001 commented Jun 16, 2022

LukeWood left a comment

LukeWood Jun 23, 2022

LukeWood Jun 23, 2022

LukeWood Jun 23, 2022

LukeWood commented Jun 24, 2022



		@tf.keras.utils.register_keras_serializable(package="keras_cv")
		class RandomResizedCrop(tf.keras.__internal__.layers.BaseImageAugmentationLayer):

Adds Random Resized Crop #457

Adds Random Resized Crop #457

Conversation

beresandras commented May 28, 2022 • edited Loading

beresandras commented May 28, 2022

LukeWood left a comment

Choose a reason for hiding this comment

LukeWood May 31, 2022

Choose a reason for hiding this comment

LukeWood May 31, 2022

Choose a reason for hiding this comment

LukeWood commented May 31, 2022

LukeWood commented Jun 2, 2022

LukeWood commented Jun 2, 2022

LukeWood commented Jun 7, 2022

beresandras commented Jun 8, 2022

beresandras commented Jun 14, 2022

LukeWood commented Jun 14, 2022

AdityaKane2001 commented Jun 16, 2022

LukeWood left a comment

Choose a reason for hiding this comment

LukeWood Jun 23, 2022

Choose a reason for hiding this comment

LukeWood Jun 23, 2022

Choose a reason for hiding this comment

LukeWood Jun 23, 2022

Choose a reason for hiding this comment

LukeWood commented Jun 24, 2022

beresandras commented May 28, 2022 •

edited

Loading