HEDJitter Transform

[Ycblue]

For digital pathology applications, color augmentation in hematoxylin & eosin space has been shown to be a useful tool (https://arxiv.org/abs/1902.06543).

Essentially, the transform disentangles an RGB image into hematoxylin and eosin channels, perturbes the channels independently and then transforms the image back into RGB space.

A Pytorch version was already implemented (https://github.com/gatsby2016/Augmentation-PyTorch-Transforms) and I'm wondering if it could be integrated into MONAI.

If there is no ongoing project on this front, I can help implement this feature.
I don't have any experience contributing to MONAI yet, so I would need some assistance if that were the case.

Cheers.

[Nic-Ma]

Welcome to your contribution.

Hi @drbeh ,

Could you please help also take a look at this feature request?

Thanks in advance.

[drbeh]

Hi @Ycblue,
Thanks for your interest in contributing to MONAI. It looks an interesting augmentation transform for H&E histopathology images. There is no ongoing effort on this from our side and I'd be happy to help you to implement it in MONAI.
However, do you know if using HEDJitter provides a better training performance than just using ColorJitter?
I couldn't find any head-to-head comparison in the paper and it seems that they haven't test it on the images with DAB stain (only H&E). Did I missed it in the paper?

cc @JHancox

[Ycblue]

Hi @drbeh,

indeed they don't mention any direct head-to-head comparison. However, they did compare HED to HSV augmentation, which includes shifting hue and saturation values (ColorJitter also perturbs Brightness and Contrast.) In the paper HED-light and HSV-strong produce similar results, with HED-light scoring a bit better in most cases.
In a previous paper, the same authors use the seperated DAB channel to identify mitotic candidates. Actually, the algorithm for color deconvolution into HED space was proposed a while ago here and is also used in the scikit function color.rgb2hed .

As far as I know, there is also no ColorJitter function implemented for MONAI and I am actually unsure about the compatibility of MONAIs Transform class with torchvision transforms.

Maybe there is interest in both?

[drbeh]

@Ycblue, thank you very much for your answers.

1. All torch vision transforms including ColorJitter can be directly used in MONAI through TorchVisiond and RandTorchVisiond wrapper transforms. You can find an example here.
2. You can also find an implementation of using deconvolution to extract HED from RGB in MONAI here. In this implementation unlike the skimage implementation the stain matrix is not fixed and is being derived from the image itself. You can find more details in this paper. We have used this for stain normalization.
3. For this HEDJitter transform, we can use a derived stain matrix too but I think it will slow it down. What do you think?

[Ycblue]

@drbeh thanks for the info on the wrapper transforms and the existing deconvolution implementations!

3. I think for augmentation purposes, deriving individual stain matrix is overkill. The values will be shifted randomly anyway and it would just slow it down as you mentioned.

Where do you propose I start with the implementation? Should I use class ExtractHEStains(Transforms) as a basis?

[drbeh]

@Ycblue, sure you can use ExtractHEStains (or any other transform in MONAI) as an example.
Before that it is worth taking a look at MONAI contributing guide and let me know if you have any question.

For implementation,

1. Please start with adding a new class named HEDJitter in monai/apps/pathology/transforms/stain/array.py. Similar to ExtractHEStain or any other transfrom in MONAI.
2. Then you should write test cases in a new file in tests/test_pathology_hed_jitter.py similar to this.
   It is important to add meaningful tests to evaluate the proper functionality of the transform. Be aware that this part might take much more time than the implementing the transform.
3. Next, you should do the same thing for the dictionary-based version of your transform and name your calss HEDJitterd (just adding d at the end). Similar to ExtractHEStaind
4. Add the unittests for the dictionary-based version at a new file called ``tests/test_pathology_hed_jitter_d.py`
5. You should do all the implementation in a new branch at a fork of MONAI repo into your personal github. see the contributing guideline for more details.
6. You can make a DRAFT PR before the completion of all parts (for instance after step 1). Although you can ask for our feedback at any time during your development, the formal review will be done once all the parts are ready and the PR is ready (not in draft mode).
7. Please do not hesitate to reach out if you have any question.

[Ycblue]

Thanks for the guidelines! I will check out the contribution guide.
A couple of questions.

1. I will basically reimplement the implementation from the repository linked above. Do I need to contact the original owners and tell them about it (even if just out of courtesy?)
2. Does MONAI use the skimage library or should I implement color.rgb2hed seperately?

[drbeh]

Great!

1. It is your responsibility to make sure that you are not violating any copyright or licensing, so contacting the authors and asking for permission might be necessary in this care. Consider that MONAI license is Apache 2.0, which is very permissive and any one should be able to use for academic and industrial use. You may need to communicate it with the authors. I don't see any license file in their repo so if they can add one similar to this to their repo would be great.
2. It is fine to use skimage in MONAI but you should import it optionally like this:

   MONAI/monai/apps/pathology/transforms/post/array.py

   Line 35 in 9efd54d

   disk, _ = optional_import("skimage.morphology", name="disk")

[Ycblue]

Hi @drbeh,

It's been a while since I've worked on this.
I wanted to include test data and according to the guideline, it should be uploaded here, but I couldn't figure out how to do it. Can you help me with that?

Best!

[drbeh]

Hi @drbeh,

It's been a while since I've worked on this. I wanted to include test data and according to the guideline, it should be uploaded here, but I couldn't figure out how to do it. Can you help me with that?

Best!

Hi @Ycblue, absolutely! However, we try to keep the test data to a minimum necessary and avoid duplicates so could you please let us know what the test data that you want to include is and how it is obtained (to check the licensing)?
Thanks

[Ycblue]

I really just want to add this sample image, similar to the original Repo.

[drbeh]

I really just want to add this sample image, similar to the original Repo.

Hi @Ycblue, thanks for your effort on HEDJitter. I was wondering where are you with your implementing unit tests?

We can add a sample image but we should make sure that the image does not have any copyright (the license is compatible with Apache 2.0 license of MONAI).

What I suggest to do is to implement the unittests using that sample image and test them locally (don't worry about the unittest failure in the CI/CD tests for now). We can validate it locally and use an image with compatible license for the CI/CD to pass. What do you think?