Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add class mapping for multi class segmentation #401

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

chaymaaBOUSNAH
Copy link

No description provided.

@Mattwmaster58
Copy link

This doesn't seem to be working for me, at least using 3 channel pngs as masks

@chaymaaBOUSNAH
Copy link
Author

This doesn't seem to be working for me, at least using 3 channel pngs as masks

how many classes do you have? you must ensure that you have mapped all classes, can you share the mapping dict you have set?

@Mattwmaster58
Copy link

Mattwmaster58 commented Nov 6, 2022

Upon closer investigation, the issue appears to be related to my mask image format as its in palette mode already. As I understand it, no further conversion should be necessary then (ie, no need to use mask_to_class). Although even after converting with convert("RGB"), other things break that I'm not sure are related. Here's one mask, I'm not sure that GH won't re-encode.
00000

Here's the code I added in __getitem__:

        mask_mode = mask.mode

        assert img.size == mask.size, \
            f'Image and mask {name} should be the same size, but are {img.size} and {mask.size}'

        img = self.preprocess(img, self.scale, is_mask=False)
        mask = self.preprocess(mask, self.scale, is_mask=True)

        if mask_mode != "P":  # if pallete mode, no need to convert a RGB color because its already correct at this point
            # mapping the class colors
            mask = self.mask_to_class(mask, self.mapping)

The error I get when trying to convert to RGB first, then using mask_to_class:

Traceback (most recent call last):
  File "I unet-multi-seg/train.py", line 213, in <module>
    train_net(net=net,
  File "unet-multi-seg/train.py", line 91, in train_net
    for batch in train_loader:
  File ".venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File ".venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 671, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File ".venv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File ".venv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "unet-multi-seg/utils/data_loading.py", line 90, in __getitem__
    mask = self.mask_to_class(mask, self.mapping)
  File "unet-multi-seg/utils/data_loading.py", line 67, in mask_to_class
    validx = (idx.sum(0) == 3)
AttributeError: 'bool' object has no attribute 'sum'
python-BaseException

I don't think PIL is handling the conversion to RGB correctly, the mask np.array has absurdly large values in it

@chaymaaBOUSNAH
Copy link
Author

chaymaaBOUSNAH commented Nov 7, 2022

I have removed /255 in preprocessing, I think it must be the cause of your issue, can you retry now?
if it doesn't work check your mask shape and values after conversion to RGB !

@sevimcengiz
Copy link

how do we need to arrange for mapping values if we have more than 3 classess?

@chaymaaBOUSNAH
Copy link
Author

how do we need to arrange for mapping values if we have more than 3 classess?

You must map each class color code to an index, for example: if you have 5 classes with the following colors: Black, blue, red, aqua, and magenta. the mapping dict must be: mapping = {(0, 0, 0): 0, (0, 0, 255): 1, (255, 0, 0): 2, (0, 255, 255): 3, (255, 0, 255): 4}

@sevimcengiz
Copy link

sevimcengiz commented Nov 10, 2022

Thank u so much for the quick reply. Mapping values worked for me for my custom dataset.

I have another question when I load any mask from Carvana datasets. It shows 1280x1918 uint8. It is not 1280x1918x3. The intensity value of the car is 1 and the background is 0. In multi-class segmentation, the mask should has w,h,c. for ex: 1280x1918x3 uint8, and a blue label which represents car should have intensity value 0 in r, 0, g, 255 in blue channel. or for magenta channel red 255, green 0 and blue 255.

Question2: why does data_loading.py only take the transpose of the images instead of both mask and images? Because code says "if it is not mask, it takes transpose ((2,0,1))"?,

it gives me an error. Error:

File "train.py", line 200, in
amp=args.amp)
File "train.py", line 98, in train_net
loss = criterion(masks_pred, true_masks)
File "/home/sevim/anaconda3/envs/pytorch37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/sevim/anaconda3/envs/pytorch37/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 1165, in forward
label_smoothing=self.label_smoothing)
File "/home/sevim/anaconda3/envs/pytorch37/lib/python3.7/site-packages/torch/nn/functional.py", line 2996, in cross_entropy
return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
RuntimeError: input and target batch or spatial sizes don't match: target [1, 959, 3], input [1, 5, 640, 959]
"

Question 3: why did you remove /255? what will happen to the normalization?

sorry for the long comment. I'm trying to understand.

@chaymaaBOUSNAH
Copy link
Author

I didn't test the Carnava dataset, but if you are talking about the mask shape after mapping, it must be (h, w) and there will be no channel dim.
we do the transpose to have the shape (channels, height, width) instead of (height, width, channels) because most Pytorch functions must have this input format, like the Cross-Entropy loss ...
I removed /255 because mapping do already the job (after mapping masks contain only indexes of each class color).
I hope my answer is clear!

@onway666
Copy link

onway666 commented May 1, 2023

how do we need to arrange for mapping values if we have more than 3 classess?

how do we need to arrange for mapping values if we have more than 3 classess?

You must map each class color code to an index, for example: if you have 5 classes with the following colors: Black, blue, red, aqua, and magenta. the mapping dict must be: mapping = {(0, 0, 0): 0, (0, 0, 255): 1, (255, 0, 0): 2, (0, 255, 255): 3, (255, 0, 255): 4}

Doesn't this operation create some kind of association between different classes? What if my classes have no relationship with each other?

@onway666
Copy link

onway666 commented May 1, 2023

What should I do if my original mask is a four-channel image where each channel is a binary image?

@BenjaminSchaar
Copy link

I am not 100 sure how to use more than 2 classes:
Should I do more than changing the parameter classes ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants