Trained with custom dataset model results

[Muratoter]

I prepared custom dataset that 7000 images has only floor class and added to ade20k dataset. Now i have dataset with 28000 images. I changed floor annotations color with #040404 in my dataset, because floor class had that color in ADE20K dataset. So my floor annotation looks like this;

(#040404 color is very close to black. If you look carefully you will see the floor annotation)

and original images;

It seems everything normal my dataset. Then i changed start_epoch and num_epoch values in config yaml file,
num_epoch: 23 start_epoch: 20 epoch_iters: 5000 .
Training process done with successfully and i have encoder and decoder model that names are encoder_epoch_23.pth, decoder_epoch_23.pth. Everything is seems normal here as well

I got results using theses models but result was not as expected.
I got this result when i download model from here decoder_epoch_20.pth

And i got this result when using i trained model;

Results seem to be getting worse. What could i be doing wrong?