Image normalization

[ieee8023]

I'm working on running the models and I'm not sure about the image normalization. I have a notebook here that loads the MaskedAutoencoderCNN with the densenet121_CXR_0.3M_mae.pth weights and reconstructs an image. But the image does not appear reconstructed correctly. I've tried many normalizations but the area outside of the lungs appears blurry and the lungs appear reconstructed ok. Any idea what is wrong?

https://github.com/mlmed/torchxrayvision/blob/c44435011d97288e4af7d458dab95dc7ed6e6790/scripts/medical_mae_example.ipynb

Here is the PR to integrate it into torchxrayvision:
mlmed/torchxrayvision#150

[lambert-x]

The image normalization settings are here:

medical_mae/main_pretrain_multi_datasets_xray.py

Lines 147 to 154 in 5744fa5

	mean_dict = {'chexpert': [0.485, 0.456, 0.406],
	'chestxray_nih': [0.5056, 0.5056, 0.5056],
	'mimic_cxr': [0.485, 0.456, 0.406]
	}
	std_dict = {'chexpert': [0.229, 0.224, 0.225],
	'chestxray_nih': [0.252, 0.252, 0.252],
	'mimic_cxr': [0.229, 0.224, 0.225]
	}

Could you please try it?

[ieee8023]

In that notebook I used

torchvision.transforms.Normalize([0.5056, 0.5056, 0.5056], [0.252, 0.252, 0.252])

[lambert-x]

For plotting, I think you need to denormalize the output with the used mean and std.

[ieee8023]

matplotlib does its own linear normalization. The images should look the same with any mean and stdev adjustment.

Do you think you can modify that notebook to get the normalization correct?

[lambert-x]

Please refer to this function (I used for visualization before):

chestxray_mean = np.array([0.5056, 0.5056, 0.5056])
chestxray_std = np.array([0.252, 0.252, 0.252])

def show_image(image, title='', mean=chestxray_mean, std=chestxray_std):
    # image is [H, W, 3]
    assert image.shape[2] == 3
    plt.imshow(torch.clip((image * std + mean) * 255, 0, 255).int())
    plt.title(title, fontsize=16)
    plt.axis('off')
    print(title,'min:', image.min().item(),'max:', image.max().item())
    print(title,'mean:', image.mean().item(),'std:',  image.std().item())
    return

[ieee8023]

What does the rest of your code look like? The output of the densenet121_CXR_0.3M_mae.pth model was NCHW so I had to swap the axis. Are you using a different model?

[ieee8023]

Also, with the VIT model using unpatchify I'm getting a patchy artifact and blurry reconstructions.

https://github.com/mlmed/torchxrayvision/blob/4fa907a53b9eb3ce722f833f1409f6099c38a63f/scripts/medical_mae_vit_example.ipynb

[lambert-x]

Just uploaded a notebook for visualization. Please check https://github.com/lambert-x/medical_mae/blob/main/visualization.ipynb