Discriminator's impact on recons loss.

[NITHISHM2410]

I used VQ f4 config ( 256 -> 64 ) in this repo to train VQGAN(taming transformers) model on COCO dataset. My recons loss steadily decreases until the discriminator kicks in. I trained my model for 50,000 iterations (54 epochs ) without disc, the reconstructions were good but bit blurry. So I decided to bring in Discriminator around 51000 iterations, since then the recons loss started to increase losing image quality.
Before the disc entry, the d_weight value was around 25-30 and recons loss (mae + lpips) was around 0.15 & I set the disc_weight to 0.75 , after disc entry the recons loss increased to 0.4 and oscillates around.
Anyone know a solution ?