Training Results

[Huage001]

Dear authors,

Thanks for your excellent work in autoregressive image generation!

I try to reproduce the training of GPT-B-256 following the instruction provided here. The specific command I used is:

torchrun \
--nnodes=1 --nproc_per_node=8 --node_rank=0 \
--master_addr=127.0.0.1 --master_port=26667 \
train_c2i.py --cloud-save-path ckpt/GPT_B --code-path dataset/imagenet_code_c2i_flip_256_ten_crop/ --image-size 256 --gpt-model GPT-B 

However, after training ~150 epoch on ImageNet1k, it seems that the generated results are still meaningless:

My environment is 8xA5000 GPUs, which is different from yours (8xA100). I wonder whether the results are sensitive to such a difference, and whether the problem would be alleviated after full training (300 epochs).

Thanks for your help in advance :)

[PeizeSun]

Hi~
This is not normal. In general, you can see meaningful images after 10 epochs training.

Can you first try to check whether the image code is correct? You can use this script here

[Huage001]

Dear Peize,

Thanks for your very prompt reply! I have checked the image code using the script. Yes, it seems that the image code is normal. Here is the reconstruction results of '0.npy' and the results seem reasonable:

In addition, I can also produce reasonable images using your released checkpoints.

[PeizeSun]

Did you change the batch size, learning rate or other training parameters ?

[Huage001]

Dear Peize,

I didn't change anything in .py files. I assumed that the command for 256 resolution is the same as that for 384 and used the script above.