Sudden stop while training

[kawaiitakagi]

When I was training the three-view net, after some epoches( not a specific number, encountered at 7, 20, 90, or other numbers) the train process stopped, without any warnings and errors. The process just stopped and through tqdm model, I could see that the trainer stopped to read data from the dataset( tqdm displayed 0 it/s). I was using a SSH client, with an NVIDIA 4060 GPU.

[layumi]

Hi @kawaiitakagi

Could you provide me more details? or try to use another machine?

[kawaiitakagi]

Hi @kawaiitakagi

Could you provide me more details? or try to use another machine?

Thank you for your reply. I found that this issue was caused by several classes lacking Google images.

[layumi]

Thank you @kawaiitakagi
I think I have fixed it by https://github.com/layumi/University1652-Baseline/blob/master/folder.py#L102
The issue still exists? Do you find any new solution?

[fangqi99]

Thank you @kawaiitakagi I think I have fixed it by https://github.com/layumi/University1652-Baseline/blob/master/folder.py#L102 The issue still exists? Do you find any new solution?

I also had a pause at a certain epoch when I ran train_no_street.py, here is my command line python train_no_street.py --name pretrained_two_view_my_model --share --views 2 --droprate 0.75 --stride 1 --h 256 --w 256 --gpu_ids 4