analyze_convergence.py error #200
Unanswered
robinbinchen
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I try to run the analyze_convergence.py to monitor the convergence. I run the tutorial data and everything is good, but on my own data, it show the error:
Traceback (most recent call last):
File "/home/hendlab/software/cryodrgn/utils/analyze_convergence.py", line 1262, in
main(parser.parse_args())
File "/home/hendlab/software/cryodrgn/utils/analyze_convergence.py", line 1129, in main
plot_loss(logfile, outdir, E, LOG)
File "/home/hendlab/software/cryodrgn/utils/analyze_convergence.py", line 198, in plot_loss
loss = analysis.parse_loss(logfile)
File "/home/hendlab/miniconda3/envs/cryodrgn1/lib/python3.9/site-packages/cryodrgn/analysis.py", line 26, in parse_loss
assert m is not None
AssertionError
I check the log file for training, the average loss goes to nan after 10 epoch:
2023-01-17 16:04:14 # =====> Epoch: 1 Average gen loss = 0.839674, KLD = 6.498083, total loss = 0.839737; Finished in 0:31:33.799551
2023-01-17 16:37:43 # =====> Epoch: 2 Average gen loss = 0.838753, KLD = 10.751885, total loss = 0.838858; Finished in 0:32:22.311160
2023-01-17 17:12:40 # =====> Epoch: 3 Average gen loss = 0.838204, KLD = 13.523651, total loss = 0.838335; Finished in 0:32:28.599590
2023-01-17 17:47:36 # =====> Epoch: 4 Average gen loss = 0.838009, KLD = 14.628567, total loss = 0.838151; Finished in 0:32:30.066586
2023-01-17 18:22:41 # =====> Epoch: 5 Average gen loss = 0.837887, KLD = 14.887753, total loss = 0.838032; Finished in 0:32:37.819370
2023-01-17 18:58:26 # =====> Epoch: 6 Average gen loss = 0.837816, KLD = 15.666186, total loss = 0.837968; Finished in 0:33:02.426848
2023-01-17 19:34:00 # =====> Epoch: 7 Average gen loss = 0.837833, KLD = 15.824527, total loss = 0.837987; Finished in 0:32:59.442240
2023-01-17 20:07:46 # =====> Epoch: 8 Average gen loss = 0.861164, KLD = 252.961207, total loss = 0.863624; Finished in 0:32:06.815892
2023-01-17 20:42:53 # =====> Epoch: 9 Average gen loss = 1.83694, KLD = 1087.186825, total loss = 1.847512; Finished in 0:32:52.103806
2023-01-17 21:17:56 # =====> Epoch: 10 Average gen loss = nan, KLD = 1333946638782691.500000, total loss = nan; Finished in 0:32:32.592444
2023-01-17 21:45:41 # =====> Epoch: 11 Average gen loss = 1.68057, KLD = 87498.612832, total loss = 2.531595; Finished in 0:25:19.137182
2023-01-17 22:03:23 # =====> Epoch: 12 Average gen loss = nan, KLD = 37164288497926.007812, total loss = nan; Finished in 0:17:09.494457
2023-01-17 22:21:04 # =====> Epoch: 13 Average gen loss = nan, KLD = 40503254724743666973580197888.000000, total loss = nan; Finished in 0:17:08.811192
2023-01-17 22:38:46 # =====> Epoch: 14 Average gen loss = nan, KLD = 274543.686224, total loss = nan; Finished in 0:17:09.640120
2023-01-17 22:56:30 # =====> Epoch: 15 Average gen loss = nan, KLD = 452516251163459.250000, total loss = nan; Finished in 0:17:12.092153
2023-01-17 23:14:31 # =====> Epoch: 16 Average gen loss = nan, KLD = 42104758555416106239000576.000000, total loss = nan; Finished in 0:17:28.314350
do you have any suggestion to solve the issue?
Thank you!
Beta Was this translation helpful? Give feedback.
All reactions