Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The model is not convergent. #7

Open
TrackingBird opened this issue May 8, 2018 · 1 comment
Open

The model is not convergent. #7

TrackingBird opened this issue May 8, 2018 · 1 comment

Comments

@TrackingBird
Copy link

When I fine tuning the network with my dataset, The logs are as follows. I think the model is not convergent. The disparity computed by the fine-tuned model is a white image with a little texture.
Can you give me some suggestion about how to train the model? Or how to get a real reliable training dataset. Thanks.

I0508 15:53:59.874593  6385 solver.cpp:245]     Train net output #73719: blob65 = 74.7856
I0508 15:53:59.874598  6385 solver.cpp:245]     Train net output #73720: blob65 = 74.7856
I0508 15:53:59.874603  6385 solver.cpp:245]     Train net output #73721: blob65 = 74.7857
I0508 15:53:59.874608  6385 solver.cpp:245]     Train net output #73722: blob65 = 74.7857
I0508 15:53:59.874614  6385 solver.cpp:245]     Train net output #73723: blob65 = 74.7857
I0508 15:53:59.874619  6385 solver.cpp:245]     Train net output #73724: blob65 = 74.7856
I0508 15:53:59.874624  6385 solver.cpp:245]     Train net output #73725: blob65 = 74.7856
I0508 15:53:59.874629  6385 solver.cpp:245]     Train net output #73726: blob65 = 74.7856
I0508 15:53:59.874634  6385 solver.cpp:245]     Train net output #73727: blob65 = 74.7856
I0508 15:53:59.874646  6385 solver.cpp:245]     Train net output #73728: flow_loss1 = 0.0113061 (* 1 = 0.0113061 loss)
I0508 15:53:59.874652  6385 solver.cpp:245]     Train net output #73729: flow_loss2 = 174.297
I0508 15:53:59.874658  6385 solver.cpp:245]     Train net output #73730: flow_loss3 = 170.482
I0508 15:53:59.874665  6385 solver.cpp:245]     Train net output #73731: flow_loss4 = 163.814
I0508 15:53:59.874670  6385 solver.cpp:245]     Train net output #73732: flow_loss5 = 154.755
I0508 15:53:59.874675  6385 solver.cpp:245]     Train net output #73733: flow_loss6 = 141.625
I0508 15:53:59.874682  6385 solver.cpp:245]     Train net output #73734: loss_down = 0.00255099 (* 0.125 = 0.000318874 loss)
I0508 15:53:59.874689  6385 solver.cpp:245]     Train net output #73735: loss_est = 0.00268216 (* 0.125 = 0.00033527 loss)
I0508 15:53:59.874701  6385 solver.cpp:245]     Train net output #73736: loss_up = 0.00255099 (* 0.125 = 0.000318874 loss)
I0508 15:53:59.874707  6385 solver.cpp:245]     Train net output #73737: loss_west = 0.00268216 (* 0.125 = 0.00033527 loss)
I0508 15:53:59.874717  6385 sgd_solver.cpp:106] Iteration 11950, lr = 3.125e-06
I0508 15:55:18.302011  6385 solver.cpp:456] Snapshotting to binary proto file disp_iter_12000.caffemodel
I0508 15:55:18.302299  6385 net.cpp:867] Serializing 137 layers
I0508 15:55:21.718309  6385 sgd_solver.cpp:273] Snapshotting solver state to binary proto file disp_iter_12000.solverstate
I0508 15:55:29.855378  6385 solver.cpp:318] Iteration 12000, loss = 0.00952777
I0508 15:55:29.855406  6385 solver.cpp:323] Optimization Done.
I0508 15:55:29.855428  6385 caffe.cpp:222] Optimization Done.````
@AlessioTonioni
Copy link
Member

If you are using our train.prototxt the model is trying to minimize only flow_loss1 (i.e. the higher resolution disparity) so I think that your model is working.
If you take a look at train.prototxt for KITTI each flow_loss except the last have the parameter loss_weight=0 meaning that is not considered during loss computation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants