Runtime Error (Traceback Attached) #106
Comments
|
Yea im unsure of this error and havent had time to look into on my end,i made the colab you linked lol just happened to check here if anyone else had issues or it was solved, im sure it has be something along the lines of dependency change, which though is hard to say i need run it locally and get the full ffmpeg error output as it likes through all of its build configuration flags before that actual error. And yea it seems its broken across the board for last week or so |
|
Yes, I saw that colab on your github. I really appreciate you taking the time to look into it. I can tell by your contributions that you're very busy. This is a very important utility for a lot of users who I am representing with this official bug post. So your efforts in tracking down the issue and either deprecating or updating the dependencies is going to be very helpful to our community of animators. Thank you very much. |
|
Full traceback on a sample file: I feel this could possibly be some version issue with pytorch and pickled checkpoint of model. Maybe a new checkpoint could be created after training on the latest version of pytorch (1.10.0+cu111 on colab)? |
Currently taking a look at this as I have CoLab Pro+:
|
|
I'm assuming that this isn't right:
Specifically, the Peak Signal-to-Noise ratio drops from ~30 to 11.5 suddenly at the ~75th epoch, going to let it finish and test out the resulting checkpoints. |
Thank you very much for working on this issue. This interpolator is a very valuable tool for many of us in the animation community. Cheers! |
No problem, I'm in the same boat as you using this for animations up until this stopped working a couple of weeks ago. Currently it appears some versions of |
|
I've made some changes to This may not be the root cause of the problem, however, I shall continue to plug away at it. I'm starting to gain a better understanding of the data structures within the pre-trained model. |
|
Right, so making more progress this, it's at epoch 106 and still going strong:
I suspect the data issue caused the GPU to crash, I suspect this is going to take another 24 hours or so to run. |
|
The next thing I have run into is the fact that once the 24 hours on Google Colab+ is up, and I restart the training using the
I'm going to look into Latest (untested) checkpoint is here: https://www.dropbox.com/s/bt0z5lah0gtp82q/SuperSloMo-Python37%2B1.10.0%2Bcu111-epoch150.ckpt?dl=0 |
|
@MSFTserver @Craig-Leko @karanpanjabi - Initial testing seems to suggest that despite my issues above getting all 200 iterations completed in Colab the following checkpoint works with the original author's YAML configs in my heavily edited version of A.I Whisperer's Intermediate Animation VQGAN+CLIP notebook:
Can I ask that you don't share the link widely as DropBox will only allow for ~130 downloads of a single link in a day; I'll have a think about hosting this in a more sensible way. |
|
Awesome i shall test tomorrow and i can reupload it to my dropbox, i have pro which i believe allows unlimited downloads and 400gb bandwidth a day. Why they have both caps im not sure as it seems limit is 400gb anyway. Sent from my SM-G988U using FastHub |
|
It is working for me. Thanks again. |
That would be fantastic. I only have Dropbox Plus. 400GB is 2,600 downloads a day which I think should be enough. I'll leave the above links available as backups. If bandwidth becomes a problem in the future we can look at other options.
Fantastic to hear! are you posting what you are doing publically? if so drop me a link, I'd be interested to see what you are doing. |
|
This was amazing! Thanks a lot @RichardSlater for fixing this and creating the new checkpoint file. I think we can also put the checkpoint file on git itself through git-lfs for long term purposes but maybe that can be brought up later through another issue ticket. |
|
@karanpanjabi gitlfs is better for private repositories, i found this out the hard way as anyone that forks or proceeds to make changes to thier fork also reflects on the original owners repository lfs usage. Sent from my SM-G988U using FastHub |
|
feel free to use this URL in projects |
|
I found another issue which seems trivial now. Maybe a retraining was not required 🙈. Sorry for not catching this earlier. <!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 400 (Bad Request)!!1</title>
<style>
*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
</style>
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>
<p><b>400.</b> <ins>That’s an error.</ins>
<p>Your client has issued a malformed or illegal request. <ins>That’s all we know.</ins>The first '<' is what we see in the UnpicklingError above. |
|
@karanpanjabi you can upload the new |
|
|
@karanpanjabi wierd because it was working, wonder why the curl commands all of sudden started to fail for that, google drive does have way lower thresholds for downloads so maybe we tapped it out for api requests vs manually through browser. In any event my notebooks have been updated and i tested it. Sent from my SM-G988U using FastHub |
😆 If nothing else we have gained two things:
Credit to @avinashpaliwal for putting this out there in the first place, and of course for the immense effort the paper authors put in architecting the neural network int he first place. That's awesome, I'm mainly on TikTok. |
Thank you very much for this project. I love it, and I've been using it successfully since early January. However, in the past week or so, it has stopped working on every colab fork I've tried. Perhaps there has been an incompatible version update in a dependency somewhere?
Describe the bug
After loading the dependencies, I input the file path to the uploaded video I want to interpolate. On execution, I get the attached error, which is a standard error being thrown by the process.communicate() method in ().
To Reproduce
Behavior happens every time. Colab is not running out of memory, as I use CO Pro+ and I have taken the images down to 100x100 pixels and 50 total frames to ensure the file is sufficiently small to rule out running out of RAM as the issue.
https://colab.research.google.com/github/MSFTserver/AI-Colab-Notebooks/blob/main/Super_SloMo.ipynb#scrollTo=Wz4BaariVdh5
Expected behavior
Up until this week when the error started, the colab output 2 files... the first being the .mkv file that the program encodes natively and then the .mp4 that results from conversion.
Interpolated results/error output
Desktop (please complete the following information):
Additional context
No additional context other than I have been having the same issue in each notebook that I am aware of which fork off the main repository, not just the one I've linked to here.
Thank you for your help, I really miss this utility!
Cheers
The text was updated successfully, but these errors were encountered: