-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Instacrash with fluid.mlpclassifier
when trying to fit
something
#364
Comments
Got a second crash (also with a really long Also, narrowed down when it happens. It seems like I got the crash when I pause the training (i.e. toggle off the toggle in the loop) then toggle it back on. I got the instacrash when toggling it back on. |
few observations:
on my machine, there is no memory leak after running it for 15 minutes without a crash - I thought of checking this since both crashes are linked to memory allocation... and I start and stop it, to no avail. so that brings us to how we can help you help us help you: are you set up for compilation? if so you could gain 2 things:
this comes at the expense of having to compile, but also maybe being confused by which version you are actually using. I have scripts to swap them in the OS, but that might not be exciting for you. in all cases, I'm happy to help. anyway, as I am unable to reproduce, we are stalled. let us know if you find something more reproducible. |
now running for 45 minutes in 'test' compile mode, starting and stopping and resetting - still no crash. |
I'll see if I can get it to crash again. Not saying the network is useful, I was just testing different structures on to see what type/direction/style was better (maybe changing structures often, via the It could just be coincidence, but happening twice with the same object/process seems unlikely. |
If you don't mind, try this version of the object (keep the other one you have for real-life use) so if it crashes we'll know better if it is fluid.verse-related and where from... |
@rconstanzo any more crash with my magic custom compile? |
Was in the UK teaching, will give it a test now. Not gotten any new crashes since though (but haven't been testing super long network structures since. |
any luck on this? |
Will test the patch above a bit more, but I'm not actively using that large structure in any patches (partially being scared from this issue). |
Turns out I hadn't tested the new version you posted. Running the above patch again (with Here are both crash reports (both are the same kind of thing as above): I will add, perhaps unhelpfully?, that the patch does not crash in a beta version of Max... |
ok reading the log it might be a memory thing again only in Max. I'll recode in debug SC and see if I can crash at all, stay tuned |
ok I'm running it in Max first, and I cannot crash... even with the default! But, as I look at your code, you know that you are running in the high priority thread, right? "but I put a deferlow" you will say... but not at the right place! [delay 60] promotes the post-process bang back to scheduler! To deal with Max's (awful) threading promotion and demotion, I usually am very disciplined now (thanks to @weefuzzy ) and put the defer right after the potential promotion objects (delay) in this instance. Will run in SC in case I hopefully manage to crash it - but I think it might still be a max memory threading thing that we are discussing in another bug (and many others actually) with @AlexHarker, only happening in Max. |
also, a few NN hints: momentum that low is not helpful, it is a huge network, and reset is not resetting the network, clear is (as per manual) |
My thinking with the Also, these specific settings aren't really working well (for a variety of reasons), but it shouldn't crash in any regard. I have managed to get the patch to crash on both of my computers go (intel and arm). |
yet it doesn't crash here - let's see if SC crashes |
[mxj WhichThread] is your friend if you are in doubt. Always try to send long jobs in the low priority thread. |
ok I'm running it in Max (no crash so far) and SC (no crash so far) at the same time - my i9 is in full leafblower mode 🤣 sc code:
|
Wait, are you I was getting a crash by not doing that. As in, I would toggle on the crunching for a bit, then toggle it off, then toggle it back on again (boom crash). I don't think I ever got a crash from just letting it run. |
ok, removing the [defer] I added, starting and stopping, this crashes, in max only. This points to @AlexHarker investigations of thread safety in Max then, which is piling up many other instacrashes... the good news for you for now, is that if you put a defer at the right place, you get a stable patch. A solution for the bigger issue is going to happen, don't worry |
As mentioned on the discourse thread got an (unrepeated) crash when trying to
fit
some data withfluid.mlpclassifier
.Attaching the isolated patch bit in question, along with the data/labels I was using at the time. Also the crash report.
This is, I believe, the crash-y bit:
crashbits.zip
The text was updated successfully, but these errors were encountered: