ComfyUI crashes a lot, black images, errors. Please help investigate.

[Silverun]

I have been learing ComfyUI for about a month and since I installed it for the first time as portable and after numerous updates I am still getting same errors. I have been researching a lot, but still nothing worked out. Ye, I am aware that same things happen to people, but there was no easy fix for that. At least I wan't to know why this is happening 😠

Also I am using Forge WebUI and it only crashed once on me in a very long time and I don't get any errors, nor blacked out images, with similar workloads compared to Comfy.

I am using GTX 1070 8GB (slight undervolt for temps) and 16 GB of ram, it is not the fastest, but still it works flawlessly in Forge, even with SDXL and loras + upcalers which means this is a software or config problem.

Comfy log and example workflow where I was crashing getting black images and another error are below.
Error rate is around 3-5 times in one hour.

• It can happed on any base model.
• I got no out of memory errors etc, temps are very normal.
• Was originally using pytorch v. 2.3.1 and still had same problems, then I ran update.bat from the folder to 2.4.1
• This can happen even on basic SD 1.5 workflow, but at reduced rate in believe.
• I am using MSI afterburner for undervolt if that is somehow conflicting

1. Error:
   Error when I get black image. You can stop generation or queue again.
   It can happen on any iteration during generation in KSampler. You can get in on 5/20 it when preview in sampler turns black and it will still iterate futher as usual, but you will still get black image, because every other step it will be black.
   
   

2. Error:
   Most annoying because you have to restart the process and ui. Can happen any time during generation process. Pressing any key does nothing.
   

Log:
comfyui.log
Workflow:
epiCRealism SD1 (1).json

I was lookig into config args to maybe debug it no idea what to tryout.
I really want to use ComfyUI, but its really getting annoying. Halp! 😸

[ltdrdata]

Have you tried using the same PyTorch version as Forge?

[Silverun]

Have you tried using the same PyTorch version as Forge?

Yes, when I first installed Comfy it came with 2.3.1 pytorch cu121, I have just recently updated it to 2.4.1. My forge uses the same version now.
On older version I had same issues.

By the way I have noticed that Forge uses different PYTORCH_CUDA_ALLOC_CONF=backend:.
https://pytorch.org/docs/stable/notes/cuda.html#cuda-memory-management

Forge uses - native instead of cudaMallocAsync.

Also it says in docs that cudaMallocAsync requires CUDA 11.4 or newer. Is this CUDA installed by default with comfy how do I check the version or even if I have it?

Also I have noticed that Forge uses these extras:

• Using pytorch attention for VAE
• VAE dtype preferences: [torch.float32] -> torch.float32
• CUDA Using Stream: False

Maybe there is something here, not sure what is this used for. How to I change it?

[Silverun]

Ok, so I have tried few different command arguments for run_gpu.bat

The most stable one right now is (which turn on native pytorch mode)
.\python_embeded\python.exe -s ComfyUI\main.py --windows-standalone-build --disable-cuda-malloc
pause

The only thing I haven't tried is --lowvram, but since I have 8gb I don't think it will be needed. Will experiment maybe.

Now I only get this new error once in a while but barelly any black images and no pause freeze and crash. Any ideas what is this new error about?

[ltdrdata]

This error typically occurs when torch is installed in a version incompatible with the GPU's CUDA,
or when xformers is installed in a version incompatible with torch.
Please check both of these.
Especially if xformers is installed, try removing it.

[Silverun]

Thanks for the answers!
So I analyzed everythign you said and checked that:
nvcc --version command was not recognized.
nvidia-smi command printed out CUDA 12.5 top right. I have updated to newest driver afterwards - now it says 12.6

So I had no CUDA Toolkit installed on the system.
Was that a problem for comfyUI?

Now I have installed CUDA Toolkit 12.1, during installation I did not check drivers and components that are part of GPU drivers like HD audio etc. Basically kept my current video drivers.

Question is:

• is this the right setup now for PyTorch (2.4.1+cu121) that my comfy uses?
• will it use 12.1 while my GPU drivers are using 12.6?

About xformers.
I ran cmd from C:\AI\ComfyUI\python_embeded
.\python.exe -m pip show xformers:

C:\AI\ComfyUI\python_embeded>.\python.exe -m pip show xformers
WARNING: Package(s) not found: xformers
Which means I don't have it installed right?

I will have to test next.

By the way:
I am using latest ComfyUI v0.2.2 Release which says pytorch 2.4.1+cu124 (portable) in release notes, but I am still on 2.4.1+cu121 does it matter?

[LukeG89]

@Silverun Just wondering, are you using Latent2RGB as preview method? Because I'm experiencing random crashes as you do, and after lots of testing I saw that using TAESD solved the problem. I'm still testing it, but I just did 70 generations in a row and still no crashes.

[Silverun]

@LukeG89 Yes I am using it now, would be nice if that solved the latest issue. But for the most part I was using no preview and was still crashing. I began using it just to see if my image was going to be black to stop generation. As far as I understand this slow TAESD method will give you best previews, but slow down pc more, not sure by how much tho.

[LukeG89]

I don't know if it can solve your crashing situation or not, but have you tried removing the space in your \Comfy UI\ folder? I saw many times people having problems because of that.

[Silverun]

Well thank you for the input, but that had no effect on my issues.

[RaskyBoi]

Any solution? i met the same problem and i can't gen any pic keep getting black image or black screen and crash or the wrost is black screen and my pc can't do anything have to force shutdown

[Silverun]

You should probably create another thread for your problem, since you cannot generate anything, where you list your logs, etc.

[Silverun]

An update - solved:

So basically I tried all above and settled on this config:
C:\AI\ComfyUI>.\python_embeded\python.exe -s ComfyUI\main.py --windows-standalone-build --force-fp32 --fp8_e5m2-unet

Turns out forcing fp32 eliminated 99% of black images and crashes. This setting, to my knowledge, sets vae, unet, and text encoder to use 32 fp which is the most accurate, but slowest option for generation. However, I found out that just setting vae to 32 does not do the trick for black images. Seem like force 32 does something else apart from all said above.

Some tests with SXDL+ loras:
fp8_e5m2-unet - uses about 2 gb less VRAM and 20% less RAM than fp16 unet and almost the same quality
fp16 unet is about 60% slower then fp8

Now I am gettign same generation speed as Forge Web UI and barelly any problems.
I have tried almost all combinations of arguments and nothing really worked that well for stability.
Maybe there is something I can do for speed but that is for another time.