Does SD uses more Vram with ControlNet than without it?

[Drakmour]

Does SD uses more Vram with ControlNet than without it? I tried to use controlner + Hires Fix and I get Cuda even on resizing from 512x512 to 768x768. Yes I have a weak 6Gb vram card, but everything is THAT bad?

[zhmou]

From my personal experience, this is true.
CtrlNet needs an extra model running in vram to control the generation step of the original model. (Maybe two, since most preprocessors also need a model, but I guess the preprocessor model is probably swapped from vram to memory or just offloaded after the guidance image is generated)
A tip(works well for me): Adding the command line args --medvram to webui-user.bat can reduce the vram usage a lot, with very little impact on performance. Also checking the low vram in the ctrlnet plugin may help you.

[Drakmour]

I already have a medvram arg. :-D lowvram makes generations too long, several minutes for 512 512.

[cosmoshe]

Try it my way?

[valden80]

I have 4GB 1650 card, and even normally need --lowvram arg. With controlnet VRAM usage definitely higher, hires-fix often cause CUDA OOM error. Also it's make generations even slower. For example - DDIM sampler on my card typically need 5s for one it (512x768 image). With controlnet it become from 7 to 11 (type of used CT means).

So yes, it's slower, and need more VRAM. But still definitely worth it. :)

[Drakmour]

Yeah it does. But to get needed result weight shoud be 0.1-0.2 or it eats all the creativity. :-)

[valden80]

It's depends on approach. If you can draw sketch (or make 3d scene) for CT by youself - high weight it's definitely what you want, so SD just assist you. :)

[Drakmour]

Nah, I usually use only the pose and let it generate the background itself.

[ClashSAN]

If you are limited in vram, use the T2I adapters (not controlnet), only an addition 0.15gb is needed.

[creeduk]

I have 2GB of VRAM and yes takes longer but running OK. I am now trying with xformers turned off and so far so good seems a little better on results as well.

[cosmoshe]

Yes, I found that once I call controlnet, it will always use video memory, I found a way to automatically release VRAM after calling controlnet.

I added this code:

Files path:
stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py [change function: def controlnet_main_entry(self, p):]
stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\supported_preprocessor.py [change class: class Preprocessor(ABC):]