[Feature Request]: Select specific regions of the controlnet image to apply the "ControlNet is more important" or "Prompt is more important" setting

[Akis-M]

The guess mode with the "ControlNet" vs "Promp" importance setting is great. However I often find myself wishing I could tell the controlnet to focus on a specific area that the prompt/model gets bad consistently, while I can give more freedom for the prompt on other areas.

Example: Hands/feet are notoriously hard for many models. I could paint over the controlnet image those areas, and select "ControlNet is more important" for the marked areas, and then "prompt is more important" for the unmarked areas. It's a much better alternative than forcing controlnet more important on the entire image and then losing freedom from your prompt.

This would basically make ControlNet perfect.

[lbeltrame]

A workaround is to use inpaint with ControlNet. In my limited testing, it works (I picked a part of the image that I wanted redone, used canny, and I used inpaint with reasonable weight, and it improved a great deal).

[Akis-M]

That requires you to generate in text 2 image then send to inpaint then mask then generate. And if you wanna generate many text2images then you have to do x99999 masks in inpainting to fix the issues. Not very convenient unfortunately.

[Chubbly]

Is this not solved using Adetailer https://github.com/Bing-su/adetailer.git?

It allows you to use ControlNets too.

[Akis-M]

@Chubbly It doesn't. Same response as Ibeltrame. You still need to first generate your image in text2image and then go to img2img before you can use adetailer to fix imperfections. Forcing 2 different workstreams, which is what I'm trying to avoid.

Ideally you'd be able to select from the txt2image screen, somewhere around the controlnet options, which part of the controlnet image to focus more on.

[dsynclabsllc]

I was looking to do this as well. This would be super helpful when it comes to posing.

[Rkkss]

Can webui even allow you to do it?
feel like you need to combine controlNET with segment anything.