please continue researching to make it work better, and work well with controlnet

[mikebilly]

I get very good results with SDXL and Hidiffusion, but when I connect with Controlnet, it doesn't follow controlnet much, please research and make them work nicely together, I really like hidiffusion

[blepping]

unfortunately, i don't really have any ideas for improving it at the moment. i did add one workaround which might have improved it but i didn't get any feedback about whether or not it helped.

further improvements probably would have to come from someone else contributing changes or even just helping me out with information about a better approach.

generally speaking, you probably don't really need to use Hidiffusion simultaneously with controlnet. if you're already guiding the model with major structural details, you're pretty much already solving the same thing Hidiffusion is for. you can possibly try using Hidiffusion and controlnet in separate phases, like the first 25% controlnet, then Hidiffusion for a while or vice versa.

using restart sampling might also help, basically give the model a second chance to either clean up problems caused by hi res generation if they exist or let the controlnet effect have more chance to apply. (note: you generally don't want to restart back into the Hidiffusion effect though)