[1.1.454] ControlNet union SDXL model

[huchenlei]

ControlNet Union is a new type of ControlNet that can handle multiple input conditions

• https://huggingface.co/xinsir/controlnet-union-sdxl-1.0
• https://github.com/xinsir6/ControlNetPlus

How to use

Download model

Model can be downloaded here: https://huggingface.co/xinsir/controlnet-union-sdxl-1.0/tree/main. After download, it is recommended to rename the file as controlnet++_union_sdxl so that the control type filter can correctly handle it.

Openpose

Do not use key map with face and hand as the union controlnet does not seem to be trained with hand/face annotation

Canny

Depth

SoftEdge

Normal

Segmentation

Note: ofade20k preprocessor does not seem to work

Multi-ControlNet

Currently the multi-controlnet is not working in the optimal way described in the original paper, but you can still try use it, as it can help you save VRAM by avoid loading another controlnet for different type of control.

Unit1 setup

Unit 2 setup

Generation result

Appendix

Input images

Mask

Generation parameters (Parameters including seed are fixed for all examples)

1girl, sitting on throne,, absurdres, highres, (masterpiece:1.2), (best quality, highest quality),
Negative prompt: (lowres, low quality, worst quality:1.2), (text:1.2), watermark, (frame:1.2), deformed, ugly, deformed eyes, blur, out of focus, blurry, deformed cat, deformed, photo, anthropomorphic cat, monochrome, photo, pet collar, gun, weapon, blue, 3d, drones, drone, buildings in background, green
Steps: 30, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 7, Seed: 1413879340, Size: 1024x1024, Model hash: 5d973c1f8a, Model: animagine-xl-2.0, ControlNet 0: "Module: none, Model: controlnet++_union_sdxl [15e6ad5d], Weight: 1.0, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.0, Guidance End: 1.0, Pixel Perfect: False, Control Mode: Balanced", ControlNet 1: "Module: canny, Model: controlnet++_union_sdxl [15e6ad5d], Weight: 1.0, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 100.0, Threshold B: 200.0, Guidance Start: 0.0, Guidance End: 1.0, Pixel Perfect: False, Control Mode: Balanced", Version: v1.9.4-249-g93c00b2a

[sdbds]

xinsir6/ControlNetPlus#1
According to my issue, union used mobile sam random split instead of ADE20K

Considering the controlnet_aux repository is now hosted by huggingface, and more new research papers will use the controlnet_aux package, I think we can talk to @Fannovel16 about unifying the preprocessor parts of the three projects to update controlnet_aux.

[Fannovel16]

@sdbds It's quite challenging as comfyui_controlnet_aux has a different DWPose implementation and embeds many third party packages's fork

[sdbds]

Yes, I'm going to start experimenting with preprocessors that don't rely on third-party packages, and I'll just use what's in the repository for the rest of it.
Also controlnet_aux's DWpose implementation seems to be faster and more suitable for video frame conversion, in my previous tests.

[slashedstar]

CAn we have a update for the promax version with tile and inpainting? 👀

[ali0une]

Yes please @huchenlei !

i currently have this error with the promax version :

2024-07-14 17:47:35,187 - ControlNet - INFO - Loading model: controlnet++_union_sdxl-promax-1.0 [9460e4db]
2024-07-14 17:47:35,196 - ControlNet - INFO - Loaded state_dict from [/whatever/stable-diffusion-webui/models/ControlNet/controlnet++_union_sdxl-promax-1.0.safetensors]
2024-07-14 17:47:35,201 - ControlNet - INFO - controlnet_sdxl_config
*** Error running process: /whatever/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py
    Traceback (most recent call last):
      File "/whatever/stable-diffusion-webui/modules/scripts.py", line 825, in process
        script.process(p, *script_args)
      File "/whatever/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1228, in process
        self.controlnet_hack(p)
      File "/whatever/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1213, in controlnet_hack
        self.controlnet_main_entry(p)
      File "/whatever/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 919, in controlnet_main_entry
        model_net, control_model_type = Script.load_control_model(p, unet, unit.model)
      File "/whatever/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 436, in load_control_model
        control_model = Script.build_control_model(p, unet, model)
      File "/whatever/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 465, in build_control_model
        control_model = build_model_by_guess(state_dict, unet, model_path)
      File "/whatever/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet_model_guess.py", line 240, in build_model_by_guess
        network = PlugableControlModel(config, state_dict)
      File "/whatever/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/cldm.py", line 24, in __init__
        self.control_model.load_state_dict(state_dict, strict=False)
      File "/whatever/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2189, in load_state_dict
        raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
    RuntimeError: Error(s) in loading state_dict for ControlNet:
    	size mismatch for task_embedding: copying a param with shape torch.Size([8, 320]) from checkpoint, the shape in current model is torch.Size([6, 320]).
    	size mismatch for control_add_embedding.linear_1.weight: copying a param with shape torch.Size([1280, 2048]) from checkpoint, the shape in current model is torch.Size([1280, 1536]).

[huchenlei]

Tile supported, but inpaint still has issue. #2998

Need to wait official inference script for inpaint from union controlnet author to see what is wrong for inpaint.

[TKoestlerx]

Edit: Outpaint is working with the preprocessor set to "none".
Inpaint is not working correctly as it seems to draw information from the masked out area.

Edit2: You can do inpainting if you use the "Effective Region Mask" Feature to upload the mask instead of the default inpainting map.

[Gavr728]

I tried to do outpainting with the preprocessor set to none, but surprisingly it also changed colors on the original picture while preserving all the other details in tact.

[sdbds]

Tile supported, but inpaint still has issue. #2998

Need to wait official inference script for inpaint from union controlnet author to see what is wrong for inpaint.

They put on other place...
https://github.com/xinsir6/ControlNetPlus/blob/main/promax/controlnet_union_test_inpainting.py