Attention Mask

[exdownloader]

Hi, thanks for your work. It's been a lot of fun training LoRAs and creating all kinds of things.

Even with perfect images and tags, It's sometimes difficult to isolate a feature or concept.
With this in mind, perhaps attention masks could allow users to indicate what parts of an image should be higher or lower attention.
An implementation of this can be found in the following repo, along with an example image:
https://github.com/7eu7d7/DreamArtist-sd-webui-extension#attention-mask

Is this something that is possible? Perhaps it would not help but I would be quite interested to find out.

Thank you

[slashedstar]

They have something like that in the original repo, but its automatic and defaults to faces:
cloneofsimo/lora#96
Can probably be adapted to custom/manual masks

[runner22k]

You mean to say, by default lora will use face recognition? or is there an option to enable/invoke it?

[recris]

I think an approach similar to the one available in webui is good and flexible enough:
AUTOMATIC1111/stable-diffusion-webui#6700

This works by using the alpha channel values of the training images for providing loss weights. This would only work for image formats that support alpha, like png. This is flexible enough to support faces or any other kind of masking / weighting, which could easily be added to the training data as a preprocessing step.

The way I see this working is by having a boolean parameter that when enabled would extract the loss weights from the input data and use those in the loss calculation step, otherwise it would fallback to current behavior (or when the input does not contain alpha).

@kohya-ss Do you think this is feasible? I can help with the development if needed.