Is Text Slider Model Fusion with Base Model Supported?

[liaoliaojun]

I have been working with the text slider model and attempting to fuse it with a base model for enhanced functionality. However, after several trials, I haven't noticed any significant effect or changes.

are there any specific configurations or steps that need to be taken to successfully merge the two models?
Could you provide any insights or documentation on how to achieve this fusion effectively?

[rohitgandikota]

Can you help me understand what you meant by fusion of two models? If you mean by mixing LoRA learnt weights into the model permanently there is a way to do it

There are two matrices in each LoRA (A and B). So for each LoRA module you need to do A@B and then add it to that layer weights by scaling it with appropriate alpha (lora scale) that you wish to do. Please note, if you mix them permanently you can't control the concepts as you can do with sliders. You will have to fix the scale and add them.

[chabrarahul]

Do you have idea about how to change lora scale for control editing. For each trained concept same value of scale can make higher editing change in the image for different concepts.

[rohitgandikota]

That's a good question! We too have noticed this trend and looking at possible normalization tricks to control the edits. Also a thing to note is that some concepts are strongly present in the model which makes their slider very robust, compared to concepts that are weakly present in the model.

[chabrarahul]

Thanks for answering. I have couple of questions here -

Q1 is there a way we can find out lora scale to edit the image. Right now for the same lora scale/alpha we get different editing strength for different concepts.

Q2 Can we normalize the scale so that per unit increase can have same variation in the image. For example if I change the slider value from 0 to 1 and lets say 3 to 4 the change should be same.

Q3 Is there a reason for restricting slider value from -4 to 4 in your demo and in the paper -3 to 3. I can see that for some of the images even at slider value 4, I cannot see the changes in image. Well in short is there a way till what slider we can go so that change in attributes remains realistic. Is there anything to do with rank or alpha ?