Workflow: UV texture map generation with ControlNet Image Segmentation #204
Replies: 11 comments 19 replies
-
|
Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
Holy shit. |
Beta Was this translation helpful? Give feedback.
-
|
That's exactly what I said out loud when I synthesized the first of those packaging textures - the orange juice one made with IMG2IMG, which by the way doesn't even tile properly because I had yet to activate the asymmetric tiling at that point ! |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
How did u export the uv textures in blender, like how did you make them segmented into different colors for each part? |
Beta Was this translation helpful? Give feedback.
-
|
This is marvelous and delicious! And I'm not talking about those gorgeous packagings haha! This is the type of SD application I want to see more of, please share your new ideas! 🥇🥇🥇🥇🥇 |
Beta Was this translation helpful? Give feedback.
-
|
Wow, incredible |
Beta Was this translation helpful? Give feedback.
-
|
@AugmentedRealityCat Would you please help me on that? |
Beta Was this translation helpful? Give feedback.
-
|
Video was rendered from Cinema4d - only the textures were synthesized with Stable Diffusion + ControlNet Image Segmentation |
Beta Was this translation helpful? Give feedback.
-
|
@AugmentedRealityCat |
Beta Was this translation helpful? Give feedback.
-
|
Yes. Stable Diffusion was used just to generate the 2d texture maps that were then applied on the 3d models in a different software, Cinema4d, and that's also where I made the animation with the box turning and the texture changing by going through all the packaging variations I had generated earlier. ControlNet's Semantic Segmentation was used to make sure the packaging images generated with Stable Diffusion would fit perfectly with the 3d model of the box afterwards. The texture generated with Stable Diffusion is then basically wrapped around the 3d model using a set of correspondance between each polygon on your model (XYZ coordinates for each vertex) and the part of your 2d texture that should be used to "paint" that polygon (the UV coordinates for each vertex). |
Beta Was this translation helpful? Give feedback.
-
|
Wonderful! Are there more complex examples? For example, a car mesh or an animal mesh? |
Beta Was this translation helpful? Give feedback.
-
|
@AugmentedRealityCat Can you provide me with the .gltf model file of this packaging box? Thanks ！！! |
Beta Was this translation helpful? Give feedback.
-
|
The packaging model and the corresponding mapping reference are assets that are provided as part of an online library that is accessible when you subscribe to a Cinema4d license. |
Beta Was this translation helpful? Give feedback.
-
|
honestly this .. shouldn't work :) If it does I will have to revisit my whole knowledge of this space and go into hiding to meditate. |
Beta Was this translation helpful? Give feedback.
-
|
but to be clear that's one clever idea, and I believe a model trained on a set of models with matching UVs could actually work. |
Beta Was this translation helpful? Give feedback.
-
It actually does work ! The thing is that when I first began playing with this there was almost no information available about that model and how to use it. So I tried different things, but I could not find any color reference chart that would tell me if that or this color was meant to represent a house or a car. And it's by experimenting that I discovered this "feature". I now understand how it all works - I made color reference charts for semantic segmentations models that I shared over here just a few weeks after this was posted, and wrote a few tutorial explaining how to use those. But if I had known all of that then, I probably would have never tried it, and I would have never discovered it could be used like this. Try it, you'll see, it works even though it should not, as we both know now. Ignorance is bliss sometimes ! No need to go hide anywhere and meditate, just laugh about my stupidity, and take advantage of what I found. Maybe you'll discover some other tricks to bend those semantic segmentation models even further. |
Beta Was this translation helpful? Give feedback.
-
|
What does the prompting do then? Is the prompting only to supply auxiliary information about the textures generated, like art style or similar? If the segmentation colors mean one thing, and you prompt for something else entirely, then what happens? |
Beta Was this translation helpful? Give feedback.
-
|
Is it possible to somehow influence what parts of the segmentation map the prompt applies to? Ie. redish colors are brick, blue is grass. |
Beta Was this translation helpful? Give feedback.
-
|
Okay, thanks for the replies. I found an extension to Stable Diffusion that uses multiple diffusion for different regions of an image, but geometrized. If one could take this idea, and apply it not to a square geometric region, but to a mask that's created by a color, then one could achieve stand-alone multi class prompting. You would then have to define a layer order, and merge the different output layers into a single image. It's quite limiting that the classes for the segmentation are pre-defined if you want to create UV textures I think. Thanks for the input though! |
Beta Was this translation helpful? Give feedback.
-
The in-between colors are actually used to separate objects that share the same semantic class. If you have a single car in your image, then it will get the "official" semantic color for cars. But if you have 5 cars at the same time, then each car will get a slight variation of that official color, This allows the system to treat each car as its own object. |
Beta Was this translation helpful? Give feedback.
-
I think you will like this extension by @continue-revolution then ! And this one from @Uminosachi ! |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @AugmentedRealityCat - I will check it out. Appreciate the replies |
Beta Was this translation helpful? Give feedback.
-
|
Here is the table for what colors correspond to which class: ufade20k: https://docs.google.com/spreadsheets/d/1se8YEtb2detS7OuPE86fXGyD269pMycAWe2mtKUj2W8/edit?gid=0#gid=0 From the segmentation section of https://stable-diffusion-art.com/controlnet/ |
Beta Was this translation helpful? Give feedback.
-
|
Where can I find the seg difference model? I can't really find it anywhere. |
Beta Was this translation helpful? Give feedback.
-
|
https://huggingface.co/lllyasviel/ControlNet-v1-1/tree/main I think I found it |
Beta Was this translation helpful? Give feedback.
-
Here is how to use ControlNet with the Image Segmentation model to generate unwrapped UV texture maps for your 3d objects.
This video shows 50 different packaging textures for this box, all synthesized in a few minutes. Textures were applied in Cinema4d, and the model used for this example was taken from its assets library, but this should work with any 3d application.
unwrappedbox_render_1.mp4
Workflow
Create or import your model and unwrap its UV coordinates. It is essential to have clean UVs for this to work.
Create a segmentation map by applying flat plain colors to the different surfaces of your object. Keep it simple. Reuse the same color for surfaces sharing the same properties.
Export that segmentation map as a PNG. It does not have to be square but it's a good idea to have it in a resolution that works well with SD. For this example, it was in 2048x1024, a nice 2:1 ratio that you can downscale easily to 1024x512 if required.
Start Stable Diffusion and enable the ControlNet extension.
Load your segmentation map as an input for ControlNet. Make sure you set the resolution to match the ratio of the texture you want to synthesize. I went for half-resolution here, with 1024x512.
Leave the Preprocessor to
None. Since we already created our own segmentation map there is no need to reprocess it.Set the ControlNet Model to the one containing the word Seg (for image SEGmentation)
You may have to adjust the weight parameter - in this example I cranked it up to 2.
. Write a simple prompt to describe the texture you want to generate, for example Grape Juice Packaging. You can also change the sampler, the steps, the CFG and the SD model. For this example I used the plain ema-only 1.5 model just to show you don't need anything fancy. You can also use other extensions, like in this case I wanted the texture to be seamless on the X axis, so I used the Asymmetric Tiling extension to achieve that.
Optional: if you want you can do this in IMG2IMG mode and use a picture to influence the texture map generation process.
Optional: if you want to synthesize many variations, it can be a good idea to use wildcards and other dynamic prompting methods.
Press the Generate button, repeat until you get an image that satisfies you. Here is one example of the results I got. I made hundreds of them over a few minutes by using
__fruit__ juice packagingas a wildcard promptImport that picture as a texture map and apply it to your 3d object using the same UV coordinates as those used to first generate the Image Segmentation map.
Press render and voila !
Beta Was this translation helpful? Give feedback.
All reactions