Could it be possible to run Pixtral-12b in the Comfy UI ? #4899
-
|
https://huggingface.co/mistral-community/pixtral-12b-240910 More details here: https://www.youtube.com/watch?v=PfzPfB3esG4 Is it technically possible to have Comfy UI support ( native or via custom node ) ? Could it give better images or have other capabilities beyond Flux ? |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 4 replies
-
|
Yes, similar models like Qwen2-VL *) and MiniCPM already have custom nodes. I don't think Pixtral is revolutionary in any way, just another late-fusion multimodal model that can interpret images, but output text only. Early-fusion models that can understand image and text and can also output image and text such as Meta's Chameleon and Transfusion is what's really exciting imo. *) https://github.com/IuvenisSapiens/ComfyUI_Qwen2-VL-Instruct |
Beta Was this translation helpful? Give feedback.
-
|
From reading the video description, I've got the impression it could also output images ....
|
Beta Was this translation helpful? Give feedback.
-
|
I think the youtuber is making stuff up. The architecture of the model is basically the same as other late-fusion vision-language models, a text generating LLM as a base (Mistral NeMo 12B in this case) and a vision adapter made to work in tandem with it afterwards. The early-fusion models are pretty much trained from scratch with image + text dataset. Some official slides from Mistral (would they really not mention it if it can actually generate images?): https://x.com/swyx/status/1833932883347865802 Huggingface employee's breakdown on Pixtral: |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, I also couldn't find other references about generating images. |
Beta Was this translation helpful? Give feedback.
-
|
Is there a node that runs it locally instead of api? |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
Yes:
https://github.com/ShmuelRonen/ComfyUI_pixtral_vision