Is Flux already usable in comfy?

[BrechtCorbeel]

Anyone tried it yet?

[CHNtentes]

4070 12G, use both t5 text encoder and flux dev unet in fp8 mode, takes about 2 min for each pic

[ltdrdata]

https://github.com/comfyanonymous/ComfyUI_examples/tree/master/flux

This feature is supported from day zero.

[tomscwills]

Using the base Flux1_dev workflow taxes my setup,
64g ram, 4090, using t5xxl_fp16. 100% Vram usage.
Will say img output is good though.

[JorgeR81]

On the other end of the hardware spectrum ...

Intel i7-7700
32 GB RAM
GTX 1070 (8GB VRAM)
Windows 10

Thanks to the "schnell" version, it's usable for me. 
This is the example workflow, ( with 1024 x 1024 image size ).
But, I'm using the [ t5xxl_fp8_e4m3fn ] text encoder, instead.

[JorgeR81]

I can also use the "dev" model, with both the model and text encoder in FP16 !

The "dev" has the same times per iteration, but it requires more steps.
With text encoder at F16, times are the same.
And with the model at FP16, it's just a little slower.

It takes a long time, but it's really nice to be able to get the maximum quality !
The preview works, so I can cancel, if I don't like the seed ...

[tpcdaz]

I have an RTX 3060 12gb edition and the schnell model works lovely.

[a3nima]

doesnt it go into lowvram mode? check the command line prompt window.

[BrechtCorbeel]

I seem to get no step closer to using it error after error with comfy right now

[AngryLoki]

For those who are interested,FLUX.1-schnell fp16 on modern CPU (Ryzen 7950x3d) is not massively slower than low-end GPU:

100%|██████████████████████████████████████████████| 4/4 [03:14<00:00, 48.66s/it]
Using split attention in VAE
Using split attention in VAE
Requested to load AutoencodingEngine
Loading 1 new model
Prompt executed in 242.54 seconds

but consumes up to 54GB RAM. Probably it casts all models to fp32 (adding --bf16-vae --bf16-unet changes nothing).

[AngryLoki]

Where does your concern come from exactly?

No concerns from my side, I'm just checking that ComfyUI does not allocate fp32 memory for bf16 model, causing OOM kills.
Safetensor files are not compressed, so my expectation is that loaded model memory consumption is the same as file size.

[CHNtentes]

what is the size of your image at the point that this happens?

1024x1024, workflow copied from https://comfyanonymous.github.io/ComfyUI_examples/flux/#flux-schnell-1

Actually, I slightly edited #3649 and with bf16 datatype (again on CPU) unet is 2 times faster:

100%|██████████████████████████████████████████████| 4/4 [01:40<00:00, 25.18s/it]
Using split attention in VAE
Using split attention in VAE
Requested to load AutoencodingEngine
Loading 1 new model
Prompt executed in 118.42 seconds

And memory consumption is 35GB, which approximately equals 23G of flux1-schnell.sft (it is already fully bfloat16) + 9.2G of t5xxl_fp16 (it should be converted from fp16 to bf16) + ae + clip_l + internal structures.

why cast fp16 to bf16? that will also cause loss, won't it?

[JorgeR81]

I've also been doing some testing on this, ( see this discussion: #4226 )
I have 32 GB RAM and 8 GB VRAM.

Even with the Comfy-org (fp8) checkpoint ( 17.2 GB ), with text encoders included, my RAM usage still reaches the limit, when loading the model, and the SSD activity spikes.
It works, but it wears down the SSD, I guess ...

[AngryLoki]

why cast fp16 to bf16? that will also cause loss, won't it?

Zen4 CPUs can't perform FP16 computations (other than convert to FP32).
Also e4m3fn has also no native support, so that's not an option. Yes, FP16->BF16 is a loss, but probably not as big as anything->e4m3fn. Ideally I need to take fp32 model and resave it as BF16, but I don't expect effect to be significant.

[RandomGitUser321]

Within the

ComfyUI/comfy/supported_models.py

Line 631 in 1c08bf3

class Flux(supported_models_base.BASE):

supported_inference_dtypes = [torch.bfloat16, torch.float32]
So it only supports them

[drulia]

ComfyUI noob here, I have downloaded fresh ComfyUI windows portable, downloaded t5xxl_fp16.safetensors and clip_l.safetensors and vae to run
FLUX.1-schnell. I'm running it using RTX 4070 Ti SUPER and system has 128GB of ram. But it takes 670 seconds to render one example image of galaxy in a bottle. https://comfyanonymous.github.io/ComfyUI_examples/flux/

Repeated runs take just as long. Running on CPU (i7-14700KF) is pretty much same story. Is this expected or I'm doing something wrong?

[JorgeR81]

There is definitely something wrong.
You should probably open an issue to get feedback from the developers.

[askmyteapot]

for a 4070ti super, you will need to downcast to fp8_e5m2 (or fp8_e4m3fn) otherwise it will be painfully slow. My guess is that you have sys ram fallback enabled too, which will slow things down even more as you exhaust your GPU Vram.

[JorgeR81]

Yeah but, there must be something else going wrong, for @drulia.
I have an older GTX 1070 (8 GB VRAM).
For me, using the fp16 model, it's slower, but only slightly.

I only have 32 GB RAM, so during model loading, I run out of RAM and the system needs to use the the page file.
#4226 (comment)

But, I can still generate over 4 times faster than that.
#4173 (comment)

[DavidSH92]

I want to test the fp8 model locally,but an error was reported ,how do I install the FluxGuidance node，comfui managr cannot find this node？

[JorgeR81]

The FluxGuidance is not a custom node.
It should be in Comfy UI, by default, if it's updated. 

[DavidSH92]

Wow~ ⊙o⊙，yesterday, I updated Comfui through ComfyUI manager and it prompted 'It has been successfully updated.' I was unable to run ComfyUI_examples successfully, but today I started Comfui and no longer reported FluxGuidance related errors. Perhaps I need to restart my computer? I'm not quite sure, thank you for your answer

[a3nima]

got prompt
[rgthree] Using rgthree's optimized recursive execution.
Requested to load FluxClipModel_
Loading 1 new model
Requested to load Flux
Loading 1 new model
loading in lowvram mode 21973.2
100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [00:39<00:00, 1.59s/it] 

can I prevent this lowvram mode (24GB VRAM) ? or doesnt it hurt? (rest of system: 64GB Sys RAM, Windows 10)

[JorgeR81]

To prevent lowvram you can try the ( smaller ) fp8 versions of the Flux models and t5 encoder ( at a slight quality loss ).
https://comfyanonymous.github.io/ComfyUI_examples/flux/
https://huggingface.co/Kijai/flux-fp8/tree/main

But you're looking pretty good ! 
Much faster than other 4090 here: #4173 (comment)

[a3nima]

thanks, Jorge! PS: 3090 to be precise :D I'm only using the workflow provided by comfyUI and updated everything incl. python before using flux.

[JorgeR81]

Hey, this user has the same GPU, and less RAM, and doesn't seem to need lowvram.
#4271 (comment)

[a3nima]

Maybe it has to do with the comfyUI version? I'm running windows standalone portable.

[JorgeR81]

I think he is using a specific --highvram cmd
You can add them, by editing your [ run_nvidia_gpu.bat ] file ( the one you click on to start Comfy UI )
#1043 (reply in thread)
#4271 (comment)

[JorgeR81]

Do you have slower generation times after the latest update ?
#4271

[JorgeR81]

Flux seems good at refining its own images.
With other models, the images tend to get burned, especially if the same seed is used. 

In this example, the skin texture looks better with 12 + 4 steps more than 25 steps.

12 + 4 steps ( workflow in the png )

25 steps

[a3nima]

doesnt this also speak for latent upscale (hiresfix) instead of using upscalers? I'm new at comfyUI, but I think I would have a workflow that has a switch like in A1111, so that I can choose to upscale with either upscaler or latent. (normally I prefer latent)

[JorgeR81]

Yes, you can add an [ Upscale Latent By ] node to upscale the latent image, between the two KSamplers ( pink wires ).
But you can also add an [ Upscale Image By ] node to do a regular upscale, after the image was generated ( blue wires ).

But you need to manually connect the nodes.
Comfy UI doesn't have "switch" nodes by default.
For that you need a custom node pack, with extra nodes, like this one.
https://github.com/ltdrdata/ComfyUI-Impact-Pack

You can install multiple custom node packs more easily with the Comfy UI Manager.
https://github.com/ltdrdata/ComfyUI-Manager

[JorgeR81]

You may need to use more steps in the first image, to get a better quality starting point, for a latent or regular upscaling,
Or you can upscale the refined image instead.