[Major Update] BitsandBytes Guidelines and Flux

[lllyasviel]

(Before we start, Forge now supports UI presets like this. Click this GIF to zoom in.)

---

(Again, before we start, to the best of my knowledge, I am the first one who made the BitsandBytes low bit acceleration actually works in a real software for image diffusion. You can cite this page if you are writing a paper/survey and want to have some nf4/fp4 experiments for image diffusion models.) -> Update Aug 12: It seems that @sayakpaul is the real first one -> they made so many contributions 😹

(BitsandBytes is a standard low-bit accelerator that are already used by most Large Language Models like LLama, Phi, etc)

@Misc{forge_bnb,
  author = {Forge},
  title  = {Forge GitHub Page about Flux, BitsandBytes, and Image Diffusion},
  year   = {2024},
}

---

Flux Checkpoints

The currently supported Flux checkpoints are

1. flux1-dev-bnb-nf4-v2.safetensors Full flux-dev checkpoint with main model in NF4. <- Recommended
2. flux1-dev-fp8.safetensors Full flux-dev checkpoint with main model in FP8.

Alternative: If you are looking for running original raw Flux, or GGUF, or any checkpoints that need loading separated modules like clip, t5, ae, etc, go to this post instead. <- If you want to run some 6GB flux unets, also use this link.

Some info about NF4:

(i) NF4 is significantly faster than FP8. To run Flux, NF4 is significantly faster than FP8 on 6GB/8GB/12GB devices and slightly faster for >16GB vram devices. For GPUs with 6GB/8GB VRAM, the speed-up is about 1.3x to 2.5x (pytorch 2.4, cuda 12.4) or about 1.3x to 4x (pytorch 2.1, cuda 12.1). I test 3070 ti laptop (8GB VRAM) just now, the FP8 is 8.3 seconds per iteration; NF4 is 2.15 seconds per iteration (in my case, 3.86x faster). This is because of less swap and partially because of NF4's native bnb.matmul_4bit rather than torch.nn.functional.linear: casts are avoided and computation is done with many low-bit cuda tricks. (Update 1: bnb's speed-up is less salient on pytorch 2.4, cuda 12.4. Newer pytorch may used improved fp8 cast.) (Update 2: the above number is not benchmark - I just tested very few devices. Some other devices may have different performances.) (Update 3: I just tested more devices now and the speed-up is somewhat random but I always see speed-ups - I will give more reliable numbers later!)

(ii) NF4 weights are about half size of FP8.

(iii) NF4 may outperform FP8 (e4m3fn/e5m2) in numerical precision, and it does outperform e4m3fn/e5m2 in many (in fact, most) cases/benchmarks.

(iv) NF4 is technically granted to outperform FP8 (e4m3fn/e5m2) in dynamic range in 100% cases.

This is because FP8 just converts each tensor to FP8, while NF4 is a sophisticated method to convert each tensor to a combination of multiple tensors with float32, float16, uint8, int4 formats to achieve maximized approximation.

(Do not confuse FP8 with bnb-int8! In large language models, when people say "8-bits better than 4 bits", they are (mostly) talking about bnb’s 8-bit implementation, which is a more sophisticated method that also involve storing chunked float32 min/max norms. The fp8 here refers to the naked e4m3fn/e5m2 without extra norms. ) <- You can say that bnb-8bit is more precise than nf4. But e4m3fn/e5m2 may not.

For example, this is the Flux FP8 state dict (you can see that each weight only has one FP8 tensor):

This is the NF4 state dict, you can see that each weight is stored in 6 different tensors, and these tensors are in different precisions including float32, uint8, etc:

So, do not be surprised if you find out that NF4 is actually more precise than FP8 despite its smaller size. And, do not argue too much if you still find FP8 more precise in some other cases ...

(The short version of the real method is that NF4 sort values in a tensor and split in into chunks and compute the abs norm of each chunk. Such norm is stored in higher precision. The “N” in NF4 means “Nested” (Update: "NormalFlow"), which means this compression happens two times. The first time it processes tensor weights, and the second time it processes extracted norms.) <- correct me if I am wrong

(The long version of the method is here)

Using Flux

If your device supports CUDA newer than 11.7 then you can use NF4. (Most RTX 3XXX/4XXX GPUs supports this.) Congratulations. Enjoy the speed. In this case, you only need to download the flux1-dev-bnb-nf4.safetensors.

If your device is GPU with GTX 10XX/20XX then your device may not support NF4, please download the flux1-dev-fp8.safetensors.

In the UI, Forge gives you an option to force the loading weight type:

However, in most cases, you can just set it to Auto and then it will use the default precision in your downloaded checkpoint.

Using this option, you can even try SDXL in nf4 and see what will happen - in my case SDXL now really works like SD1.5 fast and images are spilling out!

Important: Do Not Load FP8 checkpoint flux1-dev-fp8.safetensors with NF4 Option! If you do this, you will still get images eventually with no errors. However, you will waste 30 seconds to de-quantize the weight into fp16 and then waste 60 seconds quantize it again into NF4. And finally, your results are worse than flux1-dev-bnb-nf4.safetensors because the model is quantized twice, and quality is lost.

Getting Theoretical Upper Bound of Inference Speed

Forge's default preset should already be very fast (the default preset is already among the fastest ones, with about +/- 5% speed variations, now on my several devices even using FP8, compared to all SD-related software without special packages like TensorRT, if timed using my phone outside screen).

But you can make it even faster and really get the theoretical upper bound of inference speed for every different GPU by tuning the UI.

To begin with, this is your Flux model, and lets use the FP8 model as an example, where the base diffusion model is about 11 or 12GB.

Then lets say this is your system, you have 8GB VRAM, 32GB CPU memory, and a block called "shared GPU memory" with 16GB

Then you want to move the model to GPU. But the model is 12GB, bigger than your GPU memory 8GB. Then how?

The anwser is to split the model to two parts. One part to GPU, one other to a "swap" location.

If you select CPU as swap location, your model will load to CPU memory and GPU:

If you select "Shared" as swap location, your model will load to GPU and shared memory:

On newer devices, the "Shared" offload/swap is about 15% faster than CPU swap. However, some other devices reports crash when using "Shared" swap.

Then you can select the maximum memory to load model to GPU.

Below is an example using Flux-dev in diffusion:

Another example:

Larger GPU Weights means you get faster speed. However, if the value is too large, you will fallback to some GPU problems and the speed will decrease to like 10x slower.

Smaller GPU Weights means you get slower speed. However, you will be able to diffuse larger images, because now you have more free VRAM.

Okay then the last thing, Swap Method:

Queue: you will load a layer to GPU, and then compute, and then load another layer, and then compute, ... just like it is a queue.

ASYNC: You will have two workers. One worker always compute layers. One worker always load layers to GPU. They work at the same time.

Some experiments report that the ASYNC method is 30% faster than Queue. However, there is a drawback that one worker may mistakenly moved too many layers to GPU, making the other worker cannot have enough GPU memory to compute. In that case, the speed will be suddenly 10x slower.

Based on the above info, you can now tune the best config for your device:

Once you are done, you are on the theoretical upper bound of inference speed for your device (without using methods like TensorRT that degrade compability to like some control models).

Finally, you may need to turn on system swap if it crash:

Distilled CFG Guidance

Flux-dev is a distilled model. It is recommended to set CFG=1 and then do not use negative prompts. Using “Distilled CFG Guidance” instead. The default value is 3.5.

Note that if CFG=1, the UI of negative prompt will be greyed out.

Sanity Check

Finally, use this as a sanity check! (thanks for the prompt)

Astronaut in a jungle, cold color palette, muted colors, very detailed, sharp focus

Steps: 20, Sampler: Euler, Schedule type: Simple, CFG scale: 1, Distilled CFG Scale: 3.5, Seed: 12345, Size: 896x1152, Model: flux1-dev-bnb-nf4-v2

Make sure that you get similar image:

---

Update 1:

Hi everyone, after tested with more devices, the speed-ups of bnb are more random than I thought. I will update more reliable numbers later. Thanks and do not attack this!

[lllyasviel]

By the way, for old models, forge perfectly reproduce A1111 1.10 for all advanced features like prompt grammar, break, embeddings, loras, doras etc:

Forge:

Automatic 1111 original 1.10

[Mirabilis1729]

Hi @lllyasviel

One issue I have noticed is the 'Hires fix: show hires checkpoint' which looks like this in A1111:

Doesn't show up in Forge:

Even if you have it enabled in the uI settings.

Might not seem like an obvious need, but I've found it quite useful to be able to switch over to a different model at times in order to introduces a more stylistic look to outputs at the upscale stage and it would be very cool to see in Forge if possible.

[zethfoxster]

I too was hoping i could use it so i can switch from flux to an sd15 model and do basically img2img from main page to take advantage of fluxes extreme understanding and my million character loras lol

[Mirabilis1729]

I too was hoping i could use it so i can switch from flux to an sd15 model and do basically img2img from main page to take advantage of fluxes extreme understanding and my million character loras lol

Yeah it's a very cool feature that I don't think a lot of people either are aware of or understand, but it's really neat way of mixing things up.

[MitsuhaKitsune]

Thats a beautifull and totally unexpected (people think that forge are died) update, a lot of thanks @lllyasviel ^^

Now that we have a selector, I think it's good separate SD, XL and Flux models/loras depending on selected option (for example if I select XL only show XL/Pony checkpoints, loras, on the list and hide SD and Flux options in dropdowns).

If you consider it good idea, I don't have so much experience on gradio but I can try to do a PR for it.

[vihangasa14]

I have tested this sanity check in 6GB 3050 Vram and I took 2.5 mins

[Jonseed]

Hmm, I tried that prompt and seed on my 3060 12GB, using bnb-nf4 model, and this is what I get (took about 1.5 min). Quite a bit different... 🤔

[iamrohitanshu]

@Jonseed The only thing that seems to be different is your forge version.
Yours is Version: f2.0.1v1.10.1-previous-223-g86ee2d94
EDIT: might be the RNG.

[StanislawKarnacky]

What are your settings? Diffusion in Low Bits, Swap Method, Swap Location, GPU Weights (MB), could you share, please? In my case time goes up to 12+ minutes :(

[BerdoRod]

here is mine,

same prompt using:
NVIDIA GeForce RTX 2080 SUPER (Gigabyte) 8 Gbs
AMD Ryzen 9 3900X - RAM 32.0GB Dual-Channel DDR4 (17-17-17-39

[CCpt5]

You're ridiculously brilliant. Been following you coding run the past few days.....just updated. Really nothing to say - just other wordly skill/passion.

One thing I did want to ask (mention?) is that Ostris released a new LoRA training tool and I think there is some variation about the formatting of Flux LoRA. He used a formatting that required an update by Comfy earlier today. My first Flux LoRA works on Comfy atm but not on Forge 2.0. See his tweet here which links to the PR he submitted on ComyUI: https://x.com/ostrisai/status/1822367393555030487

Error is:

[LORA] LoRA version mismatch for KModel: d:\stable-diffusion-webui\models\Lora\Arnold-Person-LoRA-FLUX.safetensors███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:13<00:00,  1.51it/s]
Skipping unconditional conditioning when CFG = 1. Negative Prompts are ignored.
distilled_cfg_scale = 3.5
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.57it/s]
Total progress: 100%|████████████████████████████████████████████████████████████████████

If this is an easy fix I'd really appreciate it. (And if it happens to be user error on my part please forgive me)

[E2GO]

Same problem using custom made LORA from Ostrisai

[rafiislambd]

please FLUX.1-schnell model convart to NF4... thanks

[Newtrial99]

imo converting this https://huggingface.co/drbaph/FLUX.1-schnell-dev-merged/tree/main to NF4 would be even better, it somewhat struggles with text compared to dev, but it has much better prompt adherence compared to Schnell imo, that too at 4 steps.

[peppepascale]

this would be really great. i second this

[MNeMoNiCuZ]

Converting FLUX.1-schnell original model is actually still more useful, since it has different licence than the Dev-model.

If you use the schnell-dev-merged, you get the dev-licence, which is not as friendly.

[DenOfEquity]

silveroxides has done this: https://huggingface.co/silveroxides/flux1-nf4-weights/blob/main/flux1-schnell-bnb-nf4.safetensors

While it's downloading, open up the webui install directory and go to webui\backend\huggingface\black-forest-labs and make a directory FLUX.1-schnell. Copy everything from the FLUX.1-dev directory into new directory.

[Swarzox]

imo converting this https://huggingface.co/drbaph/FLUX.1-schnell-dev-merged/tree/main to NF4 would be even better, it somewhat struggles with text compared to dev, but it has much better prompt adherence compared to Schnell imo, that too at 4 steps.

any update about this?

[Panchovix]

Hi there, many thanks for the work! Any chance to support FP16 Flux?

[Ling-APE]

Could this be implemented in Comfyui?

[Panchovix]

Pardon my ignorance, but doesn't Comfy already support flux? Or do you mean the optimizations?

[Ling-APE]

I think in this case it would be the ability to load nf4 weight in ComfyUI. And also the features like async, but I guess I should wait comfyUI's developers do that instead?

[Panchovix]

Comfy devs will probably check this thread and implement it shortly, but now their backend is different to the Forge backend, so they have to adapt the ability to use nf4.

[4lt3r3go]

should be already working with new nodes i saw somewere on reddit

[Pizzawookiee]

How would I convert an SDXL checkpoint to NF4 to avoid the 60s quantizing the model from fp16?

[Jonseed]

I don't know how to convert the checkpoint, but just selecting NF4 diffusion makes SDXL models about 35% faster for me. It doesn't replicate the same seed exactly though.

[noise2014]

what gpu u have ?

[Jonseed]

I have 3060 12GB.

[arcanite24]

Is this using T5?

[WingeD123]

comparing to the file size, T5 fp8 seems included in the safetensor

[mirek190]

T5xx 8 bit is really decreasing a picture quality from my tests . Easily noticeable hands deformation , etc

[FurkanGozukara]

Yes T5 FP16 better

[Jonseed]

The model card says "T5xxl in fp8e4m3fn" so it is using T5 fp8: https://huggingface.co/lllyasviel/flux1-dev-bnb-nf4

[gonewilds]

yeah they are using T5 fp8 which decrease the quality, just give it some time.. there will be fp16 version anyway

[mercantigo]

Hi guys, Any clue why I'm getting different result in my sanity check? 4090 in Stability Matrix here, xformers disabled.

[bews]

Same results on mine 4090:

Random number generator source GPU:

Random number generator source CPU:

Both different from the sanity check.

[greendra8]

Different on my 3070

[nickaubert]

I got exactly the same image as @HMRMike above on my 3090. As far as I can tell, the sanity check settings are all the same.

[f-rank]

Almost exact same as @bews with cpu generator. Tint on helmet reflection and some other small details seem diff though.
Still a bit diff from sanity check. On a 3070 8 GB and a 4060 16 GB.
I see small variations on others too. Dunno what am not doing or doing wrong.

[HMRMike]

I got exactly the same image as @HMRMike above on my 3090. As far as I can tell, the sanity check settings are all the same.

Yeah it seems the original check just used GPU RNG source so we cannot possibly recreate it with our different cards.
CPU setting recreates for example @bews's result perfectly. So it seems all is well

[WingeD123]

I see GOAT!

[sashasubbbb]

Is there a way to check if a GPU supports NF4? (2070 Super, to be exact.)

[JorgeR81]

@Mozer You may need to update your nvidia driver
I haven't tried NF4 yet, but my 1070 seems to have the required cuda version.
Last year, Windows 10 just updated my nvidia driver, without any warning !
I only noticed because my monitor was back for a second. 
But now it seems I have the latest CUDA version available around that time.

@kyism Did your times improve with NF4 ?
Have you tried Comfy UI ?

[kyism]

@JorgeR81 i only use forge.
nf4 is 4.6 times faster than fp8 with equal image quality.

[bli22]

In my 8GB 2070s, Forge work with fp4, 1024 x 1024 pixels, the pict be generate about 320 seconds. The result is approximate to comfyui with fp8

[unalkalkan]

Hey @kyism, how long does it take for you to generate 1024x1024 on your 1070?
I have a 1080 and I was not sure if it supported nf4, since it was trying to generate in 15+ mins.

Also what is your pytorch and cuda versions. I'm at Cuda 12.0 with torch==2.3.1. Also, my nvidia driver is a bit old (525.125.06 Linux), so I was suspecting that.

[kyism]

@unalkalkan torch231+cu121, win11, outdated pc(7years)

I won't use flux again until I upgrade to a 40 series :p

[lllyasviel]

Hi everyone, after tested with more devices, the speed-ups of bnb are more random than I thought. I will update more reliable numbers later. Thanks and do not attack this!

[DexterHuang]

That's an amazing update @lllyasviel
But there seems to be a bug when trying to use the new nf4 flux model

\webui\backend\attention.py", line 314, in attention_xformers mask_out[:, :, :mask.shape[-1]] = mask RuntimeError: The expanded size of the tensor (1) must match the existing size (64) at non-singleton dimension 0. Target sizes: [1, 256, 256]. Tensor sizes: [64, 256, 256] The expanded size of the tensor (1) must match the existing size (64) at non-singleton dimension 0. Target sizes: [1, 256, 256]. Tensor sizes: [64, 256, 256]

I have used --xformer in command arg, if I don't it will crash when I try to start, not sure why as well.. below is error when I don't use --xformer:
D:\workplace\webui_forge_cu121_torch21\system\python\lib\site-packages\transformers\utils\hub.py:127: FutureWarning: Using TRANSFORMERS_CACHEis deprecated and will be removed in v5 of Transformers. UseHF_HOME instead. warnings.warn( Traceback (most recent call last): File "D:\workplace\webui_forge_cu121_torch21\webui\launch.py", line 51, in <module> main() File "D:\workplace\webui_forge_cu121_torch21\webui\launch.py", line 47, in main start() File "D:\workplace\webui_forge_cu121_torch21\webui\modules\launch_utils.py", line 544, in start import webui File "D:\workplace\webui_forge_cu121_torch21\webui\webui.py", line 21, in <module> initialize.imports() File "D:\workplace\webui_forge_cu121_torch21\webui\modules\initialize.py", line 32, in imports from modules import processing, gradio_extensions, ui # noqa: F401 File "D:\workplace\webui_forge_cu121_torch21\webui\modules\processing.py", line 19, in <module> from modules import devices, prompt_parser, masking, sd_samplers, lowvram, infotext_utils, extra_networks, sd_vae_approx, scripts, sd_samplers_common, sd_unet, errors, rng, profiling File "D:\workplace\webui_forge_cu121_torch21\webui\modules\sd_samplers.py", line 5, in <module> from modules import sd_samplers_kdiffusion, sd_samplers_timesteps, sd_samplers_lcm, shared, sd_samplers_common, sd_schedulers File "D:\workplace\webui_forge_cu121_torch21\webui\modules\sd_samplers_kdiffusion.py", line 5, in <module> from modules import sd_samplers_common, sd_samplers_extra, sd_samplers_cfg_denoiser, sd_schedulers, devices File "D:\workplace\webui_forge_cu121_torch21\webui\modules\sd_samplers_common.py", line 6, in <module> from modules import devices, images, sd_vae_approx, sd_samplers, sd_vae_taesd, shared, sd_models File "D:\workplace\webui_forge_cu121_torch21\webui\modules\sd_models.py", line 21, in <module> from backend.loader import forge_loader File "D:\workplace\webui_forge_cu121_torch21\webui\backend\loader.py", line 16, in <module> from backend.nn.vae import IntegratedAutoencoderKL File "D:\workplace\webui_forge_cu121_torch21\webui\backend\nn\vae.py", line 3, in <module> from backend.attention import attention_function_single_head_spatial File "D:\workplace\webui_forge_cu121_torch21\webui\backend\attention.py", line 12, in <module> import xformers ModuleNotFoundError: import of xformers halted; None in sys.modules

[navytut2]

Facing exactly same error as above, though with FP8

Edit: Solved it by adding --disable-xformers to webui file. Working fine now.

[DaveScream]

Hi everyone, after tested with more devices, the speed-ups of bnb are more random than I thought. I will update more reliable numbers later. Thanks and do not attack this!

Do you have plans to make TensorRT version of NF4?

[IndigoDosSantos]

Great update, thanks a lot!

Inpainting/soft-inpainting with Flux doesn't work for me yet.

[IndigoDosSantos]

Both are supported now, thanks a lot, @lllyasviel ! 💖

[1onica1]

how?

[ivanoff13]

img2img does not work((

[IndigoDosSantos]

It is supported now!

[Newtrial99]

Thanks a lot for your hard work, would it be possible for you to convert this to NF4 as well ? https://huggingface.co/drbaph/FLUX.1-schnell-dev-merged/tree/main pls, as it can do decent quality better than schnell in 4 steps

[Swarzox]

would be amazing

[ViratX]

Bump!

[tomakorea]

Did someone got Lora to work with GGUF Q8 Text encoder with Flux Dev on 24Gb GPU ? It crashes whatever the settings I'm using. I'm on Debian. I tried to setup the GPU Weights at 21Gb, then 16Gb, then 15Gb, it always do an Out of Memory error. I flushed my Vram and my System RAM (32Gb) before launching Forge. Any ideas? I'm using Forge in a Conda env.

By the way, does the flux1-dev-bnb-nf4-v2 outperform the Full Dev model + Q8 Text encoder ?

[Xenodermus]

Try with Diffusion in Low Bits set for either "Automatic (f16 LoRa)" or "bnb-nf4 (fp16 LoRA)" . Stopped having Lora Issues after that.

[tomakorea]

Try with Diffusion in Low Bits set for either "Automatic (f16 LoRa)" or "bnb-nf4 (fp16 LoRA)" . Stopped having Lora Issues after that.

I tried with bnb-nf4 (fp16 LoRA), it works, but does it lower the output quality? It seems like the result is very different compared to Automatic FP16 with the same seed. When using Automatic FP16, unfortunately it still crashes with the majority of LoRas

[dan4ik94]

I think you should disable those cmdline arguments, I'm not using them personally on my 3060 and haven't had any crashes on NF4 and other types.

[inglob]

I'm happy with the results, considering that my hardware is not close to optimal, RTX2060 6g, 32gb ram, ryzer 7 2700

[MitsuhaKitsune]

Thats a very good time for that hardware I guess

[postpunkthesmiths]

I'm getting exactly the same image, using original (not quantized) version flux1-dev and I'm concerned with it. Isn't it supposed to perform differently?) How is this possible?

Astronaut in a jungle, cold color palette, muted colors, very detailed, sharp focus
Steps: 20, Sampler: Euler, Schedule type: Simple, CFG scale: 1, Distilled CFG Scale: 3.5, Seed: 12345, Size: 896x1152, Model hash: 3f97fdc57a, Model: flux1-dev, Version: f2.0.1v1.10.1-previous-546-gf4d5e8ca, Module 1: ae, Module 2: clip_l, Module 3: t5xxl_fp16

Update: I was too quick, zooming up showed some divergences and subtracting one from another confirmed it. But anyway the level of matching is incredible. It might be a strong argument for using quantized version.

[phongthuong]

Hello! I am using forge version 231 and have this error when using controlnet. I am using windows, please help me.
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>

[ETCore7]

You probably have no certificate installed, so a secure connection cannot be made.

Open a CMD box and type:
python -m pip install certifi
and hit enter. If this doesn't work try:
pip install certifi

Source: https://www.geeksforgeeks.org/how-to-install-and-use-ssl-certificate-in-python/

[phongthuong]

thanks for your help, i found this on google, did it and it didn't work, looking forward to finding another way

[ETCore7]

Not sure how you are running Forge, but if you have an NVIDIA GPU it is also available in Pinokio. Installing AI apps requires one click, all variables and paths are set automatically.
https://pinokio.computer/

In case it is a stand alone installation you may want to try this. People had similar errors, aparently certificates could not be installed due to a missing folder.
https://www.reddit.com/r/StableDiffusion/comments/y7uqgv/comment/j2uj7hd/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

[Dean-ZhenYao-Wang]

[tmprabubiz]

I have installed Forge and all WebUI's in F: drive (due to space constraint)
==========================================================.
Saved the test.py file in C://program files//test_python_for_forge - Activated venv//scripts in Forge and executed the file and got this response
.............................................
(venv) C:\Program Files\Test_Python_For_Forge>python test.py
-- pip install --upgrade certifi
Requirement already satisfied: certifi in f:\forge\webui\venv\lib\site-packages (2024.8.30)
Traceback (most recent call last):
File "C:\Program Files\Test_Python_For_Forge\test.py", line 44, in
main()
File "C:\Program Files\Test_Python_For_Forge\test.py", line 31, in main
relpath_to_certifi_cafile = os.path.relpath(certifi.where())
File "C:\Users\REVIVE\AppData\Local\Programs\Python\Python310\lib\ntpath.py", line 744, in relpath
raise ValueError("path is on mount %r, start on mount %r" % (
ValueError: path is on mount 'F:', start on mount 'C:'
.............................................................
.
Then saved the test.py in F://Forge - ran that file and received the following response
............................................................................
(venv) F:\FORGE\Test_Python_For_Forge>python test.py
-- pip install --upgrade certifi
Requirement already satisfied: certifi in f:\forge\webui\venv\lib\site-packages (2024.8.30)
Traceback (most recent call last):
File "F:\FORGE\Test_Python_For_Forge\test.py", line 44, in
main()
File "F:\FORGE\Test_Python_For_Forge\test.py", line 31, in main
relpath_to_certifi_cafile = os.path.relpath(certifi.where())
File "C:\Users\REVIVE\AppData\Local\Programs\Python\Python310\lib\ntpath.py", line 744, in relpath
raise ValueError("path is on mount %r, start on mount %r" % (
ValueError: path is on mount 'F:', start on mount 'C:'
.....................................................................................................................................................
PLEASE GUIDE TO SOLVE THIS ISSUE,

[Dean-ZhenYao-Wang]

I have installed Forge and all WebUI's in F: drive (due to space constraint) ==========================================================. Saved the test.py file in C://program files//test_python_for_forge - Activated venv//scripts in Forge and executed the file and got this response ............................................. (venv) C:\Program Files\Test_Python_For_Forge>python test.py -- pip install --upgrade certifi Requirement already satisfied: certifi in f:\forge\webui\venv\lib\site-packages (2024.8.30) Traceback (most recent call last): File "C:\Program Files\Test_Python_For_Forge\test.py", line 44, in main() File "C:\Program Files\Test_Python_For_Forge\test.py", line 31, in main relpath_to_certifi_cafile = os.path.relpath(certifi.where()) File "C:\Users\REVIVE\AppData\Local\Programs\Python\Python310\lib\ntpath.py", line 744, in relpath raise ValueError("path is on mount %r, start on mount %r" % ( ValueError: path is on mount 'F:', start on mount 'C:' ............................................................. . Then saved the test.py in F://Forge - ran that file and received the following response ............................................................................ (venv) F:\FORGE\Test_Python_For_Forge>python test.py -- pip install --upgrade certifi Requirement already satisfied: certifi in f:\forge\webui\venv\lib\site-packages (2024.8.30) Traceback (most recent call last): File "F:\FORGE\Test_Python_For_Forge\test.py", line 44, in main() File "F:\FORGE\Test_Python_For_Forge\test.py", line 31, in main relpath_to_certifi_cafile = os.path.relpath(certifi.where()) File "C:\Users\REVIVE\AppData\Local\Programs\Python\Python310\lib\ntpath.py", line 744, in relpath raise ValueError("path is on mount %r, start on mount %r" % ( ValueError: path is on mount 'F:', start on mount 'C:' ..................................................................................................................................................... PLEASE GUIDE TO SOLVE THIS ISSUE,

The error message you sent already contains the solution. Please read your error message carefully.

[musecollaboration]

When I decided to change the flux1-dev-bnb-nf4-v2.safetensors model to another one, I got an error AssertionError: You do not have CLIP state dict!, please tell me how to fix it.

[Paharven-Salin]

for anyone coming across this, read here, solves this issue :)

#1050

[ABSOMA021085]

Hello every one, I am new to this topic of generating images and I am using webui_forge_cu121_torch231 and if I can generate images with the models and files that come by default but when I download from civitai models or tensor art it recognizes them if they appear in the list but when I try to generate them I get this error. ValueError: Failed to recognize model type!

and in the console it looks like this (please someone help me I have months trying to fix it even resorted to chat gpt already reinstalled and configured hundreds of times the 3 versions of webui_forge)

Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: f2.0.1v1.10.1-previous-561-g82eb7566
Commit hash: 82eb756
Launching Web UI with arguments: --cuda-stream
Total VRAM 24575 MB, total RAM 65461 MB
pytorch version: 2.3.1+cu121
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 3090 : native
Hint: your device supports --cuda-malloc for potential speed improvements.
VAE dtype preferences: [torch.bfloat16, torch.float32] -> torch.bfloat16
CUDA Using Stream: True
D:\webui_forge_cu121_torch231\system\python\lib\site-packages\transformers\utils\hub.py:127: FutureWarning: Using TRANSFORMERS_CACHE is deprecated and will be removed in v5 of Transformers. Use HF_HOME instead.
warnings.warn(
Using pytorch cross attention
Using pytorch attention for VAE
ControlNet preprocessor location: D:\webui_forge_cu121_torch231\webui\models\ControlNetPreprocessor
Loading additional modules ... done.
2024-10-10 13:47:09,145 - ControlNet - INFO - ControlNet UI callback registered.
Model selected: {'checkpoint_info': {'filename': 'D:\webui_forge_cu121_torch231\webui\models\Stable-diffusion\Elyse_Aly_1_6_16.TA_trained.safetensors', 'hash': '27689b84'}, 'additional_modules': ['D:\webui_forge_cu121_torch231\webui\models\VAE\ae.safetensors'], 'unet_storage_dtype': None}
Using online LoRAs in FP16: False
Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch().
Startup time: 78.3s (initial startup: 0.2s, prepare environment: 18.7s, import torch: 38.7s, initialize shared: 1.3s, other imports: 4.8s, setup gfpgan: 0.3s, list SD models: 0.7s, load scripts: 6.6s, initialize google blockly: 3.7s, create ui: 2.8s, gradio launch: 1.4s).
Environment vars changed: {'stream': False, 'inference_memory': 1024.0, 'pin_shared_memory': False}
[GPU Setting] You will use 95.83% GPU memory (23551.00 MB) to load weights, and use 4.17% GPU memory (1024.00 MB) to do matrix computation.
Loading Model: {'checkpoint_info': {'filename': 'D:\webui_forge_cu121_torch231\webui\models\Stable-diffusion\Elyse_Aly_1_6_16.TA_trained.safetensors', 'hash': '27689b84'}, 'additional_modules': ['D:\webui_forge_cu121_torch231\webui\models\VAE\ae.safetensors'], 'unet_storage_dtype': None}
[Unload] Trying to free all memory for cuda:0 with 0 models keep loaded ... Done.
Traceback (most recent call last):
File "D:\webui_forge_cu121_torch231\webui\backend\loader.py", line 270, in forge_loader
state_dicts, estimated_config = split_state_dict(sd, additional_state_dicts=additional_state_dicts)
File "D:\webui_forge_cu121_torch231\webui\backend\loader.py", line 238, in split_state_dict
guess = huggingface_guess.guess(sd)
File "D:\webui_forge_cu121_torch231\webui\repositories\huggingface_guess\huggingface_guess_init_.py", line 7, in guess
result.unet_key_prefix = [unet_key_prefix]
AttributeError: 'NoneType' object has no attribute 'unet_key_prefix'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\webui_forge_cu121_torch231\webui\modules_forge\main_thread.py", line 30, in work
self.result = self.func(*self.args, **self.kwargs)
File "D:\webui_forge_cu121_torch231\webui\modules\txt2img.py", line 123, in txt2img_function
processed = processing.process_images(p)
File "D:\webui_forge_cu121_torch231\webui\modules\processing.py", line 800, in process_images
p.sd_model, just_reloaded = forge_model_reload()
File "D:\webui_forge_cu121_torch231\system\python\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "D:\webui_forge_cu121_torch231\webui\modules\sd_models.py", line 503, in forge_model_reload
sd_model = forge_loader(state_dict, additional_state_dicts=additional_state_dicts)
File "D:\webui_forge_cu121_torch231\system\python\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "D:\webui_forge_cu121_torch231\webui\backend\loader.py", line 272, in forge_loader
raise ValueError('Failed to recognize model type!')
ValueError: Failed to recognize model type!
Failed to recognize model type!

[ABSOMA021085]

Do you know what I can do to tune the models so that they work for me, at least the ones I train?

[ETCore7]

FluxGym is a loras trainer for Flux, it's in Pinokio too. I haven't really tried it (it takes a lot of time with a 3080), but it is pretty straight forward. There are Youtube videos with demonstrations.

You add a batch of images with a certain style or thing or face, add a unique trigger word, and let FluxGym make image descriptions. You then can go through the descriptions to delete irrelevant 'noise' from the texts, and start training. After many hours (on a 3080 at least) you'll have a lora that you can use. You trigger the lora with your keyword, like "P@ul makes cake" if P@ul is your friend Paul. You may need finetuning regarding under- or overtraining. By default FluxGym saves a lora every 4 rounds. To get a feeling it's best to make it save loras every round and then test with each generation. One of the demos on Youtube was with photos of a pet goat, it looked good.

[ABSOMA021085]

FluxGym is a loras trainer for Flux, it's in Pinokio too. I haven't really tried it (it takes a lot of time with a 3080), but it is pretty straight forward. There are Youtube videos with demonstrations.

You add a batch of images with a certain style or thing or face, add a unique trigger word, and let FluxGym make image descriptions. You then can go through the descriptions to delete irrelevant 'noise' from the texts, and start training. After many hours (on a 3080 at least) you'll have a lora that you can use. You trigger the lora with your keyword, like "P@ul makes cake" if P@ul is your friend Paul. You may need finetuning regarding under- or overtraining. By default FluxGym saves a lora every 4 rounds. To get a feeling it's best to make it save loras every round and then test with each generation. One of the demos on Youtube was with photos of a pet goat, it looked good.

I have a rtx 3090 and a ryzen 9 5900X the truth is that I know I can run that and I bring as a concern that I want to train models with family photos one by one, and then make family photos and send them to my family, I do not even know how to do that but I know you can xD and that's why I'm trying at all costs to see how to do it, I'm going to check that fluxgym I had no idea of its existence but now I'm going to see it, thank you very much for that information, I hope to find there how to perfect the loras to be able to use them in flux hopefully I can do it before Christmas hahahahaha

[ABSOMA021085]

FluxGym is a loras trainer for Flux, it's in Pinokio too. I haven't really tried it (it takes a lot of time with a 3080), but it is pretty straight forward. There are Youtube videos with demonstrations.

You add a batch of images with a certain style or thing or face, add a unique trigger word, and let FluxGym make image descriptions. You then can go through the descriptions to delete irrelevant 'noise' from the texts, and start training. After many hours (on a 3080 at least) you'll have a lora that you can use. You trigger the lora with your keyword, like "P@ul makes cake" if P@ul is your friend Paul. You may need finetuning regarding under- or overtraining. By default FluxGym saves a lora every 4 rounds. To get a feeling it's best to make it save loras every round and then test with each generation. One of the demos on Youtube was with photos of a pet goat, it looked good.

I have a rtx 3090 and a ryzen 9 5900X the truth is that I know I can run that and I bring as a concern that I want to train models with family photos one by one, and then make family photos and send them to my family, I do not even know how to do that but I know you can xD and that's why I'm trying at all costs to see how to do it, I'm going to check that fluxgym I had no idea of its existence but now I'm going to see it, thank you very much for that information, I hope to find there how to perfect the loras to be able to use them in flux hopefully I can do it before Christmas hahahahaha

[ETCore7]

3090 should be great. Pick a set of 20-30 consistent photos, drag and drop to FluxGym. Give a unique keyword like 'f@mily'. Let FluxFym describe the photos, then train the lora.

PS, In case you haven't yet, load the D.O.C.P. of your memory sticks to your mainboard. You can do that in the BIOS. Default timings can be ~50% of what memory is actually capable of.

[minye26]

Hello, I have an APPLE M3 PRO AND I CAN'T GET FORGE TO RUN PROPERLY. I have 36GB of RAM
Do you have any suggestions on how I can run flux?
My regards

[JesseBarrios]

This is how I got Flux to work with Macbook M1 Pro Max 32gb. Use the correct VAE / Text encorders with Flux GGUF

[Xenodermus]

Try Pinokio, it has Forge in one click install. -------- Original message --------From: minye26 ***@***.***> Date: 24/10/2024 18:20 (GMT+00:00) To: lllyasviel/stable-diffusion-webui-forge ***@***.***> Cc: Xenodermus ***@***.***>, Manual ***@***.***> Subject: Re: [lllyasviel/stable-diffusion-webui-forge] [Major Update] BitsandBytes Guidelines and Flux (Discussion #981) Hello, I have an APPLE M3 PRO AND I CAN'T GET FORGE TO RUN PROPERLY. I have 36GB of RAM Do you have any suggestions on how I can run flux? My regards —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[zzh19701008]

实在是太棒了，感谢！

[sterbendawg]

Could someone confirm if these 2 models do not need T5, CLIP and VAE added?
I ve seen the following in another post lllyasviel entitled "[GGUF and Flux full fp16 Model] loading T5, CLIP + new VAE UI". Thank you in advance, I am a bit confused.

[igorspesh]

who has an astronaut different from the one shown, most likely you have an updated driver or something like that, I installed the Studio version and my result became similar

1st Game Redy driver

2nd Studio driver

[tomburke-rse]

I hope some still reads this and can help.

I'm using this model:

flux1-dev-bnb-nf4-v2.safetensors Full flux-dev checkpoint with main model in NF4. <- Recommended

with a fresh install from the on-click installer.
Using a Geforce RTX3060 and the most recent driver, I only get fully grey images, as if the encoder is missing.

[IndigoDosSantos]

Make sure that your settings look like this. Since you're using an RTX 3060 the value for GPU Weights (MB) will be different from the one in the screenshot. The best value should be automatically set already. If it isn't enter your maximum VRAM in MB minus 1000 MB.
The most imporant setting is to have Diffusion in Low Bits set to Automatic.

[tomburke-rse]

Got that all, strangely it doesn't work at all:

[IndigoDosSantos]

Hmm, this is really strange. According to your screenshot you're also using the latest version of Forge, so it cannot be that.
Did you try with a random seed (-1)?

[nektosu]

I have Model Hash: 70cca7d7e2
Maybe this is the reason for your problem?

Actually, I do not know what the Model Hash means
Maybe it's a hash of the .safetensors file?

PS.
Don't pay attention to the file name because I renamed it:
flux1-dev-bnb-nf4-v2 -> FLUX-nf4-v2

[ETCore7]

A flat gray rectangle in Forge when using the Flux model could result from several potential issues. Here are some common causes and steps to troubleshoot:

---

### **1. Model Incompatibility or Corruption**
- **Cause**: The Flux model file might be incompatible with Forge's setup or corrupted during download.
- **Solution**:
  - Verify the model's file integrity using a checksum, if provided by the source.
  - Ensure the model format (e.g., `safetensors`, `gguf`) matches what Forge supports.

---

### **2. Incorrect Workflow or Parameters**
- **Cause**: The Forge workflow may be missing essential nodes, or the input parameters for the model may not align with what Flux expects.
- **Solution**:
  - Check if the workflow includes the necessary nodes for preprocessing, denoising, and postprocessing.
  - Review the prompt and settings (resolution, seed, strength, CFG scale, etc.) for compatibility with the Flux model.

---

### **3. Inadequate GPU Resources**
- **Cause**: The GPU might lack sufficient memory for the Flux model, causing improper rendering.
- **Solution**:
  - Check the VRAM usage during rendering.
  - Lower the resolution or batch size and retry.

---

### **4. Missing Dependencies or Configuration Mismatches**
- **Cause**: Some dependencies or environment variables might not be correctly configured.
- **Solution**:
  - Verify that Forge and all related plugins are up to date.
  - Ensure Python libraries required by the model are installed.

---

### **5. Backend Incompatibility**
- **Cause**: Forge might be using a backend (e.g., a specific Stable Diffusion inference engine) that does not fully support Flux.
- **Solution**:
  - Check if Flux requires specific backends (like ComfyUI or InvokeAI) or settings.
  - Look into Flux's documentation for any special requirements.

---

### **6. Default Latent Space Issue**
- **Cause**: The model might not be generating proper latent spaces, resulting in a uniform output.
- **Solution**:
  - Adjust the denoising strength, or try a different sampler.
  - Experiment with a wider variety of prompts to see if the issue persists.

---

### **7. Debugging Steps**
- Enable verbose logging in Forge (if possible) to capture errors or warnings during the render.
- Test the Flux model in another environment (e.g., ComfyUI, A1111) to confirm it works outside Forge.
- If the issue persists, try a different model to isolate whether the problem is with Flux or the Forge setup.

---

If these suggestions don't resolve the issue, let me know, and we can dig deeper or approach the problem from another angle.

[gigend]

I use 8gb vram and 32gb ram, do I need to activate system swap (automatically manage paging size)?