I can get "flux1-dev-Q4_0.gguf" to work, but not "flux1-dev-Q2_K.gguf", that one throws errors (in ComfyUI)...
PS: Links to each of these files is written in a note inside the WORKFLOW I add to this post, if you wonder about where I found what file... Workflow should also be helpful in understanding the error.
This combo of GGUF and SAFETENSORS works flawless on ComfyUI v1.2.26:
flux1-dev-Q4_0.gguf
t5xxl_fp16.safetensors
clip_l.safetensors
ae.safetensors
While this combo of GGUF and SAFETENSORS gives the ERRORS BELOW on ComfyUI v1.2.26:
flux1-dev-Q2_K.gguf
t5xxl_fp8_e4m3fn.safetensors
clip_l.safetensors
ae.safetensors
So... what do I need to change to make this work in ComfyUI??
COMFYUI-ERROR:
Error occurred when executing SamplerCustomAdvanced:
module 'gguf.quants' has no attribute 'dequantize'
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\execution.py", line 316, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\execution.py", line 191, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\execution.py", line 168, in map_node_over_list
process_inputs(input_dict, i)
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\execution.py", line 157, in process_inputs
results.append(getattr(obj, func)(**inputs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\comfy_extras\nodes_custom_sampler.py", line 612, in sample
samples = guider.sample(noise.generate_noise(latent), latent_image, sampler, sigmas, denoise_mask=noise_mask, callback=callback, disable_pbar=disable_pbar, seed=noise.seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 716, in sample
output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 695, in inner_sample
samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 600, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\utils_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\sampling.py", line 143, in sample_euler
denoised = model(x, sigma_hat * s_in, **extra_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 299, in call
out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 682, in call
return self.predict_noise(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 685, in predict_noise
return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 279, in sampling_function
out = calc_cond_batch(model, conds, x, timestep, model_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 228, in calc_cond_batch
output = model.apply_model(input_x, timestep, **c).chunk(batch_chunks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\comfy\model_base.py", line 145, in apply_model
model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\comfy\ldm\flux\model.py", line 150, in forward
out = self.forward_orig(img, img_ids, context, txt_ids, timestep, y, guidance, control)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\comfy\ldm\flux\model.py", line 118, in forward_orig
img, txt = self.double_blocks[i](img=img, txt=txt, vec=vec, pe=pe)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\comfy\ldm\flux\layers.py", line 148, in forward
img_mod1, img_mod2 = self.img_mod(vec)
^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\comfy\ldm\flux\layers.py", line 110, in forward
out = self.lin(nn.functional.silu(vec))[:, None, :].chunk(self.multiplier, dim=-1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-GGUF\ops.py", line 137, in forward
weight, bias = self.get_weights(x.dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-GGUF\ops.py", line 116, in get_weights
weight = self.get_weight(self.weight, dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-GGUF\ops.py", line 108, in get_weight
weight = dequantize_tensor(tensor, dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-GGUF\dequant.py", line 21, in dequantize_tensor
new = gguf.quants.dequantize(data.cpu().numpy(), qtype)
^^^^^^^^^^^^^^^^^^^^^^
Update the custom node to get support for K quants. Q2_K will also have pretty bad image quality so I recommend Q3_K_S.
The custom node I use is:
https://github.com/city96/ComfyUI-GGUF
All other nodes are part of ComfyUI 1.2.26..
I updated GGUF, and I tried both a pip update and a fresh git clone, but same errors persist... So what "custom node" are you referring to? Comfy-GGUF or some other node?
And I use standalone ComfyUI (v1.2.26) (tried updating that too, but it was already up-to-date)
I tried something I found in the comments amongst the other community posts in here....
So I added:
--disable-smart-memory
to my startup script for ComfyUI, and now suddenly it works and runs the Q2
So it now looks like:
.\python_embeded\python.exe -s ComfyUI\main.py --windows-standalone-build --preview-method taesd --lowvram --disable-smart-memory)
Still i'd love to know what to update (name and link) and maybe how or why this added " --disable-smart-memory" fixed the problem... if possible...
File "C:\STABLE-DIFFUSION\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-GGUF\dequant.py", line 21, in dequantize_tensor
new = gguf.quants.dequantize(data.cpu().numpy(), qtype)
Thats line will never run on the newer versions of the ComfyUI-GGUF custom node, which means you were on an older version. It's not even on line 21 anymore, it's on line 23 now
I don't think --disable-smart-memory would fix that error, the more likely thing is that the manager didn't update the node or ComfyUI didn't reload it after updating it.
There's other people for whom the manager seems to do the same thing https://github.com/city96/ComfyUI-GGUF/issues/45
If you refer to "ComfyUI Manager" the custom_node that lets you easily install nodes, then I am not using it, I prefer to manually install via github/dos/cmd. So I do not have the "ComfyUI Manager".
But yes, after I changed the startup line in CUI (short for CopmfyUI) it suddenly started working, so it was not enough to just close CUI, and restart CUI, but I also had to change that setting before it properly updated your GGUF. How strange... but it works now...
I will try your flux dev Q3_K_S today (as I got the Q2 to work, but as you said, it has poor img-quality, compared to ex Q4_0 that i also have tested)
And to manually confirm I opened "dequant.py" and found it to mirror what you answered above!
(so now it has been updated proper)(and GIT PULL in GGUF folder says Im up to date too)
20_else:
21____# this is incredibly slow
22____tqdm.write(f"Falling back to numpy dequant for qtype: {qtype}")
23 > >_ new = gguf.quants.dequantize(data.cpu().numpy(), qtype)
24____return torch.from_numpy(new).to(data.device, dtype=dtype)
So, THANKS for your help, and for your GGUF - Very interesting this GGUF way of doing things, I like it!
@AlexData257 Glad it worked. Yeah, I also exclusively git clone manually, so I'm with you on that. Thought it was that ComfyUI Manager doing some caching logic i.e. only refreshing every 24 hours but the same happening with git is even stranger since now we have reports of it happening with both.