twodgirl/project-zeus-sd3-gguf

Project Zeus

The model weights belong to the Devil.

Prompt

full body, regal fantasy kingdom, white techwear jacket, black shirt, collar, young woman, curly hair, gorgeous, bokeh

Download

Download the t5xxl_fp8_e4m3fn.safetensors from camenduru.

You should have:

project_zeus_sd3.beta_x.Q8.gguf
t5xxl_fp8_e4m3fn.safetensors

Setup

pip install diffusers numpy safetensors sentencepiece transformers
pip install "gguf @ git+https://github.com/ggerganov/llama.cpp.git@master#subdirectory=gguf-py"

Cooking pot

python convert_sd3_to_diffusers.py --checkpoint_path projectZeusSD3_betaX.safetensors --output project_zeus
python convert_to_gguf.py ./project_zeus/project_zeus.Q8.gguf ./project_zeus/transformer/diffusion_pytorch_model.safetensors

Inference

from diffusers import StableDiffusion3Pipeline, SD3Transformer2DModel
import gguf
from gguf.quants import dequantize
from safetensors.torch import load_file
from transformers import T5Config, T5EncoderModel
import torch


def load_transformer_from_gguf(filepath):
    sd = {}
    reader = gguf.GGUFReader(filepath)
    for item in reader.tensors:
        xs = dequantize(item.data, item.tensor_type)
        tensor = torch.tensor(xs)
        sd[item.name] = tensor.to(dtype=torch.float16)
    config = SD3Transformer2DModel.load_config('./transformer')
    transformer = SD3Transformer2DModel.from_config(config).to(dtype=torch.float16)
    transformer.load_state_dict(sd)
    transformer.eval()

    return transformer

transformer = load_transformer_from_gguf('project_zeus_sd3.beta_x.Q8.gguf')
# Load T5 encoder in F8_E4M3 format.
# t5_config = T5Config.from_pretrained('text_encoder_3')
# t5 = T5EncoderModel(t5_config).to(dtype=torch.float16)
# t5_sd = load_file('t5xxl_fp8_e4m3fn.safetensors')
# t5.load_state_dict(t5_sd)
# t5.eval()
pipe = StableDiffusion3Pipeline.from_pretrained('stabilityai/stable-diffusion-3-medium-diffusers',
                                                transformer=transformer,
                                                # Either disable the T5 encoder.
                                                text_encoder_3=None,
                                                tokenizer_3=None,
                                                # Or enable it.
                                                # text_encoder_3=t5,
                                                torch_dtype=torch.float16)
pipe.enable_model_cpu_offload()
image = pipe('cat playing piano', num_inference_steps=20).images[0]
image.save('cat.png')

Disclaimer

Use of this code and the copy of documentation requires citation and attribution to the author via a link to their Hugging Face profile in all resulting work.