Heads up - upstream GGML produces black images on CUDA

[Green-Sky]

I just bisected upstream ggml and ggml-org/ggml@d8ba8a3 is the first bad commit (ggml-org/llama.cpp#16715).

I already tried to apply ggml-org/llama.cpp#16935 , but that does not fix the issue.

Setting GGML_CUDA_DISABLE_FUSION=1 does in fact workaround the issue.

$ result/bin/sd --diffusion-model models/SPARK.Chroma_preview-q5_k.gguf --t5xxl models/flux-extra/t5xxl_fp16.safetensors -t 8 --vae models/flux-extra/ae-f16.gguf --sampling-method dpm++2m --scheduler smoothstep --steps 20 --cfg-scale 3.8 -n "low quality, ugly, unfinished, out of focus, deformed, disfigure, blurry, smudged, restricted palette, flat colors, noisy, artifacts, fake, generated, overblown, over exposed" -p "Photograph of a lovely cat. Green rolling hills in the background." --clip-on-cpu --offload-to-cpu -v -W 512 -H 512 --diffusion-fa --chroma-disable-dit-mask

---

Params:

• image size makes no difference (512x512, 768x768 and 768x1024 are the ones I tested).
• --diffusion-fa --chroma-disable-dit-mask makes no difference.
• --offload-to-cpu neither.

I still need to determine what parameters and models are affected.

---

@am17an any insights would be helpful. Should I open an issue at llama.cpp? I am not sure if sd.cpp is at fault here or not.

[am17an]

I'll take a look, if you don't offload-to-cpu does it work? Also what is your setup, I know there is an ongoing issue with 50xx cards

[Green-Sky]

I'll take a look, if you don't offload-to-cpu does it work? Also what is your setup, I know there is an ongoing issue with 50xx cards

RTX 2070 (8gig). Will try.

--diffusion-fa --chroma-disable-dit-mask make no difference (flash attention in the DiT)

[am17an]

Thanks, can you also please tell me how to download these models?

[Green-Sky]

offload-to-cpu makes no difference.
BTW, offload-to-cpu means the model (parts) are parked in ram, when it is not used. They are separated by the different phases.

Thanks, can you also please tell me how to download these models?

Main model: https://huggingface.co/Green-Sky/SPARK.Chroma_preview-GGUF/tree/main

for ae and t5xxl: https://huggingface.co/Green-Sky/flux.1-schnell-GGUF/tree/main or https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/ae.safetensors and https://huggingface.co/comfyanonymous/flux_text_encoders/blob/main/t5xxl_fp16.safetensors

edit: more info can be found in the docs: https://github.com/leejet/stable-diffusion.cpp/blob/master/docs/chroma.md

[wbruna]

@Green-Sky , that's odd: that revision was included in Koboldcpp more than a week ago: LostRuins/koboldcpp@f77c13b .

@LostRuins , have someone noticed an issue like this?

[Green-Sky]

@wbruna there is also this: LostRuins/koboldcpp@3aec5ed

[am17an]

@Green-Sky how long should this command take? It's taking forever for me

EDIT: avoiding --clip-on-cpu makes it run fast

[am17an]

From what it looks like there is a MUL_MAT + ADD which is getting fused which shouldn't, removing that obviously fixes the issue. But I'm trying to understand why and will post a PR once I figure that out

[am17an]

Ah looks like it's

stable-diffusion.cpp/flux.hpp

Lines 612 to 614 in 8f6c5c2

	x = ggml_mul_mat(ctx->ggml_ctx, fc2, x); // [batch_size, n_token, hidden_size_x]
	x = ggml_add_inplace(ctx->ggml_ctx, x, res_x);

[wbruna]

I was able to reproduce this issue with Chroma1-HD-Flash-Q4_0 on ROCm: black image after updating ggml, and GGML_CUDA_DISABLE_FUSION=1 seems to work around it (with slight image changes).

Curiously, Flux1-Schnell-Q4_K outputs aren't affected, at all.

@am17an , the ggml_add_inplace you quoted doesn't seem to be used in this case: the NerfGLUBlock struct is specific to the Radiance variant.

[am17an]

Yeah, sorry I'm not familiar with the codebase, wherever there is a ggml_mul_mat followed by ggml_add_inplace is potentially problematic because the pointers may alias.

[stduhpf]

The only place I'm aware where a ggml_mul_mat can be directly followed by aggml_add_inplace is in the ggml_ext_linear() function with bias and no scale.
stable-diffusion.cpp/ggml_extend.hpp

[am17an]

I think ggml-org/llama.cpp#17080 should fix this, I tried locally and was able to generate a normal image

[Green-Sky]

I think ggml-org/llama.cpp#17080 should fix this, I tried locally and was able to generate a normal image

That does indeed fix it! Thanks for fixing it so fast. 🚀