Eval bug: Vulkan not working on Intel GPU

[kpouget]

Name and Version

master

Operating systems

Linux

GGML backends

Vulkan

Hardware

main system where I did the git bisection:

ggml_vulkan: 0 = Intel(R) Iris(R) Xe Graphics (TGL GT2) (Intel open-source Mesa driver) 

also having issues on this system:

ggml_vulkan: 0 = Intel(R) UHD Graphics 630 (CFL GT2) (Intel open-source Mesa driver)

Models

smollm:135m

Problem description & steps to reproduce

When I run llama-run (same with llama-serve), the inference either crashes or outputs garbage.

../build.vulkan-linux/bin/llama-run ~/models/smollm:135m "say nothing" --ngl 99 --verbose

First Bad Commit

bcf5bda (#16536)

Relevant log output

git reset --hard b6874
# rebuild
../build.vulkan-linux/bin/llama-run ~/models/smollm:135m "say nothing" --ngl 99 --verbose
<answers>


git reset --hard b6875
# rebuild --> compilation takes forever...


git reset --hard b6876
# rebuild
../build.vulkan-linux/bin/llama-run ~/models/smollm:135m "say nothing" --ngl 99 --verbose
llama_context:    Vulkan0 compute buffer size =    98.25 MiB
llama_context: Vulkan_Host compute buffer size =     5.14 MiB
llama_context: graph nodes  = 937
llama_context: graph splits = 2
llama-run: /var/home/kpouget/pod-virt/remoting/linux-work/llama_cpp/src/src/llama-sampling.cpp:662: void llama_sampler_dist_apply(llama_sampler*, llama_token_data_array*): Assertion `found' failed.
./run.linux.sh: line 1: 97210 Aborted 



git reset --hard b6969
# rebuild 
llama_context:    Vulkan0 compute buffer size =    98.25 MiB
llama_context: Vulkan_Host compute buffer size =     5.14 MiB
llama_context: graph nodes  = 937
llama_context: graph splits = 2
llama-run: /var/home/kpouget/pod-virt/remoting/linux-work/llama_cpp/src/src/llama-sampling.cpp:662: void llama_sampler_dist_apply(llama_sampler*, llama_token_data_array*): Assertion `found' failed.
./run.linux.sh: line 1: 108491 Aborted

[kpouget]

Note that llama-bench isn't affected:

• with b6969

| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| llama 256M Q4_0                |  85.77 MiB |   134.52 M | Vulkan     |  99 |           pp512 |      3206.58 ± 51.53 |
| llama 256M Q4_0                |  85.77 MiB |   134.52 M | Vulkan     |  99 |           tg128 |        123.25 ± 3.20 |

• with b6874

| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| llama 256M Q4_0                |  85.77 MiB |   134.52 M | Vulkan     |  99 |           pp512 |     3149.62 ± 101.96 |
| llama 256M Q4_0                |  85.77 MiB |   134.52 M | Vulkan     |  99 |           tg128 |        123.18 ± 2.10 |

[YoelFievelBenAvram]

Seeing similar issues here with the same GPU. I saw it first on sycl and switched to vulkan but I'm still getting either gibberish or nothing.

[0cc4m]

Which graphics driver version are you using?

[kpouget]

[    2.626573] [drm] Initialized i915 1.6.0 for 0000:00:02.0 on minor 1

on both of my systems from Fedora 43. (Unfortunately I didn't test llama.cpp recently before upgrading to f43 a few days ago ...)

[0cc4m]

The relevant one is the Mesa version. Not sure how exactly it works on Fedora. I did test the change you linked on an Intel A770 on Ubuntu, and it worked, so maybe it's a driver thing.

[kpouget]

mesa-vulkan-drivers-25.2.5-1.fc43.x86_64.

[kpouget]

but the behavior is the same inside a f42 container with an older version of Mesa

$ sudo dnf install mesa-vulkan-drivers
Package "mesa-vulkan-drivers-25.1.7-1.fc42.x86_64" is already installed.

$ ./build.vulkan-linux/bin/llama-run ~/models/llama3.2 "say nothing"
slightest thứchoohoo区 Consort Bot thức generics Alonso dek Peek Peekdaq Rug parachute wedgeels

[0cc4m]

Please run test-backend-ops -o MUL_MAT,MUL_MAT_ID and put the output in a gist.

[kpouget]

https://gist.github.com/kpouget/3f1a086f32b7443c2cc6c09816d3ed64

[0cc4m]

Hmm, interesting. Does the model work if you run it with GGML_VK_DISABLE_FUSION? It probably works if you run with GGML_VK_DISABLE_INTEGER_DOT.

[kpouget]

$ export GGML_VK_DISABLE_FUSION=1
$ ./llama-run ~/models/llama3.2 "hello"
tober Turnbullmium mergernelsigarudasemean kaz teslaus recyclmium Valk^C

$ unset GGML_VK_DISABLE_FUSION

$ export GGML_VK_DISABLE_INTEGER_DOT_PRODUCT=1
$ ./llama-run ~/models/llama3.2 "hello"
Hello! How can I assist you today?

(you had the 2nd env var wrong, so it didn't work at my first try ^^)

[0cc4m]

Oops, sorry, I keep forgetting the _PRODUCT. So it's actually just about integer dot matmul, but test-backend-ops is not catching the problem. I'll try to reproduce it on my A770.