Eval bug: ggml_gallocr_reserve_n tries to allocate beyond max buffer size

[BenPortner]

Name and Version

version: 4517 (9f7add1)
built with MSVC 19.42.34436.0 for

Operating systems

Windows

GGML backends

Vulkan

Hardware

Dell Latitude 5420
Windows 10 Enterprise
CPU: 11th Gen Intel i7-1185G7 @ 3.00GHz, 4 Cores, 8 Logical Processors x86_64
RAM: 2x16GB Hynix 3200MHz DDR4 PC4-25600
GPU: Intel Iris Xe iGPU
Storage: Western Digital PC SN530 NVMe WDC 512GB M.2 SSD

Models

tensorblock/CodeQwen1.5-7B-GGUF

Problem description & steps to reproduce

I get an OOM error when trying to use CodeQwen 1.5 7B Q4_K_M with large (e.g. 65000) context sizes. Both model and KV buffers allocate just fine. However, there is another allocation during ggml_gallocr_reserve_n, which causes the error. More specifically, galloc->buf_tallocs[0]->max_size is calculated by ggml_gallocr_alloc_graph_impl. This value is used in line 766 to allocate a new buffer. Because galloc->buf_tallocs[0]->max_size is never checked against the maximum allowed buffer size of the device, the allocation fails. In my case galloc->buf_tallocs[0]->max_size evaluates to 4428140544, whereas ggml_backend_buft_get_max_size yields 4294901760.

It seems to me that this is a bug in llama.cpp because running Llama 3.2 3B Instruct Q4_0 with the same (and even larger) context sizes works without problems.

First Bad Commit

No response

Relevant log output

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Iris(R) Xe Graphics (Intel Corporation) | uma: 1 | fp16: 1 | warp size: 32 | matrix cores: none
register_backend: registered backend Vulkan (1 devices)
register_device: registered device Vulkan0 (Intel(R) Iris(R) Xe Graphics)
register_backend: registered backend CPU (1 devices)
register_device: registered device CPU (11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz)
load_backend: failed to find ggml_backend_init in .\ggml-vulkan.dll
load_backend: failed to find ggml_backend_init in .\ggml-cpu.dll
build: 4517 (9f7add1c) with MSVC 19.42.34436.0 for  (debug)
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_load_from_file_impl: using device Vulkan0 (Intel(R) Iris(R) Xe Graphics) - 16072 MiB free
llama_model_loader: loaded meta data with 25 key-value pairs and 387 tensors from C:\Users\BNPR\Downloads\CodeQwen1.5-7B-Chat.Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = .
llama_model_loader: - kv   2:                           llama.vocab_size u32              = 92416
llama_model_loader: - kv   3:                       llama.context_length u32              = 65536
llama_model_loader: - kv   4:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   5:                          llama.block_count u32              = 32
llama_model_loader: - kv   6:                  llama.feed_forward_length u32              = 13440
llama_model_loader: - kv   7:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   8:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   9:              llama.attention.head_count_kv u32              = 4
llama_model_loader: - kv  10:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  11:                       llama.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  12:                          general.file_type u32              = 15
llama_model_loader: - kv  13:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  14:                      tokenizer.ggml.tokens arr[str,92416]   = ["<unk>", "<s>", "<|endoftext|>", "<|...
llama_model_loader: - kv  15:                      tokenizer.ggml.scores arr[f32,92416]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  16:                  tokenizer.ggml.token_type arr[i32,92416]   = [2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  17:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  18:                tokenizer.ggml.eos_token_id u32              = 4
llama_model_loader: - kv  19:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  20:            tokenizer.ggml.padding_token_id u32              = 92298
llama_model_loader: - kv  21:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  22:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  23:                    tokenizer.chat_template str              = {% for message in messages %}{% if lo...
llama_model_loader: - kv  24:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  161 tensors
llama_model_loader: - type q5_0:   16 tensors
llama_model_loader: - type q8_0:   16 tensors
llama_model_loader: - type q4_K:  177 tensors
llama_model_loader: - type q6_K:   17 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 4.41 GiB (5.23 BPW)
load: special tokens cache size = 151
load: token to piece cache size = 0.4983 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 65536
print_info: n_embd           = 4096
print_info: n_layer          = 32
print_info: n_head           = 32
print_info: n_head_kv        = 4
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: n_ff             = 13440
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 65536
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
ggml_vulkan: Compiling shadersprint_info: ssm_dt_b_c_rms   = 0
print_info: model type       = 8B
print_info: model params     = 7.25 B
print_info: general.name     = .
print_info: vocab type       = SPM
print_info: n_vocab          = 92416
.print_info: n_merges         = 0
print_info: BOS token        = 2 '<|endoftext|>'
print_info: EOS token        = 4 '<|im_end|>'
print_info: EOT token        = 2 '<|endoftext|>'
print_info: UNK token        = 0 '<unk>'
print_info: PAD token        = 92298 '<fim_pad>'
print_info: LF token         = 1396 '<0x0A>'
print_info: EOG token        = 2 '<|endoftext|>'
print_info: EOG token        = 4 '<|im_end|>'
print_info: max token length = 36
............................................Done!
load_tensors: offloading 32 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 33/33 layers to GPU
load_tensors:      Vulkan0 model buffer size =  4314.02 MiB
load_tensors:   CPU_Mapped model buffer size =   203.06 MiB
llama_init_from_model: n_seq_max     = 1
llama_init_from_model: n_ctx         = 65024
llama_init_from_model: n_ctx_per_seq = 65024
llama_init_from_model: n_batch       = 2048
llama_init_from_model: n_ubatch      = 512
llama_init_from_model: flash_attn    = 0
llama_init_from_model: freq_base     = 1000000.0
llama_init_from_model: freq_scale    = 1
llama_init_from_model: n_ctx_per_seq (65024) < n_ctx_train (65536) -- the full capacity of the model will not be utilized
llama_kv_cache_init: kv_size = 65024, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 32, can_shift = 1
llama_kv_cache_init:    Vulkan0 KV buffer size =  4064.00 MiB
llama_init_from_model: KV self size  = 4064.00 MiB, K (f16): 2032.00 MiB, V (f16): 2032.00 MiB
llama_init_from_model: Vulkan_Host  output buffer size =     0.35 MiB
ggml_vulkan: Device memory allocation of size 4428140544 failed.
ggml_vulkan: Requested buffer size exceeds device memory allocation limit: ErrorOutOfDeviceMemory
ggml_gallocr_reserve_n: failed to allocate Vulkan0 buffer of size 4428140544

[slaren]

There are no plans to support splitting the compute buffers allocated by ggml-alloc to workaround backend limitations. You can try reducing the ubatch size, eg -ub 128, to reduce the size of the compute buffer. Adding support for flash attention to Vulkan would also significantly decrease the size of the compute buffer with large contexts.

You may also get better performance on the CPU than with Vulkan on this iGPU.

[BenPortner]

Hi @slaren,

thanks for your fast answer. Indeed, it works when setting the u_batch size to 128. Now, I'm trying to figure out what this parameter does exactly. The llama-cli readme, says "physical maximum batch size. This is for pipeline parallelization.". However, in my case pipeline parallelization is dactivated (model->n_devices() evaluates to 1). So how come this parameter is used to calculate the compute buffer size? And what does "physical maximum batch size" mean in the first place? Isn't the batch size always equal to 1 when I call llama-cli (i.e. there is just one prompt)?

[slaren]

The documentation is not correct, it should be the other way around. -ub configures the physical batch size, which means the maximum number of tokens that can be evaluated at the same time. Reducing this means that the intermediate values are smaller, which results in a smaller compute buffer. #11353 should clarify this.

[BenPortner]

Thanks for the blazing fast response and the readme fix @slaren! Just to see if I understood you right:

Let's assume I want to analyze a long text document using llama.cpp. I feed the text document to the LLM using llama-cli -m some_model --file myfile.txt. Now, because the default n_ubatch value is 512, does this mean that the LLM "forgets" about the previous context every 512 tokens? I.e. by the time token 1025 is processed, it would have forgotten about token 1, 2, 3...1024?

[slaren]

No, what it means is that the tokens of myfile.txt are processed in chunks for 512 tokens. So if myfile.txt has 700 tokens, the first 512 will the processed in one batch, and the remaining 188 in another batch. Processing more tokens at a time can improve performance, at the expense of higher memory usage. What the LLM "remembers" is given by the size of the context, set with -c.

You can test the effect of different values of -ub on the performance with llama-bench, for example:

llama-bench -m model.gguf -n 0 -ub 64,128,256,512

model	size	params	backend	ngl	n_ubatch	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	64	pp512	3726.25 ± 7.07
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	128	pp512	4954.23 ± 8.90
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	256	pp512	5829.67 ± 17.94
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	512	pp512	6079.21 ± 19.90

build: 0410e03 (4528)

[BenPortner]

Thanks again for the input @slaren. I am aware of the benchmarking functionality. Unfortunately, it does not help me to wrap my head around the "quality implications". Please allow me to test my understanding again: So basically, the attention mechanism always uses all of the n_ctx tokens (the LLM "remembers"). After the attention part, the KQV-matrix is split into n_ctx/n_ubatch number of batches and each batch is passed to the feed-forward network separately. This reduces the memory demand of the feed-forward calculations. After that, the batches are assembled again and the whole is fed to the next layer's attention mechanism?

[slaren]

There are no quality implications, this is just a tradeoff between performance and memory usage. When processing a prompt the goal is to populate the KV cache, and this can be done in chunks of multiple tokens at a time by setting an attention mask such that each token only attends to the previous tokens in the sequence. The attention mechanism only uses the previous tokens sequence, the value of n_ctx is only a maximum of the sequence length.

Note that this only true for causal models, there are also non-causal models that always attend to all the tokens in the sequence, but typically are only used for embeddings.

[BenPortner]

@slaren I see, so in the attention mechanism, token n only attends to tokens n-1, n-2...n0. For the very last token, this means that it attends to all n_ctx tokens. For the other tokens, its less than that. But is my understanding of the following part (splitting the KQV matrix in to n_ubatch number of batches, then re-assembling it after the feed-forward network) more or less correct? Also: Is there an equation that we can use to estimate the size of the compute buffer beforehand?

[slaren]

There isn't any splitting, at all steps of the evaluation, each token in the batch is represented as a row in the inputs/hidden state of each layer. If you want to understand how this is implemented in llama.cpp, I would suggest looking at the gpt-2 example in ggml, it does the same thing but with fewer layers of abstraction. You could probably come up with a function to estimate the compute buffer size, but there isn't one.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[Russyyds]

I came across the same error when running the latest Gemma3 on Android device.
Here is my hardware info:
CPU:Qualcomm 8 Gen 3
GPU:QUALCOMM Adreno(TM) 750
OpenCL 3.0

Android build command:

cmake .. -G Ninja \
  -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
  -DANDROID_ABI=arm64-v8a \
  -DANDROID_PLATFORM=android-28 \
  -DBUILD_SHARED_LIBS=OFF \
  -DGGML_OPENCL=ON \
  -DGGML_OPENMP=OFF \
  -DLLAMA_CURL=OFF

run command

./build/bin/llama-gemma3-cli -mgemma-3-4b-it-qat-q4_0-gguf/gemma-3-4b-it-q4_0.gguf --mmproj gemma-3-4b-it-qat-q4_0-gguf/mmproj-model-f16-4B.gguf -p "Describe the image" --image surprise.png

detail output:

ggml_opencl: selecting platform: 'QUALCOMM Snapdragon(TM)'
ggml_opencl: selecting device: 'QUALCOMM Adreno(TM) 750 (OpenCL 3.0 Adreno(TM) 750)'
ggml_opencl: OpenCL driver: OpenCL 3.0 QUALCOMM build: commit unknown Compiler E031.45.02.05
ggml_opencl: vector subgroup broadcast support: false
ggml_opencl: device FP16 support: true
ggml_opencl: mem base addr align: 128
ggml_opencl: max mem alloc size: 1024 MB
ggml_opencl: SVM coarse grain buffer support: true
ggml_opencl: SVM fine grain buffer support: true
ggml_opencl: SVM fine grain system support: false
ggml_opencl: SVM atomics support: true
ggml_opencl: flattening quantized weights representation as struct of arrays (GGML_OPENCL_SOA_Q)
ggml_opencl: using kernels optimized for Adreno (GGML_OPENCL_USE_ADRENO_KERNELS)
register_backend: registered backend OpenCL (1 devices)
...
...
init: kv_size = 4096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 34, can_shift = 1
init:        CPU KV buffer size =   544.00 MiB
llama_context: KV self size  =  544.00 MiB, K (f16):  272.00 MiB, V (f16):  272.00 MiB
llama_context:        CPU compute buffer size =     2.16 MiB
llama_context: graph nodes  = 1435
llama_context: graph splits = 1
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
clip_ctx: CLIP using OpenCL backend
clip_model_loader: model name:
clip_model_loader: description:
clip_model_loader: GGUF version: 3
clip_model_loader: alignment:    32
clip_model_loader: n_tensors:    439
clip_model_loader: n_kv:         16

load_hparams: text_encoder:       0
load_hparams: vision_encoder:     1
load_hparams: llava_projector:    0
load_hparams: minicpmv_projector: 0
load_hparams: minicpmv_version:   2
load_hparams: glm_projector:      0
load_hparams: model size:         524616.06 MiB
load_hparams: metadata size:      0.15 MiB
ggml_backend_opencl_buffer_type_alloc_buffer: failed to allocate 1128.81 MiB
main: ../gemma-3-4b-it-qat-q4_0-gguf/gemma-3-4b-it-q4_0.gguf
ggml_gallocr_reserve_n: failed to allocate OpenCL buffer of size 1183645696

As memtioned above, I try to decrease -ub to 2 but it still occurs.
Do you have any suggestions?
Thanks