Vulkan: Odd compute buffer behaviors at specific context breakpoints version b6568 and above

[EmanuelOverride]

First became aware of issue when running the latest Koboldcpp release and previously workable configs started failing, discussed at LostRuins#1805

====

Running llama-b6567-bin-win-vulkan-x64 llama-bench with an RX 480 8GB with a --ubatch-size of 512 and an --n-prompt value of first 14592, then 14593, the Vulkan0 compute buffer size is 1022.50 MiB, 1024.69 MiB respectively.

Repeating the same process with llama-b6568-bin-win-vulkan-x64, again with the same --n-prompt values of 14592, then 14593, the compute buffer size is 1022.50 MiB, 1910.13 MiB respectively. A sizeable discrepancy.

Below n-prompt 14592, both versions behave identically all the way down. Above 14593, b6568 continues accumulating as "normal" until 15040, clocking in at 1963.75 MiB, then at 15041 jumps down to 1611.48 MiB. For comparison, b6567 at 15041 allocates 1055.31 MiB to the buffer, and grows as normal every step of the way.

Finally, the allocation discrepancy between versions continues to diminish as the context increases, and can be assumed to disappear entirely at some point -- or perhaps even result in memory savings? Nah, that'd be too good I'm sure. I've only tested it up to 20k, where it's still some 450 mb above the b6567 "standard."

Tested ClBlast and Cuda via Zluda emulation to ensure that it's a vulkan-specific issue, and it is. Don't as of yet know if it's repeatable on other hardware and would love it if someone could test this out. What changed between b6567 and b6568 that could result in such behavior, and is it expected? And, I should obviously mention that, same behavior persists with b6833.

[jeffbolznv]

#15815 is the only change in that tag, and seems like a likely candidate. CC @Acly

[EmanuelOverride]

The breakpoints are not fixed and seemingly reflect model size, but stay within a comparable range. Initially used an L3 8B 4,8gb model offloading 27/33 layers to gpu, then tested a Qwen3 8B 4,9gb 27/37 with same settings -- the first breakpoint value moved somewhere to 14200-14300. Tested a 12B 6,9gb model, same rough range as the Qwen -- didn't look too deeply into it. Tested a number of different but similarly sized models, within less than a meg, and they behave identically. Tested a few slightly larger L3 8B's and the exact breakpoint shifts, but only just. So, if it is repeatable, influence of model on the overall range should be negligible.

Offloading less layers and changing the KV and model buffer size, however, does not influence where the exact breakpoint ends up. So it's at least not a response to currently allocated memory. Changing ubatch-size from 512 to 256 or lower has a significant influence on the range, moving the breakpoints up to the 26-28k range. So, there are a number of cases where this could be a common occurrence across other hardware setups but with negligible or unnoticeable impact.

My favorite part are the huge (relative) savings at the second mark. Like, for my specific use-case, I should stay below a context of 14592 to avoid a 900mb penalty, but if I'm set on going higher, I should definitely go higher than 15041 to cut a third off that penalty. ;)

[Acly]

To summarize changes in #15815 briefly:

• Before the change, all intermediate results were allocated in a single compute buffer, which could grow indefinitely. Once it grew beyond the driver's maximum allocation limit (often 4GB, but can be lower), you got an error, even if there was technically lots of free memory left on the device.
• After the change, all intermediate tensors are batched into chunks of (by default) 1GB. Once a chunk is full a new one is started. Memory in all chunks is reused when intermediate results are no longer needed.

Even though memory is reused, chunk boundaries set an additional limitation. Memory that could be previously reused might now be at the boundary of individual buffers, and a large tensor might not fit, where previously it has.

It matches your observations quite well, especially the threshold being at 1GB memory used. To test if the change is sololy responsible you could set a different chunk size. It can be done by setting GGML_VK_SUBALLOCATION_BLOCK_SIZE environment variable to a larger size (in bytes). Be aware that will also affect weight allocation, which has always been chunked, and uses the same limit.

As for why the overhead is so large after going over the threshold... it might be a bug, or just unfortunate. Fragmentation can be an issue. If you post the full command line and the exact model you used I can try to have a look at the tensor allocations.

[EmanuelOverride]

It matches your observations quite well, especially the threshold being at 1GB memory used. To test if the change is sololy responsible you could set a different chunk size. It can be done by setting GGML_VK_SUBALLOCATION_BLOCK_SIZE environment variable to a larger size (in bytes). Be aware that will also affect weight allocation, which has always been chunked, and uses the same limit.

Cheers, that seems to be right on the money. Setting GGML_VK_SUBALLOCATION_BLOCK_SIZE to 2GB (the purported maximum of AMD's proprietary vulkan driver) restores expected behavior and allocates 1053.13 MiB to the compute buffer. It is, for whatever reason, not a solution for koboldcpp, which errors out (EDIT: On further testing, Kobold with PR 16679 seems perfectly stable with a 2GB block size.), but setting the block size to 256mb instead seemingly works fine, and only accrues a small 20mb overhead over the nominal value. On the face of it, this seems like a reasonable intermediary solution in my specific case.

Since you emphasized that this adjustment also affects weight allocation, could I ask what sort of behaviors one could expect from downward adjustments of the block size? Is there any plausible downside to working with multiple smaller chunks below 1GB?

As for the exact model used, I can reproduce the same behavior with any model, so if you're interested in taking a look, we could use the one recommended in #10879 which is llama-2-7b.Q4_0. (Wow, surprisingly memory intensive for such a small model.)

llama-2-7b.Q4_0-12200.txt

Range was good up to 12400-ish.

llama-2-7b.Q4_0-12500.txt

Then 12500 failed.

llama-2-7b.Q4_0-12500_success.txt

Lowered number of offloaded layers to make it fit. That's around the first breakpoint, at 1711.19 MiB.

llama-2-7b.Q4_0-14500.txt

Estimated the second to be somewhere within 2k context, and there we are: back at 1188.38 MiB. I previously assumed that the overhead would only continue to diminish as context increases, but from what you've told me, the expected behavior with a default block size is for this to keep recurring at every, apparently, 1024, and thus also potentially accumulating and making those breakpoints arrive sooner each time. Doesn't much matter in my case, since I'm not expecting to push my hardware much further than I already am, so I hope it's limited to me rather than some general problem. Thanks for your response.

[Acly]

Here is my attempt at illustrating how things are allocated. Mostly just documenting the reasons it behaves the way it does, might be hard to understand without knowing some internals.

xxx is allocated but unused memory, |> marks the maximum chunk size at 1024MB.

Scenario 1: --n-prompt 12200

       0 MB                    240 MB                                        1024 MB
chunk0 [ stuff | stuff | stuff |                kq-0 (776MB)                 ] |>
---
chunk0 [ stuff | stuff |               kq-1 (776MB)                 | xxxxxx ] |>
---
chunk0 [ stuff | stuff |               kq-13 (776MB)                | stuff  ] |>

The biggest tensors here are the "kq" tensors, but at 776MB they fit into a single chunk along with the rest. Things move around in memory a bit as smaller computations are mixed in with large "kq" layers.

Scenario 2: --n-prompt 12500

       0 MB                    240 MB                                        1024 MB
chunk0 [ stuff | stuff | stuff ]                                               |>
chunk1 [                     kq-0 (782MB)                     ]                |>
---
chunk0 [ stuff | xxxxxxxxxxxxx ]                                               |>
chunk1 [ stuff |                     kq-1 (782MB)                     ]        |>
---
chunk0 [ stuff |                    kq-13 (782MB)                     ]        |>
chunk1 [ stuff | stuff | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ]        |>

Here "kq" is slightly bigger, it no longer fits into one chunk with the rest, and is allocated in a separate chunk. So far no problem, both chunks still add up to ~1100MB. As things are shuffled around, other computations may take some space previously used by "kq" and the buffer grows a little more. This is fine until the point where suddenly there is enough space to fit "kq" into chunk0. Note that the already allocated but now free space in chunk1 is not enough to hold "kq", but there is enough unclaimed space still available to chunk1 that would allow it to hold "kq", and reuse all that xxx memory.

That doesn't happen because the algorithm, in the case where no chunk has enough free memory, simply picks the first chunk that can be extended to hold the tensor. 0 comes before 1.

I think it can be improved.

[Acly]

These are the results I get with #16788

n-prompt	master	#16788
--n-prompt 12200	1003.88 MiB	1003.88 MiB
--n-prompt 12500	1711.19 MiB	1026.94 MiB
--n-prompt 13500	1844.88 MiB	1106.38 MiB
--n-prompt 14500	1188.38 MiB	1188.38 MiB
--n-prompt 15500	1267.81 MiB	1267.81 MiB

...seems more reasonable. It's still possible something like this might show up when the size of a single tensor is very close to the maximum allocation size though.

[EmanuelOverride]

That's very interesting. So when I ran the same model/config with a 256mb limit, the first chunk immediately got "stretched" past that limit to fit the biggest tensor? Or does extending in your illustration apply strictly within the set limit? Whatever the case, the overhead never really went much further than 20-30mb with 256mb chunks. I feel like there's a calculable ratio between largest tensor size and chunk-size where one could expect the most severe breakpoints to emerge.

In any case, my main take-away is that this wasn't a hardware specific problem, and could potentially crop up in a number of unexpected situations, and your solution seems to tackle the issue directly and in a very promising way. Cheers!