CUDA: fix FA occupancy, optimize tile kernel #15982
Conversation
github-actions
bot
added
Nvidia GPU
|
I'll take a proper look at this, but will not be able to do so until the 17th |
|
Since you're already here, do you have an opinion on whether the HIP backend should be compiled with -ffast-math? |
|
A quick grep suggests we use inf directly (see softmax), so blanket ffast-math is out. we could use some of the ffast-math flags or use ffast-math on a per function or per translation unit basis, but im not sure its worth it. In the past on other code llvm fast-math has made things slower on hip for some reason, and before rocm 6.1 there where some bugs i have encountered where fast-math just generated plain wrong code. |
There was a problem hiding this comment.
Changes look fine to me, i can also confirm the performance delta on gfx1030. When making gfx908 use this code path i cant reproduce the same magnitude of performance improvement as @JohannesGaessler dose on gfx906, but find no regression. I noticed that this pr reduced the amount of spilled vgprs (altho some instances still spill like _ZL15flash_attn_tileILi64ELi32ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiiiiiiiiiiiiiliiliiiiil) so its possible that some of the extra improvement on gfx906 comes from reduced spills to scratch, where gfx908 can spill to agprs which has a lower performance impact.
Side note:
some of the vector fattn kernels spill to high heaven:
Function Name: _ZL22flash_attn_vec_ext_f32ILi128ELi8EL9ggml_type8ELS0_8ELb1EEvPKcS2_S2_S2_S2_PKiPfP15HIP_vector_typeIfLj2EEffffjfiiiiiiiiiiiiiliiliiiiil
TotalSGPRs: 88
VGPRs: 63
AGPRs: 64
ScratchSize [bytes/lane]: 1100
Dynamic Stack: False
Occupancy [waves/SIMD]: 4
SGPRs Spill: 0
VGPRs Spill: 298
LDS Size [bytes/block]: 10240
thats 362 vector registers spilled in this kernel, as the AGPRs are also spills in this case.
|
One of my current efforts is to make the kernel parameters more configurable as a function of hardware. I intend to soon procure an RDNA4 GPU so that I can implement support for the AMD WMMA instructions in the mma FA kernel. In principle, if the mma kernel can be made to work it should perform best since you need to hold fewer registers than the tile kernel and unlike the WMMA kernel you don't have to go through shared memory. Can you give me a list of the AMD hardware that you have so that I can adjust my purchases for wider coverage? |
|
Sure, i have gfx803 (Fiji / GCN3), gfx900 (Vega APU / GCN5), gfx906 (MI50 / GCN5.1), gfx908 (MI100 / CDNA1), gfx1030 (RX6800XT / RDNA2). I dont have any WMMA device at all, so any device with WMMA instructions would be very helpful. I know you dont intend to buy anything for actual use but from a practical perspective the large register file RDNA3 gpus (7900xtx, 7900xt, 7800xt) tend to be better for ai inference than RDNA4, just on account of being bigger devices with more CUs, vram and bandwith. |
|
@IMbackK I have 7800xt, 7900xt, 7900xtx cards, how can I help you? |
|
@broadbit-hu not atm. For regression testing it is useful to have people around who regularly run llamacpp on a given arch. But we where talking about doing feature development. When doing feature development the dev in question really needs to have the device with the instructions to be implemented on hand in one of his machines. |
This reverts commit c959b67.
This reverts commit c959b67.
This reverts commit c959b67.
This reverts commit c959b67.
This reverts commit c959b67.
This reverts commit c959b67.
This reverts commit c959b67.
This reverts commit c959b67.
This reverts commit c959b67.
)" This reverts commit 75a3a6c. d Update cudart64_12.dll Revert "Cudart 12.9" This reverts commit f79c687. Revert "Allow compile exe, pdf features off" This reverts commit 5e1c154. Update fattn.cu Update set-rows.cu batches Revert "try fix fattn again, porting some older code. the cc detection is not working well, so its hacky" This reverts commit 7b04191. Update ggml-cuda.cu Update fattn.cu Update fattn.cu Update fattn.cu Add option to disable MMA support on Turing Author : pt13762104 GGML_CUDA_NO_PEER_COPY to try to fix a crash on Gemma 3 Deactivate SWA when Fast Forwarding, commented Wrench Fix for the SWA I borked Clean-up quantkv algo comment warp sizes for now in IQ_K MMQ Kernels KV 24 -> KV 31 Add a readme. ngxson's commented hack Try some hack for gpt-oss Update llama-vocab.cpp Bump Windows max open files from 512 to 2048 Author : Thireus CLI - Specify GGML_TYPE to quantize for the main tensors. (#91) To complement the token_embd.weight and output.weight : attn_v.weight attn_k.weight. attn_q_weight attn_output.weight attn_qkv.weight ffn_gate ffn_down ffn_up EsoCroK naming v1.99430_b6645-6_Q6-IO2346_RMv1.17.99m Disable I2_K cpu quantization. To allow compilation. MMQ code adaptation Update mmq.cuh MMQ Initial code for IQ2,3,4,5,6_K IQ_K quants first gen (4, 5, 6) Some logs back Batches Croco Bench. Double the anti-abuse limits Allow compile exe, pdf features off Revert "Allow compile exe, pdf features off" This reverts commit 5e2451f129f0bca326f74aae24df475c0410cdbf. Update koboldcpp.py Revert "Allow compile exe, pdf features off" This reverts commit 2a7e9e004e8578a05fb67967d09cf36263867b9b. Revert "Allow compile exe, pdf features off" This reverts commit b4fd7809a4f77ff18bd415fcfb2d5f435e3b63a3. quantization tweaks iq3_ks quantization tweaks Minor iq3_k tweak q2_K tweaks q3_K tweaks q4_K tweaks q5_K tweaks GGUF v14 attempt of second fix. loosen gguf restrictions. Quantization improvements #295 and #302, GGML part only Improved IQ2_XS quantization #312 Improved IQ1_M quantization #327 ggml_row_size accounting fix for GGUF v14 Credits : @ikawrakow Fighting with cmake #279 Drop the GGML count limitation limit Old markings Customize KCPP.py Croco additional chat adapters andtemplates Reinstate "skip barrier of noop" Allow q8_0 KV cache for head size 256 #330 Up FA KV modes 256 candidates (1024 with Grammar) Adapt q6_0 MMQ to llama.cpp mainline Q6_0 MMQ Kernel attempt MMQ for Q6_0 authored by Ikawrakow Add Q6_0 MMQ to template generator authored by Ikawrakow Q6_0 KVQ for KCPP/Croco -> KV22 For release. fix a few lazy-cuts and hiccups left during the merge of IQ4_NL. dequantize for q6_0 and related cpy Enable q6_0 for flash attention As with IQ4_NL, just for head size of 128 for now. Without GGML_CUDA_FA_ALL_QUANTS set, only Q6_0 + Q5_0 and Q8_0 + Q6_0 are included. With this the VRAM poor have better options for selecting the best possible (as allowed by VRAM, model size, context length) quantized KV-cache. PR by Ikawrakow on ik_llama.cpp Adding Q6_0 (#77) Rev 20240807 * Adding q6_0 - basics + AVX2/Zen4 working * Adding q6_0: CUDA dequantize works, but not mmvq * Adding q6_0: CUDA mmvq works * Adding q6_0: CUDA cpy, so Q6_0 can be used for KV-cache * Add q6_0 to CPU flash attention Disappointing result: for LlaMA-3.2-1B, q6_0 K- and V-cache gives about the same PPL as q8_0 K-cache and q4_0 V-cache, while needing the exact same RAM. I.e., what was the point? * q6_0: slightly better kv-cache result Better than q8_0+q4_0, but not as good as q8_0+iq4_nl * q6_0: works on ARM_NEON * q6_0: dequantize works on Metal, but not vector dot product * q6_0: it now works on Metal Outperforms q5_0 by a significant margin. E.g. | model | size | params | backend | ngl | threads | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | ------------: | ---------------: | | llama 8B Q6_0 | 6.08 GiB | 8.03 B | Metal | 100 | 4 | tg128 | 44.02 ± 0.08 | | llama 8B Q5_0 | 5.21 GiB | 8.03 B | Metal | 100 | 4 | tg128 | 40.13 ± 0.12 | | llama 8B Q6_0 | 6.08 GiB | 8.03 B | Metal | 100 | 4 | pp512 | 500.55 ± 0.32 | | llama 8B Q5_0 | 5.21 GiB | 8.03 B | Metal | 100 | 4 | pp512 | 448.02 ± 0.27 | * q6_0: can now be used for kv-cache on Metal -> skipped. --------- Adaptation to mainline by me! IQ4_NL KVQ for KCPP/Croco missing templates instances for KVQ IQ4_NL Update fattn.cu for KVQ IQ4_NL Update fattn-vec-f16.cuh for KVQ IQ4_NL Update fattn-vec-f32.cuh for KVQ IQ4_NL CML and Makefile FOR IQ4_NL KV_IQ4_NL uncommenting VEC16 cases KV_IQ4_NL uncommenting VEC32 cases Enable IQ4_NL for V-cache in token generation Add IQ4_NL + IQ4_NL to FA This is a better alternative than Q4_0 + Q4_0 for the VRAM poor. Comment unwanted add-in in makefile iq4_nl: faster quantization (#76) CUDA: faster float -> iq4_nl conversion (#73) * iqk_mul_mat: better iq4_nl implementation on Zen4/AVX2 PP-512 performance for LLaMA-3.1-8B goes to 162.6 t/s up from 133.2 t/s. Default Blas Batch Size = 128 Quant KV and Draft QKV, 24 modes With customizable QKV for the draft as well. And reduced Blas Batch Size for the draft model. Default Draft Amount = 4 Bench context size Max contextsize and steps Croco CML SCHED_MAX_COPIES = 1 And Croco usual additions to the CMakeList Cudart 12.9 Revert "CUDA: faster tile FA (Pascal/AMD), headsize 256 (ggml-org#15769)" This reverts commit 79bc429. Revert "HIP: use v_dot2_f32_f16 instruction for FA (ggml-org#15884)" This reverts commit 17bc5a8. Revert "CUDA: larger SRAM reads for tile FA, AMD FP16 dot (ggml-org#15927)" This reverts commit 0e6ff00. Revert "CUDA: fix FA occupancy, optimize tile kernel (ggml-org#15982)" This reverts commit c959b67. Revert "CUDA: fix compilation on CC 6.0 (ggml-org#16091)" This reverts commit 368560a. Co-Authored-By: Kawrakow <[email protected]> Co-Authored-By: Iwan Kawrakow <[email protected]>
This reverts commit c959b67.
This reverts commit c959b67.
This reverts commit c959b67.
This reverts commit c959b67.
This PR fixes a bug in the CUDA FlashAttention occupancy calculation. In rare cases too few kernels would be launched in parallel, leading to a few % less performance.
This PR also delivers what I think will be the last round of performance optimizations for the tile FA kernel: I revised the memory layout to consistently copy data in 8/16 byte chunks and delayed writing the KQ accumulators to shared memory after they have been compressed to FP16. I looked up the amount of shared memory on each AMD GPU and fit the tile sizes accordingly. One thing that could still be done is do the same GQA optimization as for the mma kernel but because the GPUs using the tile kernel are comparatively slower reducing the mask I/O has little impact; it could improve performance for small batch sizes > 1 though.
Performance changes