AMD GPU support via llama.cpp HIPBLAS

[jeromew]

Hello,

First of all thank you for your work on llamafile it seems like a great idea to simplify model usage.

It seems from the readme that at this stage llamafile does not support AMD GPUs.
The cuda.c in llamafile backend seems dedicated to cuda while ggml-cuda.h in llama.cpp has a GGML_USE_HIPBLAS option for ROCm support. ROCm support is now officially supported by llama.cpp according to their README about hipBLAS

I understand that ROCm support was maybe not priority#1 for llamafile but I was wondering if you had already tried to use the HIPBLAS llama.cpp option and have some insights on the work that would need to be done in llamafile in order to add this GPU family target.

from what I understand, the llama.cpp would take care of the GPU side of things, and llamafile would need to be modified to JIT-compile llama.cpp with the correct flags and maybe need a specific toolchain for the compilation (At least ROCm SDK).

Thanks for sharing your experience on this

[jart]

I've never used an AMD GPU before, so I can't answer questions about them. However we're happy to consider your request that llamafile support them. I'd encourage anyone else who wants this to leave a comment saying so. That'll help us gauge the interest level and determine what to focus on.

[franalbani]

I would also like this!

Thanks! Your work is mind-blowing.

[mildwood]

I'm also interested in using ROCm as I don't want to pay double for Nvidia!

[jesserizzo]

Agreed, I'd also love AMD support. In my opinion it also fits with the general mission of this project. Don't let big tech companies get a monopoly on LLMs, and also don't let Nvidia get a monopoly on AI computing. I dunno. Thanks for all the hard work, this is great.

[stlhood]

Thanks for the suggestions, folks. We're going to look into this.

Any recommendations on a specific AMD card we should use for dev and testing? We could pick up an RX 7900 XTX (i.e. basically a RTX 4090 equivalent) but I want to make sure the model we pick is broadly representative in terms of hardware support.

[lovenemesis]

AMD officially only support ROCm on one or two consumer hardware level GPU, RX7900XTX being one of them, with limited Linux distribution.

However, by following the guide here on Fedora, I managed to get both RX 7800XT and the integrated GPU inside Ryzen 7840U running ROCm perfectly fine. Those are the mid and lower models of their RDNA3 lineup. So, I think it's fair to say all RDNA3 ones would work.

Judging from other ROCm related topics on PyTorch, it seems that people with RDNA2 (RX6XXX) series cards are the majority folks. Probably due to the competitive pricing after crypto mining boom?

[jammm]

Thanks for the suggestions, folks. We're going to look into this.

Any recommendations on a specific AMD card we should use for dev and testing? We could pick up an RX 7900 XTX (i.e. basically a RTX 4090 equivalent) but I want to make sure the model we pick is broadly representative in terms of hardware support.

Any RDNA3 card should work fine. It's just the "official" support that's been navi31 based GPUs.
You can also compile and run llama.cpp just fine on Windows using the HIP SDK. My primary OS is Windows and I could get this port running myself but I'm having difficulty setting up cosmos bash (getting cosmo++ permission denied errors). If someone can help me setup cosmos bash properly, I should be able to get llamafile up and running on RDNA3 GPUs within a couple hours or so.

Ideally a cmake based pipeline would be best for Windows support, but I'd understand if the makefile paradigm is what cosmopolitan's built on.

[jammm]

So I managed to get llamafile compiling ggml-cuda.so using HIP but it fails at runtime:

building ggml-cuda with nvcc -arch=native...
/usr/bin/hipcc -march=native --shared -use_fast_math -fPIC -O3 -march=native -mtune=native -DNDEBUG -DGGML_BUILD=1 -DGGML_SHARED=1 -DGGML_CUDA_DMMV_X=32 -DGGML_CUDA_MMV_Y=1 -DK_QUANTS_PER_ITERATION=2 -DGGML_CUDA_PEER_MAX_BATCH_SIZE=128 -DGGML_USE_HIPBLAS -o /home/rpr/.llamafile/ggml-cuda.so.zmifde /home/rpr/.llamafile/ggml-cuda.cu -lhipblas -lrocblas
/home/rpr/.llamafile/ggml-cuda.so
libamdhip64.so.5: cannot enable executable stack as shared object requires: Invalid argument: failed to load library
warning: GPU offload not supported on this platform; GPU related options will be ignored
warning: you might need to install xcode (macos) or cuda (windows, linux, etc.) check the output above to see why support wasn't linked

normal dlopen() works fine with ggml-cuda.so. I'm not sure why cosmo_dlopen fails here. CC @jart any ideas?

Attaching strace log:
strace_llamafile_hip.txt

Using Ubuntu 22.04.2 LTS.

EDIT: Looks like this error goes away by setting the execstack bit on amdhip64 runtime library
sudo execstack -c /opt/rocm/lib/libamdhip64.so.5

EDIT EDIT: PR made at #122. Works fine on my navi31 machine running Ubuntu 22.04. I expect it to work fine on Windows as well, though it's not tested yet.

[franalbani]

Thanks for the suggestions, folks. We're going to look into this.

Any recommendations on a specific AMD card we should use for dev and testing? We could pick up an RX 7900 XTX (i.e. basically a RTX 4090 equivalent) but I want to make sure the model we pick is broadly representative in terms of hardware support.

I don't have enough experience to give advise, but I can contribute testing on a Thinkpad T495 with AMD Radeon Vega 10 Graphics.

[batfasturd]

I would also like this feature added since it is technically possible. I've used my 6750XT successfully with Llama.cpp and linux.