Rockchip RK3588 perf

[IngwiePhoenix]

Just did a very simple run with llama-7b-4bit. It... took a while. Had it run in a screen. But, it worked!

root@FriendlyWrt /s/o/llama.cpp (master)# time ./main --color -m models/ggml-model-q4_0.bin -p "Hello there!"
main: seed = 1680443840
llama_model_load: loading model from 'models/ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 4096
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 32
llama_model_load: n_layer = 32
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 11008
llama_model_load: n_parts = 1
llama_model_load: type    = 1
llama_model_load: ggml map size = 4017.70 MB
llama_model_load: ggml ctx size =  81.25 KB
llama_model_load: mem required  = 5809.78 MB (+ 1026.00 MB per state)
llama_model_load: loading tensors from 'models/ggml-model-q4_0.bin'
llama_model_load: model size =  4017.27 MB / num tensors = 291
llama_init_from_file: kv self size  =  256.00 MB

system_info: n_threads = 8 / 8 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 0 | VSX = 0 |
sampling: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.100000
generate: n_ctx = 512, n_batch = 8, n_predict = 128, n_keep = 0


 Hello there! I am a freelance illustrator based in New Zealand. I grew up with an appreciation for the natural world, which has inspired me to create my work through observation and playful experimentation.
My focus is on watercolour painting (in particular), as well as digital art & animation. My style is bright & bold, vibrant, dynamic & colourful - I love animals!
I am always keen to collaborate with other artists/creatives, so if you are interested in working together please feel free to drop me a line. [end of text]

llama_print_timings:        load time = 93487.23 ms
llama_print_timings:      sample time =   704.72 ms /   115 runs   (    6.13 ms per run)
llama_print_timings: prompt eval time = 92466.10 ms /     4 tokens (23116.52 ms per token)
llama_print_timings:        eval time = 11195694.23 ms /   114 runs   (98207.84 ms per run)
llama_print_timings:       total time = 11289895.19 ms

________________________________________________________
Executed in  188.18 mins    fish           external
   usr time  324.60 mins    0.00 millis  324.60 mins
   sys time   11.70 mins    1.70 millis   11.70 mins

Model was loaded from external microSD via internal bus.

Im quite amazed this worked at all, honestly.

CPU Info in detail:

# lscpu
Architecture:           aarch64
  CPU op-mode(s):       32-bit, 64-bit
  Byte Order:           Little Endian
CPU(s):                 8
  On-line CPU(s) list:  0-7
Vendor ID:              ARM
  Model name:           Cortex-A55
    Model:              0
    Thread(s) per core: 1
    Core(s) per socket: 4
    Socket(s):          1
    Stepping:           r2p0
    CPU(s) scaling MHz: 100%
    CPU max MHz:        1800.0000
    CPU min MHz:        408.0000
    BogoMIPS:           48.00
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
  Model name:           Cortex-A76
    Model:              0
    Thread(s) per core: 1
    Core(s) per socket: 2
    Socket(s):          2
    Stepping:           r4p0
    CPU(s) scaling MHz: 68%
    CPU max MHz:        2352.0000
    CPU min MHz:        408.0000
    BogoMIPS:           48.00
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
Caches (sum of all):
  L1d:                  384 KiB (8 instances)
  L1i:                  384 KiB (8 instances)
  L2:                   2.5 MiB (8 instances)
  L3:                   3 MiB (1 instance)
Vulnerabilities:
  Itlb multihit:        Not affected
  L1tf:                 Not affected
  Mds:                  Not affected
  Meltdown:             Not affected
  Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:           Mitigation; __user pointer sanitization
  Spectre v2:           Vulnerable: Unprivileged eBPF enabled
  Srbds:                Not affected
  Tsx async abort:      Not affected

(/proc/cpuinfo doesnt give any more useful details here, sadly.)

Hardware is a FriendlyElec NanoPi R6s

[prusnak]

Thanks for the info

[jimtendo]

Thanks for posting this.

Just as a heads up, the RK3588 does have NPU units on it but these are not leveraged with the llama.cpp codebase (at time of writing). If other devs are interested, the NPU API for this can be found in this file: https://github.com/rockchip-linux/rknpu2/blob/master/runtime/RK3588/Linux/librknn_api/include/rknn_api.h

Note: I'm sure I've read somewhere that INT4 tensors should be supported, but I cannot see them in that API. Also, I believe the model might have to be converted to a specific RK3588 format (toolkit link in the root README.md)?

I did actually expect far better performance even with the CPUs only though with a 7B model. I notice this is an 8GB RK3588, so maybe there was a lot of memory swapping happening that slowed it down.

I don't have any chips with RK3588 yet, but if I manage to get one, I'll try to do some testing on my side. Might make great little units for running a dedicated assistant on if it can be optimized well.

[IngwiePhoenix]

If there is a specific test you want me to run, let me know!

I don't have any swap configured, regrettably. But what could easily have happened is that because this was running literally alongside my homeserver stuff, that memory management on th e kernel side got quite hectic. :)

Also, llama.cpp has improved a lot since last time - so I might just rerun the test, to see what happens. Also, Vicuna and StableLM are a thing now. Might as well give it a shot... that said, I'd have to think of a good way to gather the output into a nice table structure - because I don't want to flood this ticket, or anyone else, with a crapton of redundant output. xD

That all said, there is one more thing:

# dmesg -l err | grep -i npu
[    3.702909] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdab0000-0xfdabffff]
[    3.702953] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdac0000-0xfdacffff]
[    3.702978] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdad0000-0xfdadffff]
[    3.707178] debugfs: Directory 'fdab0000.npu-rknpu' with parent 'vdd_npu_s0' already present!
[    3.729270] RKNPU fdab0000.npu: failed to find power_model node
[    3.729289] RKNPU fdab0000.npu: RKNPU: failed to initialize power model
[    3.729297] RKNPU fdab0000.npu: RKNPU: failed to get dynamic-coefficient

Thanks to RockChip's - at least in my experience - rather spotty documentation, I couldn't figure out if these messages were relevant or not. Though it'd actually be interesting to see INT4 on this.

[jimtendo]

I did a quick test with this on Orange Pi 5 16GB using a 7B Q5_1 model. My setup is a bit clunky, so I don't have a proper benchmark (will re-run and edit in next week when I'm setup better), but I'd estimate performance at around at almost 1 token/sec. This was using 7 threads. The heatsink became pretty hot to touch - I suspect the slower performance above might've been due to either a) memory constraints or b) thermal throttling.

Would love to see how well this could run if leveraging the NPU, but I don't think the RK SDK supports INT4 quant yet. Basically, the RK process is that the models have to be converted into an RK-compatible format using their SDK's, so the quantization probably won't be great using that approach.

I haven't looked into whether RK API is low-level enough that it might be able to support running GGML models yet, but that'd probably work better than using whatever quantization process RK SDK may eventually support

[jimtendo]

I tinkered around a bit more with this last night.

I was able to get around 500ms/token using 4 threads on a 7B Q5_1.

I also played around with the new OpenCL implementation (using CLBlast), but this was significantly slower if I transfer all layers to GPU (> 1s/token). I don't have time to thoroughly investigate but, looking at the GGML OpenCL implementation, I suspect a lot of the slowdown might be how memory is handled.

In the OpenCL implementation, it looks like the tensors might be copied to the GPU as opposed to using a pointer to the Host Memory (I noticed some loops in there that do this). This makes sense for non-iGPUs (as they have their own VRAM), but probably results in unnecessary copy op's for devices with shared RAM/VRAM like the RK3588 (and AMD APU's for that matter). I believe there are flags that can be used to simply point OpenCL to host memory, but I'm unsure whether it would be compatible with the GGML tensor format. Might be a worthy optimization to consider though if it would speed up inference on AMD APU's also.

Side-note: I have tiny heatsinks on my Orange Pi 5. These get quite hot and I notice inference time slows down quite a bit as they heat up, so assuming the device gets underclocked to maintain safe temperatures.

[jimtendo]

I probably can't recommend a specific board sorry. I haven't priced them out.

Just want to add to this though - the guy that's been doing a lot of the work on the llama.cpp GPU implementations isn't sure if optimizations to the OpenCL code will yield that much benefit for boards like this. He posted the following graph yesterday indicating that the big bottleneck appears to be memory.

[Honza-Kubat]

@spv420 here are some links - I have Orange Pi 5 & 5b plan on purchasing NanoPC-T6 & Orange Pi 5 Plus as well

Orange Pi 5 8GB
https://www.aliexpress.com/item/1005004941850323.html?spm=a2g0o.productlist.main.1.68d1155cl9LNeF&algo_pvid=6e82402a-408a-44a6-b296-6048ebc00ad4&algo_exp_id=6e82402a-408a-44a6-b296-6048ebc00ad4-0&pdp_npi=4%40dis%21CZK%212396.32%212396.32%21%21%21105.00%21%21%4021038ed816933130871764786ed591%2112000034095080979%21sea%21CZ%21184718102%21&curPageLogUid=sc5SN4Q8ijQm

Orange Pi 5 16GB
https://www.aliexpress.com/item/1005004942239582.html?spm=a2g0o.productlist.main.3.68d1155cl9LNeF&algo_pvid=1f6afb66-3fab-4538-8ad4-caf0e491625d&algo_exp_id=1f6afb66-3fab-4538-8ad4-caf0e491625d-1&pdp_npi=4%40dis%21CZK%213375.39%212396.55%21%21%21147.90%21%21%4021038ed816933130651116464ed585%2112000031099030945%21sea%21CZ%21184718102%21&curPageLogUid=45YyOUlpixe9

Orange Pi 5 Plus 16GB
https://www.aliexpress.com/item/1005005561909873.html?spm=a2g0o.productlist.main.5.68d1155cl9LNeF&algo_pvid=6e82402a-408a-44a6-b296-6048ebc00ad4&algo_exp_id=6e82402a-408a-44a6-b296-6048ebc00ad4-2&pdp_npi=4%40dis%21CZK%213637.16%213637.16%21%21%21159.37%21%21%4021038ed816933130871764786ed591%2112000033644649629%21sea%21CZ%21184718102%21&curPageLogUid=kp81clwWsITC

NanoPC T6
https://www.friendlyelec.com/index.php?route=product/product&path=69&product_id=292

[marty1885]

Not sure if this helps the discussion. I made a fork that supports the RK3588 NPU via the matrix multiplication API. Unfortunately it is not faster then just using the CPU and generates questionable output due to running in int8 mode (FP16 is too slow).

Feel free to contribute, and see if anyone can work around the accuracy issue. I have a prototype that gets up to 10% faster by chunking operations. But it's complicated and I feel not worth the work if all I'm able to get is hallucinating outputs.

I'd love to upstream the code. Please contribute if you are also interested in the subject

https://github.com/marty1885/llama.cpp/tree/rknpu2-backend

[jimtendo]

Not sure if this helps the discussion. I made a fork that supports the RK3588 NPU via the matrix multiplication API. Unfortunately it is not faster then just using the CPU and generates questionable output due to running in int8 mode (FP16 is too slow).

Feel free to contribute, and see if anyone can work around the accuracy issue. I have a prototype that gets up to 10% faster by chunking operations. But it's complicated and I feel not worth the work if all I'm able to get is hallucinating outputs.

I'd love to upstream the code. Please contribute if you are also interested in the subject

https://github.com/marty1885/llama.cpp/tree/rknpu2-backend

Thanks for this! I looked into it at one point too, but I think the bottleneck will be the RAM speed on the Pi 5? This approach might still be able to speed up prompt ingestion substantially though.

Do you know if using the NPU reduces power consumption? I'm an idiot and installed a tiny heatsink on my Pi 5, so it throttles very quickly.

Will try and give your fork a go next week when I get some time.

[marty1885]

but I think the bottleneck will be the RAM speed on the Pi 5?

No, the NPU on the RK3588 is really, really bad at matrix multiplication. It's designed for vision models thus focused on convolution. It has a pretty low FLOPS when doing matrix multiplication.

This approach might still be able to speed up prompt ingestion substantially though.

Maybe, but the inaccuracy is quite significant. I am not sure what'll happen.

Do you know if using the NPU reduces power consumption? I'm an idiot and installed a tiny heatsink on my Pi 5, so it throttles very quickly.

I think it can. But not with my backend in the current state. My backend only uses 1 thread out out all given by GGML. And GGML will spin non-working threads. It's a design flaw in GGML itself and needs major refactor. Can't just use 1 thread either. Some matrices are too large to fit on the NPU. It's possible to split the work and distribute to different NPU cores. But I it's too much work for little gain (as the model is hallucinating constantly).

To compile and run my fork. I don't recommend running more then 13 layers or a 7B model on the NPU. It starts going crazy afterwards. I develop with 10.

cmake .. -DLLAMA_RKNPU2=ON
make -j
./bin/main .... -ngl 10

Also you need a Q8_0 model. It's kinda moot for lower bits since the minimal supported by the NPU is 8.

[psyhtest]

Great thread! I have a Firefly RK3588S board, so it would be great to try this out. Don't have much hope for the NPU, but am wondering if offloading matrix multiplications to the Arm Mali GPU via Arm Computer Library might be worthwhile? Any thoughts?

[marty1885]

@prusnak I tried something similar with GGML's OpenCL backend way back. I modified it enough to get RWKV (not llama) running on the Mali GPU. it has many problems. Mainly

1. ARM's OpenCL implementation is buggy and doesn't play nice with GGML
2. For some reason, the OpenCL latency is very high on my OrangePi 5
3. Decompressing k-quants requires a lot of integer operations. But the Mali GPU has 1/4 integer capacity compared to floating point.

ACL can work. But I have question if it'll be helpful. GGML pre-transposes matrix B in matmul(A, B). Thus access pattern is already as efficient as it can. IMO most OpenCL compilers can easily optimize that (need to confirm by decompiling though). After getting GGML working on Mali. You'll have to choose. Either to not support k-quants and run into the same accuracy vs bandwidth tradeoff as I do with the NPU. Or support k-quants and make k-quant decompression fast on the Mali.

Good luck. I'd love to see more LLMs on the edge.

====

For anyone interested; progress update on my side. With RKNPU2 1.6.0. It almost makes sense to use the NPU. I'm less then 10% off to being faster then the CPU on INT8 mode with just 1 NPU core. Next step is to debug non-square matrix multiplication. Something somewhere is wrong.

I won't update every step here. Please either follow my fork or check my blog from time to time.
Latest progress: https://clehaxze.tw/gemlog/2023/12-17-update-on-ggml-rknpu2-backend-and-rknpu2-1_6_0.gmi

[ggerganov]

@marty1885 Your work is very interesting. Have you considered running Whisper models on the NPU? Could be better suited as the models are much smaller compared to 7B LLMs and would immediately have various real-world applications.

[marty1885]

@ggerganov Thanks, Already done by other people. https://github.com/usefulsensors/useful-transformers runs Whisper on the NPU. They are able to do much extensive optimizations compared to GGML though. The NPU demands a custom matrix layout for maximal performance. And they are able to eliminate a majority of layout conversions by abstracting them away.

Actually good idea. I can try targeting my work against whisper.cpp. Do you know any use cases for it? And what would be the process to upstream an entire new backend?