benchmarks?

Where are the benchmarks for various hardware - eg. apple silicon

[wizd]

M1 with 7B model: 94.24 ms per token
M1 with 13B model: 202.18 ms per token

speed with command line config -t 4. If use -t 8, half speed.

[ElRoberto538]

Using command line config -t 8, note this is in a VM assigned 42 logical cores out of the total 44, other services running on the server.
AMD EPYC 7443P 7B: 89.39 ms per token

[MLTQ]

M1 Pro 32GB, 30B model:

main: mem per token = 43387780 bytes
main: load time = 10701.85 ms
main: sample time = 279.92 ms
main: predict time = 37065.80 ms / 226.01 ms per token
main: total time = 51992.27 ms

[diimdeep]

Macbook Pro 2013, Intel i5, 2 cores, 8 GB RAM
7B 4bit model
main: mem per token = 14335844 bytes
main: load time = 8224.30 ms
main: sample time = 1918.08 ms
main: predict time = 308737.91 ms / 604.18 ms per token
main: total time = 331646.62 ms

Thank you for this awesome project.

[neuhaus]

Ryzen 7 3700X, 128GB RAM @ 3200, llama.cpp numbers:

$ ./main -m models/7B/ggml-model-q4_0.bin -t 8 -n 128
main: mem per token = 14434244 bytes
main:     load time =  1270.15 ms
main:   sample time =   325.76 ms
main:  predict time = 15147.15 ms / 117.42 ms per token
main:    total time = 17077.88 ms

$ ./main -m models/13B/ggml-model-q4_0.bin -t 8 -n 128
main: mem per token = 22439492 bytes
main:     load time =  2946.00 ms
main:   sample time =    86.11 ms
main:  predict time =  7358.48 ms / 216.43 ms per token
main:    total time = 11019.28 ms

$ ./main -m models/30B/ggml-model-q4_0.bin -t 8 -n 128
main: mem per token = 43387780 bytes
main:     load time =  6666.53 ms
main:   sample time =   332.71 ms
main:  predict time = 68779.27 ms / 533.17 ms per token
main:    total time = 77333.97 ms

$ ./main -m models/65B/ggml-model-q4_0.bin -t 8 -n 128
main: mem per token = 70897348 bytes
main:     load time = 14010.35 ms
main:   sample time =   335.09 ms
main:  predict time = 140527.48 ms / 1089.36 ms per token
main:    total time = 157951.48 ms

With the 30B model, a RTX 3090 manages 15 tokens/s using text-generation-webui

[MarkSchmidty]

1.2 tokens/s on a Samsung S22 Ultra running 4 threads.

The S22 obviously has a more powerful processor. But I do not think it is 12 times more powerful. It's likely you could get much faster speeds on the Pi.

I'd be willing to bet that the bottleneck is not the processor.

Reposting the 1.2 token/second Samsung S22 Ultra result here. (Originally posted in #58)

[ItsLogic]

I must say this running on my phone at all was surprising. Here are my results on an 8+gen1 for 4bit 7B

and results for my desktop with 13900k and 64gb ddr5
4bit quant 7B

main: mem per token = 14434244 bytes
main:     load time =   609.88 ms
main:   sample time =    36.60 ms
main:  predict time =  9487.02 ms / 71.33 ms per token
main:    total time = 10341.46 ms

full precision 7B

main: mem per token = 14434244 bytes
main:     load time = 26905.18 ms
main:   sample time =    37.78 ms
main:  predict time = 23033.74 ms / 173.19 ms per token
main:    total time = 50204.95 ms

4bit quant 65B

main: mem per token = 70897348 bytes
main:     load time = 83233.36 ms
main:   sample time =    36.90 ms
main:  predict time = 86000.03 ms / 646.62 ms per token
main:    total time = 172458.39 ms

Edit:
Did something really stupid and ran 4bit 13B on my phone. TLDR its slow, dont. (unless you have lots of ram)
My phone has 12gb ram and 7gb of manually added swap. I had to run it through an adb root shell instead of termux as the android memory manager would kill termux as soon as the model started to load. The downside to this approach is that everything else on my phone is killed meaning I couldnt even get the screen to turn on while inference was running

main: mem per token = 22357508 bytes
main:     load time = 29320.15 ms
main:   sample time =  2254.09 ms
main:  predict time = 5227881.50 ms / 39307.38 ms per token
main:    total time = 5335562.00 ms

[totoCZ]

Here is my quick look at
2x Intel Xeon Gold 5120 @ 2.20GHz, march=native

7B

main: mem per token = 14762244 bytes
main:     load time =  3378.15 ms
main:   sample time =    15.87 ms
main:  predict time =  4494.55 ms / 115.24 ms per token
main:    total time =  8328.48 ms

7B fp16

main: mem per token = 14532644 bytes
main:     load time = 27977.19 ms
main:   sample time =    24.71 ms
main:  predict time =  9378.29 ms / 275.83 ms per token
main:    total time = 38135.22 ms

13B

main: mem per token = 22562468 bytes
main:     load time = 16860.55 ms
main:   sample time =   170.45 ms
main:  predict time = 56121.11 ms / 308.36 ms per token
main:    total time = 74377.55 ms

13B fp16

main: mem per token = 22562468 bytes
main:     load time = 64448.62 ms
main:   sample time =   129.29 ms
main:  predict time = 61505.41 ms / 455.60 ms per token
main:    total time = 127347.54 ms

30B

main: mem per token = 43547620 bytes
main:     load time = 51269.82 ms
main:   sample time =    49.77 ms
main:  predict time = 41543.11 ms / 585.11 ms per token
main:    total time = 95383.98 ms

65B

main: mem per token = 71553028 bytes
main:     load time = 99438.78 ms
main:   sample time =    44.94 ms
main:  predict time = 69203.49 ms / 1017.70 ms per token
main:    total time = 218532.06 ms

This is with 14 / 28 threads.
Running with 56 threads slows it down, probably NUMA.
I think 115ms is still a good result for this CPU.

So if anyone like me was wondering, does having a million cores in a server CPU give you a 65B model?
The answer is no.

[MarkSchmidty]

So if anyone like me was wondering, does having a million cores in a server CPU give you a 65B model?

It's clear by now that llama.cpp speed mostly depends on max single core performance for comparisons within the same CPU architecture, up to a limit where all CPUs of the same architecture perform approximately the same. Memory bandwidth and memory bus chokepoints appear to be the major bottlenecks after that point.

Using more cores can slow things down for two reasons:

1. More memory bus congestion from moving bits between more places. llama.cpp is well written and easily maxes out the memory bus on most even moderately powerful systems.
2. Reducing your effective max single core performance to that of your slowest cores. This is usually the primary culprit on 4 or 6 core devices (mostly phones) which often have 2 power cores and then 2-4 balanced and/or "efficiency" cores.

With these learnings in mind, it would be good to see benchmark results from anyone who manages to find some yet unknown optimization in their configuration, OS environment, or hardware environment.

[hanvyj]

How are you getting such good performance?

I'm running an i7 10750h 32gig ram with -m ./models/7B/ggml-model-f16.bin -t 12 -n 128

main: mem per token = 14499844 bytes
main:     load time =  8892.24 ms
main:   sample time =  1988.34 ms
main:  predict time = 270018.50 ms / 2093.17 ms per token
main:    total time = 287685.50 ms

2+s per token! I get similar with the 4 bit quant, if not worse.

Edit: Running with -m ./models/7B/ggml-model-q4_0.bin -t 12 -n 128

main: mem per token = 14499844 bytes
main:     load time =  1631.32 ms
main:   sample time =  1513.06 ms
main:  predict time = 574477.00 ms / 6047.13 ms per token
main:    total time = 596436.75 ms