Misc. bug: Performance regression on aarch64 q4_0

[njsyw1997]

Name and Version

llama-cli --version
version: 5615 (f470bc3)
built with Android (13324770, +pgo, +bolt, +lto, +mlgo, based on r530567d) clang version 19.0.0 (https://android.googlesource.com/toolchain/llvm-project 97a699bf4812a18fb657c2779f5296a4ab2694d2) for x86_64-unknown-linux-gnu

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-bench

Command line

Problem description & steps to reproduce

Q4_0 performance significantly dropped after this commit
/build-android-f470bc36/llama-bench -m ../gemma-2-2b-q4_0.gguf -p 512 -n 0

model	size	params	backend	threads	test	t/s
gemma2 2B Q4_0	1.51 GiB	2.61 B	CPU	8	pp512	15.84 ± 0.01

build: f470bc3 (5615)
/build-android-8f47e25f/llama-bench -m ../gemma-2-2b-q4_0.gguf -p 512 -n 0

model	size	params	backend	threads	test	t/s
gemma2 2B Q4_0	1.51 GiB	2.61 B	CPU	8	pp512	138.02 ± 8.88

build: 8f47e25 (5614)

First Bad Commit

No response

Relevant log output

[ggerganov]

Could you provide CMake logs + llama-cli logs?

cc @xctan

[njsyw1997]

For f470bc3

export ANDROID_NDK=/home/syw/android/android-ndk-r28b
commit_id=$(git rev-parse --short HEAD)
echo "Building for commit: $commit_id"
cmake -S . -B build-android-$commit_id -GNinja   -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake   \
    -DCMAKE_BUILD_TYPE=Release \
    -DANDROID_ABI=arm64-v8a   -DANDROID_PLATFORM=android-34 \
    -DCMAKE_CXX_FLAGS="-march=armv8.7a" -DCMAKE_C_FLAGS="-march=armv8.7a" \
    -DBUILD_SHARED_LIBS=OFF -DLLAMA_CURL=OFF -DGGML_LLAMAFILE=OFF -DGGML_OPENMP=OFF
cmake --build build-android-$commit_id --config Release -j8

-- The C compiler identification is Clang 19.0.0
-- The CXX compiler identification is Clang 19.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/syw/android/android-ndk-r28b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/syw/android/android-ndk-r28b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /usr/bin/git (found version "2.34.1") 
-- Setting GGML_NATIVE_DEFAULT to OFF
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE  
-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF
-- CMAKE_SYSTEM_PROCESSOR: aarch64
-- GGML_SYSTEM_ARCH: ARM
-- Including CPU backend
-- ARM detected
-- Performing Test GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E
-- Performing Test GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E - Failed
-- Adding CPU backend variant ggml-cpu:  
-- Configuring done
-- Generating done
-- Build files have been written to: /home/syw/llama.cpp/build-android-f470bc36

llama-cli -m ../gemma-2-2b-q4_0.gguf --prompt "Where is the capital of France?" -c 512 -n 64                          <
build: 5615 (f470bc36) with Android (13324770, +pgo, +bolt, +lto, +mlgo, based on r530567d) clang version 19.0.0 (https://android.googlesource.com/toolchain/llvm-project 97a699bf4812a18fb657c2779f5296a4ab2694d2) for x86_64-unknown-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_loader: loaded meta data with 33 key-value pairs and 288 tensors from ../gemma-2-2b-q4_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Gemma 2 2b
llama_model_loader: - kv   3:                           general.basename str              = gemma-2
llama_model_loader: - kv   4:                         general.size_label str              = 2B
llama_model_loader: - kv   5:                            general.license str              = gemma
llama_model_loader: - kv   6:                               general.tags arr[str,1]       = ["text-generation"]
llama_model_loader: - kv   7:                      gemma2.context_length u32              = 8192
llama_model_loader: - kv   8:                    gemma2.embedding_length u32              = 2304
llama_model_loader: - kv   9:                         gemma2.block_count u32              = 26
llama_model_loader: - kv  10:                 gemma2.feed_forward_length u32              = 9216
llama_model_loader: - kv  11:                gemma2.attention.head_count u32              = 8
llama_model_loader: - kv  12:             gemma2.attention.head_count_kv u32              = 4
llama_model_loader: - kv  13:    gemma2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  14:                gemma2.attention.key_length u32              = 256
llama_model_loader: - kv  15:              gemma2.attention.value_length u32              = 256
llama_model_loader: - kv  16:                          general.file_type u32              = 2
llama_model_loader: - kv  17:              gemma2.attn_logit_softcapping f32              = 50.000000
llama_model_loader: - kv  18:             gemma2.final_logit_softcapping f32              = 30.000000
llama_model_loader: - kv  19:            gemma2.attention.sliding_window u32              = 4096
llama_model_loader: - kv  20:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  21:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  22:                      tokenizer.ggml.tokens arr[str,256000]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv  23:                      tokenizer.ggml.scores arr[f32,256000]  = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  24:                  tokenizer.ggml.token_type arr[i32,256000]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  25:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  26:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  27:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  28:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  29:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  30:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  31:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  32:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  105 tensors
llama_model_loader: - type q4_0:  182 tensors
llama_model_loader: - type q6_K:    1 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_0
print_info: file size   = 1.51 GiB (4.97 BPW)
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 249
load: token to piece cache size = 1.6014 MB
print_info: arch             = gemma2
print_info: vocab_only       = 0
print_info: n_ctx_train      = 8192
print_info: n_embd           = 2304
print_info: n_layer          = 26
print_info: n_head           = 8
print_info: n_head_kv        = 4
print_info: n_rot            = 256
print_info: n_swa            = 4096
print_info: is_swa_any       = 1
print_info: n_embd_head_k    = 256
print_info: n_embd_head_v    = 256
print_info: n_gqa            = 2
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 6.2e-02
print_info: n_ff             = 9216
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 10000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 8192
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = 2B
print_info: model params     = 2.61 B
print_info: general.name     = Gemma 2 2b
print_info: vocab type       = SPM
print_info: n_vocab          = 256000
print_info: n_merges         = 0
print_info: BOS token        = 2 '<bos>'
print_info: EOS token        = 1 '<eos>'
print_info: EOT token        = 107 '<end_of_turn>'
print_info: UNK token        = 3 '<unk>'
print_info: PAD token        = 0 '<pad>'
print_info: LF token         = 227 '<0x0A>'
print_info: EOG token        = 1 '<eos>'
print_info: EOG token        = 107 '<end_of_turn>'
print_info: max token length = 48
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors:   CPU_Mapped model buffer size =  1548.25 MiB
load_tensors:   CPU_REPACK model buffer size =  1085.91 MiB
........................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 512
llama_context: n_ctx_per_seq = 512
llama_context: n_batch       = 512
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: freq_base     = 10000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (512) < n_ctx_train (8192) -- the full capacity of the model will not be utilized
llama_context:        CPU  output buffer size =     0.98 MiB
llama_kv_cache_unified_iswa: creating non-SWA KV cache, size = 512 cells
llama_kv_cache_unified:        CPU KV buffer size =    26.00 MiB
llama_kv_cache_unified: size =   26.00 MiB (   512 cells,  13 layers,  1 seqs), K (f16):   13.00 MiB, V (f16):   13.00 MiB
llama_kv_cache_unified_iswa: creating     SWA KV cache, size = 512 cells
llama_kv_cache_unified:        CPU KV buffer size =    26.00 MiB
llama_kv_cache_unified: size =   26.00 MiB (   512 cells,  13 layers,  1 seqs), K (f16):   13.00 MiB, V (f16):   13.00 MiB
llama_context:        CPU compute buffer size =   504.50 MiB
llama_context: graph nodes  = 1154
llama_context: graph splits = 1
common_init_from_params: setting dry_penalty_last_n to ctx_size = 512
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
main: llama threadpool init, n_threads = 8

system_info: n_threads = 8 (n_threads_batch = 8) / 8 | CPU : NEON = 1 | ARM_FMA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | REPACK = 1 |

sampler seed: 1342888035
sampler params:
        repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
        dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 512
        top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.800
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampler chain: logits -> logit-bias -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
generate: n_ctx = 512, n_batch = 2048, n_predict = 64, n_keep = 1

Where is the capital of France?
What is the country’s capital city?
Where is the capital of the USA?
What is the capital of France?
What is the capital of the Czech Republic?
What is the capital of Hungary?
What is the capital of the Netherlands?
What is the capital of the Philippines?


llama_perf_sampler_print:    sampling time =      13.37 ms /    72 runs   (    0.19 ms per token,  5386.80 tokens per second)
llama_perf_context_print:        load time =    1543.02 ms
llama_perf_context_print: prompt eval time =     715.20 ms /     8 tokens (   89.40 ms per token,    11.19 tokens per second)
llama_perf_context_print:        eval time =   13007.56 ms /    63 runs   (  206.47 ms per token,     4.84 tokens per second)
llama_perf_context_print:       total time =   13749.33 ms /    71 tokens

[njsyw1997]

For 8f47e25

Building for commit: 8f47e25f
-- The C compiler identification is Clang 19.0.0
-- The CXX compiler identification is Clang 19.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/syw/android/android-ndk-r28b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/syw/android/android-ndk-r28b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /usr/bin/git (found version "2.34.1") 
-- Setting GGML_NATIVE_DEFAULT to OFF
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE  
-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF
-- CMAKE_SYSTEM_PROCESSOR: aarch64
-- GGML_SYSTEM_ARCH: ARM
-- Including CPU backend
-- ARM detected
-- Performing Test GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E
-- Performing Test GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E - Failed
-- Adding CPU backend variant ggml-cpu:  
-- Configuring done
-- Generating done
-- Build files have been written to: /home/syw/llama.cpp/build-android-8f47e25f

build-android-8f47e25f/llama-cli -m ../gemma-2-2b-q4_0.gguf --prompt "Where is the capital of France?" -c 512 -n 64    <
build: 5614 (8f47e25f) with Android (13324770, +pgo, +bolt, +lto, +mlgo, based on r530567d) clang version 19.0.0 (https://android.googlesource.com/toolchain/llvm-project 97a699bf4812a18fb657c2779f5296a4ab2694d2) for x86_64-unknown-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_loader: loaded meta data with 33 key-value pairs and 288 tensors from ../gemma-2-2b-q4_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Gemma 2 2b
llama_model_loader: - kv   3:                           general.basename str              = gemma-2
llama_model_loader: - kv   4:                         general.size_label str              = 2B
llama_model_loader: - kv   5:                            general.license str              = gemma
llama_model_loader: - kv   6:                               general.tags arr[str,1]       = ["text-generation"]
llama_model_loader: - kv   7:                      gemma2.context_length u32              = 8192
llama_model_loader: - kv   8:                    gemma2.embedding_length u32              = 2304
llama_model_loader: - kv   9:                         gemma2.block_count u32              = 26
llama_model_loader: - kv  10:                 gemma2.feed_forward_length u32              = 9216
llama_model_loader: - kv  11:                gemma2.attention.head_count u32              = 8
llama_model_loader: - kv  12:             gemma2.attention.head_count_kv u32              = 4
llama_model_loader: - kv  13:    gemma2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  14:                gemma2.attention.key_length u32              = 256
llama_model_loader: - kv  15:              gemma2.attention.value_length u32              = 256
llama_model_loader: - kv  16:                          general.file_type u32              = 2
llama_model_loader: - kv  17:              gemma2.attn_logit_softcapping f32              = 50.000000
llama_model_loader: - kv  18:             gemma2.final_logit_softcapping f32              = 30.000000
llama_model_loader: - kv  19:            gemma2.attention.sliding_window u32              = 4096
llama_model_loader: - kv  20:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  21:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  22:                      tokenizer.ggml.tokens arr[str,256000]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv  23:                      tokenizer.ggml.scores arr[f32,256000]  = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  24:                  tokenizer.ggml.token_type arr[i32,256000]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  25:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  26:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  27:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  28:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  29:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  30:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  31:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  32:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  105 tensors
llama_model_loader: - type q4_0:  182 tensors
llama_model_loader: - type q6_K:    1 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_0
print_info: file size   = 1.51 GiB (4.97 BPW)
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 249
load: token to piece cache size = 1.6014 MB
print_info: arch             = gemma2
print_info: vocab_only       = 0
print_info: n_ctx_train      = 8192
print_info: n_embd           = 2304
print_info: n_layer          = 26
print_info: n_head           = 8
print_info: n_head_kv        = 4
print_info: n_rot            = 256
print_info: n_swa            = 4096
print_info: is_swa_any       = 1
print_info: n_embd_head_k    = 256
print_info: n_embd_head_v    = 256
print_info: n_gqa            = 2
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 6.2e-02
print_info: n_ff             = 9216
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 10000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 8192
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = 2B
print_info: model params     = 2.61 B
print_info: general.name     = Gemma 2 2b
print_info: vocab type       = SPM
print_info: n_vocab          = 256000
print_info: n_merges         = 0
print_info: BOS token        = 2 '<bos>'
print_info: EOS token        = 1 '<eos>'
print_info: EOT token        = 107 '<end_of_turn>'
print_info: UNK token        = 3 '<unk>'
print_info: PAD token        = 0 '<pad>'
print_info: LF token         = 227 '<0x0A>'
print_info: EOG token        = 1 '<eos>'
print_info: EOG token        = 107 '<end_of_turn>'
print_info: max token length = 48
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors:   CPU_Mapped model buffer size =  1548.25 MiB
load_tensors:  CPU_AARCH64 model buffer size =  1085.91 MiB
........................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 512
llama_context: n_ctx_per_seq = 512
llama_context: n_batch       = 512
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: freq_base     = 10000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (512) < n_ctx_train (8192) -- the full capacity of the model will not be utilized
llama_context:        CPU  output buffer size =     0.98 MiB
llama_kv_cache_unified_iswa: creating non-SWA KV cache, size = 512 cells
llama_kv_cache_unified:        CPU KV buffer size =    26.00 MiB
llama_kv_cache_unified: size =   26.00 MiB (   512 cells,  13 layers,  1 seqs), K (f16):   13.00 MiB, V (f16):   13.00 MiB
llama_kv_cache_unified_iswa: creating     SWA KV cache, size = 512 cells
llama_kv_cache_unified:        CPU KV buffer size =    26.00 MiB
llama_kv_cache_unified: size =   26.00 MiB (   512 cells,  13 layers,  1 seqs), K (f16):   13.00 MiB, V (f16):   13.00 MiB
llama_context:        CPU compute buffer size =   504.50 MiB
llama_context: graph nodes  = 1154
llama_context: graph splits = 1
common_init_from_params: setting dry_penalty_last_n to ctx_size = 512
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
main: llama threadpool init, n_threads = 8

system_info: n_threads = 8 (n_threads_batch = 8) / 8 | CPU : NEON = 1 | ARM_FMA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | AARCH64_REPACK = 1 |

sampler seed: 1555840794
sampler params:
        repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
        dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 512
        top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.800
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampler chain: logits -> logit-bias -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
generate: n_ctx = 512, n_batch = 2048, n_predict = 64, n_keep = 1

Where is the capital of France?

It is located in the middle of the country.

What is the capital of the United Kingdom?

London

What is the capital of China?

Beijing

Where is the capital of the United States?

Washington

What is the capital of Russia?

Moscow

What is the capital of the Philippines?

llama_perf_sampler_print:    sampling time =      14.02 ms /    72 runs   (    0.19 ms per token,  5134.42 tokens per second)
llama_perf_context_print:        load time =   10072.70 ms
llama_perf_context_print: prompt eval time =      79.81 ms /     8 tokens (    9.98 ms per token,   100.24 tokens per second)
llama_perf_context_print:        eval time =    2522.41 ms /    63 runs   (   40.04 ms per token,    24.98 tokens per second)
llama_perf_context_print:       total time =    2630.35 ms /    71 tokens

[xctan]

I've reproduced the issue and can confirm this is a regression when compiling with the Android NDK toolchain. I am now investigating the root cause and will post an update here.

[xctan]

It seems I inadvertently broke the old build configuration (BUILD_SHARED_LIBS=OFF). This results in linking problems that cause the native implementations to be excluded from the build. I will address this in a separate PR.

[xctan]

I've made progress on this regression and have identified the cause and have a working partial fix.

The issue is a classic static linking problem. When BUILD_SHARED_LIBS is OFF, the linker relies on the order of object files to resolve symbols. My refactoring work split implementations across new files, but didn't update the linking order to match the expected outcome of the linking, which is the direct cause.

[ericcurtin]

Looks like @xctan has got this @zhouwg

[chaxu01]

It seems I inadvertently broke the old build configuration (BUILD_SHARED_LIBS=OFF). This results in linking problems that cause the native implementations to be excluded from the build.

I've confirmed @xctan 's finding. Interestingly enough there seems no performance regression for the build with -DGGML_CPU_KLEIDIAI=ON.

[FanShupei]

I've made progress on this regression and have identified the cause and have a working partial fix.

The issue is a classic static linking problem. When BUILD_SHARED_LIBS is OFF, the linker relies on the order of object files to resolve symbols. My refactoring work split implementations across new files, but didn't update the linking order to match the expected outcome of the linking, which is the direct cause.

I observe GGML_CPU_NATIVE_IMPL is implemented by weak symbol . It's known that weak symbol has complicated interactions when linking with static/dynamic library. Is this the root cause?

It looks we already use GGML_CPU_GENERIC macro to choose between generic/native impl. I suggest maybe we should refactor to get rid of weak symbol tricks here completely.

[xctan]

@FanShupei

It's known that weak symbol has complicated interactions when linking with static/dynamic library. Is this the root cause?

Yes.

It looks we already use GGML_CPU_GENERIC macro to choose between generic/native impl. I suggest maybe we should refactor to get rid of weak symbol tricks here completely.

I've added the apple-fallback.h file to circumvent the weak symbol trick on Darwin targets. The main advantage of weak alias would be avoiding the need to create and maintain explicit alias tables for each architecture, which significantly improves long-term maintainability. Do you have any other suggestions on how we could best handle this?

[slaren]

I suggest maybe we should refactor to get rid of weak symbol tricks here completely.

I agree with this. I suspect that dealing with subtle issues introduced by this trick will be a much bigger maintenance burden than keeping the table of aliases up to date.

[xctan]

I suggest maybe we should refactor to get rid of weak symbol tricks here completely.

Admittedly, while the weak alias approach is promising, the cross-toolchain compatibility issues are more significant than I initially expected. I'll create another PR to refactor this part.