Metal failure after early March versions of server startup loading the model

[groovybits]

Version: 8030da7

Running on Mac M2 Ultra Studio with 192gig ram and MacOS. Model dolphin-2.7-mixtral-8x7b.Q5_K_M.gguf . This works up to the last week or so to version c2101a2, I haven't tracked down which commit breaks after that one running it on my system like this. It works when I use versions around the first week of March / End of Feb.

#!/bin/bash

server \
    -m /Volumes/BrahmaSSD/LLM/models/GGUF/dolphin-2.7-mixtral-8x7b.Q5_K_M.gguf \
    -c 0 \
    -np 2 \
    --port 8080 \
    -ngl 60 \
    -t 24 \
    --host 0.0.0.0 $@

{"build":2408,"commit":"8030da7a","function":"main","level":"INFO","line":2732,"msg":"build info","tid":"0x1dccadc40","timestamp":1710254740}
{"function":"main","level":"INFO","line":2739,"msg":"system info","n_threads":24,"n_threads_batch":-1,"system_info":"AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | ","tid":"0x1dccadc40","timestamp":1710254740,"total_threads":24}
llama_model_loader: loaded meta data with 24 key-value pairs and 995 tensors from /Volumes/BrahmaSSD/LLM/models/GGUF/dolphin-2.7-mixtral-8x7b.Q5_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = cognitivecomputations_dolphin-2.7-mix...
llama_model_loader: - kv   2:                       llama.context_length u32              = 32768
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   4:                          llama.block_count u32              = 32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv   9:                         llama.expert_count u32              = 8
llama_model_loader: - kv  10:                    llama.expert_used_count u32              = 2
llama_model_loader: - kv  11:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  12:                       llama.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  13:                          general.file_type u32              = 17
llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,32002]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  16:                      tokenizer.ggml.scores arr[f32,32002]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,32002]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  18:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  19:                tokenizer.ggml.eos_token_id u32              = 32000
llama_model_loader: - kv  20:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  21:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  22:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
llama_model_loader: - kv  23:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type  f16:   32 tensors
llama_model_loader: - type q8_0:   64 tensors
llama_model_loader: - type q5_K:  833 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens definition check successful ( 261/32002 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32002
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 14336
llm_load_print_meta: n_expert         = 8
llm_load_print_meta: n_expert_used    = 2
llm_load_print_meta: causal attm      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = Q5_K - Medium
llm_load_print_meta: model params     = 46.70 B
llm_load_print_meta: model size       = 30.02 GiB (5.52 BPW)
llm_load_print_meta: general.name     = cognitivecomputations_dolphin-2.7-mixtral-8x7b
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 32000 '<|im_end|>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.76 MiB
ggml_backend_metal_buffer_from_ptr: allocated buffer, size = 30649.58 MiB, (30649.64 / 147456.00)
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors:        CPU buffer size =    85.94 MiB
llm_load_tensors:      Metal buffer size = 30649.58 MiB
....................................................................................................
llama_new_context_with_model: n_ctx      = 32768
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M2 Ultra
ggml_metal_init: picking default device: Apple M2 Ultra
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: loading '/usr/local/bin/ggml-metal.metal'
ggml_metal_init: error: Error Domain=MTLLibraryErrorDomain Code=3 "program_source:3:10: fatal error: 'ggml-common.h' file not found
#include "ggml-common.h"
         ^~~~~~~~~~~~~~~
" UserInfo={NSLocalizedDescription=program_source:3:10: fatal error: 'ggml-common.h' file not found
#include "ggml-common.h"
         ^~~~~~~~~~~~~~~
}
llama_new_context_with_model: failed to initialize Metal backend
llama_init_from_gpt_params: error: failed to create context with model '/Volumes/BrahmaSSD/LLM/models/GGUF/dolphin-2.7-mixtral-8x7b.Q5_K_M.gguf'
{"function":"load_model","level":"ERR","line":678,"model":"/Volumes/BrahmaSSD/LLM/models/GGUF/dolphin-2.7-mixtral-8x7b.Q5_K_M.gguf","msg":"unable to load model","tid":"0x1dccadc40","timestamp":1710254740}

If the bug concerns the server, please try to reproduce it first using the server test scenario framework.

[ggerganov]

Give #6015 a try

[josephst]

@groovybits how did you compile and install llama-cpp? I suspect the binary is in /usr/local/bin, right? And is this built from source? I think the line ggml_metal_init: default.metallib not found, loading from source indicates the error.

I've had a similar error and tracked the problem down to the compiled default.metallib not being included in the bin/ dir of the compiled project. I've got a PR linked above if you want to try that branch and let me know if it works. There's also a few changes to the Nix build process but that shouldn't affect you - mkdir build && cd build && cmake .. && cmake --build . --config Release is also working.

[groovybits]

@groovybits how did you compile and install llama-cpp? I suspect the binary is in /usr/local/bin, right? And is this built from source? I think the line ggml_metal_init: default.metallib not found, loading from source indicates the error.

I've had a similar error and tracked the problem down to the compiled default.metallib not being included in the bin/ dir of the compiled project. I've got a PR linked above if you want to try that branch and let me know if it works. There's also a few changes to the Nix build process but that shouldn't affect you - mkdir build && cd build && cmake .. && cmake --build . --config Release is also working.

No that didn't fix it alone. yet this did after using your branch...

sudo cp ../ggml-common.h /usr/local/bin/

Then I switched back and with the newest main branch in llama.cpp that seems to be all I needed to do? Odd, now I can run the newest version it seems without the complaint about that file missing. So the missing binary is needing to be in the bin directory and also it is needing the header file in the bin directory? Seems to be that way here, thank you so much for pointing this branch out and having it heal my llama.cpp :D.

[groovybits]

Also note that the main branch in llama.cpp does copy this file for me into bin. Yet nothing copied the header over that it seems to need in the bin directory.

[stuaxocabinetoffice]

Needing a header file in /usr/local/bin seems a little odd - is something using the wrong path somewhere ?

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.