Significantly different results (and WRONG) inference when GPU is enabled.

[phishmaster]

I am running llama_cpp version 0.2.68 on Ubuntu 22.04LTS under conda environment. Attached are two Jupyter notebooks with ONLY one line changed (use CPU vs GPU). As you can see for exact same environmental conditions switching between CPU/GPU gives vastly different answers where the GPU is completely wrong. Some pointers on how to debug this I would appreciate it.

The only significant difference between the two files is this one liner
#n_gpu_layers=-1, # Uncomment to use GPU acceleration

The model used was openhermes-2.5-mistral-7b.Q5_K_M.gguf

mistral_llama_large-gpu.pdf
mistral_llama_large-cpu.pdf

[JohannesGaessler]

Are you getting correct results when you use the llama.cpp binaries directly without any Python bindings? If not, are you getting correct results when you compile with LLAMA_CUDA_FORCE_MMQ?

[phishmaster]

Thanks for your response. First I recompiled llama.cpp with the suggested flag LLAMA_CUDA_FORCE_MMQ e.g. LLAMA_CUDA=1 LLAMA_CUDA_FORCE_MMQ=1 make Once completed I ran the binary as follow ------- When I do not have "-ngl 40" it seems to give the correct answer. " ./main_gpu -m /data/DemoV1/Model4Demo/openhermes-2.5-mistral-7b.Q5_K_M.gguf \ -c 16192 -b 1024 -n 256 --keep 48 \ --repeat_penalty 5.0 --color -i \ -r "User:" -f prompts/chat-with-bob.txt ... User: Hello, Bob. Bob: Hello. How may I help you today? User: Please tell me the largest city in Europe. Bob: Sure. The largest city in Europe is Moscow, the capital of Russia. User:what is the capital city of France? Bob: The Capital City Of Paris.<|im_end|> " However when I ran it with -ngl 40 this is the response ... User:what is the capital city of France? ################################################################################################################## ################################################################################################################## ###################

…

________________________________ From: Johannes Gäßler ***@***.***> Sent: Friday, May 3, 2024 4:15 AM To: ggerganov/llama.cpp ***@***.***> Cc: Huy Vu ***@***.***>; Author ***@***.***> Subject: Re: [ggerganov/llama.cpp] Significantly different results (and WRONG) inference when GPU is enabled. (Issue #7048) Caution: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Are you getting correct results when you use the llama.cpp binaries directly without any Python bindings? If not, are you getting correct results when you compile with LLAMA_CUDA_FORCE_MMQ? — Reply to this email directly, view it on GitHub<#7048 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BIHUORKZ5IDGM7ZSNYYI2BLZANBRPAVCNFSM6AAAAABHEIE4XSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJSGUZDSOBWGQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[JohannesGaessler]

Sorry, I wanted to check this issue but forgot. Did you download a ready-made GGUF file from Huggingface or did you convert it yourself? If it's the former, can you provide a link to the exact file you downloaded?

[phishmaster]

No problem and thanks for responding. I downloaded the model from Huggingface at https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-GGUF/blob/main/openhermes-2.5-mistral-7b.Q5_K_M.gguf Huy

…

________________________________ From: Johannes Gäßler ***@***.***> Sent: Wednesday, May 8, 2024 11:32 AM To: ggerganov/llama.cpp ***@***.***> Cc: Huy Vu ***@***.***>; Author ***@***.***> Subject: Re: [ggerganov/llama.cpp] Significantly different results (and WRONG) inference when GPU is enabled. (Issue #7048) Caution: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Sorry, I wanted to check this issue but forgot. Did you download a ready-made GGUF file from Huggingface or did you convert it yourself? If it's the former, can you provide a link to the exact file you downloaded? — Reply to this email directly, view it on GitHub<#7048 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BIHUORPU7HZDYA53C354M53ZBJARPAVCNFSM6AAAAABHEIE4XSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBQHA2TGMZQGY>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[JohannesGaessler]

I cannot reproduce the issue on master. Can you re-download the model and check that this issue isn't due to a corrupted file?

[phishmaster]

Here is my git master

* 83330d8c - (HEAD -> master, origin/master, origin/HEAD) main : add --conversation / -cnv flag (#7108) (2 hours ago) [Dawid Potocki]
* 465263d0 - sgemm : AVX Q4_0 and Q8_0 (#6891) (2 hours ago) [Eve]
* 911b3900 - server : add_special option for tokenize endpoint (#7059) (4 hours ago) [Johan]

clean and rebuild with

LLAMA_CUDA=1 LLAMA_CUDA_FORCE_MMQ=1 make clean
LLAMA_CUDA=1 LLAMA_CUDA_FORCE_MMQ=1 make

re-downloaded the model (also matches with my previously downloaded file)

(base) hvu@Kaui:/data/DemoV1/Model4Demo$ md5sum openhermes-2.5-mistral-7b.Q5_K_M_v1.gguf

f7faa7e315e81c3778aae55fcb5fc02c openhermes-2.5-mistral-7b.Q5_K_M_v1.gguf

(pytorch_py39_cu11.8) hvu@Kaui:~/Demo_v1/llama.cpp_v3$ ./main_gpu -m /data/DemoV1/Model4Demo/openhermes-2.5-mistra
l-7b.Q5_K_M.gguf \                                                                                                
    -c 16192 -b 1024 -n 256 --keep 48 \                                                                           
    --repeat_penalty 5.0 --color -i \                                                                             
    -r "User:" -f prompts/chat-with-bob.txt  
...
llm_load_print_meta: EOT token        = 32000 '<|im_end|>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   yes
ggml_cuda_init: CUDA_USE_TENSOR_CORES: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes
  Device 1: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes
llm_load_tensors: ggml ctx size =    0.15 MiB
llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/33 layers to GPU
llm_load_tensors:        CPU buffer size =  4893.00 MiB
...................................................................................................
llama_new_context_with_model: n_ctx      = 16384
llama_new_context_with_model: n_batch    = 1024
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:  CUDA_Host KV buffer size =  2048.00 MiB
llama_new_context_with_model: KV self size  = 2048.00 MiB, K (f16): 1024.00 MiB, V (f16): 1024.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.12 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =  1147.00 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    40.01 MiB
llama_new_context_with_model: graph nodes  = 1030
llama_new_context_with_model: graph splits = 356

system_info: n_threads = 24 / 48 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 0 | AVX512_VNNI 
= 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 
| VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | 
main: interactive mode on.
...
ob: Sure. The largest city in Europe is Moscow, the capital of Russia.
User:what is the capital city of France?
Bob: The Capital City Of france Is Paris.<|im_end|>

===========================================
I tried various values of -ngl and all seems to return garbage

(pytorch_py39_cu11.8) hvu@Kaui:~/Demo_v1/llama.cpp_v3$ ./main_gpu -m /data/DemoV1/Model4Demo/openhermes-2.5-mistra
l-7b.Q5_K_M.gguf     -c 16192 -b 1024 -n 256 --keep 48     --repeat_penalty 5.0 --color -i     -r "User:" -f promp
ts/chat-with-bob.txt -ngl 40
Log start
main: build = 2817 (83330d8c)
...
ggml_cuda_init: found 2 CUDA devices:
  Device 0: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes
  Device 1: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes
llm_load_tensors: ggml ctx size =    0.44 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors:        CPU buffer size =    85.94 MiB
llm_load_tensors:      CUDA0 buffer size =  2352.25 MiB
llm_load_tensors:      CUDA1 buffer size =  2454.81 MiB
...
ob: Sure. The largest city in Europe is Moscow, the capital of Russia.
User:what is the capital city of France?
##################################################################################################################
##################################################################################################################
###################
------------------------------------------------------

Other values of -ngl 16

...
llm_load_tensors: ggml ctx size =    0.44 MiB                                                                     
llm_load_tensors: offloading 16 repeating layers to GPU                                                           
llm_load_tensors: offloaded 16/33 layers to GPU                                                                   
llm_load_tensors:        CPU buffer size =  4893.00 MiB                                                           
llm_load_tensors:      CUDA0 buffer size =  1160.19 MiB                                                           
llm_load_tensors:      CUDA1 buffer size =  1192.06 MiB                                                           
.....
User:what is the capital city of France?
#<s>▅

$<s>#"
      "!<s> 
</s>

        $
!<s>!!"

       "
$#
                 
""<s>

Pretty much the same for -ngl 8

[JohannesGaessler]

If I remember correctly the output

#<s>▅

$<s>#"
      "!<s> 
</s>

        $
!<s>!!"

       "
$#
                 
""<s>

is effectively what you get when a NO_DEVICE_CODE isn't being correctly triggered. My intuition is that this issue is specific to a V100 GPU (and maybe also the CUDA version). If possible, please check the following:

• Results after running export CUDA_VISIBLE_DEVICES=0 (makes it so that only the first GPU is used).
• Results on a non-V100 GPU.
• Results using another model, ideally with n_vocab of 32000 (check the console log, Mistral base model should have that vocab size).
• Results using CUDA 12.

[JohannesGaessler]

Also: with SXM your V100s are effectively NVLinked, right? Can you check results when compiling with LLAMA_CUDA_PEER_MAX_BATCH_SIZE=0 and LLAMA_CUDA_PEER_MAX_BATCH_SIZE=999999?

[phishmaster]

Remake with suggested flags

LLAMA_CUDA=1 LLAMA_CUDA_FORCE_MMQ=1 CUDA_VISIBLE_DEVICES=0 LLAMA_CUDA_PEER_MAX_BATCH_SIZE=0 LLAMA_CUDA_PEER_MAX_BATCH_SIZE=999999 make

I think I already have CUDA 12

(llama_cpp_py39) hvu@Kaui:~/Demo_v1/llama.cpp_v3$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

Now for the run

(pytorch_py39_cu11.8) hvu@Kaui:~/Demo_v1/llama.cpp_v3$ export CUDA_VISIBLE_DEVICES=0                              
(pytorch_py39_cu11.8) hvu@Kaui:~/Demo_v1/llama.cpp_v3$ ./main_gpu -m /data/DemoV1/Model4Demo/openhermes-2.5-mistral-7b.Q5_K_M.gguf     -c 16192 -b 1024 -n 256 --keep 48     --repeat_penalty 5.0 --color -i     -r "User:" -f prompts/chat-with-bob.txt -ngl 8      
...
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   yes
ggml_cuda_init: CUDA_USE_TENSOR_CORES: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes
llm_load_tensors: ggml ctx size =    0.30 MiB
llm_load_tensors: offloading 8 repeating layers to GPU
llm_load_tensors: offloaded 8/33 layers to GPU
llm_load_tensors:        CPU buffer size =  4893.00 MiB
llm_load_tensors:      CUDA0 buffer size =  1192.06 MiB
...................................................................................................
llama_new_context_with_model: n_ctx      = 16384
llama_new_context_with_model: n_batch    = 1024
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:  CUDA_Host KV buffer size =  1536.00 MiB
llama_kv_cache_init:      CUDA0 KV buffer size =   512.00 MiB
llama_new_context_with_model: KV self size  = 2048.00 MiB, K (f16): 1024.00 MiB, V (f16): 1024.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.12 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =  1147.00 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    40.01 MiB
llama_new_context_with_model: graph nodes  = 1030
llama_new_context_with_model: graph splits = 268
...
User:what is the capital city of France?
 ▅</s>"!"!

          " 

</s></s▅"$#<s># "<s>

Same results

[JohannesGaessler]

When checking the NVCC version your shell prefix is (llama_cpp_py39). When you actually run the model the prefix is (pytorch_py39_cu11.8). Are you sure that in both cases CUDA 12 is being used?

[JohannesGaessler]

Also, I didn't mean to compile with both LLAMA_CUDA_PEER_MAX_BATCH_SIZE=0 and LLAMA_CUDA_PEER_MAX_BATCH_SIZE=999999 at the same time. I meant to test either option individually. But if you still get incorrect results with CUDA_VISIBLE_DEVICES=0 that's not going to be the problem anyways.

[phishmaster]

Remake in base conda environment (default nvcc)

(base) hvu@Kaui:~/Demo_v1/llama.cpp_v3$ LLAMA_CUDA=1 LLAMA_CUDA_FORCE_MMQ=1 CUDA_VISIBLE_DEVICES=0 LLAMA_CUDA_PEER_MAX_BATCH_SIZE=0 make
....
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   yes                                                                        
ggml_cuda_init: CUDA_USE_TENSOR_CORES: no                                                                         
ggml_cuda_init: found 2 CUDA devices:                                                                             
  Device 0: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes                                                
  Device 1: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes                                                
llm_load_tensors: ggml ctx size =    0.44 MiB                                                                     
llm_load_tensors: offloading 8 repeating layers to GPU                                                            
llm_load_tensors: offloaded 8/33 layers to GPU                                                                    
llm_load_tensors:        CPU buffer size =  4893.00 MiB                                                           
llm_load_tensors:      CUDA0 buffer size =   588.06 MiB                                                           
llm_load_tensors:      CUDA1 buffer size =   604.00 MiB            
...

Same results

</s>What is the capital city of France?

 #"<s> </s>
<s>$
 #▅"#
!<s><s>/s><s$<s>" <s>"

llama_print_timings:        load time =    2860.74 ms
llama_print_timings:      sample time =      70.35 ms /   161 runs   (    0.44 ms per token,  2288.46 tokens per second)
llama_print_timings: prompt eval time =     360.26 ms /     9 tokens (   40.03 ms per token,    24.98 tokens per second)
llama_print_timings:        eval time =   12501.69 ms /   160 runs   (   78.14 ms per token,    12.80 tokens per second)
llama_print_timings:       total time =   13594.98 ms /   169 tokens
(base) hvu@Kaui:~/Demo_v1/llama.cpp_v3$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0
(base) hvu@Kaui:~/Demo_v1/llama.cpp_v3$ 

[slaren]

Do you get any errors with compute-sanitizer? Run compute-sanitizer ./main -m ..

[phishmaster]

No errors, and the "compute-sanitizer" didn't seem to help; however, it seems to work better if I use -ngl -1 instead of any specific values. Does that help?