[BUG] DeepSeek V3 weight_scale_inv tensor mapping not supported in converter

[howeirdo]

🚨 DeepSeek V3 weight_scale_inv Tensor Mapping Issue

Model: Kimi K2 (DeepSeek V3)
Architecture: DeepSeek V3 with FP8 quantization
Issue: Converter fails on weight_scale_inv tensors
Priority: HIGH - Blocking model integration

📋 Technical Details

Environment

• llama.cpp commit: 36c1532 (latest master)
• Python version: 3.13.5
• Platform: macOS (Apple Silicon M4)
• Model size: ~1TB (61 files)

Error Details

ValueError: Can not map tensor 'model.layers.0.mlp.down_proj.weight_scale_inv'

Root Cause: The llama.cpp converter doesn't support the weight_scale_inv tensor mapping that DeepSeek V3 uses for FP8 weight dequantization.

Model Information

• Model: Kimi K2 (DeepSeek V3)
• Architecture: DeepSeek V3 with FP8 quantization
• Size: ~2TB (78 files)
• Target: Convert to GGUF format for Apple Silicon deployment

🔍 Investigation Results

✅ What We Verified

1. llama.cpp is current - Pulled latest master branch
2. Model architecture recognized - DeepSeek V3ForCausalLM detected correctly
3. Basic conversion starts - Model loading and initial tensor processing works
4. Hardware ready - max2 with 128GB RAM and 4TB storage available

❌ What We Found Missing

1. No DeepSeek V3 support - llama.cpp converter lacks weight_scale_inv handling
2. No experimental branches - Checked dev and other development branches
3. No community patches - No existing workarounds in GitHub issues/PRs
4. No alternative converters - Standard tools don't support this architecture

🔍 Technical Investigation

• llama.cpp version: Latest master (commit 36c1532)
• DeepSeek V3 class: Uses DeepseekV2Model class (line 5901-5902)
• Tensor mapping: Inherits from TextModel without weight_scale_inv support
• FP8 tensors: Model uses torch.float8_e4m3fn format

🛠️ Attempted Solutions

1. Standard Conversion

python3 convert_hf_to_gguf.py /Volumes/4TB/ai-models/kimi-k2-original \
  --outfile /Volumes/4TB/ai-models/kimi-k2.gguf \
  --outtype bf16

Result: Failed at weight_scale_inv tensor mapping

2. Alternative Output Types

• Tried --outtype auto, f16, f32
• Result: Same failure point

3. Direct Model Loading

# Attempted with transformers + MPS backend
model = AutoModelForCausalLM.from_pretrained("/Volumes/4TB/ai-models/kimi-k2-original")

Result: Failed due to model size (~1.9TB buffer error)

4. MLX Environment

• Checked for DeepSeek V3 support in MLX
• Result: Model class not available

5. GitHub Investigation

• Searched llama.cpp issues and PRs for DeepSeek V3 support
• Checked experimental branches (dev, wip branches)
• Result: No existing support or workarounds found

🎯 Request for Assistance

We are seeking guidance from the llama.cpp community on how to proceed with DeepSeek V3 model conversion. Specifically:

1. Is DeepSeek V3 support planned or in development?
2. Are there any experimental branches with weight_scale_inv support?
3. What would be the best approach to add this support?
4. Are there alternative conversion methods we should consider?

📊 Impact

This issue is blocking the integration of a significant AI model (Kimi K2) into our development lab. The model represents a substantial investment and would be a valuable addition to our AI team.

🔗 Additional Context

• Model Location: Available for testing if needed
• Environment Details: macOS with Apple Silicon, 128GB RAM available

---

Status: BLOCKED - Awaiting community guidance
Priority: HIGH - Model integration critical for lab expansion

Thank you for your time and assistance!

[howeirdo]

This is Howard...I would just like to add that this report was generated by ai working the problem in our lab. They consulted each other (transcripts available) and agreed that the best way forward was to seek help from the community. So I said "do it" and here we are!

Looking forward to working this through....thanks!

[howeirdo]

For reference, here is the list of files we are working from:

(mlx-env) wizard@max2-2 kimi-k2-original % ls -l
total 2010184368
-rw-r--r-- 1 wizard staff 369 Jul 19 22:28 check_dtype.py
-rw-r--r-- 1 wizard staff 1447 Jul 19 21:37 config.json
-rw-r--r-- 1 wizard staff 1725 Jul 18 14:21 config.json.backup
-rw-r--r-- 1 wizard staff 315 Jul 18 14:56 configuration_deepseek.py
-rw-r--r-- 1 wizard staff 1799 Jul 19 22:31 convert_resume.py
drwxr-xr-x 11 wizard staff 352 Jul 19 22:38 DeepSeek-V3
-rwxr-xr-x@ 1 wizard staff 2378 Jul 17 22:17 download_broken_52.sh
-rwxr-xr-x@ 1 wizard staff 2375 Jul 17 20:36 download_broken.sh
-rwxr-xr-x@ 1 wizard staff 2374 Jul 17 19:35 download_remaining.sh
-rw-r--r-- 1 wizard staff 883 Jul 18 14:54 kernel.py
drwxr-xr-x 7 wizard staff 224 Jul 19 22:51 mlx-env
-rw-r--r-- 1 wizard staff 2846451040 Jul 16 20:57 model-1-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066593104 Jul 16 22:25 model-10-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 16 22:35 model-11-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 16 22:48 model-12-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 16 22:59 model-13-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 16 23:08 model-14-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 16 23:18 model-15-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 22:01 model-16-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 05:35 model-17-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 05:48 model-18-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 05:58 model-19-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066593104 Jul 17 21:58 model-2-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 06:09 model-20-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 06:19 model-21-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 06:27 model-22-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 06:37 model-23-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 06:45 model-24-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 06:55 model-25-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 07:04 model-26-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 07:12 model-27-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 07:21 model-28-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 07:29 model-29-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066593104 Jul 16 21:21 model-3-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 07:40 model-30-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 07:48 model-31-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 07:56 model-32-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 08:04 model-33-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 08:12 model-34-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 08:20 model-35-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 08:28 model-36-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 08:37 model-37-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 08:45 model-38-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 08:54 model-39-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066593104 Jul 16 21:29 model-4-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 09:02 model-40-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 09:10 model-41-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 09:18 model-42-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 09:27 model-43-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 10:36 model-44-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 10:49 model-45-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 11:02 model-46-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 11:13 model-47-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 11:23 model-48-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 11:34 model-49-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066593104 Jul 16 21:39 model-5-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 11:45 model-50-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 11:56 model-51-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 18 00:22 model-52-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066595432 Jul 17 15:59 model-53-of-61.safetensors
-rw-r--r-- 1 wizard staff 17066595432 Jul 17 20:03 model-54-of-61.safetensors
-rw-r--r-- 1 wizard staff 17066595432 Jul 17 20:31 model-55-of-61.safetensors
-rw-r--r-- 1 wizard staff 17066595432 Jul 17 21:47 model-56-of-61.safetensors
-rw-r--r-- 1 wizard staff 17066595432 Jul 17 22:30 model-57-of-61.safetensors
-rw-r--r-- 1 wizard staff 17066595432 Jul 17 23:00 model-58-of-61.safetensors
-rw-r--r-- 1 wizard staff 17066595432 Jul 17 23:34 model-59-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066593104 Jul 16 21:47 model-6-of-61.safetensors
-rw-r--r-- 1 wizard staff 17066595432 Jul 18 00:26 model-60-of-61.safetensors
-rw-r--r-- 1 wizard staff 19415420696 Jul 18 00:58 model-61-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066593104 Jul 16 21:56 model-7-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066593104 Jul 16 22:06 model-8-of-61.safetensors
-rw-r--r--@ 1 wizard staff 17066593104 Jul 16 22:15 model-9-of-61.safetensors
-rw-r--r-- 1 wizard staff 12529626 Jul 19 19:57 model.safetensors.index.json
-rw-r--r-- 1 wizard staff 33824 Jul 18 14:56 modeling_deepseek.py
-rw-r--r-- 1 wizard staff 2795286 Jul 19 19:57 tiktoken.model
-rw-r--r-- 1 wizard staff 3128 Jul 19 19:57 tokenizer_config.json
-rw-r--r-- 1 wizard staff 7847652 Jul 19 19:57 tokenizer.json
drwxr-xr-x 31 wizard staff 992 Jul 19 23:35 triton

[howeirdo]

kimi-k2-other-files.zip

if this helps!

[howeirdo]

@MoonshotAI team - FYI, we're working on converting Kimi K2 to GGUF format for local deployment. The llama.cpp converter needs support for DeepSeek V3's weight_scale_inv tensors. Any guidance on the model's FP8 architecture would be greatly appreciated! Related: We're also exploring this in the Kimi K2 repository.

[danielhanchen]

@howeirdo If you're interested, we converted Kimi K2 to GGUF here: https://huggingface.co/unsloth/Kimi-K2-Instruct-GGUF

You need to use the DeepSeek FP8->BF16 conversion script.

[howeirdo]

Thanks Daniel!

we got similar advice from others, and really appreciate it!

There is already so much to learn about, and it is all moving so fast!

I think I need to get my AI Lab Director Bob to check these issues daily!

Best regards,

Howard

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.