Eval bug: Unable to run Qwen3-Coder-480B-A35B-Instruct-UD-Q2_K_XL

[ambud]

Name and Version

./llama.cpp/build/bin/llama-cli  --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
version: 5967 (6c88b3bb)
built with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

CUDA

Hardware

AMD EPYC 7402 + 4x4060 8GB

Models

unsloth/Qwen3-Coder-480B-A35B-Instruct-UD-Q2_K_XL

Problem description & steps to reproduce

Unable to run Qwen3-Coder-480B-A35B-Instruct-UD-Q2_K_XL

Following
https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally

Related issues
QwenLM/qwen-code#48 [indicates that the issue is with llama.cpp template]
QwenLM/Qwen3-Coder#434

./llama.cpp/build/bin/llama-server     --model unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF/UD-Q2_K_XL/Qwen3-Coder-480B-A35B-Instruct-UD-Q2_K_XL-00001-of-00004.gguf   \
 --threads -1     --ctx-size 20000          --temp 0.7     --min-p 0.0     --top-p 0.8     --top-k 20  \
   --repeat-penalty 1.05 --n-gpu-layers 200 -ot ".ffn_(up|down)_exps.=CPU" --jinja

First Bad Commit

No response

Relevant log output

I'll create a simple "Hello, World!" program for you. Since you didn't specify a language, I'll
  write it in Python, which is commonly used for beginners.

  [tool_call: write_file for file_path '/Users/username/code/test/hello_world.py' with content
  'print("Hello, World!")']

Instead of executing the tool it is simply printing it.

[danielhanchen]

Are you using Qwen Coder CLI to execute the tool call, or is this just via llama-server?

[ambud]

yes @danielhanchen this is qwen-code cli and trying to have it talk to llama-server for local inference. The qwen-code cli works fine with openrouter hosted qwen coder model.

[BerryBaronBonanza]

BerryBaronBonanza@ce3b770 - minja: add missing keys() method and in/replace filters

there are still some issues. at least i don't see method errors from template anymore.

llama-server -hf unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF:Q4_K_XL --threads 32 --ctx-size 65536 --temp 0.7 --min-p 0.0 --top-p 0.8 --top-k 20 -fa --jinja

[ambud]

thanks @BerryBaronBonanza confirmed the template error is fixed above; the function calling is still broken.

[danielhanchen]

@ambud As an update - I think I managed to fix the chat template for Qwen3 480B Coder for llama-server / llama.cpp! I re-uploaded the model.

For UD-Q2_K_XL, simply download the first file at https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF/tree/main/UD-Q2_K_XL

The other option is to use the updated chat template and use --chat-template-file. See https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF?chat_template=default or https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct/raw/main/chat_template.jinja

[ambud]

thanks @danielhanchen just tried it I think it's still broken.

srv  log_server_r: request: POST /chat/completions 127.0.0.1 500
got exception: {"code":500,"message":"Value is not callable: null at row 62, column 114:\n         

Here's what I did:

1. git pull on llama.cpp and recompiled
2. downloaded the latest jinja template from HF
3. ran it with --chat-template-file

[note using CPU only inference environment here]

[BerryBaronBonanza]

there is progress!

with my fix for template processing (BerryBaronBonanza@ce3b770) i'm not getting Value is not callable error.

but i had to tweak qwen-code not to use OpenAI API completion streaming because there was runtime error during generation of response diffs in llama.cpp.

BerryBaronBonanza/qwen-code@0c1629a - Remove streaming support and refactor response handling for OpenAI content generation

running llama-server on 20GB VRAM GPU:

llama-server -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_XL --ctx-size 65536 --temp 0.7 --min-p 0.0 --top-p 0.8 --top-k 20 --repeat-penalty 1.05 --jinja -t 16 -ngl 24 --verbose --verbose-prompt

after these changes i'm able to send request to qwen-code and get reasonable output.

git clone -b qwen-code-qwen3-coder-integration https://github.com/BerryBaronBonanza/qwen-code
cd qwen-code
npm run build
npm run bundle
cd ..
git clone https://github.com/PrimalHQ/primal-web-app
cd primal-web-app
node ../qwen-code/bundle/gemini.js --telemetry false --openai-base-url http://127.0.0.1:8080 --openai-api-key xxx

╭────────────────────────────────────────────────────────────────────────╮
│  > analyze codebase and tell me how to show legends using this webapp  │
╰────────────────────────────────────────────────────────────────────────╯

✦ I'll analyze the codebase to understand how legends are implemented in this webapp.

  Let me first check what files might be related to legends by searching for relevant terms.
  <tool_call>

 ╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
 │ ✔  SearchText 'legend' within ./                                                                                                                                                                       │
 │                                                                                                                                                                                                        │
 │    Found 664 matches                                                                                                                                                                                   │
 ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
✦ Based on my analysis of the codebase, I can explain how legends are implemented in this webapp:

  Legend System Overview

  The app implements a "Legend" tier as part of its premium membership system. Here's how it works:

   1. Premium Tier: Legends are a special tier within the premium subscription system (similar to "premium" and "premium-legend").

   2. Customization: Legends can customize their profile with different visual styles:
      - Gold, Aqua, Silver, Purple, Purple Haze, Teal, Brown, Blue, Sunfire
      - These are defined in src/index.scss as CSS variables

   3. Key Components:
      - src/pages/Premium/Legends.tsx - The main legends page showing leaderboard
      - src/components/LegendCard/LegendCard.tsx - Displays legend information with customization
      - src/components/Avatar/Avatar.tsx - Shows avatar with legend styling
      - src/components/VerificationCheck/VerificationCheck.tsx - Shows verification badge for legends

   4. Data Flow:
      - Legend status is stored in app.legendCustomization context
      - Legend stats are fetched via getLegendStats() function
      - Customization data comes from LegendCustomizationConfig type

   5. Pages:
      - /legends - Shows the legend leaderboard
      - /premium/legend - Legend tier information and purchase page
      - /premium/legendary - For becoming a legend

  To see legends in action, you would navigate to the /legends route or look for users with special styling on profile pages. The visual customization is applied through CSS classes like legend_GOLD,
  legend_AQUA, etc.

  The system also includes:
   - Legend leaderboard functionality
   - Customization options for legend profiles
   - Special privileges for legends (forever premium, customizable badges)
   - Integration with the broader premium membership system

  Would you like me to explain any specific aspect of how legends are implemented in more detail?

╭────────────╮
│  > /stats  │
╰────────────╯

╭───────────────────────────────────────────────────────────────────╮
│                                                                   │
│  Session Stats                                                    │
│                                                                   │
│  Interaction Summary                                              │
│  Tool Calls:                 1 ( ✔ 1 ✖ 0 )                        │
│  Success Rate:               100.0%                               │
│                                                                   │
│  Performance                                                      │
│  Wall Time:                  7m 32s                               │
│  Agent Active:               8m 53s                               │
│    » API Time:               8m 53s (100.0%)                      │
│    » Tool Time:              198ms (0.0%)                         │
│                                                                   │
│                                                                   │
│  Model Usage                  Reqs   Input Tokens  Output Tokens  │
│  ───────────────────────────────────────────────────────────────  │
│                                  3         52,631            575  │
│  qwen3-coder-max                 2         32,476            493  │
│                                                                   │
╰───────────────────────────────────────────────────────────────────╯

[danielhanchen]

Wait apologies @ambud I had to fix it again after some debugging from folks - could you try https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct/raw/main/chat_template.jinja to see if it works (for 480B or 30B)

[danielhanchen]

@BerryBaronBonanza Nice work on supporting .keys() -> as a temp fix I removed keys() and transformed it to a loop via |items

[3dlabsio]

Wait apologies @ambud I had to fix it again after some debugging from folks - could you try https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct/raw/main/chat_template.jinja to see if it works (for 480B or 30B)

Downloaded the template after building latest llama.cpp. Used --chat-template-file and --jinja with it and it seems to respond in chinese and very random. During warmup the following message was displayed:

Failed to infer a tool call example (possible template bug)

[Nindaleth]

Used --chat-template and --jinja

My understanding of llama-server options is that for a custom chat template file you should use --jinja --chat-template-file, would that make any difference?

[ambud]

Thank you @BerryBaronBonanza confirmed that your fixes work and qwen-code is able to run.

[3dlabsio]

Used --chat-template and --jinja

My understanding of llama-server options is that for a custom chat template file you should use --jinja --chat-template-file, would that make any difference?

Yes that's what was used - corrected the typo