Add support for CogVLM model #15002

Tianyue-Zhao · 2025-08-01T01:50:42Z

This addresses the requests for CogVLM in #4387 and #4350.
CogVLM is a pretty popular model that now adds in cleanly after the recent additions to libmtmd.
I've converted a GGUF here: Link to GGUF files

Sample command and output:

build/bin/llama-mtmd-cli -m ../cogvlm-chat-hf/cogvlm-13B-chat-v1.1-F16.gguf --mmproj ../cogvlm-chat-hf/mmproj-cogvlm-chat-hf --image ./community.png --chat-template vicuna -p "Describe the picture"

load_hparams: model size:         8448.53 MiB
load_hparams: metadata size:      0.36 MiB
alloc_compute_meta:        CPU compute buffer size =   142.02 MiB
main: loading model: ../cogvlm-chat-hf/cogvlm-13B-chat-v1.1-F16.gguf
encoding image slice...
image slice encoded in 16135 ms
decoding image batch 1/1, n_tokens_batch = 1227
image decoded (batch 1/1) in 54065 ms

1. The image showcases a futuristic urban landscape with a mix of architectural styles. The buildings are multi-storied and have a combination of traditional and modern elements. There's a prominent tree in the foreground, suggesting a blend of nature and urban development. The scene appears to be bustling with activity, with various signs and billboards, indicating commercial or residential zones.


llama_perf_context_print:        load time =  108969.65 ms
llama_perf_context_print: prompt eval time =   85229.27 ms /  1241 tokens (   68.68 ms per token,    14.56 tokens per second)
llama_perf_context_print:        eval time =   19843.15 ms /    83 runs   (  239.07 ms per token,     4.18 tokens per second)
llama_perf_context_print:       total time =  126951.23 ms /  1324 tokens
llama_perf_context_print:    graphs reused =          0

I have read the contributing guidelines
Self-reported review complexity:
- Low
- Medium
- High

Tianyue-Zhao · 2025-08-02T16:47:30Z

I think I've fixed the typecheck and format check workflows that were failing before, can someone approve the workflows to run again?
Also, is there a way to run these Github workflows locally or without needing approval from a reviewer?
It would be good to run these CI/CD checks myself before posting the PR.

CISC · 2025-08-02T18:36:51Z

Also, is there a way to run these Github workflows locally or without needing approval from a reviewer? It would be good to run these CI/CD checks myself before posting the PR.

You can run flake8, pyright and editorconfig locally (or via IDE plugins), the build tests can be run manually with ctest.

CISC

This is not a complete review as I don't know enough about mtmd, just commenting...

CISC · 2025-08-02T20:20:42Z

convert_hf_to_gguf.py

+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.hparams['num_attention_heads'] = self.hparams['num_heads']
+
+    def set_gguf_parameters(self):


Suggested change

def __init__(self, *args, **kwargs):

super().__init__(*args, **kwargs)

self.hparams['num_attention_heads'] = self.hparams['num_heads']

def set_gguf_parameters(self):

def set_gguf_parameters(self):

Add num_heads to the list here instead:

llama.cpp/convert_hf_to_gguf.py

Line 1258 in de22157

self.gguf_writer.add_vision_head_count(self.find_vparam(["num_attention_heads"]))

So I didn't end up doing this, because after I add the "num_heads" to the list, self.find_vparam(["num_attention_heads"]) will still fail due to not finding the key.
I think this workaround would need to stay... Unless there's some other way?

In case it wasn't clear, I meant modifying the list like this: self.find_vparam(["num_attention_heads", "num_heads"])

Then it should just work and you don't need to change hparams.

CISC · 2025-08-02T20:23:51Z

convert_hf_to_gguf.py

+        if "query_key_value" in name:
+            # Split tensor into three along first axis
+            q, k, v = data_torch.split(data_torch.shape[0] // 3, dim=0)
+            return [
+                (self.map_tensor_name(name.replace("query_key_value", "query")), q),
+                (self.map_tensor_name(name.replace("query_key_value", "key")), k),
+                (self.map_tensor_name(name.replace("query_key_value", "value")), v),
+            ]
+
+        return [(self.map_tensor_name(name), data_torch)]


Suggested change

if "query_key_value" in name:

# Split tensor into three along first axis

q, k, v = data_torch.split(data_torch.shape[0] // 3, dim=0)

return [

(self.map_tensor_name(name.replace("query_key_value", "query")), q),

(self.map_tensor_name(name.replace("query_key_value", "key")), k),

(self.map_tensor_name(name.replace("query_key_value", "value")), v),

]

return [(self.map_tensor_name(name), data_torch)]

return [(self.map_tensor_name(name), data_torch)]

Create Q/K/V views at build time instead (check other (non-mm) models for examples).

Good point, I've changed it to do the split in llama-model.cpp instead.

CISC · 2025-08-02T20:24:30Z

convert_hf_to_gguf.py

+    def set_gguf_parameters(self):
+        super().set_gguf_parameters()
+
+    def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:


Suggested change

def set_gguf_parameters(self):

super().set_gguf_parameters()

def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:

def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:

CISC · 2025-08-02T20:25:55Z

convert_hf_to_gguf.py

+        if "query_key_value.weight" in name:
+            # Slice tensor into three along first axis
+            q, k, v = data_torch.split(data_torch.shape[0] // 3, dim=0)
+            return [
+                (self.map_tensor_name(name.replace("query_key_value", "query")), q),
+                (self.map_tensor_name(name.replace("query_key_value", "key")), k),
+                (self.map_tensor_name(name.replace("query_key_value", "value")), v),
+            ]
+
+        return [(self.map_tensor_name(name), data_torch)]


Suggested change

if "query_key_value.weight" in name:

# Slice tensor into three along first axis

q, k, v = data_torch.split(data_torch.shape[0] // 3, dim=0)

return [

(self.map_tensor_name(name.replace("query_key_value", "query")), q),

(self.map_tensor_name(name.replace("query_key_value", "key")), k),

(self.map_tensor_name(name.replace("query_key_value", "value")), v),

]

return [(self.map_tensor_name(name), data_torch)]

return [(self.map_tensor_name(name), data_torch)]

Same as above comment - both CLIP and the main model split the tensors in the graph instead.

CISC · 2025-08-02T20:34:12Z

src/llama-model.cpp

+                Qcur = ggml_rope(ctx0, Qcur, inp_pos, n_embd_head, GGML_ROPE_TYPE_NEOX);
+                Kcur = ggml_rope(ctx0, Kcur, inp_pos, n_embd_head, GGML_ROPE_TYPE_NEOX);


Update llama_model_rope_type instead and use rope_type.

Good point, I've change it to use rope_type instead.

Tianyue-Zhao · 2025-08-06T00:12:09Z

Also, is there a way to run these Github workflows locally or without needing approval from a reviewer? It would be good to run these CI/CD checks myself before posting the PR.

You can run flake8, pyright and editorconfig locally (or via IDE plugins), the build tests can be run manually with ctest.

Thanks for the info! That's something I've been wondering about for a while.

CISC · 2025-08-08T07:44:07Z

src/llama-model.cpp

+                // split qkv into Q, K, V along the first dimension
+                ggml_tensor * Qcur = ggml_cont(ctx0, ggml_view_2d(ctx0, qkv, n_embd, n_tokens,
+                    qkv->nb[1], 0));
+                ggml_tensor * Kcur = ggml_cont(ctx0, ggml_view_2d(ctx0, qkv, n_embd, n_tokens,
+                    qkv->nb[1], n_embd * ggml_element_size(qkv)));
+                ggml_tensor * Vcur = ggml_cont(ctx0, ggml_view_2d(ctx0, qkv, n_embd, n_tokens,
+                    qkv->nb[1], 2 * n_embd * ggml_element_size(qkv)));
+
+                Qcur = ggml_reshape_3d(ctx0, Qcur, n_embd_head, n_head,    n_tokens);
+                Kcur = ggml_reshape_3d(ctx0, Kcur, n_embd_head, n_head_kv, n_tokens);


Suggested change

// split qkv into Q, K, V along the first dimension

ggml_tensor * Qcur = ggml_cont(ctx0, ggml_view_2d(ctx0, qkv, n_embd, n_tokens,

qkv->nb[1], 0));

ggml_tensor * Kcur = ggml_cont(ctx0, ggml_view_2d(ctx0, qkv, n_embd, n_tokens,

qkv->nb[1], n_embd * ggml_element_size(qkv)));

ggml_tensor * Vcur = ggml_cont(ctx0, ggml_view_2d(ctx0, qkv, n_embd, n_tokens,

qkv->nb[1], 2 * n_embd * ggml_element_size(qkv)));

Qcur = ggml_reshape_3d(ctx0, Qcur, n_embd_head, n_head, n_tokens);

Kcur = ggml_reshape_3d(ctx0, Kcur, n_embd_head, n_head_kv, n_tokens);

// split qkv into Q, K, V along the first dimension

ggml_tensor * Qcur = ggml_view_3d(ctx0, qkv, n_embd_head, n_head, n_tokens, n_embd_head * sizeof(float),

qkv->nb[1], 0);

ggml_tensor * Kcur = ggml_view_3d(ctx0, qkv, n_embd_head, n_head_kv, n_tokens, n_embd_head * sizeof(float),

qkv->nb[1], n_embd * ggml_element_size(qkv));

ggml_tensor * Vcur = ggml_cont(ctx0, ggml_view_2d(ctx0, qkv, n_embd, n_tokens,

qkv->nb[1], 2 * n_embd * ggml_element_size(qkv)));

All backends can handle non-contiguous RoPE, so you don't need to ggml_cont Q/K here, and as a bonus you can directly create 3D views.

CISC · 2025-08-08T07:52:52Z

tools/mtmd/clip.cpp

+        // Apply silu
+        gate = ggml_silu_inplace(ctx0, gate);
+
+        // Multiply together
+        cur = ggml_mul(ctx0, gate, h_to_4h);


Suggested change

// Apply silu

gate = ggml_silu_inplace(ctx0, gate);

// Multiply together

cur = ggml_mul(ctx0, gate, h_to_4h);

// Apply swiglu

cur = ggml_swiglu_split(ctx0, gate, h_to_4h);

github-actions bot added examples python python script changes labels Aug 1, 2025

Tianyue-Zhao marked this pull request as ready for review August 1, 2025 02:15

Tianyue-Zhao force-pushed the cogvlm_support branch from 42113d1 to de22157 Compare August 2, 2025 16:45

CISC reviewed Aug 2, 2025

View reviewed changes

Tianyue-Zhao added 16 commits August 7, 2025 23:57

Added GGUF mappings for CogVLM model

3d98595

Add tensor mapping for CogVLM visual encoder

acc60c6

Add CogVLM to conversion script, no vision part yet

bd79e32

Added CogVLM vision model to conversion script

84ec245

Add graph for CogVLM CLIP model

c671c72

Add graph for CogVLM

722f90e

Fixes for CogVLM. Now compiles.

39ea549

Model now runs

b694293

Fixes for cogvlm graph

7ffa864

Format changes

4d57805

Fix broken rebase

9c06dc6

Rebase and fix conflicts and type check

0811eef

Use QKV tensor rather than splitting at convert time

a1c5697

Change to combined QKV in CLIP for CogVLM

40b07b2

Change to combined QKV for CogVLM LLM portion

60739b8

Changes according to comments

a571d9a

Tianyue-Zhao force-pushed the cogvlm_support branch from de22157 to a571d9a Compare August 8, 2025 00:35

CISC reviewed Aug 8, 2025

View reviewed changes

		Qcur = ggml_rope(ctx0, Qcur, inp_pos, n_embd_head, GGML_ROPE_TYPE_NEOX);
		Kcur = ggml_rope(ctx0, Kcur, inp_pos, n_embd_head, GGML_ROPE_TYPE_NEOX);

Add support for CogVLM model #15002

Are you sure you want to change the base?

Add support for CogVLM model #15002

Conversation

Tianyue-Zhao commented Aug 1, 2025

Uh oh!

Tianyue-Zhao commented Aug 2, 2025

Uh oh!

CISC commented Aug 2, 2025

Uh oh!

CISC left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tianyue-Zhao commented Aug 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!