Misc. bug: Finetuning yields different and worse results using CPU backend vs. CUDA backend

[AmyFromSonic]

Name and Version

$./build/bin/llama-cli --version
version: 5358 (10d2af0)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0 for x86_64-linux-gnu

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

Other (Please specify in the next section)

Command line

CPU: ./build/bin/llama-finetune --file ./gsm8k_with_newlines_first_125.txt --model smollm2-135M-base.gguf -b 512 -ub 512 -np 1 --device none

CUDA: CUDA_VISIBLE_DEVICES=0 ./build/bin/llama-finetune --file ./gsm8k_with_newlines_first_125.txt --model smollm2-135M-base.gguf -b 512 -ub 512 -np 1 -ngl 999

Problem description & steps to reproduce

Hey all,

I've noticed that examples/training/finetune.cpp code yields different and worse results when using the CPU backend.

As a minimal example to show this, I used finetune.cpp in the following situation:

• Model: SmolLM2-135M Base (https://huggingface.co/HuggingFaceTB/SmolLM2-135M). This was converted to GGUF using llama.cpp/convert_hf_to_gguf.py specifying f32 as the --outtype.
• Dataset: First 125 samples from GSM8K (https://huggingface.co/datasets/openai/gsm8k), stored in a single newline-delimited file (attached below). This small subset enables quick testing, since CPU training can be quite slow.
• Hyperparameters: Defaults used by the examples/training code - except the following:
  • Epochs to train for: Modified from 2 -> 10
  • Learning rate upped from 1e-7 -> 1e-6

I would expect SmolLM2-135M Base to quickly learn on (and maybe overfit) this small amount of data. Finetuning using the CUDA backend shows this is the case, with training loss decreasing over time.

I would expect similar behavior when finetuning using the CPU backend given that this is the same model, same dataset, and same hyperparameters. The only difference is the backend. However, this is showing that models finetuned with the CPU aren't learning.

I believe there's a bug in the CPU backend of the finetuning code specifically that is causing this discrepancy.

Results

I've attached the logfiles from running below:

llamacpp_finetune_smollm2135mbase_on_gsmk_125samples_cpu.log
llamacpp_finetune_smollm2135mbase_on_gsmk_125samples_gpu.log

I've attached charts showing this behavior below:

Data Used

I've attached the data used below:

gsm8k_with_newlines_first_125.txt

First Bad Commit

Commit 10d2af0

Relevant log output

[conscell]

I've experienced a similar issue in #14424. In the current version (b6424), finetuning is completely broken:

llama.cpp-b6424/ggml/src/ggml.c:6560: GGML_ASSERT(!node->view_src || node->op == GGML_OP_CPY || node->op == GGML_OP_VIEW || node->op == GGML_OP_RESHAPE || node->op == GGML_OP_PERMUTE || node->op == GGML_OP_TRANSPOSE) failed

[graehl]

I worked on finetune a bit a few months ago and at least while I was using it (took quite a while to get merged!) it appeared to work fine incl on cpu. I'll briefly look at the cpu issue since I was intending to use finetune for some work.
re: the breakage it's quite clearly due to insufficient CI testing - tuning is model specific since I guess backward/gradient isn't 100% for all ops at all times

I am investigating off an 8/28 branch since a fairly large and hard to revert change by @ggerganov removes support for the LLAMA_SET_ROWS env var. my understanding of the codebase doesn't let me easily repair the regression. perhaps @JohannesGaessler understand better (finetune is his baby)

5ef321d

d498af3 Georgi Gerganov 2025-07-18 14:31:15 +0300 graph : avoid huge
warm-up graphs for MoE models (#14753)

which greatly decreased graph_max_nodes
has been responsible for finetune failing on reasonably sized models
for the past two months. partially reverting the decrease (maybe
larger models still fail)

note: env LLAMA_SET_ROWS=0 is needed also or else:
GML_ASSERT(!node->view_src || node->op == GGML_OP_CPY || node->op == GGML_OP_VIEW || node->op == GGML_OP_RESHAPE || node->op == GGML_OP_PERMUTE || node->op == GGML_OP_TRANSPOSE) failed
(the node->op in question is indeed a rows op)
unfortunately a git revert on:

8a4280c Georgi Gerganov 2025-08-28 12:27:02 +0300 kv-cache : remove
LLAMA_SET_ROWS checks (#15505)

is not straightforward, so this branch is behind that.

[JohannesGaessler]

I'm aware of the issue but have given it comparatively low priority.