Eval bug: data corruption on CUDA experts offload

[pwilkin]

Name and Version

pwilkin@SYN-PC-11:/devel/models$ llama-cli --version
load_backend: loaded BLAS backend from /devel/tools/llama.cpp/build/bin/libggml-blas.so
register_backend: registered backend BLAS (1 devices)
register_device: registered device BLAS (OpenBLAS)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes
Device 1: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes
load_backend: loaded CUDA backend from /devel/tools/llama.cpp/build/bin/libggml-cuda.so
register_backend: registered backend CUDA (2 devices)
register_device: registered device CUDA0 (NVIDIA GeForce RTX 5070 Ti)
register_device: registered device CUDA1 (NVIDIA GeForce RTX 5070 Ti)
ggml_backend_load_best: /devel/tools/llama.cpp/build/bin/libggml-cpu-alderlake.so score: 128
ggml_backend_load_best: /devel/tools/llama.cpp/build/bin/libggml-cpu-icelake.so score: 0
ggml_backend_load_best: /devel/tools/llama.cpp/build/bin/libggml-cpu-haswell.so score: 64
ggml_backend_load_best: /devel/tools/llama.cpp/build/bin/libggml-cpu-skylakex.so score: 0
ggml_backend_load_best: /devel/tools/llama.cpp/build/bin/libggml-cpu-sapphirerapids.so score: 0
ggml_backend_load_best: /devel/tools/llama.cpp/build/bin/libggml-cpu-sandybridge.so score: 21
ggml_backend_load_best: /devel/tools/llama.cpp/build/bin/libggml-cpu-sse42.so score: 5
ggml_backend_load_best: /devel/tools/llama.cpp/build/bin/libggml-cpu-x64.so score: 1
load_backend: loaded CPU backend from /devel/tools/llama.cpp/build/bin/libggml-cpu-alderlake.so
register_backend: registered backend CPU (1 devices)
register_device: registered device CPU (Intel(R) Core(TM) i7-14700KF)
version: 6921 (eca77bf)
built with cc (Ubuntu 15.2.0-4ubuntu4) 15.2.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

CUDA

Hardware

2x 5070 Ti

Models

Minimax-M2

Problem description & steps to reproduce

This runs complex queries without problems:

llama-server -m MiniMaxAI_MiniMax-M2-IQ3_M/MiniMaxAI_MiniMax-M2-IQ3_M-00001-of-00003.gguf -ngl 99 --cpu-moe --host 0.0.0.0 -c 50000 -fa on --alias syndatis --threads 24 --chat-template-file /devel/tools/llama.cpp/models/templates/unsloth-MiniMax-M2.jinja --jinja

This starts generating corrupted outputs with any prompt of non-trivial size (say 500):

llama-server -m MiniMaxAI_MiniMax-M2-IQ3_M/MiniMaxAI_MiniMax-M2-IQ3_M-00001-of-00003.gguf -ngl 99 -ot "\.([0-9]|[0-5][0-9]|5[0-3])\.ffn_.*_exps=CPU,blk.5[4-8].*=CUDA0,blk.(6[0-2]|5[8-9]).*=CUDA1" --host 0.0.0.0 -c 50000 -fa on --alias syndatis --threads 24 --chat-template-file /devel/tools/llama.cpp/models/templates/unsloth-MiniMax-M2.jinja --jinja

Possibly related to #16935

First Bad Commit

No response

Relevant log output

N/A

[am17an]

Does #16935 cause it or fix it?

[pwilkin]

Does #16935 cause it or fix it?

What I mean is that it's probably related to the same bug that #16935 fixes, but since I tested on the unmerged PR for Minimax I haven't checked that yet. I'll try to merge the PR and see if it fixes it.

[pwilkin]

Okay, so unfortunately it's not a matter of that bug.

When I run pure --cpu-moe, everything works fine. As soon as I offload even a part of the experts to one of the GPUs, processing falls apart. It seems to be a pretty clear correlation - the more layers I offload, the worse it gets.

[pwilkin]

Found the culprit:

CPU:

ggml_debug:           ffn_moe_gate-0 = (f32) MUL_MAT_ID(blk.0.ffn_gate_exps.weight{3072, 1536, 256, 1}, ffn_norm-0 (reshaped){3072, 1, 8, 1}}) = {1536, 8, 8, 1}
                                     [
                                      [
                                       [     -7.6687,      -6.1960,      -3.1724, ...,      -3.0835,      -3.2312,      -4.8651],
                                       [     -4.7759,      -6.6228,      -5.6057, ...,      -3.4599,      -6.9503,      -4.3172],
                                       [      0.3917,      -2.9525,      -5.0509, ...,      -5.9955,      -6.3869,      -6.9655],
                                       ..., 
                                       [      0.1610,      -0.0325,      -3.6669, ...,       0.0904,      -0.0382,      -3.3468],
                                       [     -0.0050,       0.1472,       0.3272, ...,      -3.6472,      -0.3675,       0.0055],
                                       [     -3.6368,       0.1342,      -0.1055, ...,      -0.2336,      -0.0738,      -3.9070],
                                      ],
                                      [
                                       [     -0.0137,      -9.5569,      -7.9258, ...,      -9.9314,      -6.4659,      -0.1261],
                                       [     -0.1835,      -7.5041,      -0.1172, ...,      -8.2747,      -8.2225,       0.0363],
                                       [      0.2263,      -7.1513,       0.0965, ...,      -6.8539,      -6.4829,       0.1560],
                                       ..., 
                                       [     -7.4047,      -7.8707,       0.0959, ...,      -7.0869,      -6.7037,      -6.9756],
                                       [     -7.6217,      -8.7473,       0.0240, ...,      -0.0384,      -8.4723,      -8.2945],
                                       [     -7.9267,      -6.7185,       0.0782, ...,      -7.6193,       0.0492,      -8.0961],
                                      ],
                                      [
                                       [     -5.4456,      -7.6701,      -5.9626, ...,      -3.4283,      -7.5081,      -5.7541],
                                       [      0.2337,      -7.1296,      -5.8672, ...,      -0.0907,      -5.1970,      -6.6397],
                                       [      0.4396,      -7.4853,       0.2393, ...,      -7.2204,      -6.7023,       0.1432],
                                       ..., 
                                       [      0.0870,      -5.3323,      -5.5632, ...,       0.1235,       0.0105,      -7.1312],
                                       [     -7.5365,      -0.0298,      -0.0473, ...,      -7.0981,      -0.1157,      -0.0268],
                                       [     -0.1926,      -7.0820,      -0.1730, ...,      -7.9011,      -7.7972,       0.0749],
                                      ],
                                      ..., 
                                      [
                                       [     -7.9882,       0.1448,      -7.9131, ...,      -7.1935,       0.0006,       0.0508],
                                       [     -0.0462,      -8.5043,      -6.9740, ...,      -8.7574,      -5.4125,      -0.1027],
                                       [     -7.7969,      -6.6479,      -0.0420, ...,      -7.4814,       0.0318,      -8.1191],
                                       ..., 
                                       [     -7.1983,       0.0478,       0.0204, ...,      -6.6548,      -0.0530,      -0.0631],
                                       [     -4.7405,      -7.1288,      -7.9672, ...,      -8.1737,      -7.2784,      -7.2554],
                                       [      0.0043,      -6.6175,      -7.9470, ...,      -7.0365,      -7.5810,      -7.5130],
                                      ],
                                      [
                                       [     -5.8804,      -4.9585,      -5.6229, ...,      -7.6106,      -8.2512,      -3.5772],
                                       [     -4.6661,      -0.1408,      -3.1847, ...,      -4.3243,      -0.2500,      -8.9557],
                                       [     -4.4661,     -12.0820,      -4.1740, ...,      -9.8577,      -7.9137,      -4.6550],
                                       ..., 
                                       [      0.0801,      -5.9353,       0.0141, ...,       0.0868,      -0.1584,      -6.3982],
                                       [     -8.0323,       0.2582,       0.0541, ...,      -7.4233,      -0.1814,      -0.1338],
                                       [     -7.6257,      -6.7228,      -3.8659, ...,      -3.9877,      -3.8615,      -5.3955],
                                      ],
                                      [
                                       [     -5.5880,      -7.3994,       0.3604, ...,      -6.9231,      -7.5640,      -7.4724],
                                       [     -0.0771,      -0.0849,       0.2090, ...,      -0.1729,      -4.4847,      -5.0879],
                                       [     -0.3713,      -0.2392,      -4.2622, ...,       0.1209,       0.1331,      -0.1732],
                                       ..., 
                                       [     -0.1105,      -6.7346,      -6.7284, ...,       0.0588,      -6.1236,      -6.5290],
                                       [     -0.2052,      -5.8038,      -5.7952, ...,      -5.6797,       0.0482,      -5.7861],
                                       [      0.0217,      -6.5276,      -6.0358, ...,      -0.0067,      -0.0661,       0.2268],
                                      ],
                                     ]
                                     sum = -401105.406250
ggml_debug:             ffn_moe_up-0 = (f32) MUL_MAT_ID(blk.0.ffn_up_exps.weight{3072, 1536, 256, 1}, ffn_norm-0 (reshaped){3072, 1, 8, 1}}) = {1536, 8, 8, 1}
                                     [
                                      [
                                       [     -0.0649,      -0.0269,      -0.1772, ...,      -0.0185,      -0.2081,       0.2950],
                                       [      0.1176,       0.2702,      -0.2145, ...,       0.0527,       0.0861,      -0.1285],
                                       [      0.0551,       0.2376,      -0.0306, ...,       0.2060,       0.3657,      -0.1624],
                                       ..., 
                                       [      0.0801,      -0.1911,       0.0311, ...,      -0.2055,       0.0609,      -0.0267],
                                       [     -0.1092,      -0.0260,       0.0377, ...,       0.0917,      -0.0189,      -0.1785],
                                       [     -0.1659,       0.0222,      -0.0373, ...,       0.0869,       0.2371,       0.0408],
                                      ],
                                      [
                                       [     -0.0227,       0.0493,       0.0677, ...,      -0.0545,       0.0997,      -0.0753],
                                       [     -0.0217,      -0.0159,      -0.0126, ...,      -0.0228,      -0.0114,      -0.1269],
                                       [      0.2630,      -0.0994,      -0.0170, ...,       0.0266,       0.2049,      -0.0056],
                                       ..., 
                                       [     -0.0772,       0.0560,      -0.0937, ...,      -0.1082,      -0.0065,       0.1248],
                                       [      0.1828,      -0.2121,      -0.0001, ...,       0.0575,       0.0812,       0.1687],
                                       [     -0.0323,      -0.0365,      -0.0818, ...,      -0.0793,      -0.0234,      -0.0018],
                                      ],
                                      [
                                       [     -0.0629,       0.1658,      -0.0903, ...,      -0.2151,       0.1362,      -0.0097],
                                       [     -0.2778,       0.0913,      -0.0190, ...,       0.1131,       0.0373,       0.0979],
                                       [      0.5537,      -0.1390,      -0.2923, ...,       0.0488,       0.1945,       0.0002],
                                       ..., 
                                       [     -0.0124,      -0.2581,      -0.1860, ...,      -0.1146,      -0.0640,      -0.0257],
                                       [      0.0281,      -0.0469,       0.0401, ...,       0.0129,       0.0240,      -0.1028],
                                       [     -0.0743,      -0.0381,       0.0833, ...,       0.0527,      -0.0012,      -0.1685],
                                      ],
                                      ..., 
                                      [
                                       [     -0.0256,       0.0429,      -0.0116, ...,       0.1048,      -0.0908,      -0.0040],
                                       [      0.0179,       0.0617,       0.0558, ...,      -0.0949,       0.1609,      -0.0476],
                                       [      0.0895,       0.1132,      -0.0625, ...,       0.0396,      -0.0541,      -0.0400],
                                       ..., 
                                       [      0.0797,      -0.0026,       0.0284, ...,      -0.0243,       0.0731,      -0.0258],
                                       [      0.0134,      -0.0192,      -0.0606, ...,      -0.0017,      -0.0533,      -0.0915],
                                       [     -0.0532,      -0.0271,       0.0329, ...,       0.0361,      -0.0170,      -0.0282],
                                      ],
                                      [
                                       [     -0.0662,      -0.2984,      -0.1849, ...,      -0.0593,       0.0936,      -0.1455],
                                       [      0.0183,      -0.3247,       0.0433, ...,       0.1417,      -0.0377,       0.2282],
                                       [     -0.1805,       0.1212,       0.0148, ...,      -0.0531,       0.2324,      -0.0888],
                                       ..., 
                                       [     -0.2095,      -0.1012,      -0.1457, ...,      -0.1853,      -0.0221,      -0.1881],
                                       [      0.1046,      -0.0314,      -0.0435, ...,       0.0704,      -0.1183,      -0.0052],
                                       [     -0.2149,      -0.2271,       0.0415, ...,      -0.0801,       0.0086,      -0.0497],
                                      ],
                                      [
                                       [      0.2516,       0.0170,      -0.4207, ...,       0.0807,       0.0355,       0.0489],
                                       [      0.0251,      -0.2137,       0.0891, ...,       0.0987,       0.0591,      -0.0134],
                                       [     -0.1475,       0.0476,       0.0491, ...,      -0.0154,       0.0862,      -0.1002],
                                       ..., 
                                       [      0.0327,       0.1763,      -0.1341, ...,       0.0764,       0.1344,      -0.0757],
                                       [     -0.1698,      -0.1129,      -0.0780, ...,       0.0372,       0.0929,      -0.0515],
                                       [     -0.0586,       0.0151,      -0.2188, ...,      -0.1514,       0.1250,      -0.1643],
                                      ],
                                     ]
                                     sum = 11.451819

CUDA:

ggml_debug:           ffn_moe_gate-0 = (f32) MUL_MAT_ID(blk.0.ffn_gate_exps.weight{3072, 1536, 256, 1}, ffn_norm-0 (reshaped){3072, 1, 8, 1}}) = {1536, 8, 8, 1}
                                     [
                                      [
                                       [     -7.6888,      -6.2249,      -3.1697, ...,      -3.1095,      -3.2352,      -4.8580],
                                       [     -4.7974,      -6.6402,      -5.6600, ...,      -3.4737,      -6.9963,      -4.3447],
                                       [      0.3762,      -2.9918,      -5.0812, ...,      -6.0520,      -6.4382,      -7.0307],
                                       ..., 
                                       [      0.1425,      -0.0340,      -3.6835, ...,       0.0811,      -0.0290,      -3.3628],
                                       [     -0.0211,       0.1445,       0.3290, ...,      -3.6487,      -0.3838,       0.0007],
                                       [     -3.6241,       0.1457,      -0.1144, ...,      -0.2374,      -0.0774,      -3.8993],
                                      ],
                                      [
                                       [     -0.0031,      -9.5902,      -7.9343, ...,      -9.9407,      -6.4767,      -0.1204],
                                       [     -0.1579,      -7.5483,      -0.1099, ...,      -8.3190,      -8.2670,       0.0148],
                                       [      0.2163,      -7.1703,       0.0876, ...,      -6.8664,      -6.5072,       0.1509],
                                       ..., 
                                       [     -7.4225,      -7.8928,       0.0890, ...,      -7.1117,      -6.7139,      -6.9936],
                                       [     -7.6229,      -8.7505,       0.0300, ...,      -0.0378,      -8.4748,      -8.2937],
                                       [     -7.9776,      -6.7539,       0.0601, ...,      -7.6529,       0.0418,      -8.1487],
                                      ],
                                      [
                                       [     -5.4334,      -7.6766,      -5.9562, ...,      -3.4456,      -7.5015,      -5.7421],
                                       [      0.2474,      -7.1960,      -5.9197, ...,      -0.0805,      -5.2206,      -6.6769],
                                       [      0.4236,      -7.5127,       0.2278, ...,      -7.2347,      -6.7257,       0.1715],
                                       ..., 
                                       [      0.0945,      -5.3602,      -5.5770, ...,       0.1219,       0.0244,      -7.1510],
                                       [     -7.5831,      -0.0247,      -0.0560, ...,      -7.1443,      -0.1095,      -0.0255],
                                       [     -0.1857,      -7.1350,      -0.1576, ...,      -7.9709,      -7.8639,       0.0641],
                                      ],
                                      ..., 
                                      [
                                       [     -7.9493,       0.1492,      -7.8703, ...,      -7.1656,       0.0015,       0.0452],
                                       [     -0.0492,      -8.4501,      -6.9339, ...,      -8.6990,      -5.3918,      -0.1100],
                                       [     -7.7771,      -6.6261,      -0.0449, ...,      -7.4510,       0.0283,      -8.0995],
                                       ..., 
                                       [      0.0023,      -6.6370,      -7.9728, ...,      -7.0647,      -7.5908,      -7.5203],
                                       [      0.0422,      -5.6239,       0.0207, ...,      -0.0854,       0.0100,      -5.7280],
                                       [     -4.7426,      -7.1135,      -7.9444, ...,      -8.1369,      -7.2717,      -7.2561],
                                      ],
                                      [
                                       [     -5.8718,      -4.9593,      -5.6296, ...,      -7.6094,      -8.2768,      -3.5947],
                                       [     -4.6960,      -0.1401,      -3.2118, ...,      -4.3405,      -0.2606,      -9.0129],
                                       [     -4.4814,     -12.0828,      -4.1688, ...,      -9.8785,      -7.9095,      -4.6486],
                                       ..., 
                                       [      0.0144,      -0.1157,       0.0866, ...,      -0.2777,       0.3284,      -0.0220],
                                       [     -7.9904,       0.2414,       0.0569, ...,      -7.3886,      -0.1645,      -0.1285],
                                       [     -7.6365,      -6.7156,      -3.8608, ...,      -3.9834,      -3.8621,      -5.4148],
                                      ],
                                      [
                                       [     -5.6177,      -7.4522,       0.3525, ...,      -6.9675,      -7.6023,      -7.5270],
                                       [     -0.0897,      -0.0917,       0.2221, ...,      -0.1858,      -4.4651,      -5.0611],
                                       [     -0.3601,      -0.2458,      -4.2452, ...,       0.1240,       0.1526,      -0.1745],
                                       ..., 
                                       [     -0.2057,      -5.8014,      -5.8002, ...,      -5.6965,       0.0518,      -5.8028],
                                       [      0.0596,       0.0103,      -4.2733, ...,       0.2042,      -0.1150,      -4.4234],
                                       [      0.0137,      -6.4922,      -6.0004, ...,      -0.0079,      -0.0553,       0.2040],
                                      ],
                                     ]
                                     sum = -401751.562500
ggml_debug:             ffn_moe_up-0 = (f32) MUL_MAT_ID(blk.0.ffn_up_exps.weight{3072, 1536, 256, 1}, ffn_norm-0 (reshaped){3072, 1, 8, 1}}) = {1536, 8, 8, 1}
                                     [
                                      [
                                       [     -0.0742,      -0.0180,      -0.1949, ...,       0.0034,      -0.1945,       0.3162],
                                       [      0.1152,       0.2830,      -0.2203, ...,       0.0616,       0.1006,      -0.1259],
                                       [      0.0455,       0.2276,      -0.0239, ...,       0.2024,       0.3772,      -0.1422],
                                       ..., 
                                       [      0.0537,      -0.2024,       0.0293, ...,      -0.2109,       0.0673,      -0.0221],
                                       [     -0.1084,      -0.0174,       0.0507, ...,       0.0957,      -0.0187,      -0.1703],
                                       [     -0.1592,       0.0270,      -0.0323, ...,       0.0809,       0.2335,       0.0433],
                                      ],
                                      [
                                       [     -0.0168,       0.0533,       0.0751, ...,      -0.0527,       0.1014,      -0.0824],
                                       [     -0.0223,      -0.0317,      -0.0061, ...,      -0.0159,      -0.0166,      -0.1204],
                                       [      0.2518,      -0.0892,      -0.0372, ...,       0.0317,       0.1958,       0.0043],
                                       ..., 
                                       [     -0.0719,       0.0617,      -0.0893, ...,      -0.1131,       0.0010,       0.1293],
                                       [      0.1768,      -0.1974,      -0.0128, ...,       0.0769,       0.0695,       0.1666],
                                       [     -0.0477,      -0.0397,      -0.0845, ...,      -0.0785,      -0.0217,      -0.0212],
                                      ],
                                      [
                                       [     -0.0661,       0.1702,      -0.0791, ...,      -0.2166,       0.1239,      -0.0128],
                                       [     -0.3046,       0.0720,      -0.0252, ...,       0.1101,       0.0405,       0.1006],
                                       [      0.5437,      -0.1426,      -0.2945, ...,       0.0441,       0.1837,      -0.0006],
                                       ..., 
                                       [     -0.0077,      -0.2686,      -0.1746, ...,      -0.1264,      -0.0785,      -0.0218],
                                       [      0.0109,      -0.0354,       0.0311, ...,       0.0107,       0.0141,      -0.0967],
                                       [     -0.0681,      -0.0439,       0.1010, ...,       0.0587,      -0.0127,      -0.1705],
                                      ],
                                      ..., 
                                      [
                                       [     -0.0203,       0.0691,      -0.0036, ...,       0.1072,      -0.0893,      -0.0058],
                                       [      0.0113,       0.0591,       0.0556, ...,      -0.0879,       0.1608,      -0.0451],
                                       [      0.0799,       0.1017,      -0.0512, ...,       0.0407,      -0.0483,      -0.0423],
                                       ..., 
                                       [     -0.0538,      -0.0197,       0.0333, ...,       0.0430,      -0.0202,      -0.0353],
                                       [     -0.0250,      -0.0066,      -0.0272, ...,      -0.1034,       0.0260,      -0.2270],
                                       [      0.0353,      -0.0320,      -0.0575, ...,      -0.0054,      -0.0524,      -0.0788],
                                      ],
                                      [
                                       [     -0.0587,      -0.2949,      -0.1684, ...,      -0.0489,       0.1085,      -0.1462],
                                       [      0.0282,      -0.3311,       0.0236, ...,       0.1566,      -0.0517,       0.2219],
                                       [     -0.1747,       0.1210,       0.0262, ...,      -0.0772,       0.2358,      -0.1118],
                                       ..., 
                                       [     -0.0184,      -0.0035,       0.1160, ...,      -0.1333,      -0.0605,       0.0704],
                                       [      0.1186,      -0.0424,      -0.0570, ...,       0.0675,      -0.1181,      -0.0150],
                                       [     -0.2184,      -0.2347,       0.0501, ...,      -0.0892,       0.0159,      -0.0521],
                                      ],
                                      [
                                       [      0.2450,       0.0144,      -0.4135, ...,       0.0662,       0.0273,       0.0373],
                                       [      0.0324,      -0.2047,       0.0798, ...,       0.0872,       0.0574,      -0.0194],
                                       [     -0.1313,       0.0383,       0.0533, ...,      -0.0075,       0.0624,      -0.1031],
                                       ..., 
                                       [     -0.1738,      -0.1107,      -0.0842, ...,       0.0324,       0.0928,      -0.0637],
                                       [      0.1246,      -0.1921,       0.0603, ...,       0.0167,       0.0005,       0.2350],
                                       [     -0.0492,       0.0153,      -0.2130, ...,      -0.1555,       0.0977,      -0.1657],
                                      ],
                                     ]
                                     sum = 16.413723

[CISC]

Oh, hey, this looks like the issue I had here:
#15510 (comment)

[pwilkin]

@CISC I'm wondering if this is just a MUT_MAL_ID precision issue, or something else. These expert matrices are very large, but on the other hand, DeepSeek already had bigger ones and I think nothing of the sort happened there...

MUL_MAT_ID doesn't support a "force F32 precision" modifier, but maybe it would be worthwhile to add that to see if it remedies the problem?

Interestingly enough, even with just --cpu-moe we get a divergence on expert selections already. The perplexity also goes through the roof as soon as I even use --cpu-moe, even though it's not immediately visible in the outputs:

Q8_0, -ngl 0: Final estimate: PPL = 2.0373 +/- 0.05656
Q8_0, --cpu-moe: Final estimate: PPL = 35.6157 +/- 1.91603

[pwilkin]

Further research: I've disabled MUL_MAT_ID manually in ggml-cuda.cu and ran the perplexity on full GPU (though it still ran mostly on CPU because MUL_MAT_ID is most of the heavy operations) and it's perfectly fine. So it's most certainly down to MUL_MAT_ID specifically.

[pwilkin]

@JohannesGaessler @compilade adding you guys as well.

[pwilkin]

Ops checks for those exact dimensions (unless I messed up something) seem to check out fine:

  MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=256,n_used=8,b=0,m=1536,n=8,k=3072,o=1): OK
  MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=256,n_used=8,b=0,m=1536,n=8,k=3072,o=1): OK
  MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=256,n_used=8,b=0,m=1536,n=8,k=3072,o=1): OK
  MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=256,n_used=8,b=1,m=1536,n=8,k=3072,o=1): OK
  MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=256,n_used=8,b=1,m=1536,n=8,k=3072,o=1): OK
  MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=256,n_used=8,b=1,m=1536,n=8,k=3072,o=1): OK

[slaren]

Does it work with this change?

--- a/ggml/src/ggml-backend.cpp
+++ b/ggml/src/ggml-backend.cpp
@@ -1450,7 +1450,7 @@ static enum ggml_status ggml_backend_sched_compute_splits(ggml_backend_sched_t s

                 // when offloading MoE weights, we can reduce the amount of data copied by copying only the experts that are used
                 ggml_tensor * node = split->graph.nodes[0];
-                if (split->graph.n_nodes > 0 &&
+                if (false && split->graph.n_nodes > 0 &&
                     ggml_backend_buffer_get_usage(input->buffer) == GGML_BACKEND_BUFFER_USAGE_WEIGHTS &&
                     ggml_backend_buffer_is_host(input->buffer) && (
                     (node->src[0] == input_cpy && node->op == GGML_OP_MUL_MAT_ID)