[question/bug] quantized sizes.

[0wwafa]

I have noticed that while the f16 gguf of the 20b model is around 12.5 GB all other smaller quantizations are around 11 GB. There seems to be no gain in size.
Any Idea why?

[0wwafa]

wait.. this must be a bug because:

huggingface-cli download openai/gpt-oss-20b --local-dir gpt-oss-20b

Converting model to f16.
python /content/llama.cpp/convert_hf_to_gguf.py --outtype f16 gpt-oss-20b --outfile gpt-oss-20b.f16.gguf

The above is 12 GB

Quantizing q5_k
./build/bin/llama-quantize --allow-requantize --output-tensor-type f16 --token-embedding-type f16 gpt-oss-20b.f16.gguf gpt-oss-20b.q5_k.gguf q5_k 2

The above is almost 17 GB!!! (why???)

Quantizing q6_k
./build/bin/llama-quantize --allow-requantize --output-tensor-type f16 --token-embedding-type f16 gpt-oss-20b.f16.gguf gpt-oss-20b.q6_k.gguf q6_k 2

The above is 21.8 GB !!

[CISC]

Not a bug per-se, it's just that the conversion kept the MXFP4 tensors, see #15111

[0wwafa]

Even this:
Quantizing q8q4
./build/bin/llama-quantize --allow-requantize --output-tensor-type q8_0 --token-embedding-type q8_0 gpt-oss-20b.f16.gguf gpt-oss-20b.q8q4.gguf q4_k 2

Came out 14.92 GB

[0wwafa]

Anyway.. I am testing those quantizations and NONE work.

[David-AU-github]

The issue is tensor sizes in OpenAI models ; Llamacpp falls back to IQ4_NL, Q5_1 and Q8 (and mixes).

The only quants that work well are IQ4_NL, Q5_1, new "Moe quant" and Q8s.
Note: You can apply Imatrix on IQ4_NL, Q5_1 and PARTS of the new "moe quant" (output tensor / tools)

All the other quants (due to fallbacks) are subpar in terms of performance ; AS of this writing.