Feature Request: The script convert_hf_to_gguf.py supports conversion of DeepSeek-R1-0528-FP4.

[luke-8-pro]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

I'm trying to use convert_hf_to_gguf.py to convert the DeepSeek-R1-0528-FP4 safetensor files into gguf format. I hope llama.cpp can be further improved to fully support the conversion of models like DeepSeek-R1-0528-FP4 and other NVFP4-type formats.

Motivation

Adding support to convert the DeepSeek-R1-0528-FP4 model into the GGUF format via convert_hf_to_gguf.py is a critical enhancement for the llama.cpp ecosystem. DeepSeek-R1-0528-FP4 is a recently released, high-performance language model developed by DeepSeek, featuring strong reasoning capabilities and optimized for efficient inference. As it gains increasing attention within the open-source community, more and more users are interested in running it locally on consumer-grade hardware—precisely the use case that llama.cpp is designed to enable.

However, llama.cpp only supports models in the GGUF format, which is specifically designed for memory efficiency, flexible quantization, and fast CPU/GPU inference. Without a reliable conversion pipeline from Hugging Face (safetensors) to GGUF, users cannot fully leverage the potential of this model within the llama.cpp framework.

Currently, attempts to convert DeepSeek-R1-0528-FP4 using the latest version of convert_hf_to_gguf.py result in runtime errors. These issues likely stem from architectural differences or metadata handling that the script does not yet fully support—such as custom layer configurations, tensor naming conventions, or FP4-specific quantization logic.

Enabling this conversion would provide several key benefits:

Local Inference Accessibility: Users could run DeepSeek-R1-0528-FP4 directly on personal devices without relying on cloud APIs, ensuring data privacy, low latency, and offline usability.

Efficient Hardware Utilization: The advanced quantization options in GGUF allow users to run large models on systems with limited VRAM or even on CPUs, significantly lowering the hardware barrier to entry.

Enhanced Compatibility with Existing Tools: Once converted, the model can be seamlessly integrated into a wide range of llama.cpp frontends (e.g., LM Studio, Ollama, text-generation-webui), improving user experience and ecosystem interoperability.

Community Empowerment: Providing official or community-supported conversion enables broader adoption, fine-tuning, and the development of downstream applications built upon this powerful model.

In summary, supporting the conversion of DeepSeek-R1-0528-FP4 in convert_hf_to_gguf.py is not merely a technical improvement—it is a strategic step toward maintaining inclusivity, performance, and future-readiness within the local LLM community. Resolving the current conversion issues will empower users to harness one of the most promising open-source models of 2024 within the efficient and versatile llama.cpp runtime.

Possible Implementation

No response

[pwilkin]

Why on earth would you want to convert and requant a low precision quant when you already have plenty of native DeepSeek-R1-0528 GGUF quants like https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF or https://huggingface.co/bartowski/deepseek-ai_DeepSeek-R1-0528-GGUF?

[whatever1983]

@pwilkin:
Obviously, you have no understanding of the impact of half vram size and Petaflops FP4 acceleration on GB200 and MI355x. The entire llama.cpp team don't understand it.

[pwilkin]

Mate, what are you even talking about?

FP4 is a storage format for numbers (floating point 4-bit precision). Yes, some chips have hardware acceleration for FP4 operations natively. They also have hardware acceleration for other data types (like FP8, FP16 or INT8).

This does not change the fact that you do not want to requantize (unpack and repack) data from a low resolution quant such as FP4 to GGUF since this loses a lot of model quality. There are already GGUF quants created that use 4-bit precision (such as IQ4_NL quants or Q4_K quants). In fact, there are GGUF quants that pack the model to less than 20% VRAM size using 1.5bit quants (the IQ1 quants).

And the operation speed does not depend solely on quant type, it also depends on specific operations done on those quants, how well are they optimized and which hardware operations are supported.

[whatever1983]

@pwilkin

Precisely the point why you don't get it. Q4KM? That's a DIY type. Even IK's IQK IQ4K or IQ6K is deprecated in the face of FP4. It is about perfect trained weights at 4.25bpw(mxFP4) and 4.5bpw(nvFP4).

You know how much data is used to convert FP8 to FP4? Look at this repo:

amd/DeepSeek-R1-MXFP4-Preview/tree/main

Calibration Dataset: Pile which is 800GB+

Nvidia's modelopt calibration dataset is probably in the multi-Terabytes to retrain FP8 models to NVFP4.

It is a near perfect requant to FP4 even if the original model is trained in FP8. A much higher quality quantization than Q4KM. How much data is used for imatrix calibration: usually < 1MB by bartowski or mradermacher.

Then there is the gpt-oss guys who train models directly in mxfp4 using H100s, which is perfect weights without modelopt requant from FP8-> FP4.

Then there is the acceleration: NVL72 is doing 1.5million tokens/s on GPT-OSS 120B FP4. That's over 20K tokens/s per GPU. Mac laptops are doing 40 tokens/s because of the fp16 datatype recast. It is not even in the same ballpark.

I don't want to waste more time. FP4 is THE driving force behind the current US AI inferencing and training innovation. You either get it or don't.

[bitbottrap]

The original request isn't to re-quantize the data. The original request is asking to repackage the already quantized data so that llama.cpp can run it.

"convert_hf_to_gguf" to me means format conversion. (Even though it does some limited quantization duty.)

I arrived here for similar reasons - wanting to repackage NVFP4/Qwen3-Coder-480B-A35B-Instruct-FP4 into gguf so it could be used with llama.cpp.

[luke-8-pro]

The original request isn't to re-quantize the data. The original request is asking to repackage the already quantized data so that llama.cpp can run it.

"convert_hf_to_gguf" to me means format conversion. (Even though it does some limited quantization duty.)

I arrived here for similar reasons - wanting to repackage NVFP4/Qwen3-Coder-480B-A35B-Instruct-FP4 into gguf so it could be used with llama.cpp.

Yes, we encountered the same issue. Would you like to collaborate on submitting a patch to the community to improve this script?

[pwilkin]

You can't just "repackage" NVFP4 quants into GGUF. GGUF is not just a storage format, it's also a spec for quantization formats.

What you're asking here is basically "support NVFP4 quantization natively for GGUF, similar to MXFP4".

[bitbottrap]

The different fp4 formats are where some of the confusion came from. I didn't notice at first so I'm going to write this to be clear. The DeepSeek model and the one I referenced have their data in nvfp4 format. This data format is not currently supported by GGUF. gpt-oss models were released with mxfp4 format data which is (recently) supported in GGUF.

Any safetensor model with mxfp4 should be able to be relatively easily repackaged into GGUF (though convert_hf_to_gguf.py does not support this yet). Conversion from one of the NVFP4 data format safetensor models would have to undergo a quantization step. And despite the individual values being 4-bit "fp4" values they are organized and require additional scaling factors that are quite different.

You would be better off quantizing from the original model into mxfp4 rather than attempt to use an nvfp4 model as the source.

[luke-8-pro]

I'm currently just trying to use convert_hf_to_gguf.py to convert nvfp4-based model weights into the GGUF format, so that models from any source can be processed within the GGUF framework. After all, they essentially consist of the same kinds of data—model weights and scale information.

So far, I've modified convert_hf_to_gguf.py to successfully convert nvfp4 models into GGUF format. There are some warnings during the process, but it works. However, I'm not entirely sure whether the converted model remains intact or if its behavior has been distorted.

What do you think—should I submit my patch so the community can review it and help me check?

[luke-8-pro]

The different fp4 formats are where some of the confusion came from. I didn't notice at first so I'm going to write this to be clear. The DeepSeek model and the one I referenced have their data in nvfp4 format. This data format is not currently supported by GGUF. gpt-oss models were released with mxfp4 format data which is (recently) supported in GGUF.

Any safetensor model with mxfp4 should be able to be relatively easily repackaged into GGUF (though convert_hf_to_gguf.py does not support this yet). Conversion from one of the NVFP4 data format safetensor models would have to undergo a quantization step. And despite the individual values being 4-bit "fp4" values they are organized and require additional scaling factors that are quite different.

You would be better off quantizing from the original model into mxfp4 rather than attempt to use an nvfp4 model as the source.

What do you think—should I submit my patch so the community can review it and help me check?

[CISC]

@luke-8-pro Please see #14810 which will make that much simpler to implement.

[bitbottrap]

@luke-8-pro You're repacking nvfp4 into gguf? If so I believe you're ahead of what ggml in llama.cpp supports?

[luke-8-pro]

You can't just "repackage" NVFP4 quants into GGUF. GGUF is not just a storage format, it's also a spec for quantization formats.

What you're asking here is basically "support NVFP4 quantization natively for GGUF, similar to MXFP4".

Our discussion has somewhat deviated, but the core point is that NVIDIA has introduced a new quantization format, NVFP4, and we hope GGUF can support it.

[luke-8-pro]

@luke-8-pro Please see #14810 which will make that much simpler to implement.

After looking at the merge request, currently the tool supports model conversion for FP8, correct?