Add T5Gemma support #14940 #15123
Open
Add T5Gemma support #14940 #15123
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
compilade
reviewed
Aug 6, 2025
convert_hf_to_gguf.py
Outdated
Comment on lines
6449
to
6450
| # Don't call super().__init__() because it tries to find standard layer count parameters | ||
| # that don't exist in T5Gemma models (they have encoder.num_hidden_layers instead) |
There was a problem hiding this comment.
If this is the only reason, maybe instead call super().__init__() with a modified hparams?
hparams = kwargs.get("hparams") or ModelBase.load_hparams(args[0] if args else kwargs["dir_model"])
encoder_config = hparams.get("encoder", {})
hparams["num_hidden_layers"] = encoder_config.get["num_hidden_layers"]
kwargs["hparams"] = hparams
super().__init__(*args, **kwargs)There was a problem hiding this comment.
already changed all commit , this is my first time to convert model in to gguf, i just convert t5gemma-s-s-prefixlm-it ... not sure that will work on all t5gemma
convert_hf_to_gguf.py
Outdated
| for i in range(self.block_count): | ||
| # Encoder relative attention bias - shape should be (n_rel_attn_bkts, n_head) | ||
| rel_bias_enc = torch.zeros(n_rel_attn_bkts, n_head_enc, dtype=torch.float16) | ||
| yield f"enc.blk.{i}.attn_rel_b.weight", rel_bias_enc |
There was a problem hiding this comment.
Use self.format_tensor_name if possible.
Suggested change
| yield f"enc.blk.{i}.attn_rel_b.weight", rel_bias_enc | |
| yield self.format_tensor_name(gguf.MODEL_TENSOR.ENC_ATTN_REL_B, i), rel_bias_enc |
(I did not test this, but should probably work)
This also applies to the other places in this function where the output tensor names are hardcoded.
convert_hf_to_gguf.py
Outdated
Comment on lines
6703
to
6708
| # Dynamically set encoder's other parameters | ||
| for key, value in encoder_config.items(): | ||
| if key not in ["max_position_embeddings", "hidden_size", "num_hidden_layers", "intermediate_size", | ||
| "num_attention_heads", "num_key_value_heads", "head_dim", "rms_norm_eps", | ||
| "sliding_window", "attn_logit_softcapping", "final_logit_softcapping", | ||
| "rope_theta", "attention_bias", "attention_dropout", "query_pre_attn_scalar", "vocab_size"]: |
There was a problem hiding this comment.
Instead of excluding keys, maybe enumerating the included keys could be more robust.
It would also avoid adding unexpected metadata which won't necessarily be used.
…ents (ggml-org#14940) - Add T5Gemma model support with proper encoder-decoder architecture - Use super().__init__() instead of manual initialization for better inheritance - Use format_tensor_name() for consistent tensor naming - Explicitly enumerate included keys instead of excluding keys - Add proper type annotations for better type safety - Fix all trailing whitespace issues - Support relative attention bias tensors generation - Handle T5Gemma-specific post-layer normalization tensors - Implement proper tokenizer handling for BPE tokenizer - Add comprehensive tensor mapping for all T5Gemma components
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement encoder-decoder architecture with rela tive attention bias, tensor mapping, and model conversion
only tested t5gemma-s-s-prefixlm-it due to memory limitations. Please correct us if there are any mistakes.
Make sure to read the contributing guidelines before submitting a PR