-
Notifications
You must be signed in to change notification settings - Fork 13.3k
feat: nemotron thinking & toolcalling support #15676
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+333
−0
Merged
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
e725519
feat: nemotron thinking & toolcalling support
pwilkin ad89166
Trailing whitespaces
pwilkin 5801ed3
Corrected template for Nemotron
pwilkin f01caef
Template and parser fixes
pwilkin eec8af0
Final template and grammar changes
pwilkin 3753564
Whitespace
pwilkin 8edb5c4
Always do lazy grammar processing since </think> tag will always be t…
pwilkin e0d1a9d
Allow extra content after toolcall
pwilkin 0f7bfaf
Whitespace
pwilkin 82d718a
Merge branch 'ggml-org:master' into nemotron-chat
pwilkin ed596a2
New tests: thinking + tools, tools + content, thinking + tools + cont…
pwilkin 79df613
Whitespace
pwilkin 6c04c61
Remove cURL test script
pwilkin File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,162 @@ | ||
| {%- set ns = namespace(enable_thinking=true) -%} | ||
| {%- for message in messages -%} | ||
| {%- set content = message['content'] -%} | ||
| {%- if message['role'] == 'user' or message['role'] == 'system' -%} | ||
| {%- if '/think' in content -%} | ||
| {%- set ns.enable_thinking = true -%} | ||
| {%- elif '/no_think' in content -%} | ||
| {%- set ns.enable_thinking = false -%} | ||
| {%- endif -%} | ||
| {%- endif -%} | ||
| {%- endfor -%} | ||
|
|
||
| {%- if messages[0]['role'] != 'system' -%} | ||
| {%- set ns.non_tool_system_content = '' -%} | ||
| {{- '<SPECIAL_10>System | ||
| ' -}} | ||
| {%- else -%} | ||
| {%- set ns.non_tool_system_content = (messages[0]['content'] | default('', true)).replace('/think', '').replace('/no_think', '').strip() -%} | ||
| {{- '<SPECIAL_10>System | ||
| ' + ns.non_tool_system_content }} | ||
| {%- endif -%} | ||
|
|
||
| {%- if tools -%} | ||
| {%- if ns.non_tool_system_content is defined and ns.non_tool_system_content != '' -%} | ||
| {{- ' | ||
|
|
||
| ' -}} | ||
| {%- endif -%} | ||
| {{- 'You can use the following tools to assist the user if required:' -}} | ||
| {{- ' | ||
| <AVAILABLE_TOOLS>[' -}} | ||
| {%- for tool in tools -%} | ||
| {{- (tool.function if tool.function is defined else tool) | tojson -}} | ||
| {{- ', ' if not loop.last else '' -}} | ||
| {%- endfor -%} | ||
| {{- ']</AVAILABLE_TOOLS> | ||
|
|
||
| ' -}} | ||
| {{- 'If you decide to call any tool(s), use the following format: | ||
| ' -}} | ||
| {{- '<TOOLCALL>[{{"name": "tool_name1", "arguments": "tool_args1"}}, ' -}} | ||
| {{- '{{"name": "tool_name2", "arguments": "tool_args2"}}]</TOOLCALL> | ||
|
|
||
| ' -}} | ||
| {{- 'The user will execute tool-calls and return responses from tool(s) in this format: | ||
| ' -}} | ||
| {{- '<TOOL_RESPONSE>[{{"tool_response1"}}, {{"tool_response2"}}]</TOOL_RESPONSE> | ||
|
|
||
| ' -}} | ||
| {{- 'Based on the tool responses, you can call additional tools if needed, correct tool calls if any errors are found, or just respond to the user.' -}} | ||
| {%- endif -%} | ||
| {{- ' | ||
|
|
||
| ' -}} | ||
| {%- set messages = messages[1:] if messages[0]['role'] == 'system' else messages -%} | ||
| {%- if messages[-1]['role'] == 'assistant' -%} | ||
| {%- set ns.last_turn_assistant_content = (messages[-1]['content'] | default('', true)).strip() -%} | ||
| {%- set ns.last_turn_assistant_tool_calls = messages[-1]['tool_calls'] if 'tool_calls' in messages[-1] else [] -%} | ||
| {%- set messages = messages[:-1] -%} | ||
| {%- endif -%} | ||
|
|
||
| {%- for message in messages %} | ||
| {%- set content = message['content'] %} | ||
| {%- if message['role'] == 'user' -%} | ||
| {{- '<SPECIAL_11>User | ||
| ' + (content | default('', true)).replace('/think', '').replace('/no_think', '').strip() + ' | ||
| ' }} | ||
| {%- elif message['role'] == 'tool' -%} | ||
| {%- if loop.first or (messages[loop.index0 - 1].role != 'tool') -%} | ||
| {{- '<SPECIAL_11>User | ||
| ' + '<TOOL_RESPONSE>[' }} | ||
| {%- endif -%} | ||
| {{- message['content'] -}} | ||
| {{- ', ' if not loop.last and (messages[loop.index0 + 1].role == 'tool') else '' -}} | ||
| {%- if loop.last or (messages[loop.index0 + 1].role != 'tool') -%} | ||
| {{- ']</TOOL_RESPONSE>' -}} | ||
| {%- endif -%} | ||
| {%- elif message['role'] == 'assistant' -%} | ||
| {%- if content and '</think>' in content -%} | ||
| {%- set content = (content.split('</think>')[1] | default('', true)).strip() %} | ||
| {%- endif -%} | ||
| {{- '<SPECIAL_11>Assistant | ||
| ' + ((content | default('', true)).strip() if content is not none else '') }} | ||
| {%- if message.tool_calls -%} | ||
| {%- if (content | default('', true)).strip() != '' -%} | ||
| {{- ' | ||
| ' -}} | ||
| {%- endif -%} | ||
| {{- '<TOOLCALL>[' -}} | ||
| {%- for call in message.tool_calls -%} | ||
| {%- set fn = call.function if call.function is defined else call -%} | ||
| {{- '{"name": "' + fn.name + '", "arguments": ' -}} | ||
| {%- if fn.arguments is string -%} | ||
| {{- fn.arguments -}} | ||
| {%- else -%} | ||
| {{- fn.arguments | tojson -}} | ||
| {%- endif -%} | ||
| {{- '}' + (', ' if not loop.last else '') -}} | ||
| {%- endfor -%} | ||
| {{- ']</TOOLCALL>' -}} | ||
| {%- endif -%} | ||
| {{- ' | ||
| <SPECIAL_12> | ||
| ' -}} | ||
| {%- endif -%} | ||
| {%- endfor -%} | ||
|
|
||
| {%- if add_generation_prompt -%} | ||
| {{- '<SPECIAL_11>Assistant | ||
| ' -}} | ||
| {%- if ns.enable_thinking is defined and ns.enable_thinking is false -%} | ||
| {{- '<think></think>' -}} | ||
| {%- else -%} | ||
| {{- '<think> | ||
| ' -}} | ||
| {%- endif -%} | ||
| {%- if ns.last_turn_assistant_content is defined and ns.last_turn_assistant_content != '' -%} | ||
| {{- ns.last_turn_assistant_content -}} | ||
| {%- endif -%} | ||
| {%- else -%} | ||
| {%- if ns.last_turn_assistant_content is defined and ns.last_turn_assistant_content != '' -%} | ||
| {{- '<SPECIAL_11>Assistant | ||
| ' -}} | ||
| {%- if ns.enable_thinking is defined and ns.enable_thinking is false -%} | ||
| {{- '<think></think>' -}} | ||
| {%- else -%} | ||
| {{- '<think> | ||
| ' -}} | ||
| {%- endif -%} | ||
| {{- ns.last_turn_assistant_content -}} | ||
| {%- if continue_final_message is defined -%} | ||
| {%- if continue_final_message is false -%} | ||
| {{- ' | ||
| <SPECIAL_12> | ||
| ' -}} | ||
| {%- endif -%} | ||
| {%- else -%} | ||
| {{- ' | ||
| <SPECIAL_12> | ||
| ' -}} | ||
| {%- endif -%} | ||
| {%- endif -%} | ||
| {%- if ns.last_turn_assistant_tool_calls is defined and ns.last_turn_assistant_tool_calls | length > 0 -%} | ||
| {{- '<SPECIAL_11>Assistant | ||
| ' -}} | ||
| {{- '<TOOLCALL>[' -}} | ||
| {%- for call in ns.last_turn_assistant_tool_calls -%} | ||
| {%- set fn = call.function if call.function is defined else call -%} | ||
| {{- '{"name": "' + fn.name + '", "arguments": ' -}} | ||
| {%- if fn.arguments is string -%} | ||
| {{- fn.arguments -}} | ||
| {%- else -%} | ||
| {{- fn.arguments | tojson -}} | ||
| {%- endif -%} | ||
| {{- '}' + (', ' if not loop.last else '') -}} | ||
| {%- endfor -%} | ||
| {{- ']</TOOLCALL>' -}} | ||
| {{- '<SPECIAL_12> | ||
|
|
||
| ' -}} | ||
| {%- endif -%} | ||
| {%- endif -%} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.