server/public_simplechat alternate web client ui with 0 setup builtin tool calling++, reasoning - refactored, SysDateTime, rename pdftext #17038
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hanishkvc
force-pushed
the
hkvc_server_simplechat_toolcalling_v0450
branch
from
b2db6c1 to
fbb3a3d
Compare
4 tasks
Checks for toolname to be defined or not in the GenAi's response If toolname is set, then check if a corresponding tool/func exists, and if so call the same by passing it the GenAi provided toolargs as a object. Inturn the text generated by the tool/func is captured and put into the user input entry text box, with tool_response tag around it.
As output generated by any tool/function call is currently placed into the TextArea provided for End user (for their queries), bcas the GenAi (engine/LLM) may be expecting the tool response to be sent as a user role data with tool_response tag surrounding the results from the tool call. So also now at the end of submit btn click handling, the end user input text area is not cleared, if there was a tool call handled, for above reasons. Also given that running a simple arithmatic expression in itself doesnt generate any output, so wrap them in a console.log, to help capture the result using the console.log trapping flow that is already setup.
and inform the GenAi/LLM about the same
Should hopeful ensure that the GenAi/LLM will generate appropriate code/expression as the argument to pass to these tool calls, to some extent.
ie in vs code with ts-check
Move tool calling logic into tools module. Try trap async promise failures by awaiting results of tool calling and putting full thing in an outer try catch. Have forgotten the nitty gritties of JS flow, this might help, need to check.
So that when tool handler writes the result to the tc_switch, it can make use of the same, to write to the right location. NOTE: This also fixes the issue with I forgetting to rename the key in js_run wrt writing of result.
to better describe how it will be run, so that genai/llm while creating the code to run, will hopefully take care of any naunces required.
Also as part of same, wrap the request details in the assistant block using a similar tagging format as the tool_response in user block.
Instead of automatically calling the requested tool with supplied arguments, rather allow user to verify things before triggering the tool. NOTE: User already provided control over tool_response before submitting it to the ai assistant.
Instead of automatically calling any requested tool by the GenAi / llm, that is from the tail end of the handle user submit btn click, Now if the GenAi/LLM has requested any tool to be called, then enable the Tool Run related UI elements and fill them with the tool name and tool args. In turn the user can verify if they are ok with the tool being called and the arguments being passed to it. Rather they can even fix any errors in the tool usage like the arithmatic expr to calculate that is being passed to simple_calculator or the javascript code being passed to run_javascript_function_code If user is ok with the tool call being requested, then trigger the same. The results if any will be automatically placed into the user query text area. User can cross verify if they are ok with the result and or modify it suitabley if required and inturn submit the same to the GenAi/LLM.
Also avoid showing Tool calling UI elements, when not needed to be shown.
So that it can be used from different modules, if required.
Try ensure as well as verify that original console.log is saved and not overwritten. Throw an exception if things seem off wrt same. Also ensure to add a newline at end of console.log messages
The request for code to run as well as the resultant response data both need to follow a structured object convention, so that it is easy to map a request and the corresponding response to some extent.
These no longer need to worry about * setting up the console.log related redirection to capture the generated outputs, nor about * setting up a dynamic function for executing the needed tool call related code The web worker setup to help run tool calls in a relatively isolated environment independent of the main browser env, takes care of these. One needs to only worry about getting the handle to the web worker to use and inturn pass the need code wrt the tool call to it.
tools manager/module * setup the web worker that will help execute the tool call related codes in a js environment that is isolated from the browsers main js environment * pass the web worker to the tool call providers, for them to use * dont wait for the result from the tool call, as it will be got later asynchronously through a message * allow users of the tools manager to register a call back, which will be called when ever a message is got from the web worker containing response wrt previously requested tool call execution. simplechat * decouple toolcall response handling and toolcall requesting logic * setup a timeout to take back control if tool call takes up too much time. Inturn help alert the ai model, that the tool call took up too much time and so was aborted, by placing a approriate tagged tool response into user query area. * register a call back that will be called when response is got asynchronously wrt anye requested tool calls. In turn take care of updating the user query area with response got wrt the tool call, along with tool response tag around it.
Had forgotten to specify type as module wrt web worker, in order to allow it to import the toolsconsole module. Had forgotten to maintain the id of the timeout handler, which is needed to clear/stop the timeout handler from triggering, if tool call response is got well in time. As I am currently reverting the console redirection at end of handling a tool call code in the web worker message handler, I need to setup the redirection each time. Also I had forgotten to clear the console.log capture data space, before a new tool call code is executed, this is also fixed by this change. TODO: Need to abort the tool call code execution in the web worker if possible in future, if the client / browser side times out waiting for tool call response, ie if the tool call code is taking up too much time.
As the tool calling, if enabled, will need access to last few user query and ai assistant responses (which will also include in them the tool call requests and the corresponding results), so that the model can build answers based on its tool call reqs and got responses, and also given that most of the models these days have sufficiently large context windows, so the sliding window context implemented by SimpleChat logic has been increased by default to include last 4 query and their responses roughlty.
Add a pending object which maintains the pending toolcallid wrt each chat session, when ever a tool call is made. In turn when ever a tool call response is got cross check if its toolcallid matches that in the pending list. If so accept the tool call response and remove from pending list. If not just ignore the response. NOTE: The current implementation supports only 1 pending tool call at any time. NOTE: Had to change from a anonymous to arrow function so as to be able to get access to the ToolsManager instance (this) from within the function. ie make use of lexical binding semantic of arrow functions.
ie if exception raised during tool call execution and or time out occurs
Add forgotten to add , after simplechat entry. Currently I am not strictly using the importmap feature, so the error didnt create any problem, but the error was there which has been fixed.
Take the existing urltext logic including its html parser and strip it out to be simpler.
Add the meta data for the fetch xml as text tool call Implement the handler and the setup tool call plumbing logic
At simpleproxy end * Add the tag names hierarchy before contents of a tag * Remember to convert the tagDrops to small case as HTMLParser base class seems to do that by default. At the client ui end * if undefined remember to pass a empty list wrt tagDrops. * cleanup the func description and also mention possible tagDrops for RSS feeds in the tool meta
instead of the prefixing of tag heirarchy retain the xml structure while parallely allowing unwanted tags and their contents to be dropped.
Rename xmltext to xmlfiltered. This simplifies the filtering related logic as well as gives more fine grained flexibility wrt filtering bcas of re.
To make it easier for the ai model to understand that this works mainly for html pages and not say xml or pdf or so. For those one needs to use other explict tool calls provided like fetchpdftext or fetchxmltext or so The server service path renamed from urltext to htmltext. SearchWebText also updated to use htmltext now
Dont even insert skipped tags as tag blocks with empty content. This should make the resultant xml cleaner and make it use less space.
Rather a chat with gpt-oss generated a assistant response which included chat-content, chat-reasoning and chat-toolcall all in the same response. On responding to same with tool call result, the server http handshake responded with a 500 Internal server error, So added this to get more details in this case, as well as in general for future.
Contributor
Author
|
Have added logic to ignore delayed tool call results, if user and or auto trigger logic has already gone ahead without waiting for delayed tool call result. Have added xmlfiltered tool call, to allow one to fetch xml while also filtering out unwanted fields/tags from them, to help keep things from overloading the limited context window and so. Also some other general cleanup |
This simple scheme doesnt work. Rather the pdf outline seems to follow below logic If a child list is found when processing the current list, dont increment the numbering.
This increaments before itself, but we need to increment after
Pass a list to keep track of the numbering at different depths as well as to delay incrementing the numbering to the last min Dont let recursion go beyond a predefined limit
Switch from empty strings or empty list and so to undefined. undefined will be treated by Javascript and JSON to mean, not even instantiated and also dont instantiate the same
NSChatMessage implements the undefined based flow and provides helpers to check if any of the field like content or reasoning or tool_calls is available in the chat message or not. It also provides helpers to get the corresponding fields. ChatMessageEx updated to make use of NSChatMessage.
Remove the load from disk support that was previously retained wrt the old on-disk-storage format for the chat session messages. Make a note to allow non ai server handshake roles to be maintained wrt NSChatMessage. Add helper to cross check if a message belongs to such temp roles or not. Update SimpleChat class to make use of the new NSChatMessage based needed flow.
TODO: individual tool/function calls from tool_calls field, accessed using different methods in different places for now. Need to think on which is the best method to retain and use everywhere and or retain things as is.
Add a new static helper to create a ChatMessageEx for a given tool response data. Use the same when storing tool response in the chat session msgs list. This inturn avoids the need for creating a xml string with all the fields corresponding to tool response. So also no need to extract the individiual tool response fields from the all-in-one xml string and populate the tool response fields in the network structure equivalent ns data structure, when recent_chat_ns is called.
Given that the user query box no longer includes the special xml string wrt tool response data (TOOL.TEMP ROLE), so now instead set a special attribute to indicate that user query/input box is maintaining a tool response. For regular Tool responses in the chat session, now show the tool call id and tool name before the tool response data (ie content field).
Had forgotten to update these two functions wrt the tool response related new fields. This is fixed now. Also show tool-call-id and tool-name to end user as part of chat message showing. ALERT on disk structure change old saves wont work esp wrt tool responses
Allow for empty tool call results Block no content response from user role only. Also change for console.debug to console.log so people can see the blocking of empty response from user, in the browser console.
Use css conditional attribute styling to change background color of the user input textarea to match the tool role message block color, when the user input textarea is in the TOOL.TEMP mode With this user can know that the user input area is currently showing and accepting tool result data for submission.
Currently the logic doesnt allow user to send a empty message to ai, during their term. Previously this path wasnt directly alerting the end user. Now it informs the end user using placeholder property so they can see the alert, while also ensuring that once user enters something, the alert wont interfere. The logic takes care of saving any original placeholder, so that the same is restored, when user switches sessions.
Contributor
Author
|
Added support to extract outline info from pdf into text of pdftext Switch to a proper class wrt NSChatMessage, instead of the earlier typedef based flow. This helps consolidate related things into this class. Also make use of the implicit dropping of fields by JSON.stringify, wrt network handshake. Also helps prepare for the vision models support next. With above bring in fields related to tool response into NSChatMessage class and use the same for maintaining tool response rather than the earlier xml tagged and serialised response being maintained in the content field and then ... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The alternate client side web ui at tools/server/public_simplechat with 0 setup builtin tool calling ++
This PR refactors the code such that the core classes including wrt tooling (collated tools manager into a class) sit in their own module files and inturn get imported for usage in the main runtime entrypoint file as well as other places where refered. This allows developers and static tools to be aware of the data structure and flow fully.
Add system date time timestamp tool call, so that the ai model doesnt hallucinate timestamps from thin air.
Renamed pdf_to_text tool call to fetch_pdf_as_text, so that ai model understands the intent/semnatic of that tool call better.
The previous PR in this series is at #16929
Except for the python type hinting and match related python check failure (isnt it time to enable these better code structurings to be used, without rising a check failure?), other auto check errors dont relate to public_simplechat in any way and can be safely ignored.
By pointing llama-server at this alternate webui, one can use the core builtin toolcalling features without any addiitional setup. And by running the included simpleproxy.py, one can enhance tool calling to include fetching of web pages and pdfs, as well as conversion to plain text for use by the ai model.
Read public_simplechat's readme for more details.
NOTE: This is a simple minded web client ui for exploring the llama-server rest api for developers as well as allowing basic utilitarian usage for end users.