update to llama.cpp b5688 #115
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- extract updated glue code to server.hpp and utils.cpp - adapt native code in jllama.cpp to track API changes - update tags and adapt CMakeLists.txt
|
Awesome work 👍 though it'll take some time for me to review.
Yeah, if you can get the llama.cpp team to do this, it would make things much easier!
I'm a big fan of protobuf/rpc, so I'll look forward to that! |
comment says: // if the assistant message appears at the end of list, we do not add end-of-turn token so it seems the changed output should be is expected.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
note: All the code in utils.hpp and server.hpp is just lifted directly from llama.cpp, so I don't really have a good understanding of all the changes that happened there. But I've looked closely at the changes that needed to be done in jllama.cpp to track the API changes, and done a bit of testing of the resulting binary.
longer-term it would be better to get the server code in llama.cpp split so it these pieces can be used as a library directly; we are considering writing a different server wrapper with protobuf/rpc replacing JSON/http, which could use the same library.