Segfaults now with latest llama.cpp commits

[Ph0rk0z]

I pulled new commits for vendor/llama.cpp and I get terminate after throwing an instance of std::length error

The model loads fine and everything seems normal, just dies when generating. This is using the built in server.
Same error is in textgen.

[manishshettym]

Can reproduce! Here's a gdb back trace that shows the issue:

# 5 0x00007fffb935a209 in std::__throw_length_error(char const*) ()
at /opt/conda/conda-bld/gcc-compiler_1654084175708/work/gcc/libstdc++-v3/src/c++98/bitmap_allocator.cc:92
No symbol table info available.
...
# 9 0x00007fffb966b446 in llama_tokenize (ctx=0xfa66e0,
text=0x7ffff6ce9c80 "Hello, world!", **text_len=-153556112**, tokens=0xd,
n_max_tokens=0, add_bos=111)
at ....

It is due to a recent update to the tokenize APIs in llama.cpp.

Opened this PR that fixes it: #730 [cc: @abetlen]

[Ph0rk0z]

Makes sense, that's a big component.

[abetlen]

@Ph0rk0z were you using the llama.cpp version pinned in the vendor/llama.cpp directory?

[Ph0rk0z]

nope, I like to do the latest and greatest