Bug: Regular crashes due to invalid UTF-8 in output.

[kaetemi]

What happened?

Llama.cpp crashes several times a day due to invalid UTF-8 in its output in /completion endpoint.

Ideally this should just return an error code.

Name and Version

latest

What operating system are you seeing the problem on?

No response

Relevant log output

llama-server [falcon-7b:2048:10]: terminate called after throwing an instance of 'nlohmann::json_abi_v3_11_3::detail::type_error'
llama-server [falcon-7b:2048:10]:   what():  [json.exception.type_error.316] invalid UTF-8 byte at index 612: 0x5C
Error in request to /completion: socket hang up

[kaetemi]

@amir1387aht Uh, not sure if RARs on MediaFire are the right way to share bugfixes.

[kaetemi]

Hmm. Seems to be elsewhere...

[kaetemi]

So far, so good...

[ngxson]

Could you include an example of the request sent to /completion?

[kaetemi]

Requests are done in token array format. The input is sanitized. Request has Japanese unicode, and output is likely unicode but possibly not coherent (Falcon 7B is not coherent at unicode output) and might've been cut short due to generation length limit.

[ngxson]

Can you just give an example of which Japanese character / sentence which causes the crash? If we have a concrete example, it would be much simple to pin-point the error.

[kaetemi]

I don't have any log of the specific output that crashes it, no user content is logged on my inference servers.

[kaetemi]

Hmm. Just got another crash with the same error.

what(): [json.exception.type_error.316] invalid UTF-8 byte at index 264: 0x5C

[kaetemi]

Oh. The calls in completion already have replace as error handler. So it's not those then, huh.