Skip to content
This repository was archived by the owner on Jul 4, 2025. It is now read-only.

Conversation

@nguyenhoangthuan99
Copy link
Contributor

@nguyenhoangthuan99 nguyenhoangthuan99 commented Sep 20, 2024

Fix #1273

Problem 1

  1. cortex pull llama3.1-url , download is successful
  2. cortex run llama3.1
  3. Get interactive shell
  4. Send "hi"
  5. Get error
❯ cortex-nightly run llama3.1

Inorder to exit, type `exit()`

> hi

libc++abi: terminating due to uncaught exception of type nlohmann::json_abi_v3_11_3::detail::parse_error: [json.exception.parse_error.101] parse error at line 3, column 1: syntax error while parsing value - invalid literal; last read: '"chat.completion.chunk"}<U+000A><U+000A>d'; expected end of input

5![1]    93403 abort      cortex-nightly run llama3.1

This happened a few times

This problem due to nlohmann::json fail to parse content result from engine, this bug happened randomly. For that reason, we will add an try/catch logic to avoid this bug to crash the program and investigate easier.
This bug also appeared in Linux rarely and very hard to reproduce.

@nguyenhoangthuan99 nguyenhoangthuan99 marked this pull request as ready for review September 20, 2024 08:24
Copy link
Contributor

@dan-menlo dan-menlo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm.

However, if it fails to parse, what is the recovery? (we just don't show anything?)

@nguyenhoangthuan99
Copy link
Contributor Author

As the error log from ticket, this is the end of content that nlohmann::json tried to parse: '"chat.completion.chunk"}<U+000A><U+000A>d', the nlohmann::json received unexpected input so that it failed. Until now, this issue is challenge to recorver because we haven't know the exactly root cause yet. So we decided to add try catch, and add a warning log for esier to investigate later.

@nguyenhoangthuan99 nguyenhoangthuan99 merged commit 97176fd into dev Sep 23, 2024
@nguyenhoangthuan99 nguyenhoangthuan99 deleted the fix/json-parse-chat-stream-fail branch September 23, 2024 01:32
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: cortex run llama3.1 is broken

5 participants