bug: `cortex run llama3.1` is broken

[freelerobot]

Cortex version

v70

Problem 1

1. cortex pull llama3.1-url , download is successful
2. cortex run llama3.1
3. Get interactive shell
4. Send "hi"
5. Get error

❯ cortex-nightly run llama3.1

Inorder to exit, type `exit()`

> hi

libc++abi: terminating due to uncaught exception of type nlohmann::json_abi_v3_11_3::detail::parse_error: [json.exception.parse_error.101] parse error at line 3, column 1: syntax error while parsing value - invalid literal; last read: '"chat.completion.chunk"}<U+000A><U+000A>d'; expected end of input

5![1]    93403 abort      cortex-nightly run llama3.1

This happened a few times

Problem 2

1. Wait a few minutes
2. Use the same command, get a different error this time. But model loads!

❯ sudo cortex-nightly run llama3.1

Starting server ...
20240919 15:41:56.580209 UTC 3788319 INFO  Host: 127.0.0.1 Port: 3928
 - main.cc:32
Error opening file: Permission denied
Server started
Error opening file: Permission denied
Model loaded!
Inorder to exit, type `exit()`

> hi

6!0!47C<%!97++2H5B/F#8(#BA:E+FGD%'E"<9$2-8$==EF>3#B4A5.B!A0>:19)8?<9F'9D8D/<E?
A:)242FB5!$>#42,;#!37EBCH";>E%2.3'0#;"#2#<6AC!0D?19:!:/;8+)9$:7=/7C%8?+7+)/G)E=G&,G'0"!.D?
+6BC&#*!8&$34"4&?99G=!!<;-=&18,<&'1(CG=?7--H4H49;+8<&,&B(??=$/96>(.F8*3.
<!33#3G.CB0"::$HE5,5"8$A(=F;H$A+C#9,2;HH?!:;3,3:0013H))3)//5&7+(36,>AH.!%G2%4+#%:1CD(?B742;<&(0F;2?
>9(F5,H3(4A?D-B#"H18'3F1<:EG.30548.#1A=).D,A*>!:9734FG22A(/(>)57%<E!$7":94#95=(#/0,(%$50?
E*"95"74#/B"076C.E-0<2!'$>%<3.?94A2)!!#372=>H28C2%:23(,(D/0;4H5,B?3B;HE9*

Problem 3

1. Exit interactive shell
2. run cortex stop just to be sure
3. run cortex run llama3.1 again
4. computer crashes, needs a hard restart

What is your OS?

• MacOS
• Windows
• Linux

What engine are you running?

• cortex.llamacpp (default)
• cortex.tensorrt-llm (Nvidia GPUs)
• cortex.onnx (NPUs, DirectML)

[freelerobot]

llama3.1.yaml

name: Meta-Llama-3.1-8B-Instruct
model: Meta-Llama-3.1-8B-Instruct
version: 2
engine: cortex.llamacpp
prompt_template: "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{system_message}<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
id: llama3.1
top_p: 0.949999988
temperature: 0.699999988
frequency_penalty: 0
presence_penalty: 0
max_tokens: 131072
ngl: 33
ctx_len: 131072
tp: 0
stream: true
text_model: false
stop:
  - <|eot_id|>
files:
  - /Users/nicolezhu/cortexcpp-nightly/models/llama3.1/llama3.1
created: 1726745800