Skip to content

Misc. bug: Granite chat parser doesn't stream content section #15681

@shun095

Description

@shun095

Name and Version

$ ./build/bin/llama-cli --version
load_backend: loaded BLAS backend from /home/shun/Documents/dev/llama.cpp/build/bin/libggml-blas.so
register_backend: registered backend BLAS (1 devices)
register_device: registered device BLAS (OpenBLAS)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3070, compute capability 8.6, VMM: yes
load_backend: loaded CUDA backend from /home/shun/Documents/dev/llama.cpp/build/bin/libggml-cuda.so
register_backend: registered backend CUDA (1 devices)
register_device: registered device CUDA0 (NVIDIA GeForce RTX 3070)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32
load_backend: loaded ROCm backend from /home/shun/Documents/dev/llama.cpp/build/bin/libggml-hip.so
register_backend: registered backend ROCm (1 devices)
register_device: registered device ROCm0 (Radeon RX 7900 XTX)
ggml_backend_load_best: /home/shun/Documents/dev/llama.cpp/build/bin/libggml-cpu-sandybridge.so score: 21
ggml_backend_load_best: /home/shun/Documents/dev/llama.cpp/build/bin/libggml-cpu-skylakex.so score: 0
ggml_backend_load_best: /home/shun/Documents/dev/llama.cpp/build/bin/libggml-cpu-alderlake.so score: 0
ggml_backend_load_best: /home/shun/Documents/dev/llama.cpp/build/bin/libggml-cpu-sse42.so score: 5
ggml_backend_load_best: /home/shun/Documents/dev/llama.cpp/build/bin/libggml-cpu-x64.so score: 1
ggml_backend_load_best: /home/shun/Documents/dev/llama.cpp/build/bin/libggml-cpu-sapphirerapids.so score: 0
ggml_backend_load_best: /home/shun/Documents/dev/llama.cpp/build/bin/libggml-cpu-icelake.so score: 0
ggml_backend_load_best: /home/shun/Documents/dev/llama.cpp/build/bin/libggml-cpu-haswell.so score: 64
load_backend: loaded CPU backend from /home/shun/Documents/dev/llama.cpp/build/bin/libggml-cpu-haswell.so
register_backend: registered backend CPU (1 devices)
register_device: registered device CPU (AMD Ryzen 7 3700X 8-Core Processor)
version: 508 (ef47691)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0 for x86_64-linux-gnu

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

./build/bin/llama-server -hf ibm-granite/granite-3.3-8b-instruct-GGUF:Q8_0 -ngl 100 -fa -dev "CUDA0,ROCm0" -b 64 --jinja --reasoning-format deepseek -c 131072

Problem description & steps to reproduce

The problem is that reasoning_content streams token by token, while content is returned in one piece of data.

Command:

$ curl -X POST http://localhost:8080/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "llama","messages": [{"role": "user","content": "Hello"}],"stream": true }'

Response:

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"role":"assistant","content":null}}],"created":1756538758,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"<think"}}],"created":1756538758,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":"The"}}],"created":1756538758,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{}}],"created":1756538758,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" user"}}],"created":1756538758,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" has"}}],"created":1756538758,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" initiated"}}],"created":1756538758,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" a"}}],"created":1756538758,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" conversation"}}],"created":1756538758,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" with"}}],"created":1756538758,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" a"}}],"created":1756538758,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" simple"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" g"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":"reeting"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" \""}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":"Hello"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":"\"."}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" There"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" isn"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":"'t"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" a"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" specific"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" question"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" or"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" topic"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" provided"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":","}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" so"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" the"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" response"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" should"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" be"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" friendly"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" and"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" appropriate"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" for"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" an"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" initial"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" interaction"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":"."}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" It"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":"'s"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" common"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" in"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" such"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" cases"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" to"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" re"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":"ci"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":"proc"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":"ate"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" the"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" g"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":"reeting"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" and"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" perhaps"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" offer"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" assistance"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" or"}}],"created":1756538759,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" a"}}],"created":1756538760,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" general"}}],"created":1756538760,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" wel"}}],"created":1756538760,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":"coming"}}],"created":1756538760,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":" statement"}}],"created":1756538760,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":"."}}],"created":1756538760,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"Hello! \n\nHow can I assist you today? Whether you have a question, need information on a topic, or just want to chat, feel free to let me know. I'm here to help. 😊"}}],"created":1756538761,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[{"finish_reason":"stop","index":0,"delta":{}}],"created":1756538761,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk"}

data: {"choices":[],"created":1756538761,"id":"chatcmpl-u9PZEHve0NEfqnAnHe3keuU3VJ0OQuGk","model":"llama","system_fingerprint":"b508-ef476916b","object":"chat.completion.chunk","usage":{"completion_tokens":117,"prompt_tokens":189,"total_tokens":306},"timings":{"prompt_n":189,"prompt_ms":406.934,"prompt_per_token_ms":2.1530899470899474,"prompt_per_second":464.4487803919063,"predicted_n":117,"predicted_ms":2513.169,"predicted_per_token_ms":21.480076923076922,"predicted_per_second":46.5547681035378}}

data: [DONE]

First Bad Commit

#14864

Relevant log output

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions