No variance in response from /chat/completions

[mneedham]

Hi,

I'm playing around with the temperature property when calling a model from the /chat/completions API, but I can't figure out how how to get some variance in the responses. I have the temperature set to 0.8.

Here's how I start the server:

./llamafile-server-0.4 -m models/starling-lm-7b-alpha.Q4_K_M.gguf --nobrowser

And this is the way I call the API:

 curl -s http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "gpt-3.5-turbo", "temperature": 0.8,
  "messages": [
    {
      "role": "system",
      "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."
    },
    {
      "role": "user",
      "content": "Compose a poem that explains the concept of recursion in programming. A maximum of 5 lines"
    }
  ]
}' 2>/dev/null | jq -r '.choices[].message.content'

And below is a video showing the same output 3 times.

2023-12-17_10-55-15.mp4

[Humayun-glitch]

did you try to adjust the temperature value to higher values, such as 1.2 or 1.5? See if it gives you the desired level of variability in the responses.

[mneedham]

Yeh, I've tried all sorts of temperatures, but it always returns the same output:

$  curl -s http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "gpt-3.5-turbo", "temperature": 1.5,
  "messages": [
    {
      "role": "system",
      "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."
    },
    {
      "role": "user",
      "content": "Compose a poem that explains the concept of recursion in programming. A maximum of 5 lines"
    }
  ]
}' 2>/dev/null | jq -r '.choices[].message.content'
In the realm of code, a tale to unfold,
Of recursion, a concept to be told.

A function that calls itself anew,
A journey through the depths of the stack, a view.

A problem solved, a base case found,
The recursion ends, and the journey's bound.

A beautiful dance, a pattern so clear,
Recursion in code, a programmer's cheer.

[Humayun-glitch]

Oh, I see. Let me see what can be done. My first time on this repo. I'll be more than happy to help contribute.

[Muzika]

Add "cache_prompt": true to the parameters.

By default, it is false so everytime you send a curl request it assumes that it is the first time you talked to each other.

Setting it to true would make it remember that it has already sent the poem so it's gonna try generating a different one next time.

try this

curl -s http://localhost:4251/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "llava", "temperature": 0.8, "cache_prompt": true, "messages": [ { "role": "system", "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair." }, { "role": "user", "content": "Compose a poem that explains the concept of recursion in programming. A maximum of 5 lines" } ] }' 2>/dev/null | jq -r '.choices[].message.content'