Bug: Gemma2 Context switching forgets original input

[Gomez12]

What happened?

If I have a prompt like the following "<start_of_turn>user\nProductGroup: Anvil<end_of_turn><start_of_turn>user\nCan you give me the 25 most important characteristics for the previous named Productgroup? Respond in the following Json Format : [{'Characteristic':string,'Explanation':string,'ExampleValues':[string]}<end_of_turn><start_of_turn>model"
Then it starts out well, but if you set the context low (IE 512) then it starts breaking up output after context switching.
It seems like it has forgotten the Initial ProductGroup, and it just continuous outputting based on the examples from the JSON which don't mention the specific product group, so it just outputs for random products.

I don't know if this is just how it is supposed to work, or that it would be possible to add the original prompt in front when context switching.

I basically noticed this because of the server with parallel which splits the context size so you run into this much quicker, the quick-fix version is to simply up the context window

Name and Version

./llama-cli --version
version: 3281 (023b880)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu

What operating system are you seeing the problem on?

Linux

Relevant log output

No response

[matteoserva]

Yes. The "forgetting" behaviour you described is the default.
You can change it with the --keep option to keep the original prompt instead of discarding it.

If I remember correctly, in the server you have to multiply the desired context by the batch size to get the number you put in the command line.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.