Server example sampling is partially broken

[kalomaze]

At some point after Top K, the candidates size gets forcibly revoked to 1 instead of the specified value when running server.exe.
This does not happen for main.exe.

[mrsteyk]

Can't replicate this myself on e0a4002273907b2c414b6b5442d99e08bfe2df35, call to llama_sample_token almost always has size greater than min_keep. Called from mikupad with those settings:

n_probs and by extension min_keep is set to 10. This "drastic resize" for me happens after top_p. Setting top_p to 1 always gives the same size as top_k. Sampler sequence is "kfypmt".

[jart]

@kalomaze could you clarify if you're using the OpenAI endpoint? Because I encountered a similar sounding issue with that, which is being fixed by a PR I submitted.

[kalomaze]

@kalomaze could you clarify if you're using the OpenAI endpoint? Because I encountered a similar sounding issue with that, which is being fixed by a PR I submitted.

I don't remember using the OpenAI endpoint, but it's probably that. The shrinking candidates issue you mention sounds identical to what I documented