[Feature Request] Dynamic temperature sampling for better coherence / creativity

[kalomaze]

Prerequisites

• [✅] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Feature Idea

Typical sampling methods for large language models, such as Top P and Top K, (as well as alternative sampler modes that decide the Top K dynamically like Mirostat) are based off the assumption that a static temperature value (a consistently randomized probability distribution) is the ideal sampler conditioning. Mirostat, most notably, was designed to 'learn' a certain targeted level of 'entropy' over time; this helped the model find the most grammatically coherent selection of tokens to be considered by the sampler for good results. Most of these sampling implementations weren't designed to be used together. Some, like TFS, were created when the largest available models were smaller ones like GPT2. Those models struggled a lot more when attempting to generalize in different directions, and it makes sense to me that they'd need unique sampler tricks to keep them grammatically coherent.

I've tested and played around with these settings for Llama models, and while Mirostat seemed like a step in the right direction, especially for preventing repetition, I realized that nobody had made a sampler mode that would control temperature directly per token. My implementation of this would be calculated based on a simple metric; take the standard deviation of all tokens being considered by your top P / top K before applying the temperature randomization, and based on the 'confidence' of the model (as represented by the variation in choice), you can apply a temperature adjustment proportional to the variation of probability seen in the sampled set of tokens being chosen from.

The main idea is to encourage randomizing 'uncertain' probabilities (e.g, open ended writing, abstract concepts that can be represented with many words, and aren't deterministic by nature) while keeping the temperature low for more deterministic tokens without having to find the ideal selection of candidates for sampling per token (which I believe is how Mirostat was designed to work).

List of possible advantages could be:

• Having a definable range between the 'Base Temperature' and 'Maximum Temperature' could generally improve the creative problem solving ability of the model.
• Certain tokens are highly important to the context and are more important than others. For example, if the probability was randomized too far for at least one token that represents something deterministic like a certain character in a programming syntax, this leads to a higher failure rate for the rest of the generation.
• Could help prevent the model's generations from trending towards repetition due to a much broader range of probabilities that could be considered without impacting the model's intelligence as broadly (e.g a max temperature of 1.5 might not impact the model as strongly compared to if every token was sampled with that value). If this is the case, biasing against repeated tokens artificially through the Repetition Penalty would become less necessary.

List of possible disadvantages could be:

• A lot of faith is being put in the idea that strong variations of possibilities have a correlation with a high amount of acceptable / reasonable tokens. If the correlation is mild, the default range values would have to be adjusted to accomodate this, but that could be mitigated by testing different values for the base/max temp range, or through benchmarking them individually.
• The rate at which a model becomes more certain might not be linear; there might be a very short gap between 'low deviation' and 'high deviation' on unsampled probabilities.
• Reproducability might be more difficult, but I'm unsure of this. I'm guessing you could just use the same seed for every temperature value variation.

[kalomaze]

cc @KerfuffleV2
I think this is a more realistic sampler modification to implement compared to my last issue, do you have any opinions on this?

[Zhuyuqii]

https://arxiv.org/abs/2309.02772

[KerfuffleV2]

I think this is a more realistic sampler modification to implement compared to my last issue, do you have any opinions on this?

Definitely way more practical and the use case is also clearer to me than what you were talking about before.

I also had the idea to do something similar with word boundaries. I.E. if you're generating something like "that is wh" then the temperature for tokes like en, ere, at shouldn't necessarily be the same as dog since [wh]en], [wh]ere, etc. Also if you have that is a token like en or ere shouldn't necessarily have the same temperature as something like n't, what, etc. So it matters if you're in the middle of a word, and whether the token under consideration is something that would complete a word or start a new one.

[kalomaze]

I've been working on drafting this. Here's an interesting example of the Declaration of Independence and the measured top token probability for the next sentence when I gave it half of the first paragraph on Mistral 7b:

As you can see it is not deterministic enough with a low-ish temp sampler config to prevent hallucinations in quotations from being reasonably possible; some tokens are more like 95%, 90%, rather than 99.9% as someone I talked to theorized would be the case (and that it would only be that instead of 100% because it has to avoid dividing by zero).

Curiously, here's a natural language prompt:

Quite bizarrely the variance is incredibly high for natural language. Some are quite undecided and go as low as 13% when it comes to their top token probability; some are very obvious (89%) in comparison.

Standard deviation was proposed at first here to help measure 'confidence' and scale temperature accordingly, but that was more of a hunch and not necessarily the 'best' idea on how to implement the dynamic temp.

There are a multitude of ways we could measure and score how confident a model is at predicting:

• Standard deviation as already mentioned
• A 'greedy' and simplistic method that only uses the top tokens percentage and scales it. I was thinking an exponential curve where 100% probability and 90% probability would both be close to zero, but the lower you go the more aggressively it scales the temperature value up.
• First-order derivatives (differences between adjacent probabilities) are calculated to determine the rate of change of the probabilities, and this is used as a general metric for 'confidence'. For instance, [80%, 20%] would be considered less confident compared to [80%, 10%, 10%]

I will continue to update this if I make significant progress.

[KerfuffleV2]

Are those values after softmax? If not, comparing the absolute values between different runs might not really be meaningful. It's the logit value relative to other logits that determines which token gets picked, not really the absolute value.

You didn't show the code or process you used to generate that output, so it's hard to comment.

[kalomaze]

It was called right before temp sampling and with the other samplers (top p, etc) disabled but that might not have been completely accurate still, you're right on that. Though I didn't change the sampling settings between those two responses...

[kalomaze]

I have a test implementation of this feature hard-coded right now in this GUI fork of llama.cpp (koboldcpp):
LostRuins#464

I am calling it 'greedy dynamic temperature'.
This is because I'm only taking the top token's probability % and scaling the temperature value based on that with an exponential curve. It's applying that scaling curve to decide the temperature value, so that high probability values like 90% and 100% are both close to the minimum temperature value, but 40% and 50% is a more pronounced difference (the lower you go in confidence, the closer you are to the max temperature value)
I've labeled this approach as 'greedy' as it solely relies on the top token's probability for the adjustment. But that could be all we need...

It seems to be doing decently well so far with the provided test values (min temp 0.1, max temp 1.5) when I tried its ability to continue long-form text. By that, I'm referring to completing partial passages of text that LLMs have 'memorized perfectly' (things like the Declaration of Independence as I mentioned), on a non-instruct model (just for testing). It is also doing creative / open ended text generation properly and I'm not seeing much repetition there.

Will do more tests and a more 'proper' implementation of it so that this is its own option and not hardcoded. Then, if it has a good reception, I will consider a PR on the main repository here. If not, I will rethink the approach of using only the top token.

[KerfuffleV2]

I'm a bit confused by that code. The candidates aren't sorted until you call either llama_sample_softmax or llama_sample_top_k. Also there are other samplers that can change the order so they'll only be sorted if one of those two functions got called and no other sampler that changes the order was called afterwards. There's also a candidates->sorted flag which you can use to check if they're sorted. You can't necessarily assume they'll just always be sorted. For example, when using mirostat samplers everything else except temperature gets skipped, so in that case the temperature sampler will get called first, then the mirostat sampler - so the logits won't be sorted at that point.

What I'd do is just call the softmax sampler since you want the softmax value anyway. Then you'll know the logits are sorted and candidates->data[i].p will have the softmax value.

Also, from the pull:

float prob_max_token_before_temp = expf(max_l - max_l) / sum_exp;

In other words:

float prob_max_token_before_temp = expf(0) / sum_exp;

Right? max_l - max_l has to be 0 (except if it was NaN but that shouldn't be a case you run into).

[kalomaze]

I was not intending for this to have full compatibility with the other samplers until I could confirm that it was working to some degree, at that point I was going to make sure that it worked in tandem with different samplers (I'm prototyping without knowing a very good knowledge of the general codebase)
Also I'm currently investigating a potential issue with how it scales (in terms of the curve) so that will have to wait. But thank you for pointing it out still. As soon as I ensure that I'm scaling in the way I initially intended I'll try to figure out how to make sure Mirostat is disabled when this is on and that softmax has been called when measuring.

Also yeah that max_l minus max_l is probably redundant

[KerfuffleV2]

I'll try to figure out how to make sure Mirostat is disabled when this is on

You don't need to do that, you just need to run llama_sample_softmax rather than assuming the logits are already sorted when your sampler is reached.

In other words just add llama_sample_softmax(nullptr, candidates); after the line near the top when you start timing the sampler.

Then you'll have the logits nicely sorted and the softmax values available in .p

[kalomaze]

I'll try to figure out how to make sure Mirostat is disabled when this is on

You don't need to do that, you just need to run llama_sample_softmax rather than assuming the logits are already sorted when your sampler is reached.

In other words just add llama_sample_softmax(nullptr, candidates); after the line near the top when you start timing the sampler.

Then you'll have the logits nicely sorted and the softmax values available in .p

Thank you!

Also, this test is not very comprehensive and isn't the most accurate, especially with a GPT4 judge, but the intention was to see if there was any discernable general trends even without a lot of data (and quickly without having to research quote origins...)

However, it has useful data, if a bit misleading if taken too literally (like most LLM benchmarks).
The rate at which it just completely makes up quotes was much higher in non-dynamic sampling (of course, both attempts had a bunch of well known misleading quotes - but for the purposes of what this test was benchmarking, that wasn't strictly important)

I used a pretty uninteresting Mistral finetune that typically suffers from sampling issues for my testing, because I notice with a lower temp (e.g 0.6), it starts repeating and hallucinating. I chose 0.9 to avoid that here for the non-dynamic temp.
2.0 max temp with 2 k scaling value doesn't struggle with that nearly much on the same exact model, from other anecdotal testing (again, not thoroughly benchmarked, just testing the waters here)

Here's how that scale looks on a graph:

And the formula for it:

[kalomaze]

The latest commit takes a different approach for the formula now, where it is being represented as a sigmoid function.
Two presets are mapped to '1.93' and '1.94' values of temperature for now, and will be overridden if those values are used (temporarily) until a proper full implementation is put into place.

This is the basic test preset for 1.93.
1.94 uses a max of 2.0 instead, and will be more dramatic temp scaling wise.

[kalomaze]

The test build of Koboldcpp is up:
https://github.com/kalomaze/koboldcpp/releases/tag/dynamic-test

I've been getting a very positive reception so far, but the actual values used probably need better calibration.

[kalomaze]

Entropy sampling!
My math might be wrong here in some fashion (the idea of Shannon entropy is new to me, as well as C++ in general... so bear with me lol) but I noticed that this implementation is working well so far from basic testing.

The concept is essentially to ensure that, when there's total evenness in the probability distribution, let's say 'perfect' evenness, that should theoretically scale all the way to 2.0 temperature (or whatever you set as maxTemp).
The inverse of this, which is full confidence in just the top token / next to no variation at all, would have nearly 0 temperature.

This is to avoid the pitfalls of relying on a single top token:

I haven't pushed it yet, but I'll double check to see if my implementation for this is better than what I came up with before or if it needs adjustments implementation wise.

[kalomaze]

Static Top K of 40 was used, no Top P (or other samplers) were used whatsoever.

This comparison is somewhat misleading, because whether or not it uses the pattern of Celebrity - Birthday vs Birthday - Celebrity absolutely matters semantically in terms of how the model learned... but even that proves that just one bad prediction that's allowed to happen because of a sensitive temperature, can eventually lead to many bad predictions / compounding failure.
Also notice how it ends abruptly when using 1.0 temp.