Sampling (top-p, top-k)
Sampling is how a model picks each next token from its list of candidates: temperature reshapes the odds, top-p and top-k trim the list, and together they set how predictable or adventurous the output is.
At every step a model does not produce a word — it produces a probability for every token in its vocabulary. Sampling is the rule that turns that list into one choice. The simplest rule, always take the most likely token, gives repetitive, flat text; real generation draws from the list at random, weighted by those probabilities, and the settings decide how wide the draw is.
Top-k keeps only the k most likely tokens and draws among them; top-p keeps the smallest set whose probabilities add up to p — say 0.9 — so the list shrinks when the model is confident and widens when it is not. Temperature is applied before either: below 1 it sharpens the odds toward the favourites, above 1 it flattens them. Local runtimes expose all three, usually with sensible defaults such as temperature 0.7 and top-p 0.9.
Pick settings by job. Code, extraction and anything checked by a machine want low temperature and a tight top-p; brainstorming and fiction want the opposite. A seed makes a sampled run repeatable — same seed, same settings, same tokens — which matters when you compare two quantizations of one model fairly.
Where you see it on the radar
Sampling settings change what a model says, not what it needs: memory and fit are identical at temperature 0 and 1.2. That is why the finder asks about hardware and context and never about sampling.
Check what fits, whatever your settings →Related terms