03 — Sampling Knobs and Output Rails: Temperature, Top-K, Top-P, and the Rest

There's a dial labeled temperature and a dozen more next to it, and I used to treat each as a separate magic setting. The frame that collapsed them all: every sampling parameter is a different way of reshaping the next-token probability distribution before you sample from it. [1] The model always produces the same distribution for the same input; the parameters decide how that distribution becomes an actual token, and that decision is where "creative" versus "deterministic" actually lives.

The framing that finally landed is a two-stage picture. Stage one is the model: given the prompt, it outputs a probability for every token in its vocabulary. Stage two is _you_: the sampling step picks one token from that distribution, and the parameters I tune all act on this step — flattening the distribution, truncating it, or penalizing tokens that already appeared. Once I saw stage two as separate from stage one, every parameter stopped being a magic dial and became a precise operation.

Temperature: flatten or sharpen

Temperature is the one I reached for first, and it's the bluntest instrument. It rescales the distribution before sampling [2][3]. Low temperature (say 0.2) sharpens it — the most probable token becomes even more likely, and the output gets deterministic and conservative. High temperature (say 1.0 or above) flattens it — less probable tokens get a real chance, and the output gets varied and unpredictable.

The mental shortcut I use: temperature trades _reliability_ for _variety_. A coding assistant that must produce correct syntax runs at low temperature. A brainstorming partner that should surprise me runs hot. There's no "right" value; there's the right value _for the task_, and the cost of getting it wrong is either boring repetition or unreliable output.

Top-K and Top-P: trimming the menu

Temperature reshapes the whole distribution. Top-K and Top-P truncate it instead — they decide which tokens are even in the running.

Top-K keeps only the _K_ most probable tokens and discards everything else, then samples from that reduced set [4]. Low K (1–10) is conservative and factual — you're effectively considering only the obvious continuations. Medium K (20–50) balances creativity and coherence. High K (50+) lets stranger tokens in. The fixed count is the thing to notice: K is the same number of candidates no matter how flat or sharp the distribution is.

Top-P, or nucleus sampling, is the dynamic cousin [5]. Instead of a fixed count, it keeps the smallest set of tokens whose cumulative probability reaches _P_. If the model is very confident, that set might be two tokens; if it's uncertain, it might be fifty. Top-P adapts to the shape of the distribution in a way Top-K can't.

In practice I rarely set both. I reach for Top-P when I want the trim to track the model's confidence, and Top-K when I want a hard cap on candidates regardless of confidence.

Repetition penalties: breaking loops

Sometimes a model fixates — it repeats a phrase verbatim, or leans on the same word over and over. Repetition penalties fix this by reducing the probability of tokens that already appeared in the output [6]. Two flavors show up:

• Frequency penalty scales with how many times a token has been used — the more it appears, the more it's penalized.
• Presence penalty applies equally to any token used at least once — a flat nudge against repeating _anything_.

These are the params I reach for when the symptom is "it's stuck in a loop," not as default tuning. Cranking them too high produces output that contorts itself to avoid common words, which reads worse than the repetition did.

Fine-tuning: changing the model itself

Every parameter above acts on the distribution _after_ the model produced it. Fine-tuning is the one that changes the model [7]. It takes a pre-trained LLM and continues training on a smaller, task-specific dataset, nudging the weights so the model is better suited to a particular domain or style.

The caveat I had to learn the hard way: fine-tuning is expensive, slow, and not always the right tool. Before I even consider it, I exhaust the cheaper options — better context, better retrieval (RAG), a better system prompt, or a smaller specialized model [7]. Fine-tuning earns its place when the task needs a _style_ or _behavior_ baked in (tone, format, domain vocabulary) that no amount of context will reliably produce. For "the model doesn't know my data," that's RAG, not fine-tuning.

Prompt engineering: shaping the input

The other side of the output is the input. Prompt engineering is the craft of designing the instruction so the model's distribution lands where I want [8]. It's not a knob on the distribution — it changes the distribution itself by changing what the model conditions on. A precise, well-structured prompt with examples shifts the next-token probabilities in ways no sampling parameter can.

Prompt engineering and sampling parameters work together: the prompt decides _what kind_ of output the model wants to produce, and the sampling parameters decide _how varied_ the selection from that intent will be. The roadmap points to a dedicated prompt engineering track for this, which I treat as its own skill — but the core move is always the same: be specific, give examples, separate instructions from data, and iterate.

How I use this

My defaults are now deliberate, not inherited. For factual or code tasks I drop temperature low (0.0–0.3) and leave Top-P high; I want determinism. For exploratory or creative tasks I raise temperature and set a moderate Top-P, and I let repetition penalties stay at zero unless the model loops. I never reach for fine-tuning until I've exhausted context and retrieval. And when output quality is off, my first question is no longer "what's the temperature?" — it's "what does the prompt actually tell the model to do?" The knobs are the last lever, not the first.

References

[1] The New Stack, "What Temperature Means in Natural Language Processing and AI," 2024. [Online]. Available: https://thenewstack.io/what-temperature-means-in-natural-language-processing-and-ai/

[2] IBM, "What is LLM Temperature?," 2024. [Online]. Available: https://www.ibm.com/think/topics/llm-temperature

[3] DocsBot, "How Temperature Settings Transform Your AI Agent's Responses," 2024. [Online]. Available: https://docsbot.ai/article/how-temperature-settings-transform-your-ai-agents-responses

[4] DataAnnotation, "Top-K Sampling: The Complete Token Selection Guide," 2024. [Online]. Available: https://www.dataannotation.tech/blog/top-k-sampling

[5] "What are the LLM's Top-P + Top-K?," YouTube, 2024. [Video]. Available: https://www.youtube.com/watch?v=aDmp2Uim0zQ

[6] dev.to/superorange0707, "Stop the LLM From Rambling: Using Penalties to Control Repetition," 2024. [Online]. Available: https://dev.to/superorange0707/stop-the-llm-from-rambling-using-penalties-to-control-repetition-5h8

[7] IBM, "What is fine-tuning?," 2024. [Online]. Available: https://www.ibm.com/think/topics/fine-tuning

[8] AWS, "What is Prompt Engineering? — AI Prompt Engineering Explained," 2024. [Online]. Available: https://aws.amazon.com/what-is/prompt-engineering/