--- title: "03 — Sampling Knobs and Output Rails: Temperature, Top-K, Top-P, and the Rest" uid: llm-knobs-and-rails tags: ["prompt-engineering", "llm", "temperature", "top-p", "top-k", "roadmap:ai-engineer", "sampling", "fine-tuning"] excerpt: "Temperature, top-p, max tokens: each knob looks like an isolated setting. The frame that collapsed them: every generation parameter reshapes the same next-token probability distribution before you sample." date: 2026-08-13T03:28:39+0000 source: https://www.aveshina.my.id/en/blog/llm-knobs-and-rails --- There's a dial labeled temperature and a dozen more next to it, and I used to treat each as a separate magic setting. The frame that collapsed them all: **every sampling parameter is a different way of reshaping the next-token probability distribution before you sample from it.** [1] The model always produces the same distribution for the same input; the parameters decide how that distribution becomes an actual token, and that decision is where "creative" versus "deterministic" actually lives. The framing that finally landed is a two-stage picture. Stage one is the model: given the prompt, it outputs a probability for every token in its vocabulary. Stage two is _you_: the sampling step picks one token from that distribution, and the parameters I tune all act on this step — flattening the distribution, truncating it, or penalizing tokens that already appeared. Once I saw stage two as separate from stage one, every parameter stopped being a magic dial and became a precise operation. ```figure ``` ## Temperature: flatten or sharpen **Temperature** is the one I reached for first, and it's the bluntest instrument. It rescales the distribution before sampling [2][3]. Low temperature (say 0.2) sharpens it — the most probable token becomes even more likely, and the output gets deterministic and conservative. High temperature (say 1.0 or above) flattens it — less probable tokens get a real chance, and the output gets varied and unpredictable. The mental shortcut I use: temperature trades _reliability_ for _variety_. A coding assistant that must produce correct syntax runs at low temperature. A brainstorming partner that should surprise me runs hot. There's no "right" value; there's the right value _for the task_, and the cost of getting it wrong is either boring repetition or unreliable output. ## Top-K and Top-P: trimming the menu Temperature reshapes the whole distribution. **Top-K** and **Top-P** truncate it instead — they decide which tokens are even in the running. **Top-K** keeps only the _K_ most probable tokens and discards everything else, then samples from that reduced set [4]. Low K (1–10) is conservative and factual — you're effectively considering only the obvious continuations. Medium K (20–50) balances creativity and coherence. High K (50+) lets stranger tokens in. The fixed count is the thing to notice: K is the same number of candidates no matter how flat or sharp the distribution is. **Top-P**, or **nucleus sampling**, is the dynamic cousin [5]. Instead of a fixed count, it keeps the smallest set of tokens whose cumulative probability reaches _P_. If the model is very confident, that set might be two tokens; if it's uncertain, it might be fifty. Top-P adapts to the shape of the distribution in a way Top-K can't. In practice I rarely set both. I reach for Top-P when I want the trim to track the model's confidence, and Top-K when I want a hard cap on candidates regardless of confidence. ## Repetition penalties: breaking loops Sometimes a model fixates — it repeats a phrase verbatim, or leans on the same word over and over. **Repetition penalties** fix this by reducing the probability of tokens that already appeared in the output [6]. Two flavors show up: - **Frequency penalty** scales with how many times a token has been used — the more it appears, the more it's penalized. - **Presence penalty** applies equally to any token used at least once — a flat nudge against repeating _anything_. These are the params I reach for when the symptom is "it's stuck in a loop," not as default tuning. Cranking them too high produces output that contorts itself to avoid common words, which reads worse than the repetition did. ## Fine-tuning: changing the model itself Every parameter above acts on the distribution _after_ the model produced it. **Fine-tuning** is the one that changes the model [7]. It takes a pre-trained LLM and continues training on a smaller, task-specific dataset, nudging the weights so the model is better suited to a particular domain or style. The caveat I had to learn the hard way: fine-tuning is expensive, slow, and not always the right tool. Before I even consider it, I exhaust the cheaper options — better context, better retrieval (RAG), a better system prompt, or a smaller specialized model [7]. Fine-tuning earns its place when the task needs a _style_ or _behavior_ baked in (tone, format, domain vocabulary) that no amount of context will reliably produce. For "the model doesn't know my data," that's RAG, not fine-tuning. ## Prompt engineering: shaping the input The other side of the output is the input. **Prompt engineering** is the craft of designing the instruction so the model's distribution lands where I want [8]. It's not a knob on the distribution — it changes the distribution itself by changing what the model conditions on. A precise, well-structured prompt with examples shifts the next-token probabilities in ways no sampling parameter can. Prompt engineering and sampling parameters work together: the prompt decides _what kind_ of output the model wants to produce, and the sampling parameters decide _how varied_ the selection from that intent will be. The roadmap points to a dedicated prompt engineering track for this, which I treat as its own skill — but the core move is always the same: be specific, give examples, separate instructions from data, and iterate. ## How I use this My defaults are now deliberate, not inherited. For factual or code tasks I drop temperature low (0.0–0.3) and leave Top-P high; I want determinism. For exploratory or creative tasks I raise temperature and set a moderate Top-P, and I let repetition penalties stay at zero unless the model loops. I never reach for fine-tuning until I've exhausted context and retrieval. And when output quality is off, my first question is no longer "what's the temperature?" — it's "what does the prompt actually tell the model to do?" The knobs are the last lever, not the first. ## References [1] The New Stack, "What Temperature Means in Natural Language Processing and AI," 2024. [Online]. Available: [https://thenewstack.io/what-temperature-means-in-natural-language-processing-and-ai/](https://thenewstack.io/what-temperature-means-in-natural-language-processing-and-ai/) [2] IBM, "What is LLM Temperature?," 2024. [Online]. Available: [https://www.ibm.com/think/topics/llm-temperature](https://www.ibm.com/think/topics/llm-temperature) [3] DocsBot, "How Temperature Settings Transform Your AI Agent's Responses," 2024. [Online]. Available: [https://docsbot.ai/article/how-temperature-settings-transform-your-ai-agents-responses](https://docsbot.ai/article/how-temperature-settings-transform-your-ai-agents-responses) [4] DataAnnotation, "Top-K Sampling: The Complete Token Selection Guide," 2024. [Online]. Available: [https://www.dataannotation.tech/blog/top-k-sampling](https://www.dataannotation.tech/blog/top-k-sampling) [5] "What are the LLM's Top-P + Top-K?," YouTube, 2024. [Video]. Available: [https://www.youtube.com/watch?v=aDmp2Uim0zQ](https://www.youtube.com/watch?v=aDmp2Uim0zQ) [6] dev.to/superorange0707, "Stop the LLM From Rambling: Using Penalties to Control Repetition," 2024. [Online]. Available: [https://dev.to/superorange0707/stop-the-llm-from-rambling-using-penalties-to-control-repetition-5h8](https://dev.to/superorange0707/stop-the-llm-from-rambling-using-penalties-to-control-repetition-5h8) [7] IBM, "What is fine-tuning?," 2024. [Online]. Available: [https://www.ibm.com/think/topics/fine-tuning](https://www.ibm.com/think/topics/fine-tuning) [8] AWS, "What is Prompt Engineering? — AI Prompt Engineering Explained," 2024. [Online]. Available: [https://aws.amazon.com/what-is/prompt-engineering/](https://aws.amazon.com/what-is/prompt-engineering/) ```quiz Q: Temperature, Top-K, and Top-P all act on… - the model's trained weights - the next-token probability distribution at sampling time, after the model produced it correct: 1 explain: The model always outputs the same distribution for the same input. These parameters reshape that distribution before a token is sampled. Q: A low temperature (0.2) will make output… - more deterministic and conservative, favoring the most probable tokens - more varied and creative correct: 0 explain: Low temperature sharpens the distribution toward the top token. High temperature flattens it, giving lower-probability tokens a chance. Q: What's the key difference between Top-K and Top-P (nucleus) sampling? - Top-K keeps a fixed number of tokens; Top-P keeps the smallest set whose cumulative probability reaches P - Top-K sharpens the distribution; Top-P flattens it correct: 0 explain: Top-K is a fixed count. Top-P adapts: the set size varies with how confident the model is. Q: Fine-tuning should usually be considered… - first, before anything else - last, after better context, retrieval, prompt design, and smaller specialized models correct: 1 explain: Fine-tuning changes the model itself — it's expensive and slow. The cheaper, reversible levers come first. Q: A model is repeating the same phrase over and over. The most targeted fix is… - raising temperature very high - applying a frequency or presence penalty correct: 1 explain: Repetition penalties specifically downweight tokens that already appeared, breaking loops without scrambling the rest of the output. ```