--- title: "02 — How LLMs Work: Tokens, Training, Inference, and the Context Window" uid: how-llms-work tags: ["llm", "context", "training", "inference", "tokens", "roadmap:ai-engineer", "embeddings", "fundamentals"] excerpt: "The next-token trick, unpacked. An LLM is a trained statistical engine: text becomes tokens, a transformer reads them, and the model samples the next token — again and again." date: 2026-08-13T03:28:39+0000 source: https://www.aveshina.my.id/en/blog/how-llms-work --- "It just predicts the next word" is the standard one-liner for LLMs, and it's true but undersold. The mechanics behind it — **a trained statistical engine that turns text into tokens, reads them through a transformer, and samples the next token, then repeats** [1][2] — are what explain every behavior I care about as an engineer: why it's fluent, why it's confident, why it's wrong. The framing that finally landed is a pipeline, not a black box. Text comes in, gets chopped into tokens, flows through a transformer that has been trained on billions of tokens to learn statistical relationships, and out the other side comes a probability distribution over what the next token should be. Sampling picks one, it gets appended to the input, and the whole thing runs again. Generation is just that loop. ```figure ``` ## Tokens: the unit the model actually sees The first thing I had to unlearn is that an LLM reads _words_. It doesn't. It reads **tokens** — chunks of text that might be a word, a part of a word, a punctuation mark, or even a single character [3]. "Tokenization" splits the input along boundaries the model's vocabulary defined during training. The word "tokenization" itself might be four or five tokens; "cat" is one. Three reasons tokens matter for anyone building on an LLM: - **Cost is priced in tokens.** API bills are tokens-in plus tokens-out. A verbose prompt is a verbose bill. - **There's a maximum token limit** on input plus output per model — the context window. Exceed it and the model truncates or errors. - **The model predicts tokens, not words.** A weirdly split word can behave oddly; that's a tokenization artifact, not a "thought" failure. Once I internalized that the model's whole world is tokens, the rest of the mechanics made more sense. ## The transformer: how relationships get learned Inside the model is a **transformer** architecture — stacked layers of a mechanism called _attention_ that lets each token weigh which other tokens are relevant to it [1][2]. Attention is what lets the model connect a pronoun to the noun it refers to several sentences back, or a closing bracket to its opening. Through training on enormous text corpora, the transformer's weights encode statistical relationships between tokens: which tokens tend to follow which, which are substitutes, which belong together. The part I had to take seriously: the model has no concept of meaning in the human sense. It has a very precise sense of _distribution_ — what tends to appear where. That distribution is enough to produce startlingly coherent text, and it's also enough to produce confident nonsense when the prompt pushes it off the distribution it learned. Hallucinations are a property of this architecture, not a bug to be patched away. ## Training vs inference: two different modes This distinction is load-bearing and I conflated it for too long. **Training** is the expensive, one-time process that produces the model's weights. The model is fed enormous datasets, makes predictions, and its weights are adjusted — through gradient descent — to reduce the error between prediction and reality [4]. The output is a fixed set of weights: the trained model. This is what AI Researchers and ML Engineers do, and it costs real money and compute. **Inference** is what runs when I actually _use_ the model. The weights are frozen; the model takes my tokens, runs them through the transformer, and produces the next-token distribution. A self-driving car recognizing a stop sign on a road it has never seen is inference — applying learned weights to new input [5]. As an AI Engineer, almost everything I touch is inference. Training is someone else's job, and the only time I reach for it is fine-tuning, which is a light version of training on top of an existing model. ## The context window The model can only consider a fixed amount of text at once — its **context window**, measured in tokens [6]. Everything I give the model — system prompt, conversation history, retrieved documents, the user's question — competes for that window. When the window fills, the oldest content falls off. The implication that took a while to sink in: the context window is a _budget_, not a feature. Cramming everything in "just in case" makes the prompt expensive, slow, and sometimes _worse_ — models pay less attention to any single thing when overwhelmed. Managing the window — what goes in, what gets summarized, what gets cut — is a first-class engineering problem, and it's the bridge to the next idea worth a note: embeddings, which exist precisely to let a model reason about far more text than fits in its window. ## Embeddings and vector databases: a one-paragraph preview Text that won't fit in the context window can still be _searchable_. An **embedding** is a dense vector — a list of numbers — that captures the semantic content of a piece of text [7]. Similar text lands at nearby points in the vector space. A **vector database** stores those vectors and, given a query, returns the closest matches fast [8]. The payoff: instead of stuffing ten thousand documents into the context window, I embed them all, embed the user's query, retrieve the few most relevant chunks, and put only _those_ in the window. That's the shape of retrieval-augmented generation (RAG), and none of it works without the token-level mechanics above. ## AI vs AGI: where this stops being narrow One last boundary worth naming. Everything above describes _narrow AI_ — systems highly specialized to tasks they were trained for, however impressive those tasks are [9]. **AGI** (Artificial General Intelligence) is the theoretical system that could learn and reason across arbitrary tasks at a human level. Today's LLMs are narrow, however broad they feel. Keeping that line clear stops me from attributing reasoning the model doesn't have, and from trusting it in domains where its training distribution doesn't reach. ## How I use this The way of thinking pays off in three concrete habits. When a response is weirdly truncated or cut off mid-word, I check the token budget before blaming the prompt. When a model hallucinates, I treat it as a distribution artifact and fix it with context, not with scolding the model. And when I'm tempted to dump an entire knowledge base into a prompt, I reach for embeddings and retrieval instead — because the context window is a budget, and inference cost scales with every token I feed it. ## References [1] Cloudflare, "What is a large language model (LLM)?," 2024. [Online]. Available: [https://www.cloudflare.com/en-gb/learning/ai/what-is-large-language-model/](https://www.cloudflare.com/en-gb/learning/ai/what-is-large-language-model/) [2] "How Large Language Models Work," YouTube, 2024. [Video]. Available: [https://www.youtube.com/watch?v=5sLYAQS9sWQ](https://www.youtube.com/watch?v=5sLYAQS9sWQ) [3] NVIDIA, "Explaining Tokens — the Language and Currency of AI," 2024. [Online]. Available: [https://blogs.nvidia.com/blog/ai-tokens-explained/](https://blogs.nvidia.com/blog/ai-tokens-explained/) [4] Domino AI, "Machine learning model training: What it is and why it's important," 2024. [Online]. Available: [https://domino.ai/blog/what-is-machine-learning-model-training](https://domino.ai/blog/what-is-machine-learning-model-training) [5] Cloudflare, "Inference vs Training," 2024. [Online]. Available: [https://www.cloudflare.com/learning/ai/inference-vs-training/](https://www.cloudflare.com/learning/ai/inference-vs-training/) [6] IBM, "What is a Context Window in AI?," 2024. [Online]. Available: [https://www.ibm.com/think/topics/context-window](https://www.ibm.com/think/topics/context-window) [7] Cloudflare, "What are Embeddings in Machine Learning?," 2024. [Online]. Available: [https://www.cloudflare.com/en-gb/learning/ai/what-are-embeddings/](https://www.cloudflare.com/en-gb/learning/ai/what-are-embeddings/) [8] Cloudflare, "Vector Databases," 2024. [Online]. Available: [https://developers.cloudflare.com/vectorize/reference/what-is-a-vector-database/](https://developers.cloudflare.com/vectorize/reference/what-is-a-vector-database/) [9] AWS, "What is AGI?," 2024. [Online]. Available: [https://aws.amazon.com/what-is/artificial-general-intelligence/](https://aws.amazon.com/what-is/artificial-general-intelligence/) ```quiz Q: An LLM generates text by… - looking up the correct answer in a database - repeatedly predicting and sampling the next token, then appending it and looping correct: 1 explain: Generation is a loop: tokenize input, run the transformer, sample one token from the output distribution, append it, repeat. Q: Why do API costs depend on token count rather than word count? - because models price by character - because tokens are the unit the model processes, and pricing follows processing work correct: 1 explain: The model's entire world is tokens. Input and output are both measured in tokens, and the bill tracks that work. Q: Training and inference differ in that… - training freezes the weights; inference adjusts them - training adjusts the weights to learn; inference runs frozen weights on new input correct: 1 explain: Training is the expensive one-time process that produces weights. Inference applies those fixed weights. AI Engineers mostly live in inference. Q: The context window is best treated as… - an unlimited feature — put everything in - a budget that competes for space and affects cost, latency, and attention quality correct: 1 explain: Everything in the prompt fights for the window. Overloading it raises cost, slows inference, and can dilute the model's attention. Q: Today's LLMs are best described as… - Artificial General Intelligence — they reason like humans - narrow AI — highly capable within the distribution they were trained on, but not general reasoners correct: 1 explain: LLMs are narrow AI. Their fluency can look like general reasoning, but it reflects learned token distributions, not human-like understanding across arbitrary domains. ```