02 — How LLMs Work: Tokens, Training, Inference, and the Context Window

"It just predicts the next word" is the standard one-liner for LLMs, and it's true but undersold. The mechanics behind it — a trained statistical engine that turns text into tokens, reads them through a transformer, and samples the next token, then repeats [1][2] — are what explain every behavior I care about as an engineer: why it's fluent, why it's confident, why it's wrong.

The framing that finally landed is a pipeline, not a black box. Text comes in, gets chopped into tokens, flows through a transformer that has been trained on billions of tokens to learn statistical relationships, and out the other side comes a probability distribution over what the next token should be. Sampling picks one, it gets appended to the input, and the whole thing runs again. Generation is just that loop.

Tokens: the unit the model actually sees

The first thing I had to unlearn is that an LLM reads _words_. It doesn't. It reads tokens — chunks of text that might be a word, a part of a word, a punctuation mark, or even a single character [3]. "Tokenization" splits the input along boundaries the model's vocabulary defined during training. The word "tokenization" itself might be four or five tokens; "cat" is one.

Three reasons tokens matter for anyone building on an LLM:

• Cost is priced in tokens. API bills are tokens-in plus tokens-out. A verbose prompt is a verbose bill.
• There's a maximum token limit on input plus output per model — the context window. Exceed it and the model truncates or errors.
• The model predicts tokens, not words. A weirdly split word can behave oddly; that's a tokenization artifact, not a "thought" failure.

Once I internalized that the model's whole world is tokens, the rest of the mechanics made more sense.

The transformer: how relationships get learned

Inside the model is a transformer architecture — stacked layers of a mechanism called _attention_ that lets each token weigh which other tokens are relevant to it [1][2]. Attention is what lets the model connect a pronoun to the noun it refers to several sentences back, or a closing bracket to its opening. Through training on enormous text corpora, the transformer's weights encode statistical relationships between tokens: which tokens tend to follow which, which are substitutes, which belong together.

The part I had to take seriously: the model has no concept of meaning in the human sense. It has a very precise sense of _distribution_ — what tends to appear where. That distribution is enough to produce startlingly coherent text, and it's also enough to produce confident nonsense when the prompt pushes it off the distribution it learned. Hallucinations are a property of this architecture, not a bug to be patched away.

Training vs inference: two different modes

This distinction is load-bearing and I conflated it for too long.

Training is the expensive, one-time process that produces the model's weights. The model is fed enormous datasets, makes predictions, and its weights are adjusted — through gradient descent — to reduce the error between prediction and reality [4]. The output is a fixed set of weights: the trained model. This is what AI Researchers and ML Engineers do, and it costs real money and compute.

Inference is what runs when I actually _use_ the model. The weights are frozen; the model takes my tokens, runs them through the transformer, and produces the next-token distribution. A self-driving car recognizing a stop sign on a road it has never seen is inference — applying learned weights to new input [5]. As an AI Engineer, almost everything I touch is inference. Training is someone else's job, and the only time I reach for it is fine-tuning, which is a light version of training on top of an existing model.

The context window

The model can only consider a fixed amount of text at once — its context window, measured in tokens [6]. Everything I give the model — system prompt, conversation history, retrieved documents, the user's question — competes for that window. When the window fills, the oldest content falls off.

The implication that took a while to sink in: the context window is a _budget_, not a feature. Cramming everything in "just in case" makes the prompt expensive, slow, and sometimes _worse_ — models pay less attention to any single thing when overwhelmed. Managing the window — what goes in, what gets summarized, what gets cut — is a first-class engineering problem, and it's the bridge to the next idea worth a note: embeddings, which exist precisely to let a model reason about far more text than fits in its window.

Embeddings and vector databases: a one-paragraph preview

Text that won't fit in the context window can still be _searchable_. An embedding is a dense vector — a list of numbers — that captures the semantic content of a piece of text [7]. Similar text lands at nearby points in the vector space. A vector database stores those vectors and, given a query, returns the closest matches fast [8]. The payoff: instead of stuffing ten thousand documents into the context window, I embed them all, embed the user's query, retrieve the few most relevant chunks, and put only _those_ in the window. That's the shape of retrieval-augmented generation (RAG), and none of it works without the token-level mechanics above.

AI vs AGI: where this stops being narrow

One last boundary worth naming. Everything above describes _narrow AI_ — systems highly specialized to tasks they were trained for, however impressive those tasks are [9]. AGI (Artificial General Intelligence) is the theoretical system that could learn and reason across arbitrary tasks at a human level. Today's LLMs are narrow, however broad they feel. Keeping that line clear stops me from attributing reasoning the model doesn't have, and from trusting it in domains where its training distribution doesn't reach.

How I use this

The way of thinking pays off in three concrete habits. When a response is weirdly truncated or cut off mid-word, I check the token budget before blaming the prompt. When a model hallucinates, I treat it as a distribution artifact and fix it with context, not with scolding the model. And when I'm tempted to dump an entire knowledge base into a prompt, I reach for embeddings and retrieval instead — because the context window is a budget, and inference cost scales with every token I feed it.

References

[1] Cloudflare, "What is a large language model (LLM)?," 2024. [Online]. Available: https://www.cloudflare.com/en-gb/learning/ai/what-is-large-language-model/

[2] "How Large Language Models Work," YouTube, 2024. [Video]. Available: https://www.youtube.com/watch?v=5sLYAQS9sWQ

[3] NVIDIA, "Explaining Tokens — the Language and Currency of AI," 2024. [Online]. Available: https://blogs.nvidia.com/blog/ai-tokens-explained/

[4] Domino AI, "Machine learning model training: What it is and why it's important," 2024. [Online]. Available: https://domino.ai/blog/what-is-machine-learning-model-training

[5] Cloudflare, "Inference vs Training," 2024. [Online]. Available: https://www.cloudflare.com/learning/ai/inference-vs-training/

[6] IBM, "What is a Context Window in AI?," 2024. [Online]. Available: https://www.ibm.com/think/topics/context-window

[7] Cloudflare, "What are Embeddings in Machine Learning?," 2024. [Online]. Available: https://www.cloudflare.com/en-gb/learning/ai/what-are-embeddings/

[8] Cloudflare, "Vector Databases," 2024. [Online]. Available: https://developers.cloudflare.com/vectorize/reference/what-is-a-vector-database/

[9] AWS, "What is AGI?," 2024. [Online]. Available: https://aws.amazon.com/what-is/artificial-general-intelligence/