11 — LLM Fundamentals — Next-Token Prediction, Embeddings, and Vectors

I used to wave the backend roadmap's AI nodes off as ML-specialist trivia a backend dev could skip — until every AI tool in the roadmap turned out to sit on the same three ideas. The base that corrected that: an LLM is a next-token predictor over a vocabulary; embeddings are the numeric coordinates that turn meaning into something arithmetic; vectors are the space those coordinates live in. [1] The backend AI tools that fill the rest of the roadmap — Copilot, RAG, function calling, agents — are all built on these three ideas, and I can't reason about any of them without this base.

The frame that helped is that "AI" in the backend context is not one thing; it's a stack. At the bottom is the math (vectors). One layer up is the representation (embeddings — where meaning becomes vectors). One layer up is the model that consumes those representations (an LLM predicting the next token). Everything above — code completion, chatbots, agents, RAG — is plumbing around that core. Once the three foundation ideas are clear, the rest of the roadmap stops being a list of magic product names and becomes a set of engineering choices.

Next-token prediction: what an LLM actually does

Strip away the mythology and an LLM is a model that, given a sequence of tokens, outputs a probability distribution over what the next token could be [1][2]. The input is the text so far; the output is a list of candidate next tokens, each with a probability. The model then samples one (guided by parameters like temperature — high temperature spreads probability toward less likely tokens, low temperature concentrates on the most likely), appends it to the input, and repeats. Token by token, the response is built.

The thing to internalize is that this is probabilistic pattern-matching over a huge training corpus, not retrieval or reasoning in the human sense. The model has learned, from trillions of tokens of text, what tends to follow what. "The cat sat on the" is followed by "mat" more often than "chair" in the training data, so "mat" gets higher probability. The miracle is that doing this competently at scale produces behavior that looks like understanding — enough that the distinction gets blurry in practice. But the mechanism is next-token prediction, full stop.

Two consequences for backend work:

• No determinism by default. The same prompt can yield different outputs, controlled by temperature and sampling. If I need determinism (a function that returns parseable JSON), I set temperature low and use constrained decoding or structured output modes.
• Context is the only memory. The model has no persistent state between requests beyond what I put in the prompt. Every conversation history, every document, every instruction lives in the context window of the current request. That's why context management (and RAG — more in a later post) is the central backend problem of LLM applications.

Embeddings: meaning as coordinates

An embedding is a dense vector representation of a piece of data — a word, a sentence, an image — in a lower-dimensional space where similar items end up close together [3]. The classic demonstration: in a well-trained embedding space, the vector for "king" minus the vector for "man" plus the vector for "woman" lands you near "queen." The model has captured enough of the meaning of these words that analogies become arithmetic.

The reason embeddings matter for backend work is that they make similarity computable. Two pieces of text can be compared by computing the distance (cosine similarity, usually) between their embedding vectors. "Cat sat on the mat" and "the feline rested on the rug" have similar embeddings because they mean similar things, even though they share few words. That capability is the foundation of semantic search, retrieval (RAG), recommendation, and clustering — all of which are backend features, not ML research.

The practical pipeline: I take a body of text (a doc, a knowledge base), pass each chunk through an embedding model to get a vector, store those vectors in a vector database (or a Postgres + pgvector), and at query time embed the user's query and find the nearest stored vectors. The model does the magic once (turning text into coordinates); the rest is the database work I already know.

Vectors: the arithmetic underneath

A vector is a mathematical object with magnitude and direction, represented as an ordered list of numbers — its components [4]. In ML, a 768-dimensional vector is just a list of 768 floats. The vector is the substrate embeddings and model internals all live on; "embedding" is a vector whose coordinates happen to encode meaning, and an LLM's internal state is a vector whose coordinates encode the evolving context.

The operations that matter for backend AI work are few:

• Distance / similarity. Cosine similarity (angle between two vectors) or Euclidean distance (straight-line). How close are two embeddings?
• Averaging. Mean of a set of vectors — sometimes used to summarize a body of text.
• Nearest-neighbor search. Given a query vector, find the closest stored vectors — the core operation of a vector database.

I don't need to implement any of these; I need to know they exist, because the APIs of every vector DB (Pinecone, Weaviate, pgvector) are built around them. "Find me the 10 documents most similar to this query" is a nearest-neighbor search over embeddings, and knowing that lets me reach for the right tool.

AI vs. traditional software development

One more framing node from the roadmap: the contrast between traditional and AI-assisted development [5]. Traditional development is deterministic and explicit — I write every step, the computer follows it exactly. AI-assisted development delegates some of the "write the step" work to a model that proposes code based on patterns it learned. The model is probabilistic; correctness comes from review, testing, and iteration rather than from the model being guaranteed right.

This isn't a replacement of traditional development; it's a tool layered on top. The backend still runs deterministic code; the AI helps write it. The shift in mindset is being comfortable with "the model suggested this, now I verify it" rather than "I wrote this, therefore it does what I intended." Both modes still demand the same end state: tested, correct code in production. The AI changes the writing speed, not the verification obligation.

How I use this

The three foundation ideas translate into three concrete habits:

• Treat LLM outputs as probabilistic. I set temperature appropriately, use structured output modes when I need parseable results, and never assume the same prompt gives the same answer twice. For deterministic flows I constrain the model and validate its output.
• Reach for embeddings when I need semantic similarity. Keyword search is brittle; embedding search captures meaning. For any "find related X" feature, the first question is now "should this be embedding-based?"
• Context window is a backend budget. Everything the model needs must fit in the prompt, which makes context selection (what to include, what to truncate) a real engineering problem. RAG, summarization, and tool-calling are all solutions to the context-budget problem.

The framing — predict, embed, vector — is the load-bearing idea. Every AI feature in the rest of the roadmap (Copilot suggesting code, RAG retrieving docs, an agent calling a function) is an application of these three. Knowing the foundation makes the features legible.

References

[1] Cloudflare, "What is a large language model (LLM)?," 2024. [Online]. Available: https://www.cloudflare.com/en-gb/learning/ai/what-is-large-language-model/

[2] "New to LLMs? Start Here," Towards Data Science. [Online]. Available: https://towardsdatascience.com/new-to-llms-start-here/

[3] Cloudflare, "What are Embeddings in Machine Learning?," 2024. [Online]. Available: https://www.cloudflare.com/en-gb/learning/ai/what-are-embeddings/

[4] "A Gentle Introduction to Vectors for Machine Learning," Machine Learning Mastery. [Online]. Available: https://machinelearningmastery.com/gentle-introduction-vectors-machine-learning/

[5] "What is vibe coding?," IBM, 2024. [Online]. Available: https://www.ibm.com/think/topics/vibe-coding