--- title: "12 — RAG in Practice: Chunking, Retrieval, Generation, and the Fine-tuning Alternative" uid: rag-implementation tags: ["chunking", "similarity-search", "rag", "retrieval-augmented-generation", "vector-database", "roadmap:ai-engineer", "fine-tuning"] excerpt: "When RAG answers are wrong, the model usually isn't the culprit. The pipeline that reshaped my debugging: chunk, embed, retrieve, generate — with quality problems concentrated in the first two stages." date: 2026-08-13T03:28:37+0000 source: https://www.aveshina.my.id/en/blog/rag-implementation --- The first time a RAG answer came back confidently wrong, I blamed the model. The reframe that reshaped every debugging session since: **RAG** (Retrieval-Augmented Generation — answering from passages you fetch instead of training the model on them) **is a pipeline of four stages — chunk, embed, retrieve, generate — and almost every quality problem lives in the first two, not the last.** [1] When the answer is wrong, my instinct used to be to blame the model. The model is usually fine; the retrieval fed it the wrong passage, or the chunking butchered the passage before retrieval even ran. The framing that finally landed is a strict order of operations. Documents come in, get split into chunks, each chunk gets embedded into a vector, the vectors get stored. At query time, the query is embedded, the closest chunk vectors are retrieved, and those chunks are handed to the generator (the LLM) along with the question. The LLM produces an answer _grounded in_ the retrieved chunks. Each stage has its own failure modes, and naming the stage that failed is the whole skill of building RAG well. ```figure ``` ## Stage one: chunking The pipeline starts with **chunking** — breaking large documents into smaller, manageable pieces [2]. The reason is mechanical and worth stating plainly: the model has a context window limit, and even within the limit, retrieving whole documents wastes the window on irrelevant passages. Chunking lets the retriever return the _specific_ paragraph or section that matches the query. Chunking is also where most RAG quality problems originate, and I underestimated it for a long time. The decisions that matter: - **Chunk size.** Too large and retrieval pulls in irrelevant context that dilutes the signal; too small and each chunk loses the surrounding meaning it needs to be understood. There's no universal right size — it depends on the document structure. - **Overlap.** Adjacent chunks usually share some tokens at the boundaries so a concept split across a boundary still appears intact in at least one chunk. - **Structure-aware splitting.** Splitting on headings, paragraphs, or sentences beats splitting on a fixed character count, because it preserves semantic units. A naive character split can break a sentence mid-thought. The roadmap points to strategies like recursive character splitting and structure-aware chunking, and the consistent lesson across all of them is: chunking is a first-class design decision, not a preprocessing afterthought [2]. ## Stage two: embedding and indexing Each chunk gets passed through an embedding model and becomes a vector [3]. That vector, paired with the original chunk text and any metadata (source document, section, page), gets stored in a vector database and indexed for fast similarity search. The indexing is what makes retrieval over millions of chunks feel instant. The embedding step inherits every decision about the embedding model — dimensionality, language coverage, quality — so the model choice made back in the embeddings post is load-bearing here. A model that produces mediocre similarity signal will produce mediocre retrieval no matter how good the chunking was. Indexing the embeddings with metadata alongside them is what later enables **dynamic filters** — restricting retrieval to a subset (a document collection, a date range, a user's permissions) before the similarity search even runs. ## Stage three: retrieval At query time the user's question is embedded using the _same_ embedding model, and that query vector is compared against the stored chunk vectors to find the most similar ones [4]. This is the **similarity search** step, typically powered by approximate nearest neighbor over the vector index. Two things I had to learn the hard way about retrieval: - **The query and the chunks live in the same space only if they were embedded the same way.** Using a different embedding model for queries than for chunks silently breaks retrieval — the distances become meaningless. Same model, always. - **Top-K is a knob, not a constant.** Retrieving too few chunks risks missing the answer; retrieving too many floods the generator with irrelevant context that competes for attention. The right K depends on chunk size and the model's context budget, and it's worth tuning. Retrieval quality is the single biggest lever on end-to-end RAG quality. If the right passage isn't in the retrieved set, no generator can produce a correct grounded answer. ## Stage four: generation The final stage hands the retrieved chunks and the user's question to a generative model — typically an LLM — which produces an answer [5]. The model's job is to read the retrieved context and synthesize a response that's grounded in it, ideally with the discipline to say "I don't know" when the context doesn't contain the answer. The generation step is where constraints earn their keep. A system prompt like "Answer only from the provided context; if the answer isn't there, say you don't know" turns a confident hallucinator into a reliable one. Structured outputs can force the model to return a citation alongside each claim, which makes the answer auditable. Generation quality matters, but it's the _last_ thing to tune, not the first — because generation can only work with what retrieval gave it. ## RAG use cases: where this actually pays off RAG earns its place anywhere an LLM needs current, private, or domain-specific knowledge it couldn't have been trained on [6]. The canonical use cases: - **Customer support** that answers from your actual knowledge base and policies. - **Document Q&A** over internal wikis, PDFs, or codebases. - **Semantic search** that finds meaning, not just keywords. - **Chatbots** that must stay grounded in real, up-to-date information rather than the model's stale training data. The pattern across all of them: the knowledge changes faster than training cycles, or it's private and could never be in training data. That's the signal that RAG — not fine-tuning — is the right tool. ## RAG versus fine-tuning: the call I kept getting wrong This is the distinction I conflated for the longest time, and it's worth stating sharply. **Fine-tuning** adapts a model's behavior or style by continuing training on a specific dataset; it bakes _how_ the model should respond into its weights. **RAG** leaves the model alone and gives it the _what_ — the specific facts — at query time [7]. The rule I use now: - If the problem is "the model doesn't know my data," that's **RAG**. Add retrieval, don't fine-tune. - If the problem is "the model doesn't respond in the right format, tone, or style," that's **fine-tuning** (or often just better prompting). Fine-tuning is the wrong tool for knowledge injection — the model can't memorize a changing knowledge base, and retraining every time a document changes is absurd. RAG is the wrong tool for style — better prompts or fine-tuning handle that. Getting this call wrong wastes enormous effort, which is why almost every RAG guide (and the roadmap) hammers on it [7]. ## How I use this My debugging order is now strict and stage-driven. When a RAG answer is wrong, I first ask: _was the right passage even retrieved?_ I log the retrieved chunks for every query and check. If the right passage isn't there, the bug is in chunking, embedding, or retrieval — and I fix the input, not the model. If the right passage _is_ there but the answer is still wrong, _then_ the bug is in generation, and I tune the prompt, constraints, or model. I default to RAG over fine-tuning for any knowledge problem, reach for fine-tuning only for style and format, and treat chunking as a first-class design decision rather than a preprocessing step. The picture of four named stages is what turned RAG from "it sometimes works" into something I can actually debug. ## References [1] AWS, "What is RAG?," 2024. [Online]. Available: [https://aws.amazon.com/what-is/retrieval-augmented-generation/](https://aws.amazon.com/what-is/retrieval-augmented-generation/) [2] Pinecone, "Chunking Strategies for LLM Applications," 2024. [Online]. Available: [https://www.pinecone.io/learn/chunking-strategies/](https://www.pinecone.io/learn/chunking-strategies/) [3] LlamaIndex, "Indexing & Embeddings," 2024. [Online]. Available: [https://developers.llamaindex.ai/python/framework/understanding/rag/indexing/](https://developers.llamaindex.ai/python/framework/understanding/rag/indexing/) [4] TrueFoundry, "What is Similarity Search & How Does it work?," 2024. [Online]. Available: [https://www.truefoundry.com/blog/similarity-search](https://www.truefoundry.com/blog/similarity-search) [5] IBM, "What is Retrieval-Augmented Generation?," YouTube, 2024. [Video]. Available: [https://www.youtube.com/watch?v=T-D1OfcDW1M](https://www.youtube.com/watch?v=T-D1OfcDW1M) [6] Glean, "Retrieval augmented generation use cases: Transforming data into insights," 2024. [Online]. Available: [https://www.glean.com/blog/retrieval-augmented-generation-use-cases](https://www.glean.com/blog/retrieval-augmented-generation-use-cases) [7] Monte Carlo, "RAG vs Fine Tuning: How to Choose the Right Method," 2024. [Online]. Available: [https://www.montecarlodata.com/blog-rag-vs-fine-tuning/](https://www.montecarlodata.com/blog-rag-vs-fine-tuning/) ```quiz Q: RAG's four pipeline stages, in order, are… - train, validate, deploy, monitor - chunk, embed, retrieve, generate correct: 1 explain: Documents are chunked, embedded and stored, then at query time the closest chunks are retrieved and handed to a generator. Q: When a RAG answer is wrong, the first place to look is… - the generator model — swap it for a bigger one - the retrieved chunks — was the right passage even retrieved? correct: 1 explain: Generation can only work with what retrieval gave it. Log retrieved chunks per query and check whether the right passage was in the set before touching the model. Q: "The model doesn't know my private, constantly-updated knowledge base" is best solved by… - fine-tuning the model on the knowledge base - RAG, because the model can't memorize a changing knowledge base correct: 1 explain: Fine-tuning is wrong for knowledge injection. RAG gives the model the specific facts at query time, so updates don't require retraining. Q: "The model responds in the wrong tone and format" is best solved by… - RAG with more retrieved context - fine-tuning (or better prompting) for style and format correct: 1 explain: Style and format are behavioral. Fine-tuning bakes them into the weights; RAG only injects facts and won't change tone. Q: Why must the query be embedded with the same model as the chunks? - different models produce vectors in different spaces, so distances become meaningless - it doesn't matter; any embedding model works correct: 0 explain: Similarity only works if query and chunks share the same vector space. Using a different embedding model for queries silently breaks retrieval. ```