RAG Tutorial: Build a Reliable Retrieval-Augmented Generation App

Overview

Retrieval-augmented generation, or RAG, combines a language model with a searchable collection of external documents. Instead of asking the model to answer entirely from its trained knowledge, the application retrieves relevant passages at request time and includes them in the model prompt. This approach is useful for internal knowledge assistants, product documentation search, support tools, policy question answering, and other LLM app development projects that depend on changing information.

A dependable RAG application has several distinct stages:

1. Ingestion: collect source files, web pages, database records, or other approved content.
2. Normalization: remove irrelevant markup, preserve useful structure, and attach metadata.
3. Chunking: divide documents into passages that are large enough to preserve meaning but focused enough to retrieve accurately.
4. Embedding: convert each passage into a numerical representation for semantic search.
5. Indexing: store embeddings, text, and metadata in a vector database or another retrieval system.
6. Retrieval: find passages that match the user’s question, optionally using filters, keyword search, reranking, or hybrid search.
7. Generation: provide the selected context to an LLM with instructions to answer within the available evidence.
8. Evaluation: test retrieval quality, answer accuracy, citation coverage, latency, and cost.

The most important design principle is separation of responsibilities. The retriever should find useful evidence; the model should interpret that evidence and produce a clear response. If the wrong passage is retrieved, changing the system prompt alone will rarely solve the underlying problem. For prompt design patterns, see How to Write Effective System Prompts.

A practical ingestion pattern

Start with a small, representative document set rather than indexing everything at once. For each source, store the original text, a stable document identifier, a title, section or heading information, source location, access metadata, and a revision marker when available. Keep the original documents outside the vector index as the canonical record. The index should be treated as a derived artifact that can be rebuilt.

Chunking should follow the structure of the material whenever possible. A heading and its associated paragraphs often make a better unit than an arbitrary character count. If a section is too long, split it into smaller passages with modest overlap so that definitions and references are not separated. Test chunk sizes with real questions: a good chunk helps the model answer without requiring excessive unrelated context.

Maintenance cycle

RAG systems need maintenance because their source collection, retrieval behavior, model configuration, and user expectations can all change. A simple recurring cycle makes maintenance predictable.

1. Review source freshness

On a scheduled review cycle, identify documents that were added, edited, removed, or moved. Reprocess only the affected records when possible, but retain a reliable full-rebuild procedure for migrations and indexing errors. Use stable document IDs and revision information to prevent duplicate passages from accumulating in the vector database.

2. Check parsing and chunking

Inspect a sample of newly indexed content. Tables, code blocks, scanned PDFs, navigation menus, repeated headers, and footnotes can all produce poor chunks. Confirm that titles and section names remain attached to the text that depends on them. If the source format changes, update the parser before rebuilding the index.

3. Run a fixed evaluation set

Maintain a small collection of representative questions with expected evidence and acceptable answer characteristics. Include straightforward lookups, questions requiring multiple passages, ambiguous requests, and questions that should receive an “I do not have enough information” response. Run this set after changes to chunking, embeddings, retrieval settings, prompts, or models.

Track retrieval recall, answer correctness, citation or source coverage, latency, and token usage as separate measures. A response can sound fluent while citing weak evidence, and a strong retrieval result can still be poorly summarized. The guide LLM Evaluation Metrics Explained provides a useful framework for reviewing these dimensions together.

4. Review operational behavior

Look at failed searches, unanswered questions, repeated user reformulations, and feedback from reviewers. Group these observations by cause: missing source material, poor parsing, incorrect filters, weak retrieval, unclear prompting, or an unsupported user request. Fix the highest-impact cause rather than changing several components at once.

5. Rebuild when necessary

A partial update is efficient, but a clean rebuild is appropriate after changing the embedding model, metadata schema, chunking method, text extraction logic, or distance configuration. Record the configuration used for each index so results can be compared and rolled back when needed.

Signals that require updates

Not every disappointing answer requires a new model or a larger vector database. The following signals help identify what should change.

• Correct information is not retrieved: review chunk boundaries, query wording, metadata filters, embedding compatibility, and the number of retrieved passages.
• Retrieved passages are relevant but incomplete: inspect document structure and consider section-aware chunking, neighboring passage expansion, or a second retrieval step.
• The answer uses outdated content: verify source ownership, update detection, deletion handling, and index rebuild procedures.
• Answers contain unsupported details: strengthen the system instructions, require evidence-based responses, and test whether the prompt clearly separates retrieved text from instructions.
• Users receive irrelevant context: reduce noisy sources, improve metadata filters, tune retrieval limits, or add reranking.
• Latency or cost increases: measure each stage separately, including parsing, embedding, retrieval, reranking, and generation. Avoid sending more context to the model than the task requires.
• Search intent shifts: expand the evaluation set when users begin asking new question types, such as comparisons, procedural requests, or questions spanning several documents.

Security deserves a separate review. Retrieved text should be treated as data, not as trusted instructions. Documents can contain text that attempts to redirect the assistant, expose hidden information, or manipulate tool use. Apply access controls before retrieval, limit the fields passed into prompts, and validate any proposed action outside the model. Use the Prompt Injection Prevention Checklist for AI Apps when reviewing a system that handles untrusted content.

Common issues

Overly large or overly small chunks

Large chunks may contain useful context but dilute similarity signals and consume more model context. Tiny chunks can match a query precisely while removing the explanation needed to answer it. Compare several chunking strategies against the evaluation set instead of choosing a size by convention.

Missing metadata

Text without source, title, section, date, permissions, or document identifiers is difficult to filter, cite, debug, and update. Metadata should support both retrieval and operations. Store enough information to explain where every passage came from.

Duplicate and stale passages

Repeated indexing can cause multiple versions of the same content to compete in retrieval. Use deterministic identifiers, deletion handling, and index audits. When a document is replaced, ensure the previous version is no longer eligible unless version history is an intentional feature.

Prompt-only troubleshooting

A longer prompt does not compensate for missing evidence. Inspect the retrieved passages first, then adjust retrieval and source processing. When the evidence is present, use a concise system prompt that defines the assistant’s scope, citation behavior, uncertainty handling, and response format. For broader automation designs, compare the trade-offs in AI Agent vs Workflow Automation.

Evaluation based only on fluency

Readable answers can still be wrong. Ask reviewers to check whether the answer is supported by the retrieved evidence, whether important qualifications were preserved, and whether the cited source actually contains the claim. Include negative tests where the correct behavior is to ask for clarification or decline to answer.

When to revisit

Schedule a lightweight review regularly, even when the application appears stable. A monthly or quarterly cadence may be appropriate depending on how quickly the source collection changes, but the interval should be based on operational risk rather than a fixed industry rule. Review sooner when search intent shifts, users report outdated answers, a source format changes, or the application’s model, embedding configuration, prompt, or access rules are modified.

Use this practical checklist for each review:

1. Sample recently changed documents and confirm that their indexed text is accurate.
2. Run the fixed evaluation set and compare results with the previous version.
3. Inspect failed and low-confidence queries for recurring retrieval patterns.
4. Verify that deleted or restricted content is no longer retrievable.
5. Check citations, answer boundaries, latency, and token usage.
6. Record configuration changes, test results, and the next review date.

Keep this checklist with the application code and treat the vector index as rebuildable infrastructure. That practice makes future changes safer and gives the team a clear way to distinguish a data problem from a retrieval problem or a generation problem. For a complete implementation path covering ingestion, embeddings, retrieval, and evaluation, continue with How to Build a RAG AI Assistant.