RAG Tutorial: Build a Document Question-Answering App Step by Step

Overview

Retrieval-augmented generation (RAG) connects a language model to a collection of external documents. Instead of asking the model to answer entirely from its trained knowledge, your application retrieves relevant passages and places them in the model's context before generating a response. This approach is useful when an AI assistant must work with internal documentation, product manuals, support articles, policies, or other content that changes over time.

A dependable RAG system is more than a vector database and a prompt. It is a pipeline with several checkpoints:

1. Ingestion: collect, parse, clean, and identify source documents.
2. Chunking: divide content into passages that preserve enough context to be useful.
3. Embedding: convert passages and user queries into vectors for semantic matching.
4. Retrieval: find and rank candidate passages for a question.
5. Generation: instruct the model to answer using the retrieved context.
6. Evaluation: test whether retrieval and answers are accurate, relevant, safe, and appropriately cited.
7. Maintenance: handle document additions, edits, deletions, and embedding changes.

Before choosing a framework or database, define the assistant's job. A narrow question-answering app is usually easier to evaluate than a general-purpose AI agent. For broader implementation context, see this guide to building a reliable retrieval-augmented generation app and the practical RAG AI assistant tutorial.

Checklist by scenario

Scenario 1: You are preparing a new document collection

• Define which questions the assistant should answer and which it should decline.
• List every document source, including file types, URLs, owners, and update frequency.
• Assign stable identifiers to documents so versions can be tracked.
• Preserve useful metadata such as title, section, author, publication date, access scope, and source URL.
• Remove navigation labels, repeated headers, broken characters, and unrelated page elements before indexing.
• Decide how tables, lists, code blocks, images, and scanned pages will be handled.
• Record whether a document is current, archived, restricted, or superseded.

Document quality affects every later stage. If the parser loses headings or merges unrelated columns in a table, a strong language model may still produce a fluent but unsupported answer. Treat extraction as a testable part of the application rather than an invisible preprocessing step.

Scenario 2: You are choosing a chunking strategy

• Start with chunks that follow natural boundaries such as headings, paragraphs, list groups, or sections.
• Keep the source title and relevant heading path with each chunk.
• Use overlap only when it helps preserve context across boundaries.
• Store the original text alongside the vector so retrieved content can be inspected.
• Test short factual questions, multi-part questions, and questions that depend on surrounding context.

There is no universal chunk size that works for every corpus. Smaller chunks can improve precision but may omit definitions or qualifications. Larger chunks provide more context but can make ranking less focused and consume more model context. Compare several strategies against a small set of representative questions instead of selecting a setting by habit.

Scenario 3: You are implementing retrieval

• Generate embeddings with the same general approach for indexed passages and incoming queries.
• Retrieve more candidates than you intend to show the model, then apply ranking or filtering.
• Consider hybrid retrieval when exact terms, identifiers, product names, or code symbols matter.
• Filter by permissions and metadata before content reaches the model.
• Remove duplicate or near-duplicate passages from the final context.
• Log the query, selected documents, ranking scores where available, and final response.

A vector database is a storage and search component, not a complete retrieval strategy. Your application still needs rules for access control, freshness, ranking, deduplication, and empty results. A useful AI developer tools overview can help you compare the surrounding components, but validate each choice against your corpus and operational needs.

Scenario 4: You are writing the answer prompt

• Tell the model to use the supplied context as the primary evidence.
• Instruct it to say when the context is insufficient instead of guessing.
• Require citations that point to document names, headings, page numbers, or stable source links.
• Separate system instructions, retrieved content, and the user's question clearly.
• Specify the desired response format for lists, comparisons, procedures, or quoted material.
• Include an instruction to treat retrieved documents as data, not as higher-priority instructions.

Prompt design should support the retrieval policy rather than compensate for poor retrieval. Review system prompt examples for developers when defining the assistant's boundaries, and use the prompt injection prevention checklist when documents may contain untrusted or user-generated text.

Scenario 5: You are preparing the app for real users

• Display the sources used for each answer in a way users can inspect.
• Return a clear response when no relevant passage is found.
• Protect document access at retrieval time, not only in the user interface.
• Set limits for query length, retrieved context, response length, and request frequency.
• Track latency, retrieval failures, model errors, and user feedback separately.
• Provide a path for reporting incorrect answers or missing documents.

What to double-check

Retrieval quality

Create a small evaluation set before making changes. Each example should include a question, the expected source passage or passages, and an acceptable answer. Check whether the relevant passage appears in the retrieved results and whether irrelevant passages crowd it out. Test alternate wording, abbreviations, misspellings, and questions that combine information from multiple sections.

Answer grounding and citations

Read answers for unsupported details, missing conditions, and conclusions that go beyond the retrieved text. A citation is useful only if it leads to evidence that supports the specific claim. If the application cannot identify reliable evidence, the safer behavior is to ask for clarification, state that the documents do not answer the question, or direct the user to an appropriate source.

Permissions and sensitive content

Model context should contain only content the requesting user is allowed to access. Check permissions during ingestion and retrieval, and decide how access changes affect previously indexed chunks. Also review logs: queries, retrieved passages, and generated responses may contain confidential information.

Freshness and versioning

Confirm that an updated document replaces or supersedes the intended version. A simple re-indexing job can create stale duplicates if old chunks are not removed or marked inactive. Keep enough metadata to trace an answer back to the exact document version used at retrieval time.

Common mistakes

• Indexing everything without defining scope: a large collection does not automatically produce broader or better answers. Start with a clearly bounded knowledge base.
• Ignoring document structure: flattening headings, tables, and lists can remove the relationships that make a passage understandable.
• Using only semantic search: exact identifiers, error codes, filenames, and technical terms may require keyword or hybrid retrieval.
• Passing too many chunks to the model: excessive context can increase cost and make the relevant evidence harder to use.
• Letting the model fill gaps: fluent language is not evidence. Add explicit refusal and uncertainty behavior.
• Testing only successful examples: include ambiguous questions, unsupported questions, outdated documents, and permission-sensitive requests.
• Treating citations as decoration: validate that citations are generated from retrieved metadata and point to inspectable sources.
• Changing several components at once: update chunking, retrieval, prompts, and models independently where possible so you can identify what improved or degraded results.

When to revisit

Review the RAG pipeline whenever its inputs or operating conditions change. At minimum, revisit it before a major planning cycle, when the document collection grows substantially, when source systems change, or when users report repeated missing or unsupported answers.

Use this maintenance checklist:

1. Sample new and recently edited documents to verify extraction and metadata.
2. Run the evaluation set after changing the embedding model, chunking rules, retrieval filters, ranking logic, or answer prompt.
3. Check for stale, duplicate, deleted, or permission-inconsistent records.
4. Review unanswered questions to distinguish retrieval gaps from missing source content.
5. Compare answer quality, citation completeness, latency, and resource usage before and after a change.
6. Update user-facing instructions when the assistant's scope or source coverage changes.

Keep a short change log for indexing rules, prompts, retrieval settings, and evaluation results. That record makes future debugging faster and gives the team a reliable baseline when tools or workflows change. A RAG application becomes easier to trust when every answer can be connected to a document, a retrieval decision, and a repeatable test.