LLM Evaluation Guide: Build a Repeatable Test Set for Prompts and AI Apps

Overview

When a prompt, model, retrieval pipeline, or tool-calling workflow changes, the central question is simple: did the change improve the product for real users? A repeatable evaluation process turns that question into something you can investigate rather than guess.

LLM evaluation is different from ordinary unit testing. Many valid answers can have different wording, and a response may be fluent while still being inaccurate, unsafe, incomplete, or poorly formatted. A useful evaluation therefore tests several dimensions at once:

• Correctness: Does the response satisfy the task and agree with the available evidence?
• Completeness: Does it include the required facts, steps, fields, or caveats?
• Instruction following: Does it follow the system prompt, user request, format, and constraints?
• Grounding: For RAG applications, are claims supported by retrieved content rather than invented?
• Safety and refusal behavior: Does the application handle disallowed, ambiguous, or adversarial requests appropriately?
• Operational quality: Are latency, token use, tool calls, and structured-output failures acceptable for the workflow?

Start by defining the behavior your feature is supposed to deliver. Then build a test set that reflects actual inputs, edge cases, and known failures. Run the same cases against a baseline and a candidate version, record both scores and examples, and investigate trade-offs before shipping.

This process applies to a support assistant, document question-answering application, text extraction workflow, agent, or smaller feature such as summarization. For related implementation context, see the AI text utilities guide and the RAG tutorial for document question answering.

Checklist by scenario

1. Build the evaluation set

Collect examples from several sources instead of relying only on synthetic prompts or developer intuition. Include anonymized production-like requests when permitted, support tickets, manually written cases, historical failures, and boundary conditions.

• Define the feature’s supported task and out-of-scope requests.
• Group cases by user intent, difficulty, language, input length, and relevant domain.
• Include normal cases, ambiguous requests, missing information, malformed inputs, and adversarial instructions.
• Preserve a small, stable regression set that changes rarely.
• Maintain a separate development set for experimenting with prompts and examples.
• Record the expected behavior, not necessarily one exact expected sentence.

A practical test-case record can use this shape:

{
  "id": "billing_014",
  "category": "ambiguous_request",
  "input": "Can I get a refund for this?",
  "context": "Relevant account or policy text, if applicable",
  "expected_behavior": "Ask for the missing purchase details before deciding",
  "must_include": ["a clarifying question"],
  "must_not_include": ["an unsupported approval or denial"],
  "risk": "medium",
  "notes": "Tests uncertainty handling"
}

2. Evaluate a prompt revision

Keep the model, settings, tools, and input data as consistent as practical. Compare the current prompt with the proposed prompt on the same test set. If you change several variables at once, you may not know what caused the result.

• Save the complete prompt or message structure, including system instructions and examples.
• Record model identifier, relevant generation settings, retrieved context, and tool definitions.
• Compare overall scores and scores by category.
• Review every severe failure, even if the aggregate score improves.
• Check whether a gain in helpfulness caused a decline in safety, precision, or format compliance.

Prompt changes often work best when evaluated alongside the design principles in this guide to reliable system prompts. If your workflow uses multiple model calls, evaluate both the final answer and important intermediate outputs. A prompt-chaining design can fail because one early step produces incomplete or incorrectly structured data.

3. Test a RAG application

Separate retrieval quality from answer quality. A correct answer cannot be expected when the relevant document was never retrieved, and a good retrieved passage can still be misinterpreted by the generation step.

• Test whether the correct source appears in the retrieved results.
• Measure whether retrieved passages contain enough information to answer the question.
• Check citations, source references, or document identifiers when your interface exposes them.
• Include questions with no answer in the knowledge base.
• Test conflicting, outdated, duplicate, and irrelevant documents.
• Verify that the assistant says when evidence is insufficient instead of filling gaps confidently.

For implementation details, compare the evaluation stage with the retrieval, chunking, and embedding choices described in the RAG AI assistant tutorial.

4. Test an agent or tool-using workflow

Evaluate the sequence, not just the final prose. Record whether the agent selected the correct tool, supplied valid arguments, handled tool errors, respected authorization boundaries, and stopped when the task was complete.

• Include cases that require one tool call, several calls, and no call.
• Test invalid, partial, and contradictory tool results.
• Verify that sensitive actions require the expected confirmation or permission.
• Set limits for loops, retries, and unnecessary tool calls.
• Inspect traces for hidden failures masked by a plausible final response.

5. Choose an evaluation method

Use more than one method for important workflows. Automated checks are excellent for repeatability; human review is valuable for validating the rubric and investigating disagreements. An evaluator model should be treated as an assistant to the evaluation process, not an unquestionable authority.

What to double-check

Define a scoring rubric

A rubric makes judgments more consistent. Use a small scale with observable descriptions rather than vague labels such as “good” or “bad.” For example:

• 3 — Pass: Correct, complete, grounded where required, and compliant with the requested format.
• 2 — Minor issue: Generally useful but missing a non-critical detail, using weak wording, or requiring a small correction.
• 1 — Major issue: Partly relevant but materially incomplete, unsupported, or incorrectly formatted.
• 0 — Fail: Incorrect, unsafe, irrelevant, fabricated, or unusable.

Define which failures are release blockers. A high average score should not hide one severe privacy, authorization, or factual error. Track pass rate, average score, critical-failure count, format-validity rate, and category-level results. For production planning, also record latency, token consumption, and tool-call counts where those affect the user experience or operating cost.

Make evaluator-model grading more reliable

Give the evaluator the task, response, relevant context, and rubric. Ask for structured output such as a score, failure category, and short rationale. Avoid asking it to judge criteria that are not represented in the input. Calibrate the evaluator against examples reviewed by people, and periodically sample its decisions for review.

Pair broad semantic grading with deterministic checks. For example, an evaluator can judge whether a summary preserves meaning, while a program checks whether the output is valid JSON and contains required keys.

Protect the test set

Do not tune prompts repeatedly against the same small set without a holdout. Otherwise, the feature may appear to improve while simply becoming specialized to familiar examples. Keep a private or less frequently inspected test subset, and add newly discovered failures to the regression set only after the current change has been assessed.

Common mistakes

• Testing only happy paths: Real applications encounter incomplete, ambiguous, long, and adversarial inputs.
• Using one metric: Better fluency does not necessarily mean better correctness or safety.
• Changing too many variables: Prompt, model, retrieval settings, and tools should be versioned so comparisons remain interpretable.
• Accepting a single reference answer: Open-ended tasks often have several valid responses. Evaluate required properties and facts instead.
• Trusting an evaluator without calibration: Automated grading can be inconsistent, especially when criteria are vague or evidence is missing.
• Ignoring intermediate steps: Agents and RAG systems may produce a plausible final response after a flawed retrieval or tool call.
• Reporting only averages: Segment results by intent, risk, language, input length, and failure type.
• Failing to test security behavior: Include prompt injection, unauthorized requests, and attempts to override instructions. Use the prompt injection prevention checklist as a separate review track.

When to revisit

Run the evaluation suite before releasing a prompt, model, retrieval, tool, or application change. Revisit it when user feedback reveals a new failure pattern, when the knowledge base changes substantially, when a tool schema changes, or when production behavior differs from development results.

At regular planning points, review whether the test set still represents the product. Remove obsolete cases only with a recorded reason; otherwise, retain them as historical regression checks. Add cases for new user intents, newly supported languages, changed output formats, and incidents that could recur.

Use this final release checklist:

1. Confirm that the test set covers common tasks, edge cases, and known failures.
2. Version the prompt, model, settings, context, tools, and evaluation code.
3. Run the baseline and candidate on the same cases.
4. Compare overall, category-level, and critical-failure results.
5. Review representative passes, failures, and evaluator disagreements.
6. Check structured output, grounding, refusal behavior, and tool traces.
7. Record latency and resource measures relevant to the feature.
8. Document the release decision, remaining risks, and next test cases to add.

A repeatable LLM evaluation workflow does not eliminate judgment. It makes judgment easier to apply consistently, gives developers evidence for prompt and model changes, and creates a clear record when an AI feature improves—or quietly regresses.