Prompt Injection Defense for AI Apps: A Practical Security Checklist and Test Plan

Overview

Prompt injection occurs when text supplied to an AI application attempts to alter the model’s behavior in a way the application did not intend. The text may come from a user, a retrieved document, an email, a web page, a file, a tool response, or another model. It can look like an ordinary request, hidden instruction, misleading document passage, or request to reveal internal context.

The practical security question is not whether a model can always distinguish instructions from data. Models interpret language, so boundaries expressed only in natural language may be unreliable. A safer design treats the model as one component inside a larger control system. The application should decide which inputs are trusted, which actions are permitted, what data is available, and when a person must approve an operation.

Use the following layers together:

• Input separation: identify system rules, developer instructions, user requests, retrieved content, and tool results as different classes of data.
• Retrieval boundaries: control which documents can be retrieved and avoid treating retrieved text as authority over application behavior.
• Tool permissions: expose only the operations required for the current task, with strict schemas and server-side authorization.
• Output checks: validate structured responses and inspect sensitive or high-impact results before using them.
• Observability and review: record relevant decisions, failures, and approvals without storing secrets unnecessarily.

For implementation context, pair this checklist with a production guide for LLM tool calling, validation, retries, and human approval. Prompt design remains useful, but it should support these controls rather than replace them.

What to track

1. Trust boundaries and data sources

Maintain an inventory of every place text enters the application. Include chat messages, uploaded files, retrieved passages, web content, CRM records, email threads, tool responses, and model-generated intermediate results. For each source, record whether it is trusted, untrusted, or conditionally trusted.

Then document the intended role of each input. A customer message may describe a task but should not be allowed to redefine authorization rules. A retrieved policy document may provide facts, but it should not automatically gain permission to call a tool. A tool response may contain data, but it can still include content that attempts to redirect the model.

Track changes to these boundaries in code review. Adding a new connector or retrieval source should trigger a security review even if the visible prompt remains unchanged.

2. Instructions and prompt structure

Keep system and developer instructions concise, explicit, and focused on the application’s job. State what the model may do, what it must not do, and how it should handle conflicts. Tell it to treat retrieved and user-provided text as content to analyze rather than instructions to follow, while recognizing that this wording is only one layer of defense.

Use clear delimiters and labels for different content types. Avoid placing secrets, credentials, private keys, or unnecessary internal configuration in prompts. Review whether each instruction is still needed, because obsolete rules make failures harder to diagnose.

If your application uses a system prompt, see this practical guide to writing system prompts for reliable AI assistants. The same review principles apply to agent instructions and prompt chains.

3. Retrieval and RAG controls

For a retrieval-augmented generation application, track document ownership, ingestion paths, access-control metadata, freshness, and deletion behavior. Retrieval should respect the user’s authorization before passages reach the model. Filters applied only after generation are not an adequate substitute for controlling what the model can see.

Test documents that contain instructions such as “ignore previous rules,” requests for hidden context, or attempts to trigger external actions. The expected result should be that the assistant treats those passages as untrusted material and continues following application policy. A RAG tutorial can help with the retrieval pipeline, but security decisions still belong in application code and access-control systems.

4. Tool permissions and side effects

List every tool available to the model and classify its impact. Reading a public record is different from sending a message, changing a database row, issuing a refund, deleting a file, or modifying access. Use the narrowest possible tool scope, validate arguments on the server, and re-check the user’s authorization at execution time.

Prefer explicit confirmation for consequential actions. A model can prepare an email or draft a database update without being allowed to send or commit it. Rate limits, idempotency controls, allowlists, and transaction boundaries reduce the cost of mistakes and repeated calls.

5. Outputs, logs, and red-team cases

Define what a safe output looks like before testing. For structured responses, validate the schema, types, permitted values, and references. For natural-language responses, check for accidental disclosure, unsupported claims, unsafe instructions, and attempted policy overrides.

Keep a test set of direct and indirect injection cases. Include requests to reveal system instructions, follow malicious retrieved text, misuse tools, bypass authorization, expose private data, or continue after a refusal. Record the input source, model and prompt version, retrieved documents, tool calls, final result, and whether a human intervened. Redact sensitive content in logs and set a retention period appropriate to the application.

Use a repeatable evaluation set rather than relying on a few manual demonstrations. The LLM evaluation guide provides a useful structure for maintaining regression tests as prompts and workflows change.

Cadence and checkpoints

A prompt injection review should happen at several points, not only during the initial build. Use a lightweight monthly or quarterly checkpoint for stable systems, and a release-gated review for material changes.

• Before launch: map data flows, identify tools, define approval points, establish baseline attack cases, and confirm that authorization is enforced outside the model.
• Before each release: rerun the injection test set, compare refusals and tool behavior with the previous version, and inspect changes in retrieval or output validation.
• When adding a data source: review ownership, access controls, document trust assumptions, ingestion transformations, and indirect-injection cases.
• When changing a tool: reassess permissions, argument validation, side effects, confirmation requirements, retries, and failure handling.
• Monthly or quarterly: sample logs, review blocked and successful tool calls, remove obsolete tests, add incidents to the regression set, and verify that owners still understand the escalation path.

Maintain a simple change record with the application version, model configuration, system-prompt version, retrieval configuration, tool inventory, test results, open risks, and reviewer. This makes security decisions easier to reproduce when behavior changes later.

How to interpret changes

Do not reduce prompt injection defense to a single pass rate. A lower failure count may reflect a narrower test set, a less capable workflow, or a model that refuses more requests without completing legitimate tasks. Review security and utility together.

Classify findings by impact and repeatability. A response that reveals harmless formatting instructions is different from one that exposes private context. A failed attempt to draft a tool call is different from an unauthorized external action. Prioritize issues that cross a trust boundary, disclose sensitive data, bypass authorization, or create an irreversible side effect.

When a result changes, compare the full execution path: input normalization, retrieved context, prompt assembly, model response, validation, authorization, tool execution, and user confirmation. This prevents teams from “fixing” the visible answer while leaving the underlying permission problem intact.

Also watch for false positives. A security filter that blocks ordinary technical content can encourage users to bypass the application or cause developers to disable controls. Adjust the boundary, tool design, or workflow where appropriate instead of adding increasingly broad prompt instructions.

When to revisit

Revisit this checklist before every production release that changes prompts, models, retrieval settings, tools, permissions, or output handling. Repeat it immediately after an incident, unexpected tool call, data-source change, or material change in the assistant’s scope.

At the recurring monthly or quarterly review, start with the inventory rather than the wording of the prompt. Confirm that every input source has an owner, every tool has a defined permission boundary, every high-impact action has an approval path, and every important failure is represented in the regression set. Retire tests only when the associated behavior is no longer possible or relevant, and document why.

A reusable red-team test template can use this format:

1. Scenario: describe the user goal and the untrusted content source.
2. Attack: provide the injection text or behavior being tested.
3. Expected control: state whether the application should refuse, ignore the instruction, ask for confirmation, or continue safely.
4. Observed path: record retrieved context, model output, validation, tool calls, and approvals.
5. Impact: classify disclosure, unauthorized access, unsafe output, or side effect.
6. Decision: fix, accept with documented rationale, or add a compensating control.

Use the same template whenever you build an AI assistant, extend a RAG workflow, or introduce automation. Secure AI applications are maintained through repeated boundary checks, constrained permissions, meaningful tests, and clear human ownership—not through a prompt that is assumed to be impossible to override.