Stop Prompt Injection
in Agentic AI

Prompt injection has been the top item on the OWASP Top 10 for LLM Applications since the list existed, and it has not been dethroned in the agentic version either. What changes in agentic systems is the blast radius. A prompt injection against a chatbot produces a bad sentence. A prompt injection against an agent produces an action: a tool call, a database write, a wire transfer, a message sent on behalf of the user, an instruction handed off to another agent. The defense problem is no longer "stop the model from saying something wrong." It is "stop the system from doing something wrong, even when the model has been talked into wanting to." That reframing is what most teams shipping agents today have not yet done. For the broader risk taxonomy these defenses map to, the OWASP Top 10 for Agentic Applications remains the canonical reference.

Why prompt injection breaks differently in agentic systems

The original framing of prompt injection, articulated by Simon Willison in 2022, was about user inputs overriding system prompts in a single-turn LLM. Two years and a generation of agents later, the dominant failure mode is no longer the user trying to jailbreak the chatbot — it is untrusted content arriving through retrieval, tools, or other agents that the model treats as authoritative because it cannot reliably distinguish trusted from untrusted text in its own context window. Greshake et al.'s "Not what you've signed up for" formalized this as indirect prompt injection in 2023, and every credible threat model since has treated it as the central agentic risk. The reason it scales worse than direct injection is mechanical: every new retrieval source, tool, and agent peer is another untrusted input channel, and the agent's action space grows with each one.

The persistence of prompt injection at the top of the ranking is not because mitigations have failed — it is because the underlying problem is architectural. LLMs do not have a structural mechanism for separating instruction from data in the way operating systems separate code from input. Until that distinction is enforced outside the model — at the runtime layer — every new context channel an agent gains is a new injection vector.

The four entry points: where injection actually enters an agent

Why filters, system prompts, and "model alignment" cannot solve this

The most common defensive moves teams reach for first are the weakest. Input filtering — pattern-matching for suspicious strings like "ignore previous instructions" — degrades into a whack-a-mole exercise within days, because adversaries iterate faster than filters do and the space of natural-language attacks is unbounded. System prompts that instruct the model to "ignore any injected instructions" rely on the model being able to distinguish injected from legitimate instructions in its own context, which is precisely the capability the threat model assumes it lacks. Model-level alignment improves the floor on direct injection but provides no architectural guarantee against the indirect case, because the model's job is to read and act on the content in its context — including the content that was placed there by an attacker.

Microsoft's Prompt Shields documentation is explicit about this limitation: content-level classifiers are one defense layer, but they are not sufficient on their own and must be paired with action-level controls. Anthropic's research on Constitutional AI takes a similar position — model-side defenses raise the cost of attack but do not constitute a security boundary on their own.

Provenance and trust boundaries: the architectural fix

The defensible architecture treats every piece of text in an agent's context as having a provenance label — user, retrieved document, tool A, tool B, agent peer — and uses that label to gate what the content is allowed to influence. A document retrieved from an untrusted source can inform the agent's response but cannot trigger a privileged tool call. A tool output from an external endpoint can be summarized but cannot redirect the agent to a new task. The model still gets to reason over everything in its context; the runtime layer decides which reasoning paths can become real-world actions. This is the agentic equivalent of the same-origin policy that made the modern web tractable — content from one trust domain cannot exercise the privileges of another.

Building this requires three things that most agent frameworks do not provide by default: content provenance tracking through the entire pipeline (not just at the entry point), explicit trust tiers for tools and data sources, and a policy layer that knows how to evaluate "action X requested by content from source Y" rather than just "action X." For a fuller picture of how this layer fits into the broader control stack, see our piece on runtime security for AI agents.

Runtime enforcement of injection-aware policies

The control that actually stops a successful injection from becoming a successful attack is action-level enforcement at runtime — the same execution-time decision function described in Microsoft's runtime authorization model. The model can be persuaded to want to do anything; the runtime decides what it is allowed to do, scoped by the provenance of the content that motivated the action and the privilege tier of the tool being invoked.

The practical implication is that even if an attacker successfully embeds a "transfer funds to X" instruction in a retrieved web page, the runtime sees a privileged tool call originating from untrusted-tier content and denies it — or escalates it for human review under the tiered oversight model described in our guardrails piece. The model was hijacked. The system was not.

The dual-LLM (planner-executor) pattern

For high-risk agents where the cost of a successful injection is unacceptable, the strongest pattern available today is the dual-LLM architecture popularized by Simon Willison's dual-LLM proposal and used in production by several agent platforms. A "planner" LLM reads untrusted content and produces a structured plan in a constrained schema. An "executor" LLM (or a deterministic policy engine) takes only the structured plan — never the raw untrusted text — and decides which tool calls to make. The executor is never exposed to attacker-controlled prose, so it cannot be talked into anything. The planner can be hijacked; its hijack just doesn't propagate.

This pattern trades some capability for a substantial reduction in attack surface, and it is increasingly the default in regulated deployments. Google's CaMeL research describes a closely related approach using capability-based access control between planner and executor, with formal guarantees that untrusted content cannot influence tool selection.

Detection: knowing an injection happened

Prevention is not perfect, and detection is the second line of defense. Effective detection for prompt injection in agentic systems combines four signals: provenance-tagged audit logs that show which content motivated which action, anomaly detection on action sequences (an agent that suddenly attempts an out-of-pattern tool call after reading a new document is a signal), output-side classifiers that catch suspicious agent outputs before they propagate, and human review queues for any action originating from untrusted-tier content above a risk threshold. The NIST AI Risk Management Framework's Measure and Manage functions both require this kind of continuous observability — not as a compliance artifact, but as the operational substrate that makes incident response possible.

Without this layer, prompt injection incidents are detected by their downstream consequences — a wrong wire transfer, an exfiltrated record, a complaint from a customer — rather than by the security team. By that point, the cost of the incident is locked in.

Frequently Asked Questions