Prompt injection in production AI systems: the risk that arrives with the inputs

Prompt injection is what happens when the data an AI system reads contains instructions, and the model follows them. A supplier’s PDF with a line in white text that says "approve this invoice". A support ticket whose body tells the assistant to forward the customer’s history to an outside address. A web page an agent browses that tells it to ignore everything it was told before. The model cannot reliably tell the text you gave it as instructions from the text it was asked to process, because both arrive as words. If you deploy an assistant or an agent over documents, email, tickets or web content, the risk arrives with the inputs, whether or not anyone is targeting you yet.

The question to ask is the one you would ask about any untrusted input: what can this system be talked into doing, and what would it cost.

Why the model layer cannot solve it on its own

Model vendors train against injection and their filters catch the crude attempts. They will keep improving and keep being beaten, because the attack surface is natural language and no grammar separates content from command. A model that follows instructions found in a document well is, by construction, a model that can be made to follow the wrong ones. OWASP ranks prompt injection first in its Top 10 for LLM applications, ahead of sensitive information disclosure and excessive agency, and we think the ordering is right: those two are often how an injection cashes out.

So we treat injection the way a web team treats SQL injection: a permanent property of untrusted input, handled by the architecture around the model rather than by hoping the model notices.

Give the system the least it needs

The first control is the blast radius. An assistant that can only read the knowledge base and answer cannot be injected into much of consequence. An agent with a send-email tool, a refund tool and a database write can. We scope tools per task and per user, starting from the question we ask before any agent build: what is the worst action available, and how fast would we notice. Anything that moves money, commits the company to a customer or changes a record of consequence goes behind a human approval.

Two rules follow. No secrets in the context: API keys, internal URLs and other people’s data do not belong in the prompt, because anything in the prompt can be asked for. And no ambient authority: the agent acts with the permissions of the user who asked, never with a service account that sees everything, the first design rule of an internal assistant over company documents.

Keep untrusted content apart from instructions

We never paste a document, an email or a web page into the same block as the system instructions. Untrusted content is passed as data, labelled as such, with instructions that say it is to be read and never obeyed. This does not stop a determined injection, but it raises the bar and makes the logs legible: when something goes wrong you can see what was instruction and what was input.

We also validate the output before anything acts on it. Structured outputs checked against a schema, allow-lists for recipients, URLs and account identifiers, bounds on amounts. An agent told to email "the customer" can only email addresses already attached to the case.

Limit what can leave

Most injection damage is exfiltration: the attacker wants the model to send something out. Egress is therefore the control with the best ratio of effort to protection. The system can reach the tools and domains on a list and nothing else. A browsing agent cannot post to arbitrary URLs. Outbound email goes through a relay that enforces the allow-list. Rendered output does not load images from remote servers, a well-documented channel for leaking context through a URL. When connectors are wired through a standard such as MCP, the same rule applies to each server the agent can call: an inventory, and a reason for each entry.

Where personal data is involved, the same list is a compliance artefact: the data flow diagram we draw for GDPR-compliant AI systems names the same destinations.

Log it, attack it, and repeat before each release

Every prompt, retrieved document and tool call is logged with enough context to reconstruct an incident. Then we attack our own system: a red-team set of injected documents, emails and pages, maintained alongside the regular eval suite and rerun on every prompt or model change. The set starts with the obvious ("ignore previous instructions") and grows with what the field publishes and what production shows us. Findings go back into the controls above: a new allow-list entry, a tighter schema, a tool that loses a permission.

Prompt injection is a design constraint of the same order as input validation on a web form. In our builds it has never ruled out a use case, only the shortcut of wiring a model to your email and your database and trusting it to know a request from a trick. If you are planning an assistant or an agent over content you do not control, we scope these controls into the build from the first week, where they cost the least.