Design Prompt Injection Defenses for an AI Agent
Contain instructions hidden in untrusted content before an agent can use sensitive data or invoke side-effecting tools.
Problem statement
Design the safety boundary for an AI agent that reads untrusted retrieved documents and can invoke a small set of user-authorized tools.
For this exercise, assume documents may contain adversarial text that asks the agent to ignore policy, reveal private data, or take an unrelated action.
For this exercise, assume read-only search is low risk, while sending messages, changing records, or exporting data requires explicit policy authorization and sometimes user approval.
For this exercise, assume 1,500 active agent runs per second, up to 20 tool proposals per run, and a p99 policy-decision budget of 100 ms per proposal. For this exercise, assume model training, malware scanning, identity-provider design, and proving that all natural-language attacks can be detected are out of scope.
What to build
- Label untrusted content as data and keep it from changing system policy, user identity, or tool permissions.
- Validate every proposed tool call against the authenticated user, approved capability, resource scope, arguments, and current policy.
- Pause high-impact actions for a clear user approval that is bound to the exact normalized action.
- Record policy decisions and contain a compromised run without exposing secret values in model context or logs.
At scale
- For this exercise, assume 1,500 active agent runs per second and up to 20 tool proposals per run.
- For this exercise, target p99 policy decisions under 100 ms per tool proposal.
- For this exercise, policy or approval infrastructure failure must prevent side-effecting tool execution.
- For this exercise, retain tamper-evident decision metadata for 90 days without retaining raw secrets in the audit record.