Insights / AI security
Prompt injection: a briefing for decision-makers
Prompt injection is the most common vulnerability in AI applications. It occurs whenever a model processes text from a source you do not control.
Hidden instruction detected and removed
Language models follow instructions, and they cannot reliably distinguish your instructions from instructions embedded in the content they process, such as an email, a web page, a PDF or a support ticket. When an attacker places instructions in that content and the model acts on them, the result is prompt injection. It ranks first in the OWASP Top 10 for LLM Applications.
Why the risk is growing
The impact depends on what the AI system is allowed to do. A chatbot that can only reply to users can be manipulated into giving wrong answers. An assistant that can read email, search files and send messages can be manipulated into leaking data. Security researcher Simon Willison calls the dangerous combination the “lethal trifecta”: access to private data, exposure to untrusted content, and the ability to communicate externally. A system with all three can be instructed by a single malicious email to send your data to an attacker.
Questions to ask before approving an AI tool
- What content can it read? Every source of outside text, including email, web pages and uploaded files, is a possible entry point.
- What actions can it take? Sending messages, moving files and calling APIs turn a manipulated response into a breach.
- Who approves high-risk actions? A person should confirm any action that sends data externally or cannot be reversed.
- Is activity logged? Investigating an incident requires a record of what the model read and what it did.
- Has your configuration been tested? Vendors test their models. Testing your specific deployment is your responsibility.
The reliable defense is design.
Mitigation
No current technique eliminates prompt injection. Input filtering and improved models reduce the risk without removing it. Effective mitigation is architectural: limit what each AI tool can access, avoid combining all three elements of the trifecta, require approval for consequential actions, and test the deployment as an attacker would. Our AI red teaming and AI Security & Governance services cover each of these steps.