Global site search

Search guides, labs, glossary, and research

Type two or more characters to search.

Published research

The Anatomy of Indirect Prompt Injection

Threat models, retrieval, concealment, action amplification, instruction hierarchy, dual-model isolation, and execution monitoring.

Indirect influence ≈ 23 min read 41.2 KB source Download raw Markdown

This local reader uses a conservative, dependency-free Markdown renderer. Citation markers from the original report are preserved as source pills; equations and Mermaid diagrams remain text. Use the raw Markdown for exact source fidelity.

The Architectural Paradigm Shift and the Emergence of Indirect Threats

The rapid integration of Large Language Models (LLMs) into enterprise workflows has fundamentally altered the trajectory of artificial intelligence. Models have evolved from isolated, conversational text generators into autonomous, agentic systems capable of executing complex workflows. Through the adoption of paradigms such as Retrieval-Augmented Generation (RAG) and tool-calling architectures, modern AI assistants possess the capability to browse the web, ingest corporate documents, parse emails, and execute external Application Programming Interface (API) calls1. This expansion of capabilities, however, has introduced a severe and structurally inherent vulnerability to the ecosystem: the indirect prompt injection (IPI) attack4. Indirect prompt injection occurs when adversarial instructions are embedded within untrusted, third-party content—such as a webpage, an email, a calendar invite, or a PDF—and are subsequently consumed by an LLM during its routine data retrieval operations3. Unlike direct prompt injection or jailbreaking, where a malicious user actively submits adversarial queries to bypass safety filters, indirect prompt injection shifts the attack vector to the environment itself7. The adversary does not require direct access to the model's interface; they merely need to plant a payload in a location where the AI is likely to retrieve it2. When the LLM ingests this poisoned external data, it processes the embedded instructions alongside the legitimate user's query and the developer's foundational system instructions. Because LLMs operate on continuous streams of tokens and inherently struggle to differentiate between informational context and executable commands, the model may execute the malicious payload5. This dynamic forces the AI agent into the role of a "confused deputy," leveraging its elevated privileges and tool access to execute the attacker's objectives under the mistaken assumption that it is fulfilling a valid, high-priority instruction11. The consequences range from ecosystem contamination and reputation manipulation to the catastrophic exfiltration of sensitive corporate data2.

The Fundamental Conflict: Semantic Processing vs. Syntactic Isolation

To comprehend the pervasiveness of indirect prompt injection, one must analyze the foundational architecture of contemporary language models and contrast it with traditional computing paradigms. In classical software engineering, security relies on a strict, hardware-enforced or protocol-enforced separation between instructions (the control plane) and data (the data plane). Vulnerabilities like SQL injection are mitigated through parameterized queries, which syntactically isolate untrusted user input from executable database commands2. LLMs, conversely, lack this syntactic isolation. They parse and generate natural language using attention mechanisms where the boundaries between system rules, user requests, and retrieved data are purely semantic and highly porous2. When an LLM processes a combined context window, a conflict arises among three distinct entities competing for the model's attention. The system developer sets the overarching behavioral guardrails, attempting to enforce policies such as data privacy and ethical alignment11. The end-user dictates the immediate task, such as requesting a summary of recent emails. Finally, the external content provides the raw data, which may harbor an adversarial payload instructing the model to ignore previous instructions and forward passwords to an external server11. Because the LLM relies on stochastic attention weights to determine the relevance of tokens, a strongly worded, imperative command embedded in the retrieved text can overwhelmingly capture the model's attention9. The model subsequently discards its original developer and user objectives, succumbing to a state of instruction hijack. This represents a fundamental structural limitation of transformer-based architectures operating on boundary-less token streams10.

Empirical Prevalence and the Scale of the Threat

While indirect prompt injection was initially theorized as a conceptual vulnerability, recent large-scale empirical studies have demonstrated that the threat is pervasive, highly developed, and actively deployed in the wild. The ecosystem of poisoned external content extends far beyond isolated academic demonstrations, encompassing a heterogeneous network of disruptive prompts, reputation manipulation schemes, and AI-bot detection mechanisms6. An extensive empirical analysis examining 1.2 billion URLs across 24.8 million hosts identified tens of thousands of validated indirect prompt injection instances actively embedded in the web ecosystem6. The data reveals a growing source of tension between LLM-driven automation and the website owners whose content is consumed. Site owners, contributors, and malicious adversaries alike are actively utilizing recurring templates to manipulate downstream model behavior6. The scale of model vulnerability has been further quantified through large-scale public red-teaming competitions. In a comprehensive evaluation spanning 41 behavioral scenarios, researchers analyzed over 270,000 attack attempts submitted against frontier models16. The findings indicate that vulnerability is a universal characteristic of modern LLMs, independent of their overall reasoning capabilities. The following table synthesizes the findings from these large-scale empirical evaluations, illustrating the breadth and nature of the threat landscape.

Empirical Study FocusScale of AnalysisKey Findings and MetricsPrimary Objectives Observed
Web Ecosystem Prevalence \[cite: 6\]1.2 billion URLs across 24.8 million hosts.15.3K validated injection instances across 11.7K distinct webpages. Compliance rates reach up to 8% for smaller models on plain-text inputs.AI-bot detection, reputation manipulation, content-protection directives, disruptive prompts targeting crawlers and search pipelines.
Model Vulnerability (Red Teaming) \[cite: 16\]272,000 attack attempts by 464 participants against 13 frontier models.8,648 successful attacks. Universal vulnerability across models. Attack Success Rates (ASR) ranged from 0.5% (Claude Opus) to 8.5% (Gemini 2.5 Pro).Tool calling hijack, malicious coding instructions, and unauthorized computer use. Weak correlation between model capability and robustness.

The empirical data demonstrates that high capability does not equate to high security; in fact, models exhibiting superior reasoning and tool-calling capabilities often demonstrate higher vulnerability to complex, multi-step indirect injections, as they are more adept at following the nuanced, adversarial instructions embedded in the external content16.

The Lifecycle of an Indirect Prompt Injection Attack

For an indirect prompt injection to successfully compromise an LLM-integrated system, the adversary must meticulously engineer the payload to survive three distinct operational phases: retrieval, concealment, and execution1. The failure of the payload at any of these stages neutralizes the attack.

Phase 1: Overcoming the Retrieval Barrier

The most sophisticated adversarial payload is entirely inert if the LLM's retrieval architecture never selects it for ingestion. In production RAG systems, user queries are converted into dense vector embeddings, and a similarity search is performed against a database of document chunks. Unoptimized indirect prompt injections are rarely retrieved under natural user queries because the semantic meaning of the malicious instruction (e.g., "exfiltrate data") does not match the semantic intent of the user's query (e.g., "summarize my quarterly earnings")1. To overcome this retrieval barrier, advanced adversaries decompose their payloads into two distinct elements. The first element is a "trigger fragment," explicitly engineered to guarantee a high cosine similarity score with anticipated target queries1. The second element is the "attack fragment," which encodes the arbitrary malicious objectives. By leveraging black-box optimization algorithms against public embedding APIs, attackers can synthesize highly compact trigger fragments that append seamlessly to the attack fragment. Research indicates that this optimization is highly cost-efficient, requiring as little as $0.21 per target query, while achieving near-100% retrieval success across a wide array of proprietary and open-source embedding models1. Once the optimized payload is planted in a shared corpus—such as an enterprise SharePoint drive or a public code repository—the RAG system is statistically guaranteed to retrieve it when related queries are issued.

Phase 2: Payload Concealment and Persistence

Because the output of the RAG system and the final response of the agent are often visible to human users, the injection must be heavily concealed to prevent detection and manual intervention16. If a human operator observes raw malicious instructions rendering on a webpage or within a document summary, the operation is compromised. Consequently, concealment is a critical dimension of the threat16. Attackers rely on two primary strategies to maintain invisibility while remaining highly legible to the automated parsers feeding the LLM. The predominant strategy, accounting for approximately 70% of wild instances, involves hiding the payload by construction6. Adversaries place the instructions within non-rendered HTML elements, such as HTTP response headers, document metadata fields, or standard HTML comment tags. While web browsers omit these elements during visual rendering for human users, headless crawlers and LLM ingestion tools parse the raw document source, seamlessly delivering the instructions into the model's context window6. When payloads must be placed within the visible body of a document, attackers utilize UI-rendered concealment techniques. These include font color manipulation—matching the text color identically to the background color—or manipulating font sizes and opacity to render the text imperceptible6. Furthermore, these payloads are frequently positioned at the absolute beginning of the document structure. This strategic placement ensures that the LLM processes the adversarial instructions early in its inference cycle, effectively overriding any downstream structural cues or safety filters before they can be evaluated6.

Phase 3: Execution and Action Amplification

The final phase of the attack lifecycle depends entirely on the privileges granted to the targeted LLM agent. If the model is restricted to a read-only conversational interface, the execution is limited to output manipulation. The attacker may force the model to generate factually incorrect summaries, fabricate scientific citations, disseminate political misinformation, or engage in targeted defamation disguised as objective reporting2. However, the threat escalates exponentially when the LLM possesses action amplification capabilities through API integrations and tool usage4. Modern assistants are designed to read sensitive emails, query private databases, and write external communications. When a tool-enabled agent processes a concealed attack fragment, the payload coerces the model into generating specific tool-call JSON structures4. The agent, acting on behalf of the authenticated user, executes the unauthorized API calls without triggering standard network anomaly detection, as the traffic originates from a trusted, authorized service account7.

Taxonomy of Threats and Systemic Impact

The integration of LLMs blurs the traditional boundaries between passive data and executable code, exposing systems to a wide array of security risks2. The impact of successful indirect prompt injections can be categorized into a comprehensive taxonomy of threats, each leveraging the unique reasoning and generative capabilities of the model. Data exfiltration and information gathering represent the most immediate threat to enterprise security. An attacker can instruct the compromised agent to query private data stores, read the user's historical conversations, extract sensitive credentials, or compile personal identifiable information2. Once the data is aggregated in the model's memory, the agent autonomously formats it and transmits it to an attacker-controlled server, often bypassing content filters by encoding the data into URL parameters or obfuscated text blocks2. Fraud and sophisticated social engineering constitute another severe impact category. Because the compromised LLM possesses intimate access to the user's communication history, writing style, and ongoing professional context, it can be weaponized to autonomously craft highly persuasive, personalized phishing campaigns2. The model can distribute convincing scam messages and fraudulent promotions that appear entirely legitimate to the recipients, leveraging its natural language generation capabilities to create compelling deceptions that traditional spam filters fail to detect2. Perhaps the most insidious theoretical threat introduced by IPI is the concept of malware distribution via self-propagating AI worms2. In this scenario, the initial payload instructs the LLM not only to execute a malicious action but also to embed a copy of the prompt injection instructions into any new content it generates. For instance, if the agent is tasked with summarizing a poisoned document and emailing the summary to a colleague, the agent stealthily appends the injection payload into the outgoing email. When the colleague's AI assistant reads that email, it becomes compromised, executes the payload, and subsequently forwards the infection to further contacts2. This creates an autonomous, ecosystem-wide contamination that spreads laterally through LLM-integrated applications without requiring traditional software vulnerabilities.

Case Study in Enterprise Exploitation: EchoLeak (CVE-2025-32711)

The theoretical impact of indirect prompt injection materialized into a critical enterprise threat with the disclosure of EchoLeak (CVE-2025-32711), a vulnerability affecting Microsoft 365 Copilot17. Carrying a CVSS score of 9.3, EchoLeak represents the first widely reported zero-click indirect prompt injection attack against a production GenAI productivity tool17. The exploit allowed remote, unauthenticated attackers to exfiltrate highly sensitive corporate data—including SharePoint files, Teams chats, and private emails—without requiring any user interaction beyond the receipt of a crafted email12. EchoLeak did not rely on a single flaw; rather, it succeeded by chaining together multiple bypasses against state-of-the-art AI security guardrails and traditional web security policies21. The attack initiated when a benign-appearing email containing a hidden payload was delivered to a user's Outlook inbox. When the user engaged in normal Copilot interactions, the RAG engine retrieved the email, pulling the invisible payload into the LLM's context window12. The following table details the sequential bypass mechanisms utilized in the EchoLeak exploit chain, demonstrating the profound complexity of defending against AI-native threats.

Exploit PhaseEnterprise Security Control TargetedAttacker Bypass MechanismOutcome
Evasion of AI FiltersCross Prompt Injection Attempt (XPIA) ClassifierAvoided explicit trigger phrases (e.g., "ignore previous instructions") by utilizing natural, benign business phrasing.The payload successfully evaded the AI firewall, tricking the LLM into initiating data compilation without triggering safety alerts12.
Output Redaction EvasionMarkdown Inline Link Redaction FilterInstructed the LLM to format the outbound data using reference-style Markdown links (e.g., \[text\]\[1\] combined with \[1\]: url?data=\<secret\>) rather than standard inline links.The output filter failed to recognize the reference syntax, allowing the malicious URL containing the exfiltrated data to be generated in the chat history21.
Zero-Click ExecutionUser Click Interaction RequirementsCommanded the LLM to format the reference link as an image tag (e.g., \!\[alt\]\[1\]).Upon rendering the chat, the client UI automatically attempted to fetch the image, triggering a zero-click HTTP GET request that silently exfiltrated the stolen data22.
Network Egress EvasionBrowser Content Security Policy (CSP)Leveraged an explicitly trusted Microsoft Teams asynchronous preview API to proxy the outbound image request.The request was routed through a highly trusted domain, fully bypassing CSP restrictions designed to block unrecognized external servers12.

EchoLeak illustrates how attackers weaponize the very way large language models interpret text, executing an "LLM scope violation" where the AI is coerced into breaching its trust boundaries17. The attack's stealth is amplified by its ability to instruct the model to never explicitly reference the malicious email in its output, rendering the data exfiltration practically invisible to the end user and difficult for incident response teams to trace using conventional telemetry12.

Modern Defense Mechanisms and Mitigation Architectures

The complexity of securing non-deterministic models operating on natural language has necessitated the rapid development of specialized AI defense architectures. Recognizing that no single solution is foolproof against adaptive adversaries, the industry has adopted a layered, defense-in-depth approach spanning prompt engineering, architectural isolation, cryptographic validation, and execution monitoring25.

Prompt Engineering and Spotlighting Mitigations

The most immediate line of defense involves prompt engineering techniques designed to help the LLM visually and semantically distinguish between trusted system instructions and untrusted external data27. These techniques, collectively referred to as "Spotlighting," focus on modifying the input structure and system prompt without altering the underlying model weights27. Delimiting is the foundational spotlighting technique, wherein the untrusted text is enclosed within predefined boundary tokens (e.g., \<\<\< data \>\>\>). The system prompt explicitly instructs the LLM to disregard any commands located within these boundaries10. However, because attackers can easily guess or brute-force standard delimiters, researchers developed Datamarking. This technique interleaves a special token continuously throughout the untrusted input. For instance, replacing every whitespace character with a unique symbol like a caret (^), transforming the input into a continuous string of marked text. The system prompt is then informed that any text containing this marker is untrusted and must be summarized but never executed as an instruction27. The most robust spotlighting variant is Encoding, which leverages the LLM's inherent understanding of translation algorithms. The untrusted input is transformed using algorithms such as Base64 or ROT13. The system prompt instructs the model to decode the text in memory to answer questions about it, but strictly ignore any embedded commands. This one-way transformation disrupts the syntactical structure of the injection, making it highly difficult for an attacker to craft a payload that survives the encoding process while retaining its imperative force27. While spotlighting techniques successfully reduce baseline attack success rates from over 50% to under 2% against static attacks, rigorous red-teaming has revealed their fragility against determined adversaries10. Studies utilizing adaptive, search-based attacks, reinforcement learning, and gradient optimization have demonstrated that adversaries can consistently bypass spotlighting defenses, achieving success rates exceeding 95% against these boundaries28. Consequently, spotlighting is considered a necessary foundational layer but critically insufficient as a standalone defense.

The Instruction Hierarchy and Privilege Prioritization

To address the root cause of instruction conflicts, researchers have introduced the "Instruction Hierarchy," an architectural paradigm that explicitly models the input channels of an LLM and enforces a rigid priority ordering29. Modern LLMs receive input from multiple sources, but without explicit training, they treat a command from a website with the same obedience as a command from the application developer14. The instruction hierarchy resolves this by defining a strict trust-ordered policy. System messages provided by the developer are assigned the highest priority (Priority 0 \- Critical). User messages are assigned secondary priority (Priority 10 \- High). Finally, text retrieved from tools, web browsing, or documents is assigned the lowest priority (Priority 30 \- Low)14. When conflicts arise in the context window, the model is expected to maintain a running feasible set of responses, rejecting lower-tier instructions that attempt to override, ignore, or conflict with higher-tier directives14. Because LLMs do not natively enforce this priority queue, they require extensive fine-tuning. Researchers utilize synthetic data generation, context distillation, and Reinforcement Learning from Human Feedback (RLHF) to train models on highly nuanced conflict scenarios30. In these datasets, aligned instructions (which adhere to the system prompt) must be followed, while misaligned instructions (which contradict the system prompt) must be ignored or explicitly refused14. The introduction of the IH-Challenge dataset and the subsequent fine-tuning of frontier models like GPT-5-Mini demonstrate the efficacy of this approach. Training with online adversarial example generation improved instruction hierarchy robustness by an average of 10% across diverse benchmarks, reducing unsafe behavior from 6.6% to 0.7% while maintaining high benign utility31. Comprehensive evaluation suites like IHEval, containing thousands of examples of instruction alignments and conflicts, are now utilized to quantify a model's adherence to these priorities, revealing that even competitive models require targeted optimization to surpass a 48% accuracy rate in resolving complex conflicts32.

Architectural Isolation: The Dual LLM Pattern

Rather than relying entirely on a single model's ability to navigate complex instruction conflicts, architectural defenses advocate for strict privilege separation. The most prominent implementation of this concept is the Dual LLM Pattern11. This framework bifurcates the agent's cognition into two distinct models. The Quarantined LLM is tasked exclusively with processing untrusted external data, such as reading and summarizing incoming emails or web pages. Crucially, the Quarantined LLM is entirely stripped of tool access and APIs11. If it processes a malicious payload and succumbs to prompt injection, it is trapped within a powerless sandbox. The Privileged LLM acts as the central controller and tool orchestrator. It possesses the authority to execute API calls, modify system state, and interact with the user. However, the Privileged LLM is structurally prohibited from directly reading raw untrusted data11. Instead, the Quarantined LLM converts the untrusted text into strictly typed values or symbolic variables (e.g., $VAR1). The Privileged LLM manipulates these opaque handles, ensuring that raw, potentially poisoned text never enters its high-authority reasoning paths33. Advanced iterations of this concept include the Code-Then-Execute pattern and the Context-Minimization pattern. In the Code-Then-Execute approach, the privileged model generates code in a custom sandboxed Domain Specific Language (DSL) that specifies how tool outputs should be passed between agents, enabling deterministic data flow analysis that tracks tainted data throughout the entire process34. Context-Minimization further reduces risk by stripping natural language from external interactions, forcing the quarantined model to convert untrusted data into strictly formatted API descriptions with rigid character limits, ensuring that complex prompt injections cannot survive the formatting process34.

Execution-Centric Validation and Masked Re-Execution (MELON)

Recognizing that sophisticated adversaries continuously innovate to bypass prompt-level and architectural defenses, researchers have developed execution-centric validation systems. These systems treat attacks as trajectory deviations, monitoring the agent's behavior in real-time to detect the distinct operational signatures of a hijacked model3. A premier methodology in this space is MELON (Masked re-Execution and TooL comparisON). The design of MELON is grounded in a profound behavioral insight: state collapse. When an agent is successfully compromised by an indirect prompt injection, its subsequent actions become conditionally independent of the legitimate user's original request, driven entirely by the malicious tasks embedded in the retrieved data35. MELON detects this state collapse by running two parallel execution threads. The primary thread processes the agent's trajectory normally, incorporating both the user prompt and the retrieved data. Concurrently, a second thread executes a "masked" trajectory, where the user's original request is mathematically masked or entirely removed using a specialized masking function, leaving only the retrieved external data in the context window35. The system then evaluates the similarity of the tool calls generated by both executions. If the agent attempts to execute identical or highly similar tool calls in both the original and the masked execution, it provides deterministic proof that the agent is acting solely upon the instructions hidden in the external data, entirely ignoring the user's intent. Upon detecting this deviation, the system halts execution35. By incorporating prompt augmentation, the MELON-Aug configuration achieves extraordinary results on rigorous benchmarks like AgentDojo, reducing the attack success rate to 0.32% while maintaining a 68.72% utility rate on benign tasks using frontier models like GPT-4o36. Other advanced execution monitors include Rennervate, which analyzes the internal mechanics of the model itself. Rennervate utilizes a two-step attentive pooling mechanism to monitor attention features at a fine-grained, token level15. By analyzing how attention heads aggregate across response tokens, the system can detect the covert influence of indirect injections and implement precise sanitization, neutralizing the attack while preserving the LLM's functionality. Extensive experiments across multiple datasets and LLMs have proven Rennervate highly effective against adaptive adversaries15. The following table provides a comparative analysis of these modern defense architectures, highlighting their operational mechanisms and inherent tradeoffs.

Defense ArchitectureOperational MechanismKey StrengthsLimitations and Tradeoffs
Spotlighting (Datamarking/Encoding) \[cite: 27, 28\]Modifies input text with continuous tokens or encoding (e.g., Base64) to visually distinguish untrusted data.Easy to implement via prompt engineering. Low inference overhead. Highly effective against static attacks.Highly vulnerable to adaptive adversaries using gradient optimization or reinforcement learning.
Instruction Hierarchy \[cite: 30, 31\]Enforces a rigid priority queue (System \> User \> Tool) through extensive RLHF and synthetic data fine-tuning.Targets the root cause of instruction conflicts. Generalizes well to unseen attacks and improves overall model safety.Requires expensive model retraining and continuous dataset updates. Can lead to over-refusal of benign user requests.
Dual LLM Pattern \[cite: 11, 33, 34\]Architecturally separates reasoning into a Privileged model (tool access) and a Quarantined model (data parsing) using symbolic variables.Establishes a deterministic trust boundary. Prevents raw poisoned text from reaching the execution layer.Significantly increases architectural complexity, inference latency, and API costs. Harder to debug multi-agent interactions.
Masked Re-Execution (MELON) \[cite: 35, 36, 37\]Compares tool calls between a normal execution and a masked execution (user prompt removed) to detect state collapse.Achieves near-zero ASR (\~0.32%) without requiring model retraining. Preserves high benign utility.Doubles the computational overhead and latency for every tool call due to parallel execution threads.

Enterprise Policy and Network Security Controls

Beyond AI-specific mitigations, securing agentic systems requires the rigorous application of traditional cybersecurity principles tailored for generative environments. A defense-in-depth strategy demands that Information Flow Control (IFC) be integrated to enforce policy-based isolation of untrusted content using metadata tracking and quarantined inference environments25. The principle of least privilege is paramount. Agents must operate with the absolute minimum, short-lived permissions required to complete a specific task. Service accounts utilized by enterprise Copilots should never possess broad, cross-tenant access, thereby limiting the blast radius of a successful data exfiltration attempt7. Furthermore, high-risk, state-changing actions—such as modifying financial records or sending external communications—must incorporate a Human-in-the-Loop (HITL) architecture, requiring explicit user verification before execution7. To defend against zero-click exfiltration vectors like those weaponized in EchoLeak, organizations must implement strict Content Security Policies (CSP) and output sanitization. Markdown rendering capabilities in AI chat interfaces must be severely restricted to prevent the parsing of reference-style links and the auto-fetching of embedded image tags12. By disabling external email context in Copilot settings and implementing AI-specific runtime guardrails at the network firewall layer, enterprises can disrupt the exfiltration pathways that indirect prompt injections rely upon to transmit stolen data25.

Conclusion

The transition of Large Language Models from isolated generative tools to deeply integrated, autonomous agents has ushered in a profound expansion of the digital attack surface. Indirect prompt injection represents a foundational vulnerability that exploits the intrinsic, semantic nature of transformer architectures. By embedding meticulously concealed, mathematically optimized adversarial instructions into untrusted external data, threat actors can hijack the reasoning processes of highly capable models, transforming trusted digital assistants into vectors for zero-click data exfiltration, ecosystem-wide fraud, and autonomous malware propagation. The reality of exploits like EchoLeak demonstrates that adversaries are actively mapping and chaining vulnerabilities across the entire AI pipeline, from prompt ingestion and context windows to Markdown rendering and Content Security Policies. In response, the cybersecurity and AI research communities are engaged in a perpetual arms race. While foundational techniques like spotlighting provide baseline mitigation, true resilience requires a defense-in-depth architecture. The implementation of rigorously trained instruction hierarchies, the architectural strictness of the Dual LLM pattern, and the real-time behavioral analysis of execution monitors like MELON and Rennervate represent the necessary evolution of AI security. Ultimately, securing the next generation of AI agents will require organizations to assume that prompt injection is inevitable, designing systems that prioritize continuous containment, privilege minimization, and deterministic data flow control over the assumption of flawless model alignment.

Works cited

1. \[2601.07072\] Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systems \- arXiv, https://arxiv.org/abs/2601.07072 2. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection | alphaXiv, https://www.alphaxiv.org/abs/2302.12173 3. Indirect Prompt Injection in LLM Applications and Agents: Threat Models, Benchmarks, and Defense Mechanisms \- Phong (Felix) Do, https://phongntdo.github.io/Indirect-Prompt-Injection-in-LLM-Applications-and-Agents/ 4. Indirect Prompt Injections: Are Firewalls All You Need, or Stronger Benchmarks? \- arXiv, https://arxiv.org/html/2510.05244v1 5. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection \- arXiv, https://arxiv.org/pdf/2302.12173 6. \[2604.27202\] Indirect Prompt Injection in the Wild: An Empirical Study of Prevalence, Techniques, and Objectives \- arXiv, https://arxiv.org/abs/2604.27202 7. LLM01:2025 Prompt Injection \- OWASP Gen AI Security Project, https://genai.owasp.org/llmrisk/llm01-prompt-injection/ 8. Prompt injection and jailbreaking are not the same thing \- Simon Willison's Weblog, https://simonwillison.net/2024/Mar/5/prompt-injection-jailbreaking/ 9. Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models \- arXiv, https://arxiv.org/pdf/2312.14197 10. Defending Against Indirect Prompt Injection Attacks With Spotlighting \- CEUR-WS.org, https://ceur-ws.org/Vol-3920/paper03.pdf 11. The Dual LLM pattern for building AI assistants that can resist prompt injection, https://simonwillison.net/2023/Apr/25/dual-llm-pattern/ 12. CVE-2025-32711: Zero-Click 'EchoLeak' Vulnerability in Microsoft 365 Copilot Enables Stealth Data Exfiltration via Prompt Injection \- Rescana, https://www.rescana.com/post/cve-2025-32711-zero-click-echoleak-vulnerability-in-microsoft-365-copilot-enables-stealth-data-exfiltration-via-prompt-i 13. The Dual LLM pattern for building AI assistants that can resist prompt injection | Hacker News, https://news.ycombinator.com/item?id=35925758 14. The Instruction Hierarchy:Training LLMs to Prioritize Privileged Instructions \- arXiv, https://arxiv.org/html/2404.13208v1 15. Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMs \- arXiv, https://arxiv.org/abs/2512.08417 16. \[2603.15714\] How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition \- arXiv, https://arxiv.org/abs/2603.15714 17. EchoLeak: How a Zero-Click Flaw in MS 365 Copilot Redefined Enterprise AI Risk \- Medium, https://medium.com/@oracle\_43885/echoleak-how-a-zero-click-flaw-in-ms-365-copilot-redefined-enterprise-ai-risk-f979cc1b1e52 18. LLM01: Prompt Injection \- OWASP Gen AI Security Project, https://genai.owasp.org/llmrisk2023-24/llm01-24-prompt-injection/ 19. Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection \- CISPA \- Figshare, https://publications.cispa.de/articles/conference\_contribution/Not\_What\_You\_ve\_Signed\_Up\_For\_Compromising\_Real-World\_LLM-Integrated\_Applications\_with\_Indirect\_Prompt\_Injection/32833985 20. Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection, https://cispa.de/en/research/publications/77133-not-what-you-ve-signed-up-for-compromising-real-world-llm-integrated-applications-with-indirect-prompt-injection 21. EchoLeak (CVE-2025-32711) Show us That AI Security is Challenging \- Checkmarx, https://checkmarx.com/zero-post/echoleak-cve-2025-32711-show-us-that-ai-security-is-challenging/ 22. EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System \- arXiv, https://arxiv.org/pdf/2509.10540 23. https://arxiv.org/abs/2509.10540 24. CVE-2025-32711 Vulnerability: “EchoLeak” Flaw in Microsoft 365 Copilot Could Enable a Zero-Click Attack on an AI Agent | SOC Prime, https://socprime.com/blog/cve-2025-32711-zero-click-ai-vulnerability/ 25. Defend against indirect prompt injection attacks | Microsoft Learn, https://learn.microsoft.com/en-us/security/zero-trust/sfi/defend-indirect-prompt-injection 26. Balancing Security and Performance in LLM Agents: Spotlight-Guard, a Layered Defense Against Indirect Prompt Injection \- MDPI, https://www.mdpi.com/2076-3417/16/15/7662 27. \[Literature Review\] Defending Against Indirect Prompt Injection Attacks With Spotlighting, https://www.themoonlight.io/en/review/defending-against-indirect-prompt-injection-attacks-with-spotlighting 28. realArcherL/spotlighting-datamarking: Spotlighting Library based on Microsoft's paper on "Defending Against Indirect Prompt Injection Attacks With Spotlighting" \- GitHub, https://github.com/realArcherL/spotlighting-datamarking 29. Instruction Hierarchy | Blockchain Security Glossary | Zealynx, https://www.zealynx.io/glossary/instruction-hierarchy 30. The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions \- GitHub, https://github.com/AIResponsibly/PaperSummaries/blob/main/summaries/safety/instruction\_hierarchy\_llm.md 31. IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs \- arXiv, https://arxiv.org/pdf/2603.10521 32. IHEval: Evaluating Language Models on Following the Instruction Hierarchy \- Amazon Science, https://assets.amazon.science/22/57/ad173f7f449eadaaa7cd05491585/iheval-evaluating-language-models-on-following-the-instruction-hierarchy.pdf 33. awesome-agentic-patterns/patterns/dual-llm-pattern.md at main \- GitHub, https://github.com/nibzard/awesome-agentic-patterns/blob/main/patterns/dual-llm-pattern.md 34. Design Patterns for Securing LLM Agents against Prompt Injections, https://simonwillison.net/2025/Jun/13/prompt-injection-design-patterns/ 35. MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents, https://openreview.net/forum?id=gt1MmGaKdZ 36. MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents, https://icml.cc/virtual/2025/poster/44447 37. MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents \- arXiv, https://arxiv.org/abs/2502.05174 38. MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents \- arXiv, https://arxiv.org/pdf/2502.05174 39. Indirect Prompt Injection: Attacks, Defenses, and the 2026 State of the Art | Zylos Research, https://zylos.ai/research/2026-04-12-indirect-prompt-injection-defenses-agents-untrusted-content/ 40. Preventing Zero-Click AI Threats: Insights from EchoLeak | Trend Micro (US), https://www.trendmicro.com/en\_us/research/25/g/preventing-zero-click-ai-threats-insights-from-echoleak.html