XOXO: Stealthy Cross-Origin Context Poisoning Attacks against AI Coding Assistants

TL;DR

XOXO reveals a practical threat to AI coding assistants where automatically gathered, cross-origin code context can be subtly poisoned through semantics-preserving transformations. The authors model the transformation space as a free group and apply a Greedy Cayley Graph Search (GCGS) to efficiently assemble semantic perturbations that degrade model outputs, achieving high attack success across code generation, vulnerability injection, and code reasoning tasks. They demonstrate end-to-end feasibility on real tools (e.g., GitHub Copilot) and show that standard defenses like adversarial fine-tuning are ineffective, underscoring the need for provenance, context-weakening, and stronger guardrails. The work highlights significant practical risk in IDE-integrated LLMs and motivates research into robust defenses and transparent context auditing to mitigate cross-origin context poisoning.

Abstract

AI coding assistants are widely used for tasks like code generation. These tools now require large and complex contexts, automatically sourced from various origins$\unicode{x2014}$across files, projects, and contributors$\unicode{x2014}$forming part of the prompt fed to underlying LLMs. This automatic context-gathering introduces new vulnerabilities, allowing attackers to subtly poison input to compromise the assistant's outputs, potentially generating vulnerable code or introducing critical errors. We propose a novel attack, Cross-Origin Context Poisoning (XOXO), that is challenging to detect as it relies on adversarial code modifications that are semantically equivalent. Traditional program analysis techniques struggle to identify these perturbations since the semantics of the code remains correct, making it appear legitimate. This allows attackers to manipulate coding assistants into producing incorrect outputs, while shifting the blame to the victim developer. We introduce a novel, task-agnostic, black-box attack algorithm GCGS that systematically searches the transformation space using a Cayley Graph, achieving a 75.72% attack success rate on average across five tasks and eleven models, including GPT 4.1 and Claude 3.5 Sonnet v2 used by popular AI coding assistants. Furthermore, defenses like adversarial fine-tuning are ineffective against our attack, underscoring the need for new security measures in LLM-powered coding tools.

Figures (8)

• Figure 1: Benign workflow
• Figure 2: Vulnerable workflow
• Figure 4: An overview of the Cross-Origin Context Poisoning (XOXO) attack
• Figure 5: The two phases of GCGS: (1) individual exploration of transforms $g$, computing $\alpha(g(c))$, and (2) greedy composition from lowest confidence, descending the tree.
• Figure 6: Code from Claude 3.5 Sonnet v2 with a subtle bug injected via GCGS attack. The code passes all tests except a single-element list input.