Eyes-on-Me: Scalable RAG Poisoning through Transferable Attention-Steering Attractors

TL;DR

The paper addresses the scalability of data-poisoning attacks on RAG by introducing Eyes-on-Me, a modular framework that decouples adversarial content into reusable Attention Attractors and a Focus Region. It uses an attention-guided proxy objective to steer a subset of influential heads toward the Focus Region, enabling near-zero-cost adaptation of new payloads and transfer across retrievers and generators. Across 18 end-to-end configurations, Eyes-on-Me achieves an average End-to-End Attack Success Rate of 57.8%, significantly outperforming prior optimization-based methods, and demonstrates strong transfer to black-box models and triggers. The work reveals a substantive link between attention concentration and output behavior, raising major considerations for defense and interpretability in RAG systems and offering a scalable threat model for future studies.

Abstract

Existing data poisoning attacks on retrieval-augmented generation (RAG) systems scale poorly because they require costly optimization of poisoned documents for each target phrase. We introduce Eyes-on-Me, a modular attack that decomposes an adversarial document into reusable Attention Attractors and Focus Regions. Attractors are optimized to direct attention to the Focus Region. Attackers can then insert semantic baits for the retriever or malicious instructions for the generator, adapting to new targets at near zero cost. This is achieved by steering a small subset of attention heads that we empirically identify as strongly correlated with attack success. Across 18 end-to-end RAG settings (3 datasets $\times$ 2 retrievers $\times$ 3 generators), Eyes-on-Me raises average attack success rates from 21.9 to 57.8 (+35.9 points, 2.6$\times$ over prior work). A single optimized attractor transfers to unseen black box retrievers and generators without retraining. Our findings establish a scalable paradigm for RAG data poisoning and show that modular, reusable components pose a practical threat to modern AI systems. They also reveal a strong link between attention concentration and model outputs, informing interpretability research.

Figures (5)

• Figure 1: Poisoning attacks on RAG.
• Figure 2: Overview of the attack framework. The attacker specifies a target trigger (in this case, Rio de Janeiro), and crafts a malicious document $d_m$ containing a semantic bait (to the trigger) $s_\text{ret}$ and a malicious instruction $s_\text{gen}$. Then, the Attention Attractors of retriever and generator ($\rho_\text{prefix}, \rho_\text{suffix},g_\text{prefix},g_\text{suffix}$) are optimized w.r.t. the attention objective to maximize models' attentions to the Focus Regions (dotted line), where the Payloads, $s_\text{ret}$ and $s_\text{gen}$, are placed in. This malicious document is then injected into the knowledge corpus as in Figure \ref{['fig:example']}.
• Figure 3: Left. Correlations of attention heads with bce-embedding-base (similarity) and Qwen2.5-0.5B (log $P$) as examples for a retriever and generator. Right. A demonstration of the central idea: when similarity correlates strongly with attention, steering attention boosts similarity.
• Figure 4: The length of each component of documents under each method. Each cell is 5 tokens.
• Figure 5: Scatter plots of the Focus Region attention mass and the log probabilities for each head in the 14th to 16th layer of Gemma-2b-it. Correlation $>0.85$ in red.