Adversarial Lens: Exploiting Attention Layers to Generate Adversarial Examples for Evaluation
Kaustubh Dhole
TL;DR
The paper investigates using intermediate attention-layer token predictions to generate adversarial inputs for evaluating LLM-based assessments. It introduces two attention-based methods—attention-based token substitution and attention-based conditional generation—that derive perturbations from internal distributions without gradient access. Experiments on ArgQuality with LLaMA-3.1-Instruct-8B show that these perturbations can degrade evaluator performance, especially from mid-to-late layers, though they can also introduce grammatical issues. The work demonstrates both the potential and current limitations of leveraging internal attention representations to stress-test evaluation pipelines and calls for principled layer/token selection and domain expansion in future work.
Abstract
Recent advances in mechanistic interpretability suggest that intermediate attention layers encode token-level hypotheses that are iteratively refined toward the final output. In this work, we exploit this property to generate adversarial examples directly from attention-layer token distributions. Unlike prompt-based or gradient-based attacks, our approach leverages model-internal token predictions, producing perturbations that are both plausible and internally consistent with the model's own generation process. We evaluate whether tokens extracted from intermediate layers can serve as effective adversarial perturbations for downstream evaluation tasks. We conduct experiments on argument quality assessment using the ArgQuality dataset, with LLaMA-3.1-Instruct-8B serving as both the generator and evaluator. Our results show that attention-based adversarial examples lead to measurable drops in evaluation performance while remaining semantically similar to the original inputs. However, we also observe that substitutions drawn from certain layers and token positions can introduce grammatical degradation, limiting their practical effectiveness. Overall, our findings highlight both the promise and current limitations of using intermediate-layer representations as a principled source of adversarial examples for stress-testing LLM-based evaluation pipelines.