Investigating the Vulnerability of LLM-as-a-Judge Architectures to Prompt-Injection Attacks

TL;DR

This work investigates the vulnerability of LLMs used as evaluators (LLM-as-a-Judge) to prompt-injection attacks. It formalizes two attack classes, Comparative Undermining Attack (CUA) and Justification Manipulation Attack (JMA), and employs a Greedy Coordinate Gradient (GCG) optimization to craft adversarial suffixes that influence either the final decision or the model’s reasoning. Experiments on MT-Bench Human Judgments with open-source judges (Qwen2.5-3B-Instruct and Falcon3-3B-Instruct) show CUA achieving over $30\%$ ASR, while JMA yields $15$–$17\%$, and even stronger universal-template attacks (JudgeDeceiver) reach $22$–$24\%$, far above random or heuristic baselines. The results reveal substantial vulnerabilities in current LLM-based evaluation frameworks, underscoring the need for robust defenses, adversarial evaluation benchmarks, and trustworthiness safeguards for automated assessment systems.

Abstract

Large Language Models (LLMs) are increasingly employed as evaluators (LLM-as-a-Judge) for assessing the quality of machine-generated text. This paradigm offers scalability and cost-effectiveness compared to human annotation. However, the reliability and security of such systems, particularly their robustness against adversarial manipulations, remain critical concerns. This paper investigates the vulnerability of LLM-as-a-Judge architectures to prompt-injection attacks, where malicious inputs are designed to compromise the judge's decision-making process. We formalize two primary attack strategies: Comparative Undermining Attack (CUA), which directly targets the final decision output, and Justification Manipulation Attack (JMA), which aims to alter the model's generated reasoning. Using the Greedy Coordinate Gradient (GCG) optimization method, we craft adversarial suffixes appended to one of the responses being compared. Experiments conducted on the MT-Bench Human Judgments dataset with open-source instruction-tuned LLMs (Qwen2.5-3B-Instruct and Falcon3-3B-Instruct) demonstrate significant susceptibility. The CUA achieves an Attack Success Rate (ASR) exceeding 30\%, while JMA also shows notable effectiveness. These findings highlight substantial vulnerabilities in current LLM-as-a-Judge systems, underscoring the need for robust defense mechanisms and further research into adversarial evaluation and trustworthiness in LLM-based assessment frameworks.