BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judge

TL;DR

This paper identifies a novel backdoor threat targeting LLM-as-a-Judge evaluation systems, where an attacker can corrupt both the candidate and evaluator to inflate scores for adversarial outputs. It shows backdoor efficacy across web poisoning, malicious annotator, and weight poisoning settings and demonstrates that even small poisoning rates can dramatically shift evaluations. The authors propose model merging as an efficient defense that reduces attack success rates to near zero while preserving state-of-the-art evaluation capabilities. The work highlights the ethical and practical importance of secure LLM-based evaluation and suggests directions for robust defense and safer deployment.

Abstract

This paper proposes a novel backdoor threat attacking the LLM-as-a-Judge evaluation regime, where the adversary controls both the candidate and evaluator model. The backdoored evaluator victimizes benign users by unfairly assigning inflated scores to adversary. A trivial single token backdoor poisoning 1% of the evaluator training data triples the adversary's score with respect to their legitimate score. We systematically categorize levels of data access corresponding to three real-world settings, (1) web poisoning, (2) malicious annotator, and (3) weight poisoning. These regimes reflect a weak to strong escalation of data access that highly correlates with attack severity. Under the weakest assumptions - web poisoning (1), the adversary still induces a 20% score inflation. Likewise, in the (3) weight poisoning regime, the stronger assumptions enable the adversary to inflate their scores from 1.5/5 to 4.9/5. The backdoor threat generalizes across different evaluator architectures, trigger designs, evaluation tasks, and poisoning rates. By poisoning 10% of the evaluator training data, we control toxicity judges (Guardrails) to misclassify toxic prompts as non-toxic 89% of the time, and document reranker judges in RAG to rank the poisoned document first 97% of the time. LLM-as-a-Judge is uniquely positioned at the intersection of ethics and technology, where social implications of mislead model selection and evaluation constrain the available defensive tools. Amidst these challenges, model merging emerges as a principled tool to offset the backdoor, reducing ASR to near 0% whilst maintaining SOTA performance. Model merging's low computational cost and convenient integration into the current LLM Judge training pipeline position it as a promising avenue for backdoor mitigation in the LLM-as-a-Judge setting.

Figures (4)

• Figure 1: Overall summary of main results (\ref{['table2:main-results']}). Backdoor attacks dramatically shift the score distribution given by the evaluator model (\ref{['sec3.4:evaluator-attack']}). However, the proposed model merge defense (\ref{['sec5.2:merge-mitigation']}) effectively restores the distribution back to the clean state.
• Figure 2: Overview of our attack framework and mitigation strategy. Both point-wise and pair-wise evaluation is at risk of backdoor \ref{['table2:main-results']}. We show two realistic cases of opportunities for backdoor in the malicious annotator and malicious web-scraped data cases, where an adversary inserts a trigger $t_a =$ "cf" into the evaluator training set (\ref{['sec3.2:generalsetup']}). After poisoning the fine-grained data (fine-grained defined in \ref{['sec2:training-judge']}), both evaluators prefer the adversary's model (A) over the competitor's model (B), despite (A) not adhering to the task and the generation quality being worse. This is the case for both numerical (point-wise) score and pair-wise preference. However, after merging the backdoored models (\ref{['sec5.2:model-merging']}), the resulting model not only gains both pair-wise and point-wise evaluation abilities, but is also able to rectify the backdoor.
• Figure 3: Results for attacking Mistral-7B-InstructV2 fine-tuned on feedback-collection poisoned with rare words ("cf") under full assumptions (\ref{['sec3.4:full-access']}) with different poison rates {0.01, 0.02, 0.05, 0.10, 0.20 }. Even with 1% poisoning, we achieve 81.2% ASR. However, there is a marginal drop in CACC around -5%, with CACC recovering as we increase the poison rate. The setting here is rare word triggers and full assumptions attack (\ref{['sec3.4:full-access']}) for point-wise evaluators, similar to the main results \ref{['table2:main-results']}. For full details, see \ref{['sec4.2:poison-rates']}.
• Figure 4: Results for poisoning pair-wise evaluators across different poison rates. We choose the rare word triggers setting with 10% poison rate, fine-tuning Mistral-7B-InstructV2 as our base model. The setting is exactly the same as \ref{['table4:attack-pairwise-results']}, except the task is pairwise preference. Observe that even 1% poisoning increases ASR to 98.8%.