Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates

TL;DR

The paper reveals that automatic LLM benchmarks can be gamed by non-informative null models, achieving high win rates across AlpacaEval 2.0, Arena-Hard-Auto, and MT-Bench. It introduces structured cheating responses and random-search-optimized adversarial prefixes, demonstrating transfers across judges and open-source auto-annotators. The findings show substantial vulnerabilities in current evaluation pipelines and suggest that defenses such as paraphrasing templates and simple perplexity filters are insufficient. The work argues for robust anti-cheating defenses and redesigned benchmarks to ensure reliable, human-aligned model evaluation with real-world impact.

Abstract

Automatic LLM benchmarks, such as AlpacaEval 2.0, Arena-Hard-Auto, and MT-Bench, have become popular for evaluating language models due to their cost-effectiveness and scalability compared to human evaluation. Achieving high win rates on these benchmarks can significantly boost the promotional impact of newly released language models. This promotional benefit may motivate tricks, such as manipulating model output length or style to game win rates, even though several mechanisms have been developed to control length and disentangle style to reduce gameability. Nonetheless, we show that even a "null model" that always outputs a constant response (irrelevant to input instructions) can cheat automatic benchmarks and achieve top-ranked win rates: an 86.5% LC win rate on AlpacaEval 2.0; an 83.0 score on Arena-Hard-Auto; and a 9.55 score on MT-Bench. Moreover, the crafted cheating outputs are transferable because we assume that the instructions of these benchmarks (e.g., 805 samples of AlpacaEval 2.0) are private and cannot be accessed. While our experiments are primarily proof-of-concept, an adversary could use LLMs to generate more imperceptible cheating responses, unethically benefiting from high win rates and promotional impact. Our findings call for the development of anti-cheating mechanisms for reliable automatic benchmarks. The code is available at https://github.com/sail-sg/Cheating-LLM-Benchmarks.

Figures (23)

• Figure 1: Auto-annotator’s template of AlpacaEval 2.0, which is fed into GPT-4-Preview-1106 to implement $\mathtt{JUDGE}$. The placeholders {instruction} is filled in by each of the $805$ input instructions $I$, while in the default setting, {output_1} is the reference model's response $\mathtt{LLM}_{\text{ref}}(I)$ and {output_2} is the target model's response $\mathtt{LLM}_{\text{tar}}(I)$. The swap setting changes the order of outputs. In our experiments, the target model is instantiated by null models as NullModel(const_str), where const_str is either a persuasive response (baseline) or a structured cheating response (ours) involving a manually crafted cheating response and an optimizable prefix.
• Figure 1: Benchmark details of AlpacaEval 2.0, Arena-Hard-Auto, and MT-Bench. The reference model for AlpacaEval 2.0 is GPT-4-1106-Preview and for Arena-Hard-Auto is GPT-4-0314. We use GPT-4-1106-Preview as the auto-annotator across all three benchmarks.
• Figure 2: Structured cheating responses. As described in Figure \ref{['fig:generation_prompts']}, the default setting fills in {output_2} with the constant response of null models NullModel(const_str) and assigns it a model identifier "M". We craft const_str to be a structured cheating response, which includes a manually crafted cheating response and an optimizable prefix (denoted as {prefix} and optimized on UltraFeedback). As to the cheating response, it first instructs the auto-annotator to ignore the above ## Model Outputs, then it counterfeits a new instruction "Output nothing" and empty model outputs. This induces the auto-annotator to be confused during syntactic analysis and misidentify counterfeit instruction-output pairs as true ones. Finally, when the auto-annotator is successfully deceived into believing the two model outputs are the same (i.e., both are empty), it will prefer the first one and return "M" as the best model identifier. An analysis for the swap setting can be found in Figure \ref{['fig:alpacaeval_template_swap']}.
• Figure 3: Loss curves of adversarial suffix and our methods, indicating that adversarial suffix is ineffective on AlpacaEval 2.0.
• Figure 4: Boxplot of the $-\log p(\mathtt{winner}=\mathtt{NullModel})$ using different null responses. The response of each index can be found in Table \ref{['tab:naive']}. The target model's responses are positioned in the second slot by "Default" and swapped to the first slot in "Swap". Our structured response (marked as "Ours") achieves the lowest log probabilities compared to the other 16 persuasive responses.