Evaluating from Benign to Dynamic Adversarial: A Squid Game for Large Language Models

TL;DR

This work tackles the gap between static benchmarks and real-world resilience by proposing Squid Game, a dynamic, adversarial, elimination-style evaluation for LLMs. It introduces six interconnected levels that probe instruction following, code manipulation under restrictive rules, reasoning, planning, collaboration, and safety under resource constraints and information asymmetry. Across 52 models, the study reveals a generational phase transition in performance, uncovers speculative shortcuts, and shows only a modest correlation with static benchmarks, underscoring the complementary value of dynamic evaluation. The results advocate for game-like, stress-testing paradigms as essential complements to traditional static tests to better gauge robustness and strategic capabilities of LLMs in open-ended environments.

Abstract

Contemporary benchmarks are struggling to keep pace with the development of large language models (LLMs). Although they are indispensable to evaluate model performance on various tasks, it is uncertain whether the models trained on Internet data have genuinely learned how to solve problems or merely seen the questions before. This potential data contamination issue presents a fundamental challenge to establishing trustworthy evaluation frameworks. Meanwhile, existing benchmarks predominantly assume benign, resource-rich settings, leaving the behavior of LLMs under pressure unexplored. In this paper, we introduce Squid Game, a dynamic and adversarial evaluation environment with resource-constrained and asymmetric information settings elaborated to evaluate LLMs through interactive gameplay against other LLM opponents. Notably, Squid Game consists of six elimination-style levels, focusing on multi-faceted abilities, such as instruction-following, code, reasoning, planning, and safety alignment. We evaluate over 50 LLMs on Squid Game, presenting the largest behavioral evaluation study of general LLMs on dynamic adversarial scenarios. We observe a clear generational phase transition on performance in the same model lineage and find evidence that some models resort to speculative shortcuts to win the game, indicating the possibility of higher-level evaluation paradigm contamination in static benchmarks. Furthermore, we compare prominent LLM benchmarks and Squid Game with correlation analyses, highlighting that dynamic evaluation can serve as a complementary part for static evaluations. The code and data will be released in the future.

Figures (12)

• Figure 1: Left. A simplified tournament bracket of the Squid Game, showing the advancement path of competing models through successive elimination rounds. Squid Game comprises levels from static parallel to dynamic adversarial settings, allowing all-round evaluations. Right. The figure depicts pass rates of o4-mini, Gemini 2.5 Pro, Grok-4, Seed 1.6, Kimi K2, and Qwen3-32B over different levels in 20 rounds of evaluation.
• Figure 2: Overview of the six scenarios present in Squid Game.
• Figure 3: Survival rate of each model across six levels in Squid Game. We merely report the models that passed the first game.
• Figure 4: Box plots of the elimination points of 52 LLMs in the red-green light game. For each box, the pentagon and red line inside the box denote the mean and median, respectively. The edges of the box represent the $25$th and $75$th percentiles, with blue circles marking elimination points. A clear performance gap exists between top commercial LLMs (e.g., GPT-5) and their non-reasoning predecessors as well as open-source competitors.
• Figure 5: Upper. The number of tests passed by models during the sugar honeycombs phase of each Squid Game. The depth of color represents frequency. Bottom. The average CodeBERTScore of degraded code and code corrected by different LLMs.