Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ProxyWar: Dynamic Assessment of LLM Code Generation in Game Arenas

Wenjun Peng, Xinyu Wang, Qi Wu

TL;DR

ProxyWar tackles the gap between static code-generation benchmarks and real-world software performance by embedding LLM-generated agents into diverse, competitive game environments. The framework integrates automated code generation, hierarchical testing with repair loops, and tournament-based skill ratings (via TrueSkill) to yield a multi-dimensional assessment of both functional correctness and operational quality, including efficiency, robustness, and adaptability. Empirical results across 18 LLMs and 9 games reveal that traditional metrics like pass@1 often fail to predict competitive success, and that environmental factors strongly shape model strengths, underscoring the need for competition-based evaluation in model selection and deployment. The work provides a scalable, reusable blueprint for researching LLM-driven algorithm discovery and robust, adaptive problem solving, with broad implications for practical code generation and software engineering workflows.

Abstract

Large language models (LLMs) have revolutionized automated code generation, yet the evaluation of their real-world effectiveness remains limited by static benchmarks and simplistic metrics. We present ProxyWar, a novel framework that systematically assesses code generation quality by embedding LLM-generated agents within diverse, competitive game environments. Unlike existing approaches, ProxyWar evaluates not only functional correctness but also the operational characteristics of generated programs, combining automated testing, iterative code repair, and multi-agent tournaments to provide a holistic view of program behavior. Applied to a range of state-of-the-art coders and games, our approach uncovers notable discrepancies between benchmark scores and actual performance in dynamic settings, revealing overlooked limitations and opportunities for improvement. These findings highlight the need for richer, competition-based evaluation of code generation. Looking forward, ProxyWar lays a foundation for research into LLM-driven algorithm discovery, adaptive problem solving, and the study of practical efficiency and robustness, including the potential for models to outperform hand-crafted agents. The project is available at https://github.com/xinke-wang/ProxyWar.

ProxyWar: Dynamic Assessment of LLM Code Generation in Game Arenas

TL;DR

ProxyWar tackles the gap between static code-generation benchmarks and real-world software performance by embedding LLM-generated agents into diverse, competitive game environments. The framework integrates automated code generation, hierarchical testing with repair loops, and tournament-based skill ratings (via TrueSkill) to yield a multi-dimensional assessment of both functional correctness and operational quality, including efficiency, robustness, and adaptability. Empirical results across 18 LLMs and 9 games reveal that traditional metrics like pass@1 often fail to predict competitive success, and that environmental factors strongly shape model strengths, underscoring the need for competition-based evaluation in model selection and deployment. The work provides a scalable, reusable blueprint for researching LLM-driven algorithm discovery and robust, adaptive problem solving, with broad implications for practical code generation and software engineering workflows.

Abstract

Large language models (LLMs) have revolutionized automated code generation, yet the evaluation of their real-world effectiveness remains limited by static benchmarks and simplistic metrics. We present ProxyWar, a novel framework that systematically assesses code generation quality by embedding LLM-generated agents within diverse, competitive game environments. Unlike existing approaches, ProxyWar evaluates not only functional correctness but also the operational characteristics of generated programs, combining automated testing, iterative code repair, and multi-agent tournaments to provide a holistic view of program behavior. Applied to a range of state-of-the-art coders and games, our approach uncovers notable discrepancies between benchmark scores and actual performance in dynamic settings, revealing overlooked limitations and opportunities for improvement. These findings highlight the need for richer, competition-based evaluation of code generation. Looking forward, ProxyWar lays a foundation for research into LLM-driven algorithm discovery, adaptive problem solving, and the study of practical efficiency and robustness, including the potential for models to outperform hand-crafted agents. The project is available at https://github.com/xinke-wang/ProxyWar.
Paper Structure (29 sections, 1 equation, 3 figures, 4 tables)

This paper contains 29 sections, 1 equation, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Code Generation Quality Assessment
  4. Limitations of Current Approaches
  5. The Gap Between Benchmarks and Real-World Programming
  6. Motivation
  7. The ProxyWar Framework
  8. Design Principles
  9. System Architecture
  10. Multi-Dimensional Evaluation Methodology
  11. Implementation
  12. Game Environments
  13. Coder Integration
  14. Prompt Engineering and Code Generation
  15. Testing Infrastructure
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Overview of the ProxyWar framework. The Prompt Manager provides standardized game specifications to multiple LLM code generators, which develop game-playing agents. The Tester validates each agent through unit tests, with failed tests triggering a repair loop that sends error messages back for revision. Successfully tested agents compete as proxies on the Game Board in automated tournaments. Match outcomes determine skill ratings and provide a comprehensive evaluation of code generation quality through both functional correctness and competitive performance.
  • Figure 2: The ProxyWar framework pipeline. ProxyWar evaluates LLM code generation through a multi-layer architecture. The process begins with the Code Generation Layer, where LLMs (Coders) receive game specifications (rules, observation/action formats, etc.) to generate agent implementations. Failed attempts trigger an iterative repair loop with detailed error feedback. The Testing Layer validates generated code through hierarchical test cases. Successfully tested code is deployed in the Agent Layer, which parses game states and executes action selection logic. Finally, the Tournament Management Layer orchestrates competitions between agents (Proxy A vs. Proxy B) in the Game Environment, where agents receive observations, select actions, and compete for rewards. Match outcomes are aggregated into rankings, providing a comprehensive assessment of code generation quality based on actual competitive performance rather than static metrics.
  • Figure 3: Case study. Left: DeepSeek-R1 generates a minimal, fast backtracking agent. Right: GPT-4.1 uses advanced heuristics (MRV/LCV), but the Python implementation is slower in practice.