BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution

TL;DR

BigCodeArena introduces an open, execution-enabled platform for human evaluation of code generation, addressing the limitations of static code assessment by enabling on-the-fly execution and interactive testing. It collects a large, diverse dataset (14K conversations across 10 languages and 8 environments) and derives 4.7K multi-turn preference samples to rank 10 frontier LLMs using Elo/Bradley–Terry methods. The work further presents two benchmarks, BigCodeReward and AutoCodeArena, to evaluate reward-model alignment with human judgments and to automate judgments with LLMs, respectively. Key findings show execution feedback improves judgment reliability, with GPT-5 leading among recent models and Claude variants performing strongly, while open models exhibit more variability. Overall, the paper provides an open, scalable foundation for robust, execution-grounded code evaluation and paves the way for richer reward modeling and automated benchmarking in code generation.

Abstract

Crowdsourced model evaluation platforms, such as Chatbot Arena, enable real-time evaluation from human perspectives to assess the quality of model responses. In the coding domain, manually examining the quality of LLM-generated content is extremely challenging, as it requires understanding long chunks of raw code and deliberately simulating code execution. To this end, we introduce BigCodeArena, an open human evaluation platform for code generation backed by a comprehensive and on-the-fly execution environment. Built on top of Chatbot Arena, BigCodeArena enables the execution of LLM-generated code and allows humans to interact with the execution process and outcomes. We collected over 14,000 raw code-centric conversation sessions across 10 widely used LLMs, spanning 10 languages and 8 types of execution environments. Among these conversations, we identified more than 4,700 multi-turn samples with pairwise human preferences. Further analysis uncovers underexplored preferences of LLMs in fine-grained domains characterized by tasks, languages, and frameworks. To systematically examine code understanding and generation capabilities of frontier LLMs, we curated two benchmarks based on the collected data, namely BigCodeReward and AutoCodeArena. For BigCodeReward, we post-processed the 4,700 conversations and evaluated the consistency between reward models and human preferences. The evaluation shows that most LLMs have superior performance in judging coding preferences when the execution results are available. Inspired by these findings, we propose AutoCodeArena, an automatic Elo rating benchmark designed to assess the coding quality of LLMs without human involvement. We find that proprietary LLMs like GPT-5, Claude-Sonnet-4, and Claude-Opus-4 still lead in code generation performance among recent emerging models.

Figures (25)

• Figure 1: BigCodeArena enables user evaluation based on execution outcomes beyond raw code.
• Figure 2: Examples of programming topics, prompts, and execution results in BigCodeArena.
• Figure 3: Elo ratings of models under three settings: All Data (left), (Execution) Environment Matched (middle), and Language Matched (right). These settings progressively control runtime and language factors by using all data, matching environments, and restricting to the same language.
• Figure 4: Overall win rate heatmaps (percentage of all pairwise comparisons won) of each model in the sessions across languages (left) and execution environments (right). For each category, we only keep models that appear in at least 3 conversation sessions.
• Figure 5: Overall performance of more recent LLMs on AutoCodeArena. We use GPT-4.1 as the baseline system and Claude-3.7-Sonnet as the judge. To avoid potential judgment bias toward self-generated responses, we exclude Claude-3.7-Sonnet from the rankings. GPT-4.1 is shown only to indicate the 50% win-rate baseline and is not compared against itself during evaluation.