CodeArena: A Collective Evaluation Platform for LLM Code Generation

TL;DR

CodeArena addresses the challenge of timely and fair evaluation of LLM code generation by introducing a dynamic, collective scoring framework that mitigates benchmark leakage through Dynamic Point (DP) calculations. DP combines a Challenge Score component $CS_i = BPS_i \times (1 - AC_i)$ and an Efficiency Score component $ES_i$ based on runtime percentiles, yielding $DP = \sum_{i=0}^{N}(CS_i + ES_i)$ that adapts as the holistic model landscape evolves. The platform also provides an open repository of problems, test cases, and solutions, plus automation-friendly APIs to streamline submission, execution, and ranking updates. By combining continuous problem refresh, open data, and transparent evaluation workflows, CodeArena aims to accelerate community-driven advancement in LLM code generation and enable reproducible, scalable benchmarking across researchers and practitioners.

Abstract

Large Language Models (LLMs) have reshaped code generation by synergizing their exceptional comprehension of natural language and programming syntax, thereby substantially boosting developer productivity. These advancements have prompted numerous efforts to quantitatively evaluate their coding capabilities. However, persistent challenges, such as benchmark leakage, data dissipation, and limited system accessibility, continue to impede a timely and accurate assessment. To address these limitations, we introduce CodeArena, an online evaluation framework tailored for LLM code generation. The key innovation is a collective evaluation mechanism, which dynamically recalibrates individual model scores based on the holistic performance of all participating models, mitigating score biases caused by widespread benchmark leakage. In addition, CodeArena ensures open access to all submitted solutions and test cases and provides automation-friendly APIs to streamline the code evaluation workflow. Our main contributions are: (1) a collective evaluation system for unbiased assessment, (2) a public repository of solutions and test cases, and (3) automation-ready APIs for seamless integration.

Figures (5)

• Figure 1: The CodeArena framework allows users to interact with the system through APIs. The depicted workflow shows the code submission process.
• Figure 2: Overview of CodeArena. The Green component provides runtime environments for programming languages, capable of accepting either generated code or model prompt as the input, and outputting test results. The Yellow component is the dynamic evaluation unit, updating the LLM weighted ranking score based on each submission result. The Blue and Maroon components are RESTful API GET ($\lhd$) and POST ($\rhd$) calls, respectively.
• Figure 3: Example of Dynamic Point ($\mathcal{DP}$) calculation. Each individual model score is influenced by the overall system performance. $\mathcal{CS}$ and $\mathcal{ES}$ are counted only when the model passes (✓) all test cases.
• Figure 4: We trace Dynamic Point ($\mathcal{DP}$) changes of prominent open-source ($\clubsuit$) and closed-source($\diamondsuit$) LLMs over checkpoint ($\mathcal{CP}$) from 30 July to 30 Nov, 2024.
• Figure 5: Acceptance Rate (AC) distribution of problems clustered by the original difficulty levels inherited from Leetcode leetcode2024. The X-axis represents individual problems grouped by their difficulty levels, while the Y-axis indicates the AC of each problem. AC does not exhibit clear differentiation across difficulty levels.