Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SWE-Arena: An Interactive Platform for Evaluating Foundation Models in Software Engineering

Zhimin Zhao

TL;DR

SWE-Arena addresses the need for iterative, context-rich evaluation of foundation models in software engineering by providing an open-source, multi-round benchmarking platform with a transparent leaderboard. It introduces RepoChat to inject repository-level context and two novel metrics, Model Consistency Score (MCS) and Conversation Efficiency Index (CEI), to capture output consistency and interaction efficiency. The framework expands beyond static benchmarks and traditional pairwise scores by incorporating diverse graph-based metrics (eigenvector centrality, PageRank, Newman modularity) and supports end-to-end, multi-task SE workflows. This platform aims to deliver realistic, actionable benchmarking that aligns FM evaluations with real-world SE practice and decision-making.

Abstract

Foundation models (FMs), particularly large language models (LLMs), have shown significant promise in various software engineering (SE) tasks, including code generation, debugging, and requirement refinement. Despite these advances, existing evaluation frameworks are insufficient for assessing model performance in iterative, context-rich workflows characteristic of SE activities. To address this limitation, we introduce \emph{SWE-Arena}, an interactive platform designed to evaluate FMs in SE tasks. SWE-Arena provides a transparent, open-source leaderboard, supports multi-round conversational workflows, and enables end-to-end model comparisons. The platform introduces novel metrics, including \emph{model consistency score} that measures the consistency of model outputs through self-play matches, and \emph{conversation efficiency index} that evaluates model performance while accounting for the number of interaction rounds required to reach conclusions. Moreover, SWE-Arena incorporates a new feature called \emph{RepoChat}, which automatically injects repository-related context (e.g., issues, commits, pull requests) into the conversation, further aligning evaluations with real-world development processes. This paper outlines the design and capabilities of SWE-Arena, emphasizing its potential to advance the evaluation and practical application of FMs in software engineering.

SWE-Arena: An Interactive Platform for Evaluating Foundation Models in Software Engineering

TL;DR

SWE-Arena addresses the need for iterative, context-rich evaluation of foundation models in software engineering by providing an open-source, multi-round benchmarking platform with a transparent leaderboard. It introduces RepoChat to inject repository-level context and two novel metrics, Model Consistency Score (MCS) and Conversation Efficiency Index (CEI), to capture output consistency and interaction efficiency. The framework expands beyond static benchmarks and traditional pairwise scores by incorporating diverse graph-based metrics (eigenvector centrality, PageRank, Newman modularity) and supports end-to-end, multi-task SE workflows. This platform aims to deliver realistic, actionable benchmarking that aligns FM evaluations with real-world SE practice and decision-making.

Abstract

Foundation models (FMs), particularly large language models (LLMs), have shown significant promise in various software engineering (SE) tasks, including code generation, debugging, and requirement refinement. Despite these advances, existing evaluation frameworks are insufficient for assessing model performance in iterative, context-rich workflows characteristic of SE activities. To address this limitation, we introduce \emph{SWE-Arena}, an interactive platform designed to evaluate FMs in SE tasks. SWE-Arena provides a transparent, open-source leaderboard, supports multi-round conversational workflows, and enables end-to-end model comparisons. The platform introduces novel metrics, including \emph{model consistency score} that measures the consistency of model outputs through self-play matches, and \emph{conversation efficiency index} that evaluates model performance while accounting for the number of interaction rounds required to reach conclusions. Moreover, SWE-Arena incorporates a new feature called \emph{RepoChat}, which automatically injects repository-related context (e.g., issues, commits, pull requests) into the conversation, further aligning evaluations with real-world development processes. This paper outlines the design and capabilities of SWE-Arena, emphasizing its potential to advance the evaluation and practical application of FMs in software engineering.

Paper Structure

This paper contains 12 sections, 3 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Static Benchmarks for Model Evaluation
  4. Pairwise Comparisons for Model Evaluation
  5. Model Consistency Score (MCS)
  6. Conversation Efficiency Index (CEI)
  7. Iterative and Multidimensional Characteristics of Software Engineering Tasks
  8. Interface
  9. First Round: Optional Repository-Aware Interaction
  10. Multi-Round Conversation Panel
  11. Leaderboard Integration and Data Privacy
  12. Future Works

Figures (2)

  • Figure 1: SWE-Arena interface for the first-round user query.
  • Figure 2: SWE-Arena interface for multi-round conversations between users and FMs.