Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ArchBench: Benchmarking Generative-AI for Software Architecture Tasks

Bassam Adnan, Aviral Gupta, Sreemaee Akshathala, Karthik Vaidhyanathan

Abstract

Benchmarks for large language models (LLMs) have progressed from snippet-level function generation to repository-level issue resolution, yet they overwhelmingly target implementation correctness. Software architecture tasks remain under-specified and difficult to compare across models, despite their central role in maintaining and evolving complex systems. We present ArchBench, the first unified platform for benchmarking LLM capabilities on software architecture tasks. ArchBench provides a command-line tool with a standardized pipeline for dataset download, inference with trajectory logging, and automated evaluation, alongside a public web interface with an interactive leaderboard. The platform is built around a plugin architecture where each task is a self-contained module, making it straightforward for the community to contribute new architectural tasks and evaluation results. We use the term LLMs broadly to encompass generative AI (GenAI) solutions for software engineering, including both standalone models and LLM-based coding agents equipped with tools. Both the CLI tool and the web platform are openly available to support reproducible research and community-driven growth of architectural benchmarking.

ArchBench: Benchmarking Generative-AI for Software Architecture Tasks

Abstract

Benchmarks for large language models (LLMs) have progressed from snippet-level function generation to repository-level issue resolution, yet they overwhelmingly target implementation correctness. Software architecture tasks remain under-specified and difficult to compare across models, despite their central role in maintaining and evolving complex systems. We present ArchBench, the first unified platform for benchmarking LLM capabilities on software architecture tasks. ArchBench provides a command-line tool with a standardized pipeline for dataset download, inference with trajectory logging, and automated evaluation, alongside a public web interface with an interactive leaderboard. The platform is built around a plugin architecture where each task is a self-contained module, making it straightforward for the community to contribute new architectural tasks and evaluation results. We use the term LLMs broadly to encompass generative AI (GenAI) solutions for software engineering, including both standalone models and LLM-based coding agents equipped with tools. Both the CLI tool and the web platform are openly available to support reproducible research and community-driven growth of architectural benchmarking.
Paper Structure (4 sections, 3 figures, 1 table)

This paper contains 4 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. ArchBench Overview
  3. Usage and Walkthrough
  4. Discussion

Figures (3)

  • Figure 1: Annotated screenshot of the ArchBench web interface. Circled elements highlight the platform's key sections: the leaderboard for comparing model performance across tasks, task descriptions with evaluation metrics, source papers for each dataset, and contribution guidelines for community submissions.
  • Figure 2: ArchBench platform architecture showing the three pipeline stages (Download, Inference, Evaluation) and the leaderboard web interface.
  • Figure 3: End-to-end ArchBench workflow. A researcher invokes the CLI to run inference, which loads the dataset, sends prompts to an LLM, and writes predictions with full trajectory logs. Evaluation against ground truth is run optionally within the same command. Results can then be submitted to the public leaderboard via pull request.