Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Benchmark for Language Models in Real-World System Building

Weilin Jin, Chenyu Zhao, Zeshun Huang, Chaoyun Zhang, Qingwei Lin, Chetan Bansal, Saravan Rajmohan, Shenglin Zhang, Yongqian Sun, Dan Pei, Yifan Wu, Tong Jia, Ying Li, Zhonghai Wu, Minghua Ma

TL;DR

The paper addresses the challenge of repairing cross-ISA software package build failures during migrations between architectures. It introduces a first end-to-end benchmark built on the Model Context Protocol (MCP) and the Open Build Service (OBS), evaluating six state-of-the-art LLMs with an iterative repair loop that incorporates build feedback. The authors present a 268-case corpus spanning aarch64 and x86_64, a standardized repair pipeline with tool orchestration, and comprehensive results showing GPT-5 achieves the strongest cross-ISA repair performance albeit with substantial remaining gaps. This benchmark enables systematic evaluation and development of architecture-aware, reasoning-efficient LLMs to bridge architectural gaps in real-world software deployment and portability.

Abstract

During migration across instruction set architectures (ISAs), software package build repair is a critical task for ensuring the reliability of software deployment and the stability of modern operating systems. While Large Language Models (LLMs) have shown promise in tackling this challenge, prior work has primarily focused on single instruction set architecture (ISA) and homogeneous programming languages. To address this limitation, we introduce a new benchmark designed for software package build repair across diverse architectures and languages. Comprising 268 real-world software package build failures, the benchmark provides a standardized evaluation pipeline. We evaluate six state-of-the-art LLMs on the benchmark, and the results show that cross-ISA software package repair remains difficult and requires further advances. By systematically exposing this challenge, the benchmark establishes a foundation for advancing future methods aimed at improving software portability and bridging architectural gaps.

A Benchmark for Language Models in Real-World System Building

TL;DR

The paper addresses the challenge of repairing cross-ISA software package build failures during migrations between architectures. It introduces a first end-to-end benchmark built on the Model Context Protocol (MCP) and the Open Build Service (OBS), evaluating six state-of-the-art LLMs with an iterative repair loop that incorporates build feedback. The authors present a 268-case corpus spanning aarch64 and x86_64, a standardized repair pipeline with tool orchestration, and comprehensive results showing GPT-5 achieves the strongest cross-ISA repair performance albeit with substantial remaining gaps. This benchmark enables systematic evaluation and development of architecture-aware, reasoning-efficient LLMs to bridge architectural gaps in real-world software deployment and portability.

Abstract

During migration across instruction set architectures (ISAs), software package build repair is a critical task for ensuring the reliability of software deployment and the stability of modern operating systems. While Large Language Models (LLMs) have shown promise in tackling this challenge, prior work has primarily focused on single instruction set architecture (ISA) and homogeneous programming languages. To address this limitation, we introduce a new benchmark designed for software package build repair across diverse architectures and languages. Comprising 268 real-world software package build failures, the benchmark provides a standardized evaluation pipeline. We evaluate six state-of-the-art LLMs on the benchmark, and the results show that cross-ISA software package repair remains difficult and requires further advances. By systematically exposing this challenge, the benchmark establishes a foundation for advancing future methods aimed at improving software portability and bridging architectural gaps.
Paper Structure (21 sections, 2 figures, 2 tables)

This paper contains 21 sections, 2 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. METHODOLOGY
  3. Overview
  4. Benchmark Construction
  5. LLM-driven repair process
  6. MCP Tools
  7. Tool Result Cache
  8. Iterative Reasoning and Verification Loop
  9. Experiments
  10. Evaluation Metrics
  11. RQ1: Performance of LLMs on the Benchmark
  12. Cross-ISA Repair Performance
  13. Time and Token Efficiency
  14. RQ2: Impact of Iterative Feedback
  15. DISCUSSION
  16. ...and 6 more sections

Figures (2)

  • Figure 1: The automatic cross-ISA build repair pipeline of the benchmark.
  • Figure 2: Error category distribution across cross-ISA build failures. The figure illustrates the proportion of five error types across both migration directions (x86_64→aarch64 and aarch64→x86_64).