Agent-Based Software Artifact Evaluation

TL;DR

ArtifactCopilot reframes artifact evaluation as a state-aware, graph-guided workflow problem and delivers the first end-to-end agent-based framework to automate the process from paper-derived instructions to badge determination. It introduces the AE Graph to capture execution-state and dependency information, and employs a Planning Agent for environment normalization together with an Evaluation Agent for robust execution and recovery. Across 48 real-world artifacts, ArtifactCopilot attains a Badge Consistency Rate of 85.42%, vastly outperforming static script generation and interactive code baselines while requiring near-zero human intervention and low per-artifact cost. The work provides empirical insights into standardizing artifact evaluation practices and outlines design principles for scalable, automated evaluation systems with broad implications for reproducibility in software engineering research.

Abstract

Artifact evaluation has been adopted in the Software Engineering (SE) research community for 15 years, substantially improving research reproducibility across major SE conferences. However, this success has introduced a growing scalability challenge, as artifact evaluation relies heavily on reviewers' manual execution and debugging, leading to escalating human effort amid rapidly increasing paper submissions. To address this problem, we investigate automated artifact evaluation. We first conduct a preliminary study on artifacts from top-tier SE conferences and identify three key challenges: perceiving execution states, maintaining stable execution environments, and recovering from execution errors. Inspired by these findings, we propose ArtifactCopilot, the first end-to-end agent-based framework for automated artifact evaluation. ArtifactCopilot automates environment construction, instruction execution, and error recovery by combining an execution normalization strategy to ensure environment stability with an artifact evaluation graph that transforms README documents into dependency-aware command graphs, enabling structured execution planning, execution-state tracking, and error recovery. Evaluation on 48 real-world artifacts shows that ArtifactCopilot matches human artifact evaluation outcomes for 85.42% of the artifacts, outperforming Claude Code by 52.09 percentage points, while costing only \$0.091 per artifact on average and requiring zero human intervention for 45 out of 48 artifacts.

Figures (7)

• Figure 1: Three commonly used artifact evaluation badges.
• Figure 2: Overview of dataset construction process.
• Figure 3: Representative examples of automation failures and manual intervention during artifact evaluation.
• Figure 4: ArtifactCopilot workflow: from paper PDF to badge determination through structured modeling and hierarchical execution.
• Figure 5: Example transformation from README documentation to an AE Graph.