Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering

Chuanzhe Guo, Jingjing Wu, Sijun He, Yang Chen, Zhaoqi Kuang, Shilong Fan, Bingjin Chen, Siqi Bao, Jing Liu, Hua Wu, Qingfu Zhu, Wanxiang Che, Haifeng Wang

TL;DR

MEnvAgent tackles the scalability and verifiability gap in polyglot SWE environment construction by introducing a Planning-Execution-Verification multi-agent framework augmented with an Environment Reuse Mechanism. The system retrieves and incrementally patches historical environments to rapidly generate executable task environments across 10 languages, validated on the 1,000-task MEnvBench and culminating in the large MEnvData-SWE verifiable dataset. Empirically, MEnvAgent improves F2P by 8.6% and Pass rates by 11.0%, while cutting time costs by 43% on average compared with strong baselines, demonstrating robust cross-language performance and efficiency. The work enables scalable, execution-based data generation for SWE and downstream model training, with open-source code, benchmarks, and datasets to accelerate future research in verifiable software engineering.

Abstract

The evolution of Large Language Model (LLM) agents for software engineering (SWE) is constrained by the scarcity of verifiable datasets, a bottleneck stemming from the complexity of constructing executable environments across diverse languages. To address this, we introduce MEnvAgent, a Multi-language framework for automated Environment construction that facilitates scalable generation of verifiable task instances. MEnvAgent employs a multi-agent Planning-Execution-Verification architecture to autonomously resolve construction failures and integrates a novel Environment Reuse Mechanism that reduces computational overhead by incrementally patching historical environments. Evaluations on MEnvBench, a new benchmark comprising 1,000 tasks across 10 languages, demonstrate that MEnvAgent outperforms baselines, improving Fail-to-Pass (F2P) rates by 8.6% while reducing time costs by 43%. Additionally, we demonstrate the utility of MEnvAgent by constructing MEnvData-SWE, the largest open-source polyglot dataset of realistic verifiable Docker environments to date, alongside solution trajectories that enable consistent performance gains on SWE tasks across a wide range of models. Our code, benchmark, and dataset are available at https://github.com/ernie-research/MEnvAgent.

MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering

TL;DR

MEnvAgent tackles the scalability and verifiability gap in polyglot SWE environment construction by introducing a Planning-Execution-Verification multi-agent framework augmented with an Environment Reuse Mechanism. The system retrieves and incrementally patches historical environments to rapidly generate executable task environments across 10 languages, validated on the 1,000-task MEnvBench and culminating in the large MEnvData-SWE verifiable dataset. Empirically, MEnvAgent improves F2P by 8.6% and Pass rates by 11.0%, while cutting time costs by 43% on average compared with strong baselines, demonstrating robust cross-language performance and efficiency. The work enables scalable, execution-based data generation for SWE and downstream model training, with open-source code, benchmarks, and datasets to accelerate future research in verifiable software engineering.

Abstract

The evolution of Large Language Model (LLM) agents for software engineering (SWE) is constrained by the scarcity of verifiable datasets, a bottleneck stemming from the complexity of constructing executable environments across diverse languages. To address this, we introduce MEnvAgent, a Multi-language framework for automated Environment construction that facilitates scalable generation of verifiable task instances. MEnvAgent employs a multi-agent Planning-Execution-Verification architecture to autonomously resolve construction failures and integrates a novel Environment Reuse Mechanism that reduces computational overhead by incrementally patching historical environments. Evaluations on MEnvBench, a new benchmark comprising 1,000 tasks across 10 languages, demonstrate that MEnvAgent outperforms baselines, improving Fail-to-Pass (F2P) rates by 8.6% while reducing time costs by 43%. Additionally, we demonstrate the utility of MEnvAgent by constructing MEnvData-SWE, the largest open-source polyglot dataset of realistic verifiable Docker environments to date, alongside solution trajectories that enable consistent performance gains on SWE tasks across a wide range of models. Our code, benchmark, and dataset are available at https://github.com/ernie-research/MEnvAgent.
Paper Structure (69 sections, 8 equations, 10 figures, 12 tables)

This paper contains 69 sections, 8 equations, 10 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. MEnvAgent Design
  4. Multi-Agent Architecture
  5. Planning Stage.
  6. Execution Stage.
  7. Verification Stage.
  8. Environment Reuse Mechanism
  9. Environment Retrieval.
  10. Verification-Driven Adaptation.
  11. MEnvBench Construction
  12. Data Collection and Filtering
  13. Phase 1: Repository Acquisition.
  14. Phase 2: Instance Extraction & Quality Assurance.
  15. Benchmark Composition
  16. ...and 54 more sections

Figures (10)

  • Figure 1: Comparison between manual environment construction and MEnvAgent (Ours). MEnvAgent leverages multi-agent collaboration to achieve automated environment construction, characterized by an efficient environment reuse mechanism.
  • Figure 2: Overview of MEnvAgent. (Top) The Environment Reuse Mechanism retrieves and adapts historical environments via incremental patching to reduce overhead. (Bottom) The Planning-Execution-Verification loop, where agents autonomously draft scripts, interactively repair build errors, and diagnose test failures to guide iterative refinement.
  • Figure 3: Performance trade-off analysis on MEnvBench. The x-axis represents the average time cost (lower is better), and the y-axis represents the pass rate (higher is better). MEnvAgent points cluster in the top-left region, indicating it achieves higher validity and success rates with significantly lower time consumption compared to baselines.
  • Figure 4: Impact of data scale on performance metrics. We illustrate the trends of (a) Reuse Success Rate, (b) Time Cost, and (c) Pass Rate as the number of instances per repository increases from 1 to 10. The results confirm that larger data scale significantly enhances reuse probability and overall efficiency.
  • Figure 5: F2P performance analysis relative to repository size.
  • ...and 5 more figures