Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository

Zhiyuan Peng, Xin Yin, Pu Zhao, Fangkai Yang, Lu Wang, Ran Jia, Xu Chen, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

TL;DR

RepoGenesis tackles the gap in evaluating end-to-end, repository-level microservice generation by providing a multilingual benchmark that spans 106 Python/Java repositories, 1,258 API endpoints, and 2,335 tests, all vetted through a rigorous review-rebuttal QA process. The benchmark assesses architecture, dependency management, and cross-file consistency via three metrics—Pass@1, API Coverage (AC), and Deployment Success Rate (DSR)—across multiple coding agents and IDEs. Key findings show a persistent gap between code synthesis and deployable system construction, with best Pass@1 scores around the mid-twenties on Python and Java, and a notable AC-DSR decoupling where API skeletons are functional but deployments fail. Fine-tuning a model with RepoGenesis data (GenesisAgent-8B) closes the gap somewhat, achieving GPT-5 mini-like performance and demonstrating the benchmark’s value for model development and benchmarking in real-world 0-to-1 microservice generation.

Abstract

Large language models and agents have achieved remarkable progress in code generation. However, existing benchmarks focus on isolated function/class-level generation (e.g., ClassEval) or modifications to existing codebases (e.g., SWE-Bench), neglecting complete microservice repository generation that reflects real-world 0-to-1 development workflows. To bridge this gap, we introduce RepoGenesis, the first multilingual benchmark for repository-level end-to-end web microservice generation, comprising 106 repositories (60 Python, 46 Java) across 18 domains and 11 frameworks, with 1,258 API endpoints and 2,335 test cases verified through a "review-rebuttal" quality assurance process. We evaluate open-source agents (e.g., DeepCode) and commercial IDEs (e.g., Cursor) using Pass@1, API Coverage (AC), and Deployment Success Rate (DSR). Results reveal that despite high AC (up to 73.91%) and DSR (up to 100%), the best-performing system achieves only 23.67% Pass@1 on Python and 21.45% on Java, exposing deficiencies in architectural coherence, dependency management, and cross-file consistency. Notably, GenesisAgent-8B, fine-tuned on RepoGenesis (train), achieves performance comparable to GPT-5 mini, demonstrating the quality of RepoGenesis for advancing microservice generation. We release our benchmark at https://github.com/pzy2000/RepoGenesis.

RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository

TL;DR

RepoGenesis tackles the gap in evaluating end-to-end, repository-level microservice generation by providing a multilingual benchmark that spans 106 Python/Java repositories, 1,258 API endpoints, and 2,335 tests, all vetted through a rigorous review-rebuttal QA process. The benchmark assesses architecture, dependency management, and cross-file consistency via three metrics—Pass@1, API Coverage (AC), and Deployment Success Rate (DSR)—across multiple coding agents and IDEs. Key findings show a persistent gap between code synthesis and deployable system construction, with best Pass@1 scores around the mid-twenties on Python and Java, and a notable AC-DSR decoupling where API skeletons are functional but deployments fail. Fine-tuning a model with RepoGenesis data (GenesisAgent-8B) closes the gap somewhat, achieving GPT-5 mini-like performance and demonstrating the benchmark’s value for model development and benchmarking in real-world 0-to-1 microservice generation.

Abstract

Large language models and agents have achieved remarkable progress in code generation. However, existing benchmarks focus on isolated function/class-level generation (e.g., ClassEval) or modifications to existing codebases (e.g., SWE-Bench), neglecting complete microservice repository generation that reflects real-world 0-to-1 development workflows. To bridge this gap, we introduce RepoGenesis, the first multilingual benchmark for repository-level end-to-end web microservice generation, comprising 106 repositories (60 Python, 46 Java) across 18 domains and 11 frameworks, with 1,258 API endpoints and 2,335 test cases verified through a "review-rebuttal" quality assurance process. We evaluate open-source agents (e.g., DeepCode) and commercial IDEs (e.g., Cursor) using Pass@1, API Coverage (AC), and Deployment Success Rate (DSR). Results reveal that despite high AC (up to 73.91%) and DSR (up to 100%), the best-performing system achieves only 23.67% Pass@1 on Python and 21.45% on Java, exposing deficiencies in architectural coherence, dependency management, and cross-file consistency. Notably, GenesisAgent-8B, fine-tuned on RepoGenesis (train), achieves performance comparable to GPT-5 mini, demonstrating the quality of RepoGenesis for advancing microservice generation. We release our benchmark at https://github.com/pzy2000/RepoGenesis.
Paper Structure (88 sections, 5 equations, 3 figures, 18 tables)

This paper contains 88 sections, 5 equations, 3 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Benchmark -- RepoGenesis
  3. Statistics of RepoGenesis
  4. Task Definition and Challenges
  5. Evaluation Pipeline
  6. Repository-level Code Generation
  7. Post-Generation Evaluation
  8. Benchmark Construction
  9. Repository Collection
  10. Repository Preprocessing
  11. Test Case & Quality Assurance
  12. Blind Review
  13. Refinement and Consensus
  14. Inter-Annotator Agreement
  15. Difficulty Classification
  16. ...and 73 more sections

Figures (3)

  • Figure 1: Capability map of agents and IDEs
  • Figure 2: Overview of benchmark construction and evaluation pipeline, which undergo rigorous quality assurance through a "Review-Rebuttal QA". Given a requirement document (README.md), various agents and IDEs generate repositories. Repositories are executed in sandboxed environments and evaluated across DSR, AC, and Pass@1.
  • Figure 3: Trajectory showing Cursor (GPT-4.1) stuck in a non-execution loop. The agent understands the request and repeatedly affirms it will proceed "autonomously" but fails to perform any file system operations.