Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Large-Scale Study on the Development and Issues of Multi-Agent AI Systems

Daniel Liu, Krishna Upadhyay, Vinaik Chhetri, A. B. Siddique, Umar Farooq

TL;DR

This paper addresses how open-source multi-agent AI systems (MAS) evolve and are maintained in practice. It employs a large-scale empirical analysis of eight MAS repositories, analyzing 42,267 commits and 4,731 resolved issues to characterize development patterns and maintenance practices. It finds three development profiles and a dominance of perfective commits, alongside heterogeneous issue landscapes with a surge in activity after 2023, highlighting momentum and fragility in the MAS ecosystem. By releasing a curated dataset and providing actionable insights, it informs tooling, testing infrastructure, and documentation efforts to improve long-term reliability of MAS.

Abstract

The rapid emergence of multi-agent AI systems (MAS), including LangChain, CrewAI, and AutoGen, has shaped how large language model (LLM) applications are developed and orchestrated. However, little is known about how these systems evolve and are maintained in practice. This paper presents the first large-scale empirical study of open-source MAS, analyzing over 42K unique commits and over 4.7K resolved issues across eight leading systems. Our analysis identifies three distinct development profiles: sustained, steady, and burst-driven. These profiles reflect substantial variation in ecosystem maturity. Perfective commits constitute 40.8% of all changes, suggesting that feature enhancement is prioritized over corrective maintenance (27.4%) and adaptive updates (24.3%). Data about issues shows that the most frequent concerns involve bugs (22%), infrastructure (14%), and agent coordination challenges (10%). Issue reporting also increased sharply across all frameworks starting in 2023. Median resolution times range from under one day to about two weeks, with distributions skewed toward fast responses but a minority of issues requiring extended attention. These results highlight both the momentum and the fragility of the current ecosystem, emphasizing the need for improved testing infrastructure, documentation quality, and maintenance practices to ensure long-term reliability and sustainability.

A Large-Scale Study on the Development and Issues of Multi-Agent AI Systems

TL;DR

This paper addresses how open-source multi-agent AI systems (MAS) evolve and are maintained in practice. It employs a large-scale empirical analysis of eight MAS repositories, analyzing 42,267 commits and 4,731 resolved issues to characterize development patterns and maintenance practices. It finds three development profiles and a dominance of perfective commits, alongside heterogeneous issue landscapes with a surge in activity after 2023, highlighting momentum and fragility in the MAS ecosystem. By releasing a curated dataset and providing actionable insights, it informs tooling, testing infrastructure, and documentation efforts to improve long-term reliability of MAS.

Abstract

The rapid emergence of multi-agent AI systems (MAS), including LangChain, CrewAI, and AutoGen, has shaped how large language model (LLM) applications are developed and orchestrated. However, little is known about how these systems evolve and are maintained in practice. This paper presents the first large-scale empirical study of open-source MAS, analyzing over 42K unique commits and over 4.7K resolved issues across eight leading systems. Our analysis identifies three distinct development profiles: sustained, steady, and burst-driven. These profiles reflect substantial variation in ecosystem maturity. Perfective commits constitute 40.8% of all changes, suggesting that feature enhancement is prioritized over corrective maintenance (27.4%) and adaptive updates (24.3%). Data about issues shows that the most frequent concerns involve bugs (22%), infrastructure (14%), and agent coordination challenges (10%). Issue reporting also increased sharply across all frameworks starting in 2023. Median resolution times range from under one day to about two weeks, with distributions skewed toward fast responses but a minority of issues requiring extended attention. These results highlight both the momentum and the fragility of the current ecosystem, emphasizing the need for improved testing infrastructure, documentation quality, and maintenance practices to ensure long-term reliability and sustainability.
Paper Structure (18 sections, 8 figures, 6 tables)

This paper contains 18 sections, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Multi-Agent AI System
  4. Related Work
  5. Methodology
  6. Study Objective and Research Questions
  7. Dataset Construction
  8. Data Collection
  9. Preprocessing
  10. Results
  11. Development and Contribution Patterns (RQ1)
  12. What are the distinct commit activity patterns across repositories?
  13. What is the distribution of commit types in MAS repositories?
  14. Issue Landscape (RQ2)
  15. How do issue patterns evolve over time across frameworks?
  16. ...and 3 more sections

Figures (8)

  • Figure 1: Development and maintenance activity across eight major multi-agent AI systems, illustrating the commits and issues used in our large-scale study.
  • Figure 2: Commit activity patterns across multi-agent AI frameworks. (a) Cumulative development growth. (b) Variation in monthly commit regularity, higher means irregular patterns.
  • Figure 3: Monthly commit activity sparklines for each repository showing temporal patterns.
  • Figure 4: Code churn patterns showing lines added, lines deleted, and files changed across different development profiles
  • Figure 5: Ecosystem-level code evolution in multi-agent frameworks. (a) Cumulative growth shows overall expansion in code and file changes. (b) Temporal distribution highlights the balance between code additions and deletions over time.
  • ...and 3 more figures