MAS-ZERO: Designing Multi-Agent Systems with Zero Supervision

TL;DR

This paper introduces MAS-ZERO, the first inference-time automatic MAS design framework that operates with zero supervision. It uses a meta-agent to iteratively design, critique, and refine problem-specific MAS configurations, enabling dynamic decomposition and the ability to revert to simpler systems without validation data. Through MAS-Init, MAS-Evolve (meta-design and meta-feedback), and MAS-Verify, MAS-ZERO constructs per-task MAS architectures and selects reliable solutions, demonstrating substantial accuracy gains across reasoning, coding, and agentic tasks while maintaining cost efficiency. The framework also builds a persistent experience library to support continual improvement and highlights the potential for stronger verifiers to unlock further gains.

Abstract

Multi-agent systems (MAS) leveraging the impressive capabilities of Large Language Models (LLMs) hold significant potential for tackling complex tasks. However, most current MAS depend on manually designed agent roles and communication protocols. These manual designs often fail to align with the underlying LLMs' strengths and struggle to adapt to novel tasks. Recent automatic MAS approaches attempt to mitigate these limitations but typically necessitate a validation set for tuning and yield static MAS designs lacking adaptability during inference, while also removing the flexibility to reduce to simpler systems. We introduce MAS-ZERO, the first self-evolved, inference-time framework for automatic MAS design. MAS-ZERO employs meta-level design to iteratively design, critique, and refine MAS configurations tailored to each problem instance, without requiring a validation set. Critically, it enables dynamic problem decomposition and agent composition through meta-feedback on solvability and completeness, and reduction to simpler systems when appropriate. Experiments across reasoning (math and graduate-level QA), coding, and agentic (search-based) benchmarks, using both closed-source and open-source LLM backbones of varying sizes, demonstrate that MAS-ZERO outperforms strong manual and automatic MAS baselines. It achieves substantial average accuracy improvements of up to 16.69% on reasoning, 16.66% on coding, and 5.45% on agentic tasks, while maintaining cost efficiency.

Figures (14)

• Figure 1: Scatter plots comparing the Pareto fronts of various GPT-4o-based systems on three benchmarks, including manual MAS, automatic MAS and MAS-Zero. MAS-Zero delivers high performance at lower cost than comparable automatic MAS methods, establishing a new frontier for accuracy vs. cost trade-off.
• Figure 2: Conceptual comparison of MAS-Zero, with existing automatic and manual MAS designs. MAS-Zero avoids tuning MAS on validation set by maintaining a self-evolving process that iteratively designs and evaluates task-specific MAS at inference time.
• Figure 3: MAS-Zero overview. Purple highlights the given input and final output. Orange highlights the components and steps in MAS-Zero. Dashed arrows indicate the information flow within Meta-feedback. MAS-Zero takes as inputs the question and building blocks, and solves the task in three key steps: MAS-Init (Sec. \ref{['sec.mas_blocks']}), MAS-Evolve (Sec. \ref{['sec.meta_iter']}) and MAS-Verify (Sec. \ref{['sec.self_verify']}).
• Figure 4: (A) Performance gains (GPT-4o) over MAS-Evolve. (B) Performance gains (GPT-4o) given an oracle verifier. Automatic MAS baselines cannot integrate external verifiers, yielding zero improvement.
• Figure F.1: Illustration for the iterations in MAS-Evolve.