Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MAS-on-the-Fly: Dynamic Adaptation of LLM-based Multi-Agent Systems at Test Time

Guangyi Liu, Haojun Lin, Huan Zeng, Heng Wang, Quanming Yao

TL;DR

Inspired by how biological systems adapt, MASFly is introduced, a novel multi-agent framework enabling dynamic adaptation at test time, and achieves state-of-the-art performance.

Abstract

Large Language Model (LLM)-based multi-agent systems (MAS) have emerged as a promising paradigm for solving complex tasks. However, existing works often rely on manual designs or "one-size-fits-all" automation, lacking dynamic adaptability after deployment. Inspired by how biological systems adapt, we introduce MASFly, a novel multi-agent framework enabling dynamic adaptation at test time. To adapt system generation, MASFly employs a retrieval-augmented SOP instantiation mechanism that leverages a self-constructed repository of successful collaboration patterns, enabling the LLM to assemble customized MASs for new queries. For adaptive execution, MASFly incorporates an experience-guided supervision mechanism, where a dedicated Watcher agent monitors system behaviors with reference to a personalized experience pool and provides real-time interventions. Extensive experiments demonstrate that MASFly achieves state-of-the-art performance, most notably a 61.7% success rate on the TravelPlanner benchmark, while exhibiting strong task adaptability and robustness.

MAS-on-the-Fly: Dynamic Adaptation of LLM-based Multi-Agent Systems at Test Time

TL;DR

Inspired by how biological systems adapt, MASFly is introduced, a novel multi-agent framework enabling dynamic adaptation at test time, and achieves state-of-the-art performance.

Abstract

Large Language Model (LLM)-based multi-agent systems (MAS) have emerged as a promising paradigm for solving complex tasks. However, existing works often rely on manual designs or "one-size-fits-all" automation, lacking dynamic adaptability after deployment. Inspired by how biological systems adapt, we introduce MASFly, a novel multi-agent framework enabling dynamic adaptation at test time. To adapt system generation, MASFly employs a retrieval-augmented SOP instantiation mechanism that leverages a self-constructed repository of successful collaboration patterns, enabling the LLM to assemble customized MASs for new queries. For adaptive execution, MASFly incorporates an experience-guided supervision mechanism, where a dedicated Watcher agent monitors system behaviors with reference to a personalized experience pool and provides real-time interventions. Extensive experiments demonstrate that MASFly achieves state-of-the-art performance, most notably a 61.7% success rate on the TravelPlanner benchmark, while exhibiting strong task adaptability and robustness.
Paper Structure (39 sections, 2 equations, 11 figures, 7 tables)

This paper contains 39 sections, 2 equations, 11 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-based Multi-Agent System
  4. Learning at Test Time for LLMs
  5. Memory in Multi-Agent System
  6. Proposed Method
  7. Overall Framework
  8. SOP-Guided System Generation
  9. Process Supervised Execution
  10. Reflective Experience Distillation
  11. Comparison with Existing Works
  12. Experiment
  13. Experimental Setup
  14. Benchmarks.
  15. Baselines.
  16. ...and 24 more sections

Figures (11)

  • Figure 1: Drawing inspiration from biological adaptation, MASFly achieves dynamic adaptation at test-time, enabling adaptive system generation via SOP instantiation and adaptive execution via process supervision.
  • Figure 2: Overview of our framework MASFly, which consists of three stages: (1) SOP-Guided System Generation, (2) Process-Supervised Execution, and (3) Reflective Experience Distillation.
  • Figure 3: The case illustration of the SOP Repository.
  • Figure 4: The case illustration of the Personalized Experience Pool.
  • Figure 5: The efficiency and performance comparison on HumanEval.
  • ...and 6 more figures