Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Collaborative Multi-Agent Test-Time Reinforcement Learning for Reasoning

Zhiyuan Hu, Yunhai Hu, Juncheng Liu, Shuyue Stella Li, Yucheng Wang, Zhen Xu, See-Kiong Ng, Anh Tuan Luu, Xinxing Xu, Bryan Hooi, Cynthia Breazeal, Hae Won Park

TL;DR

MATTRL presents a lightweight, test-time adaptation framework for multi-agent reasoning that injects structured textual experiences into dialogue without updating model weights. By forming a team of specialists, constructing a curated pool of high-value experiences from the conversation, and retrieving them to guide subsequent deliberation, MATTRL improves robustness to distribution shift across medicine, math, and education benchmarks. Key findings show that Difference rewards offer strong precision advantages for credit assignment, and an adaptive router that can switch between single-agent and MATTRL modes yields additional gains. The work highlights practical trade-offs in compute and experience drift, and it demonstrates that structured, test-time experiences can outperform both single-agent baselines and prompt engineering alone. Overall, MATTRL provides a scalable, auditable, and distribution-shift-robust path to enhanced multi-agent reasoning in real-world tasks.

Abstract

Multi-agent systems have evolved into practical LLM-driven collaborators for many applications, gaining robustness from diversity and cross-checking. However, multi-agent RL (MARL) training is resource-intensive and unstable: co-adapting teammates induce non-stationarity, and rewards are often sparse and high-variance. Therefore, we introduce \textbf{Multi-Agent Test-Time Reinforcement Learning (MATTRL)}, a framework that injects structured textual experience into multi-agent deliberation at inference time. MATTRL forms a multi-expert team of specialists for multi-turn discussions, retrieves and integrates test-time experiences, and reaches consensus for final decision-making. We also study credit assignment for constructing a turn-level experience pool, then reinjecting it into the dialogue. Across challenging benchmarks in medicine, math, and education, MATTRL improves accuracy by an average of 3.67\% over a multi-agent baseline, and by 8.67\% over comparable single-agent baselines. Ablation studies examine different credit-assignment schemes and provide a detailed comparison of how they affect training outcomes. MATTRL offers a stable, effective and efficient path to distribution-shift-robust multi-agent reasoning without tuning.

Collaborative Multi-Agent Test-Time Reinforcement Learning for Reasoning

TL;DR

MATTRL presents a lightweight, test-time adaptation framework for multi-agent reasoning that injects structured textual experiences into dialogue without updating model weights. By forming a team of specialists, constructing a curated pool of high-value experiences from the conversation, and retrieving them to guide subsequent deliberation, MATTRL improves robustness to distribution shift across medicine, math, and education benchmarks. Key findings show that Difference rewards offer strong precision advantages for credit assignment, and an adaptive router that can switch between single-agent and MATTRL modes yields additional gains. The work highlights practical trade-offs in compute and experience drift, and it demonstrates that structured, test-time experiences can outperform both single-agent baselines and prompt engineering alone. Overall, MATTRL provides a scalable, auditable, and distribution-shift-robust path to enhanced multi-agent reasoning in real-world tasks.

Abstract

Multi-agent systems have evolved into practical LLM-driven collaborators for many applications, gaining robustness from diversity and cross-checking. However, multi-agent RL (MARL) training is resource-intensive and unstable: co-adapting teammates induce non-stationarity, and rewards are often sparse and high-variance. Therefore, we introduce \textbf{Multi-Agent Test-Time Reinforcement Learning (MATTRL)}, a framework that injects structured textual experience into multi-agent deliberation at inference time. MATTRL forms a multi-expert team of specialists for multi-turn discussions, retrieves and integrates test-time experiences, and reaches consensus for final decision-making. We also study credit assignment for constructing a turn-level experience pool, then reinjecting it into the dialogue. Across challenging benchmarks in medicine, math, and education, MATTRL improves accuracy by an average of 3.67\% over a multi-agent baseline, and by 8.67\% over comparable single-agent baselines. Ablation studies examine different credit-assignment schemes and provide a detailed comparison of how they affect training outcomes. MATTRL offers a stable, effective and efficient path to distribution-shift-robust multi-agent reasoning without tuning.
Paper Structure (76 sections, 18 equations, 6 figures, 8 tables)

This paper contains 76 sections, 18 equations, 6 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-based multi-agent collaboration.
  4. Reinforcement learning for LLM reasoning.
  5. Test-time adaptation and structured experience.
  6. Credit assignment under collaboration.
  7. Methodology
  8. Multi-Expert Team Collaboration
  9. Stage I: Team formation.
  10. Stage II: Consensus via experience-augmented dialogue.
  11. Stage III: Report synthesis and final decision.
  12. Remarks.
  13. Test-Time Experience Construction
  14. Contribution ratio and terminal shared reward.
  15. Turn-level reward for each agent.
  16. ...and 61 more sections

Figures (6)

  • Figure 1: MATTRL overview. The figure uses medical diagnosis as a running example, but the framework is domain-general. Math and education instantiations are in Appendix \ref{['Math-settup']} and \ref{['education-setup']}.
  • Figure 2: GPT-5 Multi-Agent: Acc. by Team Size.
  • Figure 3: General & disease-specific experience
  • Figure 4: MATTRL in Math: Multi-Specialist Math Problem-solving Collaboration.
  • Figure 5: MATTRL in Education: Multi-Specialist Teaching Collaboration.
  • ...and 1 more figures