Scaling Multiagent Systems with Process Rewards

TL;DR

MAPPA tackles credit assignment and sample efficiency in finetuning multiple specialized agents solving long-horizon tasks. It introduces per-action process rewards provided by an AI coach, enabling dense supervision without ground-truth labels and implicit credit assignment. The method leverages REINFORCE++ with KL-penalized rewards and a PPO-style objective within a scalable distributed architecture, and demonstrates strong gains on MathChat (AIME/AMC) and DSBench (data pipelines). The results suggest that scaling the number of specialized agents with coach-guided reinforcement learning can yield substantial improvements across domains, albeit with challenges like coach bias and reward hacking that merit future work. The work advances scalable, low-supervision multiagent systems for complex, tool-augmented tasks and provides a foundation for further research into trainable coaches and reward backpropagation.

Abstract

While multiagent systems have shown promise for tackling complex tasks via specialization, finetuning multiple agents simultaneously faces two key challenges: (1) credit assignment across agents, and (2) sample efficiency of expensive multiagent rollouts. In this work, we propose finetuning multiagent systems with per-action process rewards from AI feedback (MAPPA) to address both. Through assigning credit to individual agent actions rather than only at task completion, MAPPA enables fine-grained supervision without ground truth labels while extracting maximal training signal from each rollout. We demonstrate our approach on competition math problems and tool-augmented data analysis tasks. On unseen math problems, MAPPA achieves +5.0--17.5pp on AIME and +7.8--17.2pp on AMC. For data analysis tasks, our method improves success rate by +12.5pp while quality metrics improve by up to 30%, validating that per-action supervision can lead to improvements across different multiagent system on various domains. By addressing these challenges, our work takes a first step toward scaling multiagent systems for complex, long-horizon tasks with minimal human supervision.

Figures (7)

• Figure 1: Multiagent architectures with separate weights enable specialization through end-to-end training, sidestepping catastrophic forgetting that limits single-model scaling just like mixture-of-experts (MoE) architecture.
• Figure 2: Agent execution loop with per-action coach evaluation.
• Figure 3: Behavioral metrics during MathChat training. Code Executor Tool Call % (top-left) shows the percentage of Code Executor actions that contain a tool call; the other three panels show average response length (in tokens) for each agent at each training step. Faint lines show raw data; solid lines show exponential moving average ($\alpha=1/3$). Qwen3-4B shows dramatically increased tool usage and more concise responses, while R1-Distill-Qwen-1.5B maintains stable behavior throughout training.
• Figure 4: DSBench three-agent pipeline. Each agent executes Python code via a shared sandbox, reading inputs from and writing outputs to a shared file workspace. The Data Engineer preprocesses raw CSV files into pickle artifacts; the Modeler trains and saves a model; the Analyst generates the final submission for evaluation.
• Figure 5: DSBench training dynamics over 84 steps (21 epochs). Light points show raw metrics; solid line shows EMA ($\alpha$=0.3). Dashed lines mark peak raw values. Classification metrics peak early (steps 44--48) then decline, while regression RMSE continues improving through step 84, illustrating specialization to regression tasks.