Behavior Knowledge Merge in Reinforced Agentic Models

TL;DR

The paper addresses how to merge multiple RL-trained, task-specific agents into a single generalist without eroding task-specific capabilities. It identifies a fundamental mismatch: RL induces sparse, heterogeneous task vectors that are not well served by dense, average-based merging used in SFT-based methods. The proposed Reinforced Agent Merging (RAM) disentangles shared and unique parameter updates and employs a distribution-aware rescaling (RAM+) to counteract signal dilution, showing state-of-the-art performance across coding, tool-use, and memory domains and across architectures. The results demonstrate that RAM enables synergistic cross-task improvements, often surpassing the specialized agents themselves, thus offering a practical path to scalable, multi-task RL-enabled agentic systems.

Abstract

Reinforcement learning (RL) is central to post-training, particularly for agentic models that require specialized reasoning behaviors. In this setting, model merging offers a practical mechanism for integrating multiple RL-trained agents from different tasks into a single generalist model. However, existing merging methods are designed for supervised fine-tuning (SFT), and they are suboptimal to preserve task-specific capabilities on RL-trained agentic models. The root is a task-vector mismatch between RL and SFT: on-policy RL induces task vectors that are highly sparse and heterogeneous, whereas SFT-style merging implicitly assumes dense and globally comparable task vectors. When standard global averaging is applied under this mismatch, RL's non-overlapping task vectors that encode critical task-specific behaviors are reduced and parameter updates are diluted. To address this issue, we propose Reinforced Agent Merging (RAM), a distribution-aware merging framework explicitly designed for RL-trained agentic models. RAM disentangles shared and task-specific unique parameter updates, averaging shared components while selectively preserving and rescaling unique ones to counteract parameter update dilution. Experiments across multiple agent domains and model architectures demonstrate that RAM not only surpasses merging baselines, but also unlocks synergistic potential among agents to achieve performance superior to that of specialized agents in their domains.

Figures (9)

• Figure 1: Performance comparison of RAM/RAM+ and baselines on 12 tasks across 3 agent domains. Our method achieves the best average performance and secures SOTA results on 9 out of 12 tasks, surpassing even the original specialized agents (Coding, Memory, Tool).
• Figure 2: Left: Density (1-Sparsity) of task vectors varies between agent models. Right: Non-zero elements distributions of task vector varies on the number of overlaps with other task vectors.
• Figure 3: The performance gain (%) of merging unique regions of reinforced task vectors across domains.
• Figure 4: Method Overview.(a) A base model is trained via RL to different agents, we track the distributions of obtained reinforced task vectors. (b) Probing the distribution of task vectors to shared, unique, unchanged sets. (c) Selective merging task vectors by averaging shared regions and rescaling unique regions to the base model.
• Figure 5: The performances of merging two agents across domains.