$n$-Musketeers: Reinforcement Learning Shapes Collaboration Among Language Models

TL;DR

This work tackles the challenge of combining multiple frozen, off-the-shelf language models to perform complex reasoning under RLVR without retraining the experts. It introduces soft hidden-state collaboration via a trainable Perceiver-style latent interface that converts distributed hidden representations into context tokens, conditioning a single RLVR-trained policy. Through experiments on Reasoning Gym and GSM8K, the approach yields competitive results and reveals emergent, reward-aligned expert roles, while ablations show the usefulness of the latent interface is task-dependent and benefits are not universal. The core contribution is a plug-and-play, representation-level fusion mechanism that provides observability into expert utilization and demonstrates that structured collaboration can arise under outcome-based supervision, enabling modular, multi-expert reasoning without expensive joint fine-tuning. This has practical implications for building scalable agentic systems from heterogeneous, independently developed SLMs, guiding when to leverage latent expert conditioning and how to observe its dynamics during RLVR training.

Abstract

Recent progress in reinforcement learning with verifiable rewards (RLVR) shows that small, specialized language models (SLMs) can exhibit structured reasoning without relying on large monolithic LLMs. We introduce soft hidden-state collaboration, where multiple heterogeneous frozen SLM experts are integrated through their internal representations via a trainable attention interface. Experiments on Reasoning Gym and GSM8K show that this latent integration is competitive with strong single-model RLVR baselines. Ablations further reveal a dual mechanism of expert utilization: for simpler arithmetic domains, performance gains can largely be explained by static expert preferences, whereas more challenging settings induce increasingly concentrated and structured expert attention over training, indicating emergent specialization in how the router connects to relevant experts. Overall, hidden-state collaboration provides a compact mechanism for leveraging frozen experts, while offering an observational window into expert utilization patterns and their evolution under RLVR.

Figures (6)

• Figure 1: During training with RLVR, expert LM utilization becomes increasingly structured, without routing or expert-role supervision. Left: reward during training. Right: change in expert attention from initialization, illustrating emergent expert roles.
• Figure 2: Comparison of (a) output-level textual collaboration, (b) hard expert routing, and (c) our hidden state soft prompting approach, highlighting differences in computational efficiency, expert utilization, and observability of expert utilization.
• Figure 3: Perceiver-based hidden state collaboration framework. Frozen heterogeneous SLM experts expose final-layer hidden states, which are aligned and aggregated via trainable latent query tokens using cross-attention. The resulting context tokens condition the final policy model trained with RLVR.
• Figure 4: Expert utilization dynamics and training progress under RLVR. Top row: evolution of per-expert utilization over training steps for different tasks and expert team compositions (Default Team, LLM Team replacement, and Naive Team). Shaded regions indicate standard deviation across runs. Bottom row: corresponding reward progression and routing entropy over training. The emergence of selective expert usage without routing supervision suggests that RLVR alone can induce self-organized expert role differentiation.
• Figure 5: Ablation comparing the proposed model (with $\mathcal{C}$) and a baseline without $\mathcal{C}$ on Logic and Arithmetic. Models with $\mathcal{C}$ achieve consistently higher rewards.