SOUP: Token-level Single-sample Mix-policy Reinforcement Learning for Large Language Models

TL;DR

SOUP introduces a token-level mix-policy paradigm that unifies off-policy and on-policy learning within single samples by restricting off-policy influence to the sequence prefix and continuing generation on-policy. This fine-grained approach uses token-level importance ratios to integrate historical policy information while maintaining training stability, improving exploration and final performance over standard on-policy training and previous off-policy extensions. Extensive experiments across model backbones and math benchmarks demonstrate robust gains, stable training dynamics, and enhanced inference diversity via relay sampling. The work provides a practical direction for more effective RL in large language models by balancing exploration, stability, and efficiency through token-level mix-policy design.

Abstract

On-policy reinforcement learning (RL) methods widely used for language model post-training, like Group Relative Policy Optimization (GRPO), often suffer from limited exploration and early saturation due to low sampling diversity. While off-policy data can help, current approaches that mix entire trajectories cause significant policy mismatch and instability. In this work, we propose the $\textbf{S}$ingle-sample Mix-p$\textbf{O}$licy $\textbf{U}$nified $\textbf{P}$aradigm (SOUP), a framework that unifies off- and on-policy learning within individual samples at the token level. It confines off-policy influence to the prefix of a generated sequence sampled from historical policies, while the continuation is generated on-policy. Through token-level importance ratios, SOUP effectively leverages off-policy information while preserving training stability. Extensive experiments demonstrate that SOUP consistently outperforms standard on-policy training and existing off-policy extensions. Our further analysis clarifies how our fine-grained, single-sample mix-policy training can improve both exploration and final performance in LLM RL.

Figures (6)

• Figure 1: Overview of SOUP. SOUP unifies off-policy and on-policy data at the token level within a single sample. The sequence prefix is sampled from the behavior policy and kept after truncation. The current policy then generates the remaining tokens conditioned on this prefix to form a complete sequence. Token-wise importance ratios are computed using the corresponding policies.
• Figure 2: Training rewards of different methods.
• Figure 3: Word cloud diagrams of tokens at the truncation points under different truncation strategies.
• Figure 4: The pass@$k$ difference between SOUP inference and single-model inference at the inference phase. The greater the value above the baseline of 0, the better SOUP performs.
• Figure 5: The relationship between entropy, clipping ratio, and relative position ratio (tokens are binned into 10% intervals) under different training strategies.