Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learning

TL;DR

Seer tackles the rollout bottleneck in synchronous RL for reasoning LLMs by exploiting intra-group contextual signals. It introduces Divided Rollout, Context-Aware Scheduling, and Adaptive Grouped Speculative Decoding to balance memory usage and accelerate inference without sacrificing on-policy fidelity. Across production-grade RL workloads, Seer delivers $74\%-97\%$ throughput gains and $75\%-93\%$ reductions in long-tail latency, demonstrating strong practical impact for faster RL iteration. The approach combines a global KVCache, CST-based speculation, and fine-grained scheduling to achieve scalable, lossless rollout acceleration.

Abstract

Reinforcement Learning (RL) has become critical for advancing modern Large Language Models (LLMs), yet existing synchronous RL systems face severe performance bottlenecks. The rollout phase, which dominates end-to-end iteration time, suffers from substantial long-tail latency and poor resource utilization due to inherent workload imbalance. We present Seer, a novel online context learning system that addresses these challenges by exploiting previously overlooked similarities in output lengths and generation patterns among requests sharing the same prompt. Seer introduces three key techniques: divided rollout for dynamic load balancing, context-aware scheduling, and adaptive grouped speculative decoding. Together, these mechanisms substantially reduce long-tail latency and improve resource efficiency during rollout. Evaluations on production-grade RL workloads demonstrate that Seer improves end-to-end rollout throughput by 74% to 97% and reduces long-tail latency by 75% to 93% compared to state-of-the-art synchronous RL systems, significantly accelerating RL training iterations.

Figures (12)

• Figure 1: Challenges and Seer's solution for long-generation rollout. Conventional group-level rollout treats request groups as monolithic units, leading to severe inter-instance and intra-instance load imbalance. Seer achieves dynamic load balancing and prevents preemption through divided rollout. Building upon divided rollout, Seer implements online context learning, enabling context-aware scheduling and adaptive grouped speculative decoding to further reduce rollout time.
• Figure 2: Distribution of output lengths during rollout across three reasoning tasks. The generation lengths span from hundreds to 98K tokens, demonstrating both high average length and extreme variance.
• Figure 3: KVCache utilization, number of running requests, and preemption count during a synchronous rollout phase of the Qwen2-VL-72B task. In the early stage of rollout, insufficient KVCache capacity causes frequent request preemptions; in the later stage, a small number of extremely long request groups contribute to a long-tail period that accounts for nearly half of the total rollout time.
• Figure 4: Length correlation within response groups. Each column represents a prompt group in GRPO rollout, and each cell corresponds to an individual response. The color intensity indicates output length. Responses within the same group exhibit strong length correlation, with most groups showing similar generation lengths across their requests.
• Figure 5: The overview of Seer.