Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control

Haofeng Xu, Junwei Su, Yukun Tian, Lansong Diao, Zhengping Qian, Chuan Wu

TL;DR

This work proposes GRADIENT ALIGNMENT CONTROL (GAC), a simple dynamics-aware stabilization method that regulates asynchronous RL progress along stale-aligned directions via gradient projection, and establishes convergence guarantees under bounded staleness and demonstrates empirically that GAC recovers stable, on-policy training dynamics and matches synchronized baselines even at high staleness.

Abstract

Asynchronous execution is essential for scaling reinforcement learning (RL) to modern large model workloads, including large language models and AI agents, but it can fundamentally alter RL optimization behavior. While prior work on asynchronous RL focuses on training throughput and distributional correction, we show that naively applying asynchrony to policy-gradient updates can induce qualitatively different training dynamics and lead to severe training instability. Through systematic empirical and theoretical analysis, we identify a key signature of this instability: asynchronous training exhibits persistently high cosine similarity between consecutive policy gradients, in contrast to the near-orthogonal updates observed under synchronized training. This stale-aligned gradient effect amplifies correlated updates and increases the risk of overshooting and divergence. Motivated by this observation, we propose GRADIENT ALIGNMENT CONTROL(GAC), a simple dynamics-aware stabilization method that regulates asynchronous RL progress along stale-aligned directions via gradient projection. We establish convergence guarantees under bounded staleness and demonstrate empirically that GAC recovers stable, on-policy training dynamics and matches synchronized baselines even at high staleness.

GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control

TL;DR

This work proposes GRADIENT ALIGNMENT CONTROL (GAC), a simple dynamics-aware stabilization method that regulates asynchronous RL progress along stale-aligned directions via gradient projection, and establishes convergence guarantees under bounded staleness and demonstrates empirically that GAC recovers stable, on-policy training dynamics and matches synchronized baselines even at high staleness.

Abstract

Asynchronous execution is essential for scaling reinforcement learning (RL) to modern large model workloads, including large language models and AI agents, but it can fundamentally alter RL optimization behavior. While prior work on asynchronous RL focuses on training throughput and distributional correction, we show that naively applying asynchrony to policy-gradient updates can induce qualitatively different training dynamics and lead to severe training instability. Through systematic empirical and theoretical analysis, we identify a key signature of this instability: asynchronous training exhibits persistently high cosine similarity between consecutive policy gradients, in contrast to the near-orthogonal updates observed under synchronized training. This stale-aligned gradient effect amplifies correlated updates and increases the risk of overshooting and divergence. Motivated by this observation, we propose GRADIENT ALIGNMENT CONTROL(GAC), a simple dynamics-aware stabilization method that regulates asynchronous RL progress along stale-aligned directions via gradient projection. We establish convergence guarantees under bounded staleness and demonstrate empirically that GAC recovers stable, on-policy training dynamics and matches synchronized baselines even at high staleness.
Paper Structure (71 sections, 6 theorems, 52 equations, 7 figures, 3 tables, 1 algorithm)

This paper contains 71 sections, 6 theorems, 52 equations, 7 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Limitations and Gaps in Existing Work.
  3. Contributions.
  4. Preliminaries
  5. Group-Relative Policy Optimization (GRPO)
  6. Synchronous and Asynchronous GRPO Training
  7. Training Dynamics of Asynchronous RL
  8. Staleness Induces Catastrophic Instability
  9. Gradient Alignment as a Dynamical Signature
  10. Distinct cosine-similarity behavior under synchronous and asynchronous training.
  11. Theoretical Evidence
  12. Alignment-induced instability and collapse.
  13. Gradient Alignment Control (GAC)
  14. Directional Gradient Projection
  15. Adaptive Alignment Control under Stale Rollouts
  16. ...and 56 more sections

Key Result

Theorem 3.1

Under standard smoothness and bounded-variance assumptions, and assuming locally bounded parameter drift induced by staleness, the iterates satisfy where $L^\star \triangleq \sup_\theta \mathcal{L}(\theta)$, $L$ is the smoothness constant of $\mathcal{L}(\theta)$, $\sigma^2$ bounds the variance of the gradient, $b_t$ denotes the staleness-induced bias arising from stale rollouts, and $\widehat{g}

Figures (7)

  • Figure 1: Progressive instability under increasing staleness. Panels (a--b) report training reward and validation accuracy across staleness levels. Panels (c--d) report consecutive-gradient cosine similarity $c_t$, contrasting synchronized training ($s{=}0$) with mild ($s{=}4$) and larger staleness ($s{=}8,16$). Results on MATH with Qwen3-4B.
  • Figure 2: Cross-scale learning curves under large staleness. Panels (a--d) report training reward and panels (e--h) report MinervaMath validation accuracy for Qwen3-1.7B/4B/8B and Llama-3.2-3B-Instruct. Curves compare synchronized GRPO, stale-rollout GRPO, M2PO, BAPO, and GAC.
  • Figure 3: Gradient-alignment dynamics under staleness. Consecutive-gradient cosine similarity $c_t$ for Qwen3-4B/8B, comparing synchronized GRPO, stale-rollout GRPO, M2PO, BAPO and GAC.
  • Figure 4: Robustness across staleness levels. Training reward trajectories comparing stale-rollout GRPO and GAC under $s\in\{8,16,32\}$. GAC maintains stable convergence across staleness levels, whereas stale-rollout GRPO degrades progressively as staleness increases.
  • Figure 5: Threshold robustness of GAC. Ablations around the default $(c_{\text{low}},c_{\text{high}}){=}(0.05,0.3)$ show that accuracy varies only mildly across nearby settings, supporting practical robustness without per-model retuning.
  • ...and 2 more figures

Theorems & Definitions (7)

  • Theorem 3.1: Convergence of Asynchronous GRPO, Informal
  • Proposition 4.1: Bias reduction via orthogonal projection, Informal
  • Theorem 5.1: Alignment-Aware Convergence under Staleness
  • Lemma 5.2: Alignment Expansion
  • Theorem 5.3: Convergence of Asynchronous GRPO with Alignment Persistence
  • Proposition 6.1: Bias reduction via stale-direction projection
  • proof