Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RL-VLA$^3$: Reinforcement Learning VLA Accelerating via Full Asynchronism

Zhong Guan, Haoran Sun, Yongjian Guo, Shuai Di, Xiaodong Bai, Jing Long, Tianyun Zhao, Mingxi Luo, Chen Zhou, Yucheng Guo, Qiming Yang, Wanting Xu, Wen Huang, Yunxuan Ma, Hongke Zhao, Likang Wu, Xiaotie Deng, Xi Xiao, Sheng Wen, Yicheng Gong, Junwu Xiong

TL;DR

This paper tackles the throughput bottleneck in reinforcement learning for Vision-Language-Action models by introducing RL-VLA$^3$, a fully end-to-end asynchronous framework that decouples environment interaction, rollout, and policy updates across a three-level architecture. The approach combines asynchronous training/inference, fine-grained asynchronous interaction with a dynamic batching scheduler, and a streamer-based micro-batch training pipeline to maximize hardware utilization. Empirical results on LIBERO and ManiSkill across 8–256 GPUs show substantial throughput gains (up to 59.25% in LIBERO and up to 126.67% with optimized separation) while maintaining competitive policy performance, with detailed ablations and scaling analyses. The work signals a practical pathtoward scalable, robust VLA-based embodied intelligence and outlines future directions for broader simulator support and cross-embodiment learning.

Abstract

In recent years, Vision-Language-Action (VLA) models have emerged as a crucial pathway towards general embodied intelligence, yet their training efficiency has become a key bottleneck. Although existing reinforcement learning (RL)-based training frameworks like RLinf can enhance model generalization, they still rely on synchronous execution, leading to severe resource underutilization and throughput limitations during environment interaction, policy generation (rollout), and model update phases (actor). To overcome this challenge, this paper, for the first time, proposes and implements a fully-asynchronous policy training framework encompassing the entire pipeline from environment interaction, rollout generation, to actor policy updates. Systematically drawing inspiration from asynchronous optimization ideas in large model RL, our framework designs a multi-level decoupled architecture. This includes asynchronous parallelization of environment interaction and trajectory collection, streaming execution for policy generation, and decoupled scheduling for training updates. We validated the effectiveness of our method across diverse VLA models and environments. On the LIBERO benchmark, the framework achieves throughput improvements of up to 59.25\% compared to existing synchronous strategies. When deeply optimizing separation strategies, throughput can be increased by as much as 126.67\%. We verified the effectiveness of each asynchronous component via ablation studies. Scaling law validation across 8 to 256 GPUs demonstrates our method's excellent scalability under most conditions.

RL-VLA$^3$: Reinforcement Learning VLA Accelerating via Full Asynchronism

TL;DR

This paper tackles the throughput bottleneck in reinforcement learning for Vision-Language-Action models by introducing RL-VLA, a fully end-to-end asynchronous framework that decouples environment interaction, rollout, and policy updates across a three-level architecture. The approach combines asynchronous training/inference, fine-grained asynchronous interaction with a dynamic batching scheduler, and a streamer-based micro-batch training pipeline to maximize hardware utilization. Empirical results on LIBERO and ManiSkill across 8–256 GPUs show substantial throughput gains (up to 59.25% in LIBERO and up to 126.67% with optimized separation) while maintaining competitive policy performance, with detailed ablations and scaling analyses. The work signals a practical pathtoward scalable, robust VLA-based embodied intelligence and outlines future directions for broader simulator support and cross-embodiment learning.

Abstract

In recent years, Vision-Language-Action (VLA) models have emerged as a crucial pathway towards general embodied intelligence, yet their training efficiency has become a key bottleneck. Although existing reinforcement learning (RL)-based training frameworks like RLinf can enhance model generalization, they still rely on synchronous execution, leading to severe resource underutilization and throughput limitations during environment interaction, policy generation (rollout), and model update phases (actor). To overcome this challenge, this paper, for the first time, proposes and implements a fully-asynchronous policy training framework encompassing the entire pipeline from environment interaction, rollout generation, to actor policy updates. Systematically drawing inspiration from asynchronous optimization ideas in large model RL, our framework designs a multi-level decoupled architecture. This includes asynchronous parallelization of environment interaction and trajectory collection, streaming execution for policy generation, and decoupled scheduling for training updates. We validated the effectiveness of our method across diverse VLA models and environments. On the LIBERO benchmark, the framework achieves throughput improvements of up to 59.25\% compared to existing synchronous strategies. When deeply optimizing separation strategies, throughput can be increased by as much as 126.67\%. We verified the effectiveness of each asynchronous component via ablation studies. Scaling law validation across 8 to 256 GPUs demonstrates our method's excellent scalability under most conditions.
Paper Structure (23 sections, 2 equations, 4 figures, 4 tables)

This paper contains 23 sections, 2 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language-Action Models and Supervised Baselines.
  4. Reinforcement Learning and Infrastructure for VLAs.
  5. Methodology
  6. Asynchronous training and inference
  7. Asynchronous Interaction Strategy
  8. Streamer Generation
  9. Experimental Results
  10. Experiment Setup
  11. Models.
  12. Benchmarks.
  13. Implementation and Baselines.
  14. Evaluation Metrics.
  15. Result Analysis
  16. ...and 8 more sections

Figures (4)

  • Figure 1: An illustration of the proposed asynchronous framework for VLA training. The design operates asynchronously on two levels: (1) macroscopically, between the rollout and training pipeline stages, and (2) microscopically, within each rollout epoch. The latter is governed by two trigger strategies: batch size and waiting time.
  • Figure 2: Comparative throughput analysis of different separation strategies.
  • Figure 3: Scaling behavior with increasing GPU resources for the LIBERO+$\pi_{0.5}$ configuration.
  • Figure 4: Training success rate curves for LIBERO+OpenVLA-OFT and ManiSkill+$\pi_{0.5}$ under different training strategies. Our asynchronous training pipeline shows a steady growth in success rate, matching the trend of baseline methods.