SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning

TL;DR

This work targets the data and generalization bottlenecks of vision-language-action robotics trained via SFT. It introduces SimpleVLA-RL, an online RL framework built on veRL that adds VLA-specific trajectory sampling, multi-environment rendering, and a simple outcome-based reward, together with exploration enhancements and GRPO updates. Empirical results across LIBERO and RoboTwin benchmarks show substantial gains in data efficiency, generalization to unseen tasks, and sim-to-real transfer, achieving state-of-the-art performance on several suites and outperforming prior SFT baselines. A notable finding is the emergence of pushcut behaviors during RL, illustrating the potential of RL to discover novel, efficient strategies beyond demonstration data.

Abstract

Vision-Language-Action (VLA) models have recently emerged as a powerful paradigm for robotic manipulation. Despite substantial progress enabled by large-scale pretraining and supervised fine-tuning (SFT), these models face two fundamental challenges: (i) the scarcity and high cost of large-scale human-operated robotic trajectories required for SFT scaling, and (ii) limited generalization to tasks involving distribution shift. Recent breakthroughs in Large Reasoning Models (LRMs) demonstrate that reinforcement learning (RL) can dramatically enhance step-by-step reasoning capabilities, raising a natural question: Can RL similarly improve the long-horizon step-by-step action planning of VLA? In this work, we introduce SimpleVLA-RL, an efficient RL framework tailored for VLA models. Building upon veRL, we introduce VLA-specific trajectory sampling, scalable parallelization, multi-environment rendering, and optimized loss computation. When applied to OpenVLA-OFT, SimpleVLA-RL achieves SoTA performance on LIBERO and even outperforms $π_0$ on RoboTwin 1.0\&2.0 with the exploration-enhancing strategies we introduce. SimpleVLA-RL not only reduces dependence on large-scale data and enables robust generalization, but also remarkably surpasses SFT in real-world tasks. Moreover, we identify a novel phenomenon ``pushcut'' during RL training, wherein the policy discovers previously unseen patterns beyond those seen in the previous training process. Github: https://github.com/PRIME-RL/SimpleVLA-RL

Figures (7)

• Figure 1: Overview of SimpleVLA-RL. SimpleVLA-RL is an efficient RL framework for VLA that improves long-horizon planning under data scarcity, outperforms SFT in simulation and real-world tasks, reveals a "pushcut" new-action phenomenon, and strengthens spatial/object/goal generalization.
• Figure 2: Overview of SimpleVLA-RL.
• Figure 3: The effectiveness of three key enhancements: dynamic sampling, higher rollout temperature, and clip higher.
• Figure 4: Generalization Analysis on LIBERO: Goal Unseen (Top), Object Unseen (Middle), Spatial Unseen (Bottom).
• Figure 5: "move can pot" task: Model learned to push the can to the pot (bottom) instead of grasp-move-place in the demonstration data (top).