Interactive Post-Training for Vision-Language-Action Models

TL;DR

The paper presents RIPT-VLA, a critic-free reinforcement learning framework that post-trains pretrained Vision-Language-Action models through interactive environment feedback using sparse binary rewards. Building on LOOP and PPO with a dynamic rollout sampling strategy, it achieves state-of-the-art performance across LIBERO and MetaWorld benchmarks, including dramatic gains in extreme low-data regimes from single demonstrations. The method demonstrates strong multitask, cross-scenario, and cross-goal generalization, while avoiding reliance on shaped rewards or value critics. This positions RIPT-VLA as a practical third-stage paradigm to unlock latent visuomotor skills in VLA models and enhance real-world adaptability.

Abstract

We introduce RIPT-VLA, a simple and scalable reinforcement-learning-based interactive post-training paradigm that fine-tunes pretrained Vision-Language-Action (VLA) models using only sparse binary success rewards. Existing VLA training pipelines rely heavily on offline expert demonstration data and supervised imitation, limiting their ability to adapt to new tasks and environments under low-data regimes. RIPT-VLA addresses this by enabling interactive post-training with a stable policy optimization algorithm based on dynamic rollout sampling and leave-one-out advantage estimation. RIPT-VLA has the following characteristics. First, it applies to various VLA models, resulting in an improvement on the lightweight QueST model by 21.2%, and the 7B OpenVLA-OFT model to an unprecedented 97.5% success rate. Second, it is computationally efficient and data-efficient: with only one demonstration, RIPT-VLA enables an unworkable SFT model (4%) to succeed with a 97% success rate within 15 iterations. Furthermore, we demonstrate that the policy learned by RIPT-VLA generalizes across different tasks and scenarios and is robust to the initial state context. These results highlight RIPT-VLA as a practical and effective paradigm for post-training VLA models through minimal supervision.

Figures (7)

• Figure 1: Overview of RIPT-VLA. While VLA models are typically trained with two supervised stages, we propose a third stage: Reinforcement Interactive Post-Training for VLA. RIPT-VLA sets state-of-the-art results across diverse benchmarks. It also presents remarkable improvement under the low-data regime, transforming an SFT model from near failure to 97% with one demonstration.
• Figure 2: Few-shot curve on LIBERO-LONG.
• Figure 3: Cross-scenario task generalization from Scenario A to Scenario B with the same goal.
• Figure 4: Cross-goal task generalization from Goal A to Goal B in the same scenario.
• Figure 5: Training curve analysis of dynamic sampling.