SOP: A Scalable Online Post-Training System for Vision-Language-Action Models

TL;DR

This work tackles the challenge of achieving expert-level proficiency without sacrificing generality in Vision-Language-Action (VLA) models for real-world robotics. It introduces Scalable Online Post-training (SOP), a closed-loop actor–learner framework that jointly scales data collection across a fleet with online, asynchronous policy updates in a cloud learner, enabling on-policy correction and multi-task adaptation. SOP is algorithm-agnostic and instantiated with HG-DAgger and RECAP, demonstrating substantial gains across cloth folding, box assembly, and grocery restocking tasks, with near-linear scaling as fleet size increases and adaptation achievable within hours of real-world interaction. The key contribution is showing that tightly coupling deployment and learning enables efficient, reliable, and scalable post-training of generalist robot policies in the physical world, suggesting fleets can effectively serve as scalable compute for continual policy improvement.

Abstract

Vision-language-action (VLA) models achieve strong generalization through large-scale pre-training, but real-world deployment requires expert-level task proficiency in addition to broad generality. Existing post-training approaches for VLA models are typically offline, single-robot, or task-specific, limiting effective on-policy adaptation and scalable learning from real-world interaction. We introduce a Scalable Online Post-training (SOP) system that enables online, distributed, multi-task post-training of generalist VLA models directly in the physical world. SOP tightly couples execution and learning through a closed-loop architecture in which a fleet of robots continuously streams on-policy experience and human intervention signals to a centralized cloud learner, and asynchronously receives updated policies. This design supports prompt on-policy correction, scales experience collection through parallel deployment, and preserves generality during adaptation. SOP is agnostic to the choice of post-training algorithm; we instantiate it with both interactive imitation learning (HG-DAgger) and reinforcement learning (RECAP). Across a range of real-world manipulation tasks including cloth folding, box assembly, and grocery restocking, we show that SOP substantially improves the performance of large pretrained VLA models while maintaining a single shared policy across tasks. Effective post-training can be achieved within hours of real-world interaction, and performance scales near-linearly with the number of robots in the fleet. These results suggest that tightly coupling online learning with fleet-scale deployment is instrumental to enabling efficient, reliable, and scalable post-training of generalist robot policies in the physical world.

Figures (7)

• Figure 1: Scalable Online Post-training (SOP). A fleet of robots continuously collects experience across diverse tasks, streams interaction data to a centralized cloud server, and receives updated control policies asynchronously—enabling VLA models to improve proficiency on each task while preserving generality.
• Figure 2: SOP overview. SOP is a scalable actor–learner framework for online, multi-task post-training of generalist policies. The robot fleet streams on-policy rollouts to the cloud learner. Optional human interventions are triggered in failure or uncertain cases, providing corrected trajectories or actions that are incorporated into the streaming experience buffer. The cloud learner constructs task-balanced updates by mixing an online buffer with a static offline buffer, applies a plug-in post-training module (e.g., HG-DAgger/RECAP), and asynchronously broadcasts refreshed weights back to all actors to close a low-latency online training loop.
• Figure 3: Illustrations of the three task categories. (A) Grocery Restocking scenarios: (A-1) flat-shelf restocking; (A-2) correcting misplaced items; (A-3) freezer restocking involving door manipulation; and (A-4) open-cooler restocking with carton handling. (B) Laundry Folding: a bimanual sequence where the robot flattens and folds a garment. (C) Box Assembly: a sequence showing two robot arms coordinating to fold a flattened cardboard sheet into a 3D box structure.
• Figure 4: Comparison of Success Rate and Throughput across three manipulation domains. Across all domains, our approach demonstrates superior efficiency and reliability. SOP w/ HG-DAgger consistently achieves 2-4x higher throughput and significantly reduces failure rates compared to offline methods under our evaluation protocol (policy-side throughput excluding human reset/setup time).
• Figure 5: Effect of pretraining data scale on SOP. Larger pretraining datasets yield higher initial success and higher final performance after online post-training.