Reinforcing Action Policies by Prophesying

TL;DR

This work tackles the brittleness of imitation-trained Vision–Language–Action policies by introducing Prophet, a large-scale, action-conditioned world model that can be rapidly adapted to new robots and environments. Paired with Flow-GRPO and FlowScale, Prophet enables data- and compute-efficient post-training RL (ProphRL) to optimize long-horizon, action-conditioned policies without extensive real-robot trials. The approach yields consistent performance gains across public benchmarks and real robots, and introduces an optical-flow-based evaluation protocol to measure action-conditioned fidelity beyond appearance. The results demonstrate strong generalization, few-shot adaptability, and practical RL stability, with implications for scalable, real-world robotic learning.

Abstract

Vision-Language-Action (VLA) policies excel in aligning language, perception, and robot control. However, most VLAs are trained purely by imitation, which overfits to demonstrations, and is brittle under distribution shift. Reinforcement learning (RL) directly optimizes task reward and thus addresses this misalignment, but real-robot interaction is expensive and conventional simulators are hard to engineer and transfer. We address both data efficiency and optimization stability in VLA post-training via a learned world model and an RL procedure tailored to flow-based action heads. Specifically, we introduce Prophet, a unified action-to-video robot actuation pretrained across large-scale, heterogeneous robot data to learn reusable action-outcome dynamics. It is able to few-shot adapt to new robots, objects, and environments, yielding a rollout-ready simulator. Upon Prophet, we reinforce action policies with Flow-action-GRPO (FA-GRPO), which adapts Flow-GRPO to operate on VLA actions, and with FlowScale, a stepwise reweighting that rescales per-step gradients in the flow head. Together, Prophet, FA-GRPO, and FlowScale constitute ProphRL, a practical, data- and compute-efficient path to VLA post-training. Experiments show 5-17% success gains on public benchmarks and 24-30% gains on real robots across different VLA variants.

Figures (14)

• Figure 1: ProphRL uses a world model as a real-world–facing simulator to post-train VLA policies. Our world model Prophet extends a video generator with history-aware mechanism and dual action conditioning, and is pretrained on large-scale robot trajectories to model action-to-video dynamics. The pretrained Prophet enables 'prophesying' precise, physically plausible long-horizon rollouts, and can be rapidly adapted via few-shot fine-tuning to new environments, objects, and trajectories. Upon Prophet, we introduce the FA-GRPO with FlowScale RL algorithm to more stably and efficiently improve policies. Together, our training paradigm turns diverse logged data and a single pretrained world model into a unified engine for scalable, data-efficient, and safely improvable VLA systems.
• Figure 2: ProphRL Training paradigm. Given an initial frame and instruction, the policy predicts an action chunk and Prophet generates the future robot rollout, updating the policy input, current frame, and history buffer until the episode ends. An offline reward model scores each full trajectory, and the policy is reinforced with FA-GRPO and FlowScale using these 'prophesied' and realistic rollouts.
• Figure 3: Action frame visualization. The first row shows RGB frames, the middle row shows the constructed action frames, and the last row shows the alignment between the visualized action frames and the image pixels.
• Figure 4: Real-world reward model prompt and example response for the task PulloutTissueScene on BRIDGE. The prompt is designed following the principles in Sec. \ref{['sec:rm_discussion']} (high recall with reasonably clean positive labels), while not optimal, it provides sufficiently informative supervision for our current real-world experiments.
• Figure 5: Presentation of custom data collected using a UR30e robot arm. We collect data for four tabletop manipulation tasks, including challenging cases such as pulling tissues from a box, which are impossible to simulate accurately in standard physics simulators.