SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models

TL;DR

This work introduces Self-Referential Policy Optimization (SRPO) for Vision-Language-Action (VLA) models to overcome reward sparsity without external demonstrations or hand-crafted rewards. SRPO uses in-batch successful trajectories and latent world representations from a pre-trained world model to shape progress-based rewards for failed trajectories, enabling efficient, self-supervised learning. Empirical results on LIBERO and LIBERO-Plus show state-of-the-art performance with remarkable data efficiency and robust generalization, including real-world robotic tasks. The approach also demonstrates enhanced exploration and effective reward shaping grounded in world progress, suggesting a new paradigm for autonomous VLA learning.

Abstract

Vision-Language-Action (VLA) models excel in robotic manipulation but are constrained by their heavy reliance on expert demonstrations, leading to demonstration bias and limiting performance. Reinforcement learning (RL) is a vital post-training strategy to overcome these limits, yet current VLA-RL methods, including group-based optimization approaches, are crippled by severe reward sparsity. Relying on binary success indicators wastes valuable information in failed trajectories, resulting in low training efficiency. To solve this, we propose Self-Referential Policy Optimization (SRPO), a novel VLA-RL framework. SRPO eliminates the need for external demonstrations or manual reward engineering by leveraging the model's own successful trajectories, generated within the current training batch, as a self-reference. This allows us to assign a progress-wise reward to failed attempts. A core innovation is the use of latent world representations to measure behavioral progress robustly. Instead of relying on raw pixels or requiring domain-specific fine-tuning, we utilize the compressed, transferable encodings from a world model's latent space. These representations naturally capture progress patterns across environments, enabling accurate, generalized trajectory comparison. Empirical evaluations on the LIBERO benchmark demonstrate SRPO's efficiency and effectiveness. Starting from a supervised baseline with 48.9% success, SRPO achieves a new state-of-the-art success rate of 99.2% in just 200 RL steps, representing a 103% relative improvement without any extra supervision. Furthermore, SRPO shows substantial robustness, achieving a 167% performance improvement on the LIBERO-Plus benchmark.

Figures (16)

• Figure 1: Overview of Self-Referential Policy Optimization (SRPO). Existing approaches for Vision-Language-Action (VLA) reinforcement learning face significant limitations: (a) methods like GRPO rely solely on sparse outcome rewards, providing limited learning signal, while (b) hand-crafted process reward modeling (PRM) requires costly external demonstrations and task-specific engineering. In contrast, our SRPO framework introduces a self-referential paradigm that leverages (i) in-batch successful trajectories and (ii) latent world representations to construct progress-wise rewards, enabling efficient utilization of failure trajectories. Extensive experimental evaluation demonstrates that SRPO achieves (1) state-of-the-art performance, (2) superior training efficiency, (3) stronger generalization capabilities, and (4) improved real-world performance.
• Figure 2: Overview of the SRPO method. During policy rollout, both successful and failed trajectories are collected in the Rollout Reference Set. For each trajectory, we employ a world model pre-trained on large-scale robotics video data assran2025v as an encoder to extract latent world representations. Behavioral similarity is modeled as the L2 distance between trajectory embeddings in this space to yield progress-wise rewards. These rewards are subsequently used for advantage estimation and policy optimization under KL regularization.
• Figure 3: Comparison of progress estimation methods in simulated (a-c) and real-world (d-f) environments. Our SRPO reward (a,d) provides monotonic and physically plausible progress estimation. Pixel-level rewards (b,e) show sensitivity to perceptual changes, while ImageBind rewards (c,f) exhibit erratic trends from jerky motions.
• Figure 4: Training performance comparison using different progress reward formulations. Our SRPO-based reward enables stable and efficient learning, consistently outperforming both baselines.
• Figure 5: Training efficiency comparison between SRPO and GRPO: (a) LIBERO-Long, (b) LIBERO-Object.