SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning

Abstract

Vision-language models (VLMs) have shown impressive capabilities across diverse tasks, motivating efforts to leverage these models to supervise robot learning. However, when used as evaluators in reinforcement learning (RL), today's strongest models often fail under partial observability and distribution shift, enabling policies to exploit perceptual errors rather than solve the task. To address this limitation, we introduce SOLE-R1 (Self-Observing LEarner), a video-language reasoning model explicitly designed to serve as the sole reward signal for online RL. Given only raw video observations and a natural-language goal, SOLE-R1 performs per-timestep spatiotemporal chain-of-thought (CoT) reasoning and produces dense estimates of task progress that can be used directly as rewards. To train SOLE-R1, we develop a large-scale video trajectory and reasoning synthesis pipeline that generates temporally grounded CoT traces aligned with continuous progress supervision. This data is combined with foundational spatial and multi-frame temporal reasoning, and used to train the model with a hybrid framework that couples supervised fine-tuning with RL from verifiable rewards. Across four different simulation environments and a real-robot setting, SOLE-R1 enables zero-shot online RL from random initialization: robots learn previously unseen manipulation tasks without ground-truth rewards, success indicators, demonstrations, or task-specific tuning. SOLE-R1 succeeds on 24 unseen tasks and substantially outperforms strong vision-language rewarders, including GPT-5 and Gemini-3-Pro, while exhibiting markedly greater robustness to reward hacking.

Figures (15)

• Figure 1: SOLE-R1 is a video-language reasoning model designed to guide online RL with per-timestep chain-of-thought reasoning and progress prediction. In large-scale experiments across 40 tasks, SOLE-R1 outperforms strong baseline models with zero-shot online RL.
• Figure 2: SOLE-R1 Training Data Mixture. The dataset combines foundational spatial reasoning (single-image and depth), multi-frame temporal reasoning, and our synthesized video trajectories with chain-of-thought explanations and dense progress supervision, jointly enabling reasoning over space and time for progress prediction.
• Figure 3: Zero-shot Success Rate of Online RL across 40 Tasks. We plot the mean and standard error across three random seeds (real-world experiments use a single seed, shown as a single value). In all experiments, the robot begins with a random policy and learns entirely through interaction with the task, guided only by the predicted rewards. We do not use any ground-truth rewards (sparse or dense), task-specific tuning, or demonstrations at any point of learning.
• Figure 4: Perceived vs True Success in Zero-shot RL. We compute perceived success as the average max progress predicted by each model (since LIV predicts values between -1 and 1, we re-scale to 0 to 100). RoboReward is excluded as it does not provide a dense reward. We compute true success as the average max ground-truth reward achieved. Failure types: reward-hacking (bottom-right quadrant: low true success, but high perceived success) versus signal-limited (bottom-left quadrant: low true success and low perceived success)
• Figure 5: Ablated-models zero-shot success.