See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulation

TL;DR

SPR, a progress-aware vision-language-action framework that dynamically grounds language instructions into a sequence of spatial subgoals, achieves state-of-the-art robustness with the smallest performance drop, surpassing OpenVLA-OFT and UniVLA, demonstrating superior out-of-distribution robustness.

Abstract

Measurement of task progress through explicit, actionable milestones is critical for robust robotic manipulation. This progress awareness enables a model to ground its current task status, anticipate verifiable intermediate states, and detect and recover from failures when progress stalls. To embody this capability, we introduce See, Plan, Rewind (SPR), a progress-aware vision-language-action framework that dynamically grounds language instructions into a sequence of spatial subgoals. SPR operates through a continuous core cycle, Seeing the current state and upcoming milestone, Planning a trajectory towards the next 2D waypoint, and Rewinding to a recoverable state upon failure by monitoring progress against the expected sequence. This closed-loop approach enables robust error correction without requiring additional training data or auxiliary models. Extensive experiments demonstrate the framework's effectiveness, generalization and robustness: SPR outperforms the MolmoAct baseline by 5\% on the LIBERO benchmark. On the challenging LIBERO-Plus benchmark with unseen instructions and initial states, SPR achieves state-of-the-art robustness with the smallest performance drop, surpassing OpenVLA-OFT and UniVLA, demonstrating superior out-of-distribution robustness.

Figures (16)

• Figure 1: See-Plan-Rewind framework's closed-loop execution workflow. Starting from the initial state (top-left), the model performs See-Plan reasoning to generate actions (arrow to top-right), which visualizes the subtask decomposition and trajectory planning. Under normal execution, the loop returns to top-left. When progress monitoring detects failures (bottom-right), the Rewind mechanism activates (bottom-left), returning the robot to the initial state before resuming the loop. This closed-loop design enables autonomous error recovery through explicit progress awareness.
• Figure 2: SPR framework overview. Left: Upon receiving the task description and observation, the model performs See-Plan reasoning(Sec. \ref{['sec:subtask_planning']}), which identifies remaining subtasks with 2D spatial coordinates (See) and plans a gripper trajectory to the next subtask waypoint (Plan), then outputs action tokens for execution. Each inference step also updates the state recorder, where $S_N$ denotes the predicted subtask count and $T_N$ the planned 2D trajectory at the current timestep. Right: The Rewind mechanism (Sec. \ref{['sec:rewind']}) examines the state recorder after each step: if no anomaly is detected, the original task description is retained; if sustained anomalies are identified, the task description is switched to a rewind instruction for $N$ steps before reverting to normal execution. Data generation in Sec. \ref{['sec:data_generation']}.
• Figure 3: Task success rate vs. maximum episode length across LIBERO subsets. Progress-aware models continue improving after the baseline plateaus, demonstrating the ability to leverage extended horizons for complex error recovery.
• Figure 4: Component ablation on LIBERO-Long and LIBERO-Plus variants. Performance progressively improves from w/o Rewind & Semantics (spatial coordinates only) to w/o Rewind (spatial + semantic) to Ours (full SPR with Rewind), validating the contribution of each component.
• Figure 5: Effect of rewind step count $N$ on LIBERO. $N$=3 achieves optimal performance across all subsets.