RoVer: Robot Reward Model as Test-Time Verifier for Vision-Language-Action Model

TL;DR

RoVer addresses the data- and compute-heavy bottlenecks of training-time scaling in Vision-Language-Action (VLA) robotics by introducing an external test-time verifier, the Process Reward Model (PRM), that scores candidate actions and provides a refinement direction. By sharing perception features and using direction-guided expansion, RoVer converts available test-time compute into more reliable action selection without retraining the base policies. Empirically, RoVer yields backbone-agnostic improvements on CALVIN, scales with candidate budgets, and delivers significant inference-time speedups, with real-robot experiments confirming gains in generalization for manipulation tasks. The approach offers a practical, plug-and-play pathway to enhance embodied AI systems by reallocating compute to inference while preserving model architecture and weights.

Abstract

Vision-Language-Action (VLA) models have become a prominent paradigm for embodied intelligence, yet further performance improvements typically rely on scaling up training data and model size -- an approach that is prohibitively expensive for robotics and fundamentally limited by data collection costs. We address this limitation with $\mathbf{RoVer}$, an embodied test-time scaling framework that uses a $\mathbf{Ro}$bot Process Reward Model (PRM) as a Test-Time $\mathbf{Ver}$ifier to enhance the capabilities of existing VLA models without modifying their architectures or weights. Specifically, RoVer (i) assigns scalar-based process rewards to evaluate the reliability of candidate actions, and (ii) predicts an action-space direction for candidate expansion/refinement. During inference, RoVer generates multiple candidate actions concurrently from the base policy, expands them along PRM-predicted directions, and then scores all candidates with PRM to select the optimal action for execution. Notably, by caching shared perception features, it can amortize perception cost and evaluate more candidates under the same test-time computational budget. Essentially, our approach effectively transforms available computing resources into better action decision-making, realizing the benefits of test-time scaling without extra training overhead. Our contributions are threefold: (1) a general, plug-and-play test-time scaling framework for VLAs; (2) a PRM that jointly provides scalar process rewards and an action-space direction to guide exploration; and (3) an efficient direction-guided sampling strategy that leverages a shared perception cache to enable scalable candidate generation and selection during inference.

Figures (8)

• Figure 1: RoVer overview. A frozen VLA proposes $N$ actions; an external process reward model (PRM) scores candidates and predicts a refinement direction. Candidates are expanded along the guided direction and the top‑scoring action executes. Perceptual features are cached once and reused across candidates to amortize compute.
• Figure 2: Left: verifier architecture. A shared perception cache and per‑candidate action encoder (with an action amplifier) feed a GPT‑2 backbone via reward/direction tokens, producing a scalar process reward and an action‑space refinement direction. Right: direction visualization on the PushT benchmark, where PRM is trained atop a pre‑trained Diffusion Policy and predicts 2D $(x,y)$ action directions; for CALVIN, the PRM predicts 6D pose directions. Note: on PushT, a DP pre‑trained policy already achieves near‑100% success; we primarily use PushT here for intuitive direction visualization rather than performance gains.
• Figure 3: Direction‑guided test-time scaling: average chain length versus $N{+}M$ ($N$ policy proposals, $M$ guided expansions). Panels correspond to GR-1, Dita, and MoDE. Red: unguided (random) expansion; blue: direction‑guided (DG). MoDE shows broad but less stable gains due to the chunk–step mismatch discussed in Q2.
• Figure 4: GR-1: SR@k ($k{=}1\ldots5$) as a function of $N$ (policy proposals). Increasing $N$ boosts success across all $k$, revealing the value of proposal diversity; together with Fig. \ref{['fig:tts-baselines-n']}, this shows that adding direction‑guided expansions $M$ on top of non‑trivial $N$ yields further gains.
• Figure 5: Left: visualization of our real-robot experiments. Right: the dual-arm Dobot testing platform used in our evaluation.