MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Models

TL;DR

The paper addresses reward design bottlenecks in sparse-reward robotic manipulation by leveraging Vision-Language Models (VLMs) to provide semantic guidance. It introduces MARVL, a three-part framework that (i) fine-tunes VLMs via Scene-View Decomposition to restore spatial grounding, (ii) applies Multi-Stage Decomposition with Task Direction Projection to produce monotonic progress signals across subtasks, and (iii) uses Confidence-Thresholded Shaping to filter noisy matches. Experiments on Meta-World show MARVL outperforms existing VLM-reward methods in sample efficiency and robustness, sometimes matching or surpassing an Oracle dense reward, and it generalizes to Panda-Gym and varying camera setups. Overall, MARVL demonstrates that structured grounding and stage-wise progress alignment can transform VLM signals into reliable, scalable rewards for robotic manipulation.

Abstract

Designing dense reward functions is pivotal for efficient robotic Reinforcement Learning (RL). However, most dense rewards rely on manual engineering, which fundamentally limits the scalability and automation of reinforcement learning. While Vision-Language Models (VLMs) offer a promising path to reward design, naive VLM rewards often misalign with task progress, struggle with spatial grounding, and show limited understanding of task semantics. To address these issues, we propose MARVL-Multi-stAge guidance for Robotic manipulation via Vision-Language models. MARVL fine-tunes a VLM for spatial and semantic consistency and decomposes tasks into multi-stage subtasks with task direction projection for trajectory sensitivity. Empirically, MARVL significantly outperforms existing VLM-reward methods on the Meta-World benchmark, demonstrating superior sample efficiency and robustness on sparse-reward manipulation tasks.

Figures (19)

• Figure 1: Radar plot of performance across eight Meta-World manipulation tasks. MARVL achieves consistently strong and balanced performance across all skill categories, surpassing the Oracle reward on several tasks and outperforming prior VLM-based reward methods.
• Figure 2: Reward Misalignment in VLM-Based Methods.Left: VLM reward signals along an oracle button-press-topdown trajectory. The green dashed curve denotes the environment-provided dense reward in Meta-World, whose scale differs from VLM rewards and is shown only to indicate the overall trend of task progress. Middle: t-SNE projection of image embeddings from five Meta-World tasks under three viewpoints, showing that embeddings cluster by viewpoint rather than by task identity. Right: Text–image similarity matrix for the instruction set (button-press-topdown, door-open, drawer-open, push, and window-open) in original CLIP, where rows correspond to image embeddings and columns correspond to text embeddings. Ideally, the diagonal should dominate, but weak diagonal structure and noisy off-diagonal activations indicate poor alignment between text and image embeddings.
• Figure 3: Overview of MARVL. (a) We first fine-tune the VLM encoder via Scene-View Decomposition, followed by Multi-stage Decomposition with Task Direction Projection and Confidence-Thresholded Shaping to derive calibrated VLM rewards. (b) MARVL could be easily integrated into the RL loop, where the agent interacts with the environment under dense VLM reward guidance.
• Figure 4: Effectiveness of Scene-View Decomposition.Left: Reconstructions retain critical spatial structure (e.g., arm pose) despite high-frequency pixel abstraction, verifying the encoder's geometric grounding. Right: t-SNE visualization confirms that fine-tuned embeddings cluster by task identity rather than camera viewpoint, demonstrating effective disentanglement of semantics from nuisance factors.
• Figure 5: Raw vs. thresholded MARVL rewards on the button-press-topdown task using oracle trajectories. The raw MARVL reward (left) exhibits noise and spurious positive activations at early steps, while the thresholded reward (right) suppresses these false positives and amplifies high-confidence signals, yielding a cleaner and more reliable reward signal under MARVL. The orange dashed line indicates the stage transition point. The green dashed curve shows the environment-provided dense reward.

Theorems & Definitions (4)

• Theorem 4.2
• proof
• Theorem 4.3
• proof