Seeing Farther and Smarter: Value-Guided Multi-Path Reflection for VLM Policy Optimization

TL;DR

The paper tackles long-horizon robotic manipulation with Vision-Language Models by identifying limitations of prior reflective planning approaches that rely on implicit value learning, a single greedy forecast, and high latency. It introduces a value-guided post-training framework where action-plan value is defined as the distance-to-goal reduction $\Delta d_t^{(H)}$ and estimated by a scalable critic, enabling explicit value supervision. A multi-path reflection mechanism uses beam search to explore multiple imagined futures and aggregates their guidance during decoding, while a confidence-based early exit gate reduces unnecessary reflection. Across 100 unseen tasks, the method achieves substantial gains in success rate and reductions in inference time, validating robustness and efficiency gains for real-world, long-horizon robotic manipulation with VLMs.

Abstract

Solving complex, long-horizon robotic manipulation tasks requires a deep understanding of physical interactions, reasoning about their long-term consequences, and precise high-level planning. Vision-Language Models (VLMs) offer a general perceive-reason-act framework for this goal. However, previous approaches using reflective planning to guide VLMs in correcting actions encounter significant limitations. These methods rely on inefficient and often inaccurate implicit learning of state-values from noisy foresight predictions, evaluate only a single greedy future, and suffer from substantial inference latency. To address these limitations, we propose a novel test-time computation framework that decouples state evaluation from action generation. This provides a more direct and fine-grained supervisory signal for robust decision-making. Our method explicitly models the advantage of an action plan, quantified by its reduction in distance to the goal, and uses a scalable critic to estimate. To address the stochastic nature of single-trajectory evaluation, we employ beam search to explore multiple future paths and aggregate them during decoding to model their expected long-term returns, leading to more robust action generation. Additionally, we introduce a lightweight, confidence-based trigger that allows for early exit when direct predictions are reliable, invoking reflection only when necessary. Extensive experiments on diverse, unseen multi-stage robotic manipulation tasks demonstrate a 24.6% improvement in success rate over state-of-the-art baselines, while significantly reducing inference time by 56.5%.

Figures (5)

• Figure 1: Overview of our value-guided multi-path reflection framework. Initially, the VLM policy proposes a candidate action list conditioned on the current and goal images ($\S$\ref{['sec:method-vlm']}). The action with the highest probability ("pick up purple") is treated as a preliminary plan and assessed by a confidence-based trigger ($\S$\ref{['sec:method-trigger']}). If reflection is needed, a diffusion dynamics model imagines multiple future trajectories via beam search, guided by the candidate action list. A critic evaluates each of these imagined trajectories by estimating their distance reduction to the goal state, and this evaluation is incorporated as feedback into multiple distinct contexts. These inputs are then aggregated during the decoding stage, allowing the agent to reflect on the potential futures, revise its initial plan, and generate a more proper action ("pick up yellow") ($\S$\ref{['sec:method-reflection']}).
• Figure 2: (a) Overview of the multi-path reflection mechanism during decoding. Output distributions from multiple inputs based on distinct imagined futures are dynamically combined using complementary or contrastive decoding based on their Jensen-Shannon Divergence to enhance or correct output. (b) A substantial 78.9% of optimal actions are produced in the initial proposal stage, while the reflection stage is also essential for more challenging decisions, leading to an 18% increase in overall success.
• Figure 3: Task examples.
• Figure 4: Comparison of success rates on 100 unseen long-horizon manipulation tasks. Our value-guided reflective planning framework, in both its simulation-based (sim) and diffusion-based variants, significantly outperforms all baselines, including traditional methods like MCTS and Behavioral Cloning (BC) and the prior state-of-the-art, ReflectVLM. For a fair comparison, both our method and ReflectVLM are evaluated under only a single round of post-training. The "Ours w/ oracle value" results demonstrate the upper-bound performance of our framework with perfect value estimation.
• Figure 5: (a) Inference time cost comparison. Our method significantly outperforms ReflectVLM reflectvlm, reducing inference time by over 45% via the confidence-based early-exit strategy. (b) Ablation study on multi-path aggregation strategies. Our aggregation mechanism during decoding outperforms both the single-path baseline and traditional post-hoc selection methods, such as Best-of-N and Majority Voting. (c) Qualitative analysis of the advantage distribution for revised actions. Our method's reflection is highly precise, primarily targeting low-advantage actions (peaking near 0). In contrast, the baseline's reflection is indiscriminate and less likely to trigger necessary revisions.