When to Think and When to Look: Uncertainty-Guided Lookback

TL;DR

This work analyzes when test-time thinking helps visual reasoning in LVLMs, revealing that longer reasoning chains can be detrimental and that concise, image-grounded prompts often lead to better performance. It introduces uncertainty-guided lookback, a training-free decoding strategy that injects short, image-focused prompts only when the chain drifts, optionally with parallel exploration of grounded continuations. The method yields consistent accuracy gains and token savings across MMMUval and five additional multimodal/math benchmarks, effectively shifting the accuracy–compute Pareto frontier. Overall, the approach demonstrates robust, scalable improvements in visual reasoning by adaptively routing deliberation to image-grounded, uncertain instances.

Abstract

Test-time thinking (that is, generating explicit intermediate reasoning chains) is known to boost performance in large language models and has recently shown strong gains for large vision language models (LVLMs). However, despite these promising results, there is still no systematic analysis of how thinking actually affects visual reasoning. We provide the first such analysis with a large scale, controlled comparison of thinking for LVLMs, evaluating ten variants from the InternVL3.5 and Qwen3-VL families on MMMU-val under generous token budgets and multi pass decoding. We show that more thinking is not always better; long chains often yield long wrong trajectories that ignore the image and underperform the same models run in standard instruct mode. A deeper analysis reveals that certain short lookback phrases, which explicitly refer back to the image, are strongly enriched in successful trajectories and correlate with better visual grounding. Building on this insight, we propose uncertainty guided lookback, a training free decoding strategy that combines an uncertainty signal with adaptive lookback prompts and breadth search. Our method improves overall MMMU performance, delivers the largest gains in categories where standard thinking is weak, and outperforms several strong decoding baselines, setting a new state of the art under fixed model families and token budgets. We further show that this decoding strategy generalizes, yielding consistent improvements on five additional benchmarks, including two broad multimodal suites and math focused visual reasoning datasets.

Figures (7)

• Figure 1: Pass@k accuracy on MMMUval for 10 LVLM variants from the InternVL3.5 and Qwen3-VL families. Increasing the number of samples k (breadth) yields steep early gains for all models, especially smaller ones, while enabling the built-in thinking mode consistently shifts curves upward but with diminishing returns beyond Pass@8. This illustrates that additional sampling can often substitute for deeper test-time thinking and that the benefits of thinking depend strongly on model capacity.
• Figure 2: Category-level performance heatmaps showing z-scored accuracy across disciplines for all models and their thinking variants. Left: mean accuracy z-scores; right: Pass@10 z-scores under extensive sampling. Warmer colors indicate categories where a model performs above the global average, while cooler colors highlight relative weaknesses.
• Figure 3: Mean Pass@k accuracy on MMMUval for each vision–language model and its corresponding “Thinking” variant. Boxplots summarize variation across evaluation runs, where higher medians and tighter interquartile ranges indicate stronger and more stable performance, highlighting systematic differences between standard and Thinking modes.
• Figure 4: An example where the 32B thinking model fails on all 10 passes, while the instruct model answers correctly using 1 token.
• Figure 5: Token footprint of instruct vs. thinking across difficulty and scale. Boxplots show the distribution of total generated tokens per question, split by correctness (correct vs. wrong), model family (InternVL3.5 vs. Qwen3-VL), size (4B/8B/32B), and difficulty (Easy/Medium/Hard). The top panel reports Instruct models, while the bottom reports Thinking variants. Thinking consistently inflates token usage relative to instruct—especially on medium and hard questions and on failures—illustrating the compute overhead of deliberate reasoning. At larger scales (e.g., 32B), correct thinking traces tend to be shorter than for smaller models at the same difficulty, suggesting more efficient reasoning with increased capacity.