Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Saliency-Aware Multi-Route Thinking: Revisiting Vision-Language Reasoning

Mingjia Shi, Yinhan He, Yaochen Zhu, Jundong Li

TL;DR

Vision–language models struggle with object grounding when inference-time scaling shifts reasoning toward text-dominated generation. The authors propose Saliency-Aware Principle Selection (SAP), a model-agnostic, data-free framework that searches over high-level reasoning principles and evaluates multiple parallel routes to preserve visual grounding. SAP uses saliency-guided evaluation with four ordinal signals and a population-based Evolutionary PER to refine principles, enabling scalable, parallel inference while maintaining grounding and reducing hallucinations. Empirical results across 16 VL benchmarks show SAP achieving competitive performance with lower latency than sequential LongCoT and robust grounding under various model scales and backbones, highlighting its practical impact for reliable multimodal reasoning in real-time settings.

Abstract

Vision-language models (VLMs) aim to reason by jointly leveraging visual and textual modalities. While allocating additional inference-time computation has proven effective for large language models (LLMs), achieving similar scaling in VLMs remains challenging. A key obstacle is that visual inputs are typically provided only once at the start of generation, while textual reasoning (e.g., early visual summaries) is generated autoregressively, causing reasoning to become increasingly text-dominated and allowing early visual grounding errors to accumulate. Moreover, vanilla guidance for visual grounding during inference is often coarse and noisy, making it difficult to steer reasoning over long texts. To address these challenges, we propose \emph{Saliency-Aware Principle} (SAP) selection. SAP operates on high-level reasoning principles rather than token-level trajectories, which enable stable control over discrete generation under noisy feedback while allowing later reasoning steps to re-consult visual evidence when renewed grounding is required. In addition, SAP supports multi-route inference, enabling parallel exploration of diverse reasoning behaviors. SAP is model-agnostic and data-free, requiring no additional training. Empirical results show that SAP achieves competitive performance, especially in reducing object hallucination, under comparable token-generation budgets while yielding more stable reasoning and lower response latency than CoT-style long sequential reasoning.

Saliency-Aware Multi-Route Thinking: Revisiting Vision-Language Reasoning

TL;DR

Vision–language models struggle with object grounding when inference-time scaling shifts reasoning toward text-dominated generation. The authors propose Saliency-Aware Principle Selection (SAP), a model-agnostic, data-free framework that searches over high-level reasoning principles and evaluates multiple parallel routes to preserve visual grounding. SAP uses saliency-guided evaluation with four ordinal signals and a population-based Evolutionary PER to refine principles, enabling scalable, parallel inference while maintaining grounding and reducing hallucinations. Empirical results across 16 VL benchmarks show SAP achieving competitive performance with lower latency than sequential LongCoT and robust grounding under various model scales and backbones, highlighting its practical impact for reliable multimodal reasoning in real-time settings.

Abstract

Vision-language models (VLMs) aim to reason by jointly leveraging visual and textual modalities. While allocating additional inference-time computation has proven effective for large language models (LLMs), achieving similar scaling in VLMs remains challenging. A key obstacle is that visual inputs are typically provided only once at the start of generation, while textual reasoning (e.g., early visual summaries) is generated autoregressively, causing reasoning to become increasingly text-dominated and allowing early visual grounding errors to accumulate. Moreover, vanilla guidance for visual grounding during inference is often coarse and noisy, making it difficult to steer reasoning over long texts. To address these challenges, we propose \emph{Saliency-Aware Principle} (SAP) selection. SAP operates on high-level reasoning principles rather than token-level trajectories, which enable stable control over discrete generation under noisy feedback while allowing later reasoning steps to re-consult visual evidence when renewed grounding is required. In addition, SAP supports multi-route inference, enabling parallel exploration of diverse reasoning behaviors. SAP is model-agnostic and data-free, requiring no additional training. Empirical results show that SAP achieves competitive performance, especially in reducing object hallucination, under comparable token-generation budgets while yielding more stable reasoning and lower response latency than CoT-style long sequential reasoning.
Paper Structure (67 sections, 5 theorems, 39 equations, 4 figures, 3 tables, 2 algorithms)

This paper contains 67 sections, 5 theorems, 39 equations, 4 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Methodology
  4. Principle-Guided Reasoning Generation
  5. Intractable Route Search
  6. Principle-guided reasoning
  7. Principle Selection
  8. Saliency-Aware Principle Evolution for Vision--Language Reasoning
  9. Approach overview
  10. Saliency-aware principle evaluation
  11. Evolutionary optimization
  12. Framework Analysis
  13. Comparison with long chain-of-thought
  14. Data- and gradient-free scaling
  15. Empirical Study
  16. ...and 52 more sections

Key Result

Theorem 3.1

Consider the $(\mu+\lambda)$ evolutionary procedure over a principle space $\mathcal{X}$. Let $F_t$ denote the best fitness in the population at iteration $t$, and let $T$ be the total number of iterations. Then the following properties hold:

Figures (4)

  • Figure 1: Text reliance in vision-language reasoning. Our SAP, a multi-route approach, employs visual grounding as guidance, alleviating object hallucination in text-dominated LongCoT.
  • Figure 2: Better perception (Right) is obtained by SAP (Middle) evaluated by perception-intensive benchmarks, OCRVQA and POPE-recall, about object hallucination. Left: Text reliance in reasoning from Qwen3-VL-8B-Thinking on MS-COCO. Right: Saliency-Aware Principle (SAP) selection pipeline and empirical comparison between SAP implementation with Qwen3-VL-8B serie on benchmarks.
  • Figure 3: Ablation on weights of rewarding terms in fitness (loss design): aggregation of routes for final answer reduces the reliance on consensus in multimodal reasoning. Diversity, consensus, evidence refer to weights ratio of $w_{d}, w_{c}, w_{e}$ where $\sum w=1$ and uncertainty penalty is always 1, $w_{u}=1$.
  • Figure 4: SAP supports inference-time speedup via parallelism.

Theorems & Definitions (12)

  • Theorem 3.1: Optimization and generalization with $(\mu+\lambda)$ evolution
  • Theorem 4.2: Monotone best-fitness under $(\mu+\lambda)$ selection
  • proof
  • Theorem 4.4: One-step improvement bound
  • proof
  • Remark 4.5: Small-$q_t$ regime and the role of $\lambda$
  • Corollary 4.6: At least one improvement within $T$ iterations (lower bound)
  • proof
  • Definition 4.7: Effective principle set
  • Proposition 4.9: Coverage increases with $\lambda T$
  • ...and 2 more