Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VAR: Visual Attention Reasoning via Structured Search and Backtracking

Wei Cai, Jian Zhao, Yuchen Yuan, Tianle Zhang, Ming Zhu, Haichuan Tang, Chi Zhang, Xuelong Li

TL;DR

Visual Attention Reasoning (VAR), a novel framework that recasts grounded reasoning as a structured search over a reasoning trajectory space, sets a new state-of-the-art on a comprehensive suite of hallucination and safety benchmarks, significantly outperforming existing open-source models and demonstrating competitive performance against leading proprietary systems.

Abstract

Multimodal Large Language Models (MLLMs), despite their advances, are hindered by their high hallucination tendency and heavy reliance on brittle, linear reasoning processes, leading to failures in complex tasks. To address these limitations, we introduce Visual Attention Reasoning (VAR), a novel framework that recasts grounded reasoning as a structured search over a reasoning trajectory space. VAR decomposes the reasoning process into two key stages: traceable evidence grounding and search-based chain-of-thought (CoT) generation, which incorporates a backtracking mechanism for self-correction. The search is guided by a multi-faceted reward function with semantic and geometric self-verification components, which penalize outputs that are not faithfully grounded in the visual input. We provide a theoretical analysis for our search strategy, validating its capability to find the correct solution with high probability. Experimental results show that our 7B model, VAR-7B, sets a new state-of-the-art on a comprehensive suite of hallucination and safety benchmarks, significantly outperforming existing open-source models and demonstrating competitive performance against leading proprietary systems.

VAR: Visual Attention Reasoning via Structured Search and Backtracking

TL;DR

Visual Attention Reasoning (VAR), a novel framework that recasts grounded reasoning as a structured search over a reasoning trajectory space, sets a new state-of-the-art on a comprehensive suite of hallucination and safety benchmarks, significantly outperforming existing open-source models and demonstrating competitive performance against leading proprietary systems.

Abstract

Multimodal Large Language Models (MLLMs), despite their advances, are hindered by their high hallucination tendency and heavy reliance on brittle, linear reasoning processes, leading to failures in complex tasks. To address these limitations, we introduce Visual Attention Reasoning (VAR), a novel framework that recasts grounded reasoning as a structured search over a reasoning trajectory space. VAR decomposes the reasoning process into two key stages: traceable evidence grounding and search-based chain-of-thought (CoT) generation, which incorporates a backtracking mechanism for self-correction. The search is guided by a multi-faceted reward function with semantic and geometric self-verification components, which penalize outputs that are not faithfully grounded in the visual input. We provide a theoretical analysis for our search strategy, validating its capability to find the correct solution with high probability. Experimental results show that our 7B model, VAR-7B, sets a new state-of-the-art on a comprehensive suite of hallucination and safety benchmarks, significantly outperforming existing open-source models and demonstrating competitive performance against leading proprietary systems.
Paper Structure (19 sections, 2 theorems, 8 equations, 3 figures, 3 tables)

This paper contains 19 sections, 2 theorems, 8 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Visual Attention Reasoning
  3. The Decomposed Reasoning Trajectory
  4. Search-Based Chain-of-Thought with Backtracking
  5. Experiment
  6. VAR Implementation
  7. Data Preparation
  8. Evaluated Models and Configurations
  9. Main Result
  10. Further Analysis
  11. Ablation Studies
  12. Related Works
  13. Conlusion
  14. Appendix
  15. Lemma Proof
  16. ...and 4 more sections

Key Result

Lemma 1

Let $\gamma \in (0, 1)$ and $\delta \in (0, 1/2)$ be constants. Define error tolerance $\epsilon$ and exploration budget $B$ as: Suppose a policy $\pi_\theta$ satisfies the Probabilistic Forward Progress and Reliable Trajectory Recovery conditions with these parameters, and we restrict each node expansion to at most $B$ generative attempts.Then, the probability that $\pi_\theta$ generates a corre

Figures (3)

  • Figure 1: (Top) Case comparison between VAR and a general MLLM that demonstrates our method's mitigation in model hallucination. (Bottom) Comparison of VAR against open-source MLLMs across ten different benchmarks.
  • Figure 2: An overview of our VAR framework
  • Figure 3: Correlation of Model Capabilities

Theorems & Definitions (5)

  • Lemma 1: Sufficient Conditions for High-Probability Trajectory Generation
  • proof
  • Remark 1: Relaxing the Backtracking Precision
  • Lemma 2: Characterization of the Optimal Policy via Parity
  • proof