Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Glance-or-Gaze: Incentivizing LMMs to Adaptively Focus Search via Reinforcement Learning

Hongbo Bai, Yujin Zhou, Yile Wu, Chi-Min Chan, Pengcheng Wen, Kunhao Pan, Sirui Han, Yike Guo

TL;DR

Glance-or-Gaze (GoG) tackles the mismatch between static parametric knowledge in large multimodal models and the evolving, knowledge-intensive nature of real-world queries by enabling adaptive visual planning. It introduces a Selective Gaze mechanism to filter noise before retrieval and a dual-stage training regime: Reflective GoG Behavior Alignment via supervised fine-tuning, followed by Complexity-Adaptive Reinforcement Learning to optimize iterative, multi-step search. The GoG-Instruct data and GRPO-based RL on hard samples yield state-of-the-art performance across six benchmarks, with ablations confirming the necessity of Selective Gaze and complexity-aware learning. This framework significantly improves accuracy and robustness for knowledge-intensive VQA, enabling more autonomous and reliable multimodal search in dynamic environments.

Abstract

Large Multimodal Models (LMMs) have achieved remarkable success in visual understanding, yet they struggle with knowledge-intensive queries involving long-tail entities or evolving information due to static parametric knowledge. Recent search-augmented approaches attempt to address this limitation, but existing methods rely on indiscriminate whole-image retrieval that introduces substantial visual redundancy and noise, and lack deep iterative reflection, limiting their effectiveness on complex visual queries. To overcome these challenges, we propose Glance-or-Gaze (GoG), a fully autonomous framework that shifts from passive perception to active visual planning. GoG introduces a Selective Gaze mechanism that dynamically chooses whether to glance at global context or gaze into high-value regions, filtering irrelevant information before retrieval. We design a dual-stage training strategy: Reflective GoG Behavior Alignment via supervised fine-tuning instills the fundamental GoG paradigm, while Complexity-Adaptive Reinforcement Learning further enhances the model's capability to handle complex queries through iterative reasoning. Experiments across six benchmarks demonstrate state-of-the-art performance. Ablation studies confirm that both Selective Gaze and complexity-adaptive RL are essential for effective visual search. We will release our data and models for further exploration soon.

Glance-or-Gaze: Incentivizing LMMs to Adaptively Focus Search via Reinforcement Learning

TL;DR

Glance-or-Gaze (GoG) tackles the mismatch between static parametric knowledge in large multimodal models and the evolving, knowledge-intensive nature of real-world queries by enabling adaptive visual planning. It introduces a Selective Gaze mechanism to filter noise before retrieval and a dual-stage training regime: Reflective GoG Behavior Alignment via supervised fine-tuning, followed by Complexity-Adaptive Reinforcement Learning to optimize iterative, multi-step search. The GoG-Instruct data and GRPO-based RL on hard samples yield state-of-the-art performance across six benchmarks, with ablations confirming the necessity of Selective Gaze and complexity-aware learning. This framework significantly improves accuracy and robustness for knowledge-intensive VQA, enabling more autonomous and reliable multimodal search in dynamic environments.

Abstract

Large Multimodal Models (LMMs) have achieved remarkable success in visual understanding, yet they struggle with knowledge-intensive queries involving long-tail entities or evolving information due to static parametric knowledge. Recent search-augmented approaches attempt to address this limitation, but existing methods rely on indiscriminate whole-image retrieval that introduces substantial visual redundancy and noise, and lack deep iterative reflection, limiting their effectiveness on complex visual queries. To overcome these challenges, we propose Glance-or-Gaze (GoG), a fully autonomous framework that shifts from passive perception to active visual planning. GoG introduces a Selective Gaze mechanism that dynamically chooses whether to glance at global context or gaze into high-value regions, filtering irrelevant information before retrieval. We design a dual-stage training strategy: Reflective GoG Behavior Alignment via supervised fine-tuning instills the fundamental GoG paradigm, while Complexity-Adaptive Reinforcement Learning further enhances the model's capability to handle complex queries through iterative reasoning. Experiments across six benchmarks demonstrate state-of-the-art performance. Ablation studies confirm that both Selective Gaze and complexity-adaptive RL are essential for effective visual search. We will release our data and models for further exploration soon.
Paper Structure (49 sections, 3 equations, 4 figures, 6 tables)

This paper contains 49 sections, 3 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Large Multimodal Models
  4. Search-Augmented Visual Reasoning
  5. Method
  6. Overview
  7. Stage 1: Reflective GoG Behavior Alignment
  8. GoG-Instruct Data Construction
  9. Training Objective
  10. Stage 2: Complexity-Adaptive Reinforcement Learning
  11. RL Data Construction
  12. Training Objective
  13. Experiments
  14. Experiment Setup
  15. Implementation Details
  16. ...and 34 more sections

Figures (4)

  • Figure 1: Comparison between previous baselines and our Glance-or-Gaze (GoG) framework. GoG employs an active multi-step strategy: proposing candidate regions via visual grounding, filtering relevant crops through Selective Gaze, and conducting precise search only on selected regions with iterative cross-modal reflection.
  • Figure 2: Overview of the Glance-or-Gaze (GoG) framework. Stage 1 (Left): Reflective GoG Behavior Alignment constructs GoG-Instruct data through uncertainty-aware filtering and human-verified trajectory synthesis, then performs supervised fine-tuning to instill active selection and cross-modal reflection. Stage 2 (Right): Complexity-Adaptive RL constructs complexity-stratified data at two difficulty levels and applies reinforcement learning to enhance planning capabilities for adaptive visual reasoning.
  • Figure 3: Distribution of search behavior across different training stages. "No Search" indicates samples without any search action, "One Search" represents samples using only one type of search (text, image, or crop), and "Mix Search" denotes samples combining multiple search types.
  • Figure 4: Effectiveness of Selective Gaze. Comparison of Crop Selection Accuracy between SFT and RL training stages across two model architectures.