Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models

TL;DR

PixSearch introduces an end-to-end segmenting Large Multimodal Model that autonomously triggers retrieval and grounds responses with region-level pixel masks. By emitting <search> tokens and routing queries across text, whole-image, or region modalities, it achieves pixel-grounded, multi-hop reasoning without external detectors or captioners. Two-stage supervised fine-tuning aligns retrieval timing and query selection with robust segmentation, yielding substantial improvements in factual grounding on egocentric/entity-centric VQA (up to ~26% accuracy gains on CRAG-MM ego-centric) while remaining competitive on standard VQA and text QA tasks. The approach offers a scalable path toward more factual, pixel-grounded multimodal reasoning in real-world AI assistants.

Abstract

Visual Question Answering (VQA) often requires coupling fine-grained perception with factual knowledge beyond the input image. Prior multimodal Retrieval-Augmented Generation (MM-RAG) systems improve factual grounding but lack an internal policy for when and how to retrieve. We propose PixSearch, the first end-to-end Segmenting Large Multimodal Model (LMM) that unifies region-level perception and retrieval-augmented reasoning. During encoding, PixSearch emits <search> tokens to trigger retrieval, selects query modalities (text, image, or region), and generates pixel-level masks that directly serve as visual queries, eliminating the reliance on modular pipelines (detectors, segmenters, captioners, etc.). A two-stage supervised fine-tuning regimen with search-interleaved supervision teaches retrieval timing and query selection while preserving segmentation ability. On egocentric and entity-centric VQA benchmarks, PixSearch substantially improves factual consistency and generalization, yielding a 19.7% relative gain in accuracy on CRAG-MM compared to whole image retrieval, while retaining competitive reasoning performance on various VQA and text-only QA tasks.

Figures (4)

• Figure 1: Egocentric images from wearables devices often render entities smaller than it appears because of wide-angle cameras. MM-RAG methods that rely on full-image search or caption-only queries can introduce retrieval noises and degrade QA quality. PixSearch, an end-to-end segmenting LMM, learns when to issue a query, how to route among text, whole-image, and region-level queries, and how to reason over retrieved evidence for answer generation. Our work also compares against pipeline, tool-based approaches. On CRAG-MM wang2025crag, PixSearch (full) improves accuracy by 26% and reduces hallucination by 39%.
• Figure 2: Overview of the proposed PixSearch framework. The model learns to decide when retrieval is needed, how to query (text, whole image, or segmented region), and grounds its answers in retrieved evidence while preserving mask-generation capabilities.
• Figure 3: Data construction pipeline for Stage-2 training.
• Figure 4: Search Behavior Plot from the PixSearch$_{\text{Interleaved}}$ outputs for CRAG-MM evaluation set.