DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories

TL;DR

DeepImageSearch reframes image retrieval as corpus-level, context-aware reasoning over visual histories, addressing the gap where traditional methods treat images in isolation. The authors introduce DISBench, built via a semi-automated, human-verified pipeline, and an agentic baseline, ImageSeeker, equipped with tools and memory for long-horizon exploration. Empirical results show that state-of-the-art models struggle with cross-event reasoning, achieving an Exact Match of $EM=28.7$ and an F1 of $55.0$, while direct embedding-based retrieval yields substantially lower metrics, underscoring the need for agentic planning and memory. The work offers DISBench as a challenging testbed and establishes a modular framework for advancing context-aware, multimodal retrieval in realistic visual histories, with implications for personal photo organization and memory-supporting AI systems.

Abstract

Existing multimodal retrieval systems excel at semantic matching but implicitly assume that query-image relevance can be measured in isolation. This paradigm overlooks the rich dependencies inherent in realistic visual streams, where information is distributed across temporal sequences rather than confined to single snapshots. To bridge this gap, we introduce DeepImageSearch, a novel agentic paradigm that reformulates image retrieval as an autonomous exploration task. Models must plan and perform multi-step reasoning over raw visual histories to locate targets based on implicit contextual cues. We construct DISBench, a challenging benchmark built on interconnected visual data. To address the scalability challenge of creating context-dependent queries, we propose a human-model collaborative pipeline that employs vision-language models to mine latent spatiotemporal associations, effectively offloading intensive context discovery before human verification. Furthermore, we build a robust baseline using a modular agent framework equipped with fine-grained tools and a dual-memory system for long-horizon navigation. Extensive experiments demonstrate that DISBench poses significant challenges to state-of-the-art models, highlighting the necessity of incorporating agentic reasoning into next-generation retrieval systems.

Figures (7)

• Figure 1: Evolution of image retrieval paradigms. (a) Direct retrieval matches queries to images through visual semantic alignment. (b) Reasoning-intensive retrieval requires inference over external knowledge, but still evaluates each image independently. (c) DeepImageSearch demands corpus context awareness, where models must first locate target events within the visual history and then identify qualifying images through multi-step reasoning.
• Figure 2: Two query types in DISBench. (a) Intra-Event queries locate a specific event and filter targets within it. (b) Inter-Event queries scan across events to verify recurring elements under temporal/spatial constraints.
• Figure 3: Semi-automated data construction pipeline. Starting from raw images, we first parse visual content to extract salient clues and person attributes, then mine latent associations across the corpus through retrieval and verification strategy. These elements are organized into a memory graph, from which we sample subgraphs via random walks to synthesize candidate queries for human verification.
• Figure 4: Dataset statistics of DISBench. (a) Query type distribution shows a balanced split between intra-event and inter-event queries. (b) Target images span diverse themes including portraits, nature views, daily items, and scenic spots.
• Figure 5: Effect of test-time scaling with different strategies.