MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions

TL;DR

MagicLens addresses image retrieval with open-ended textual instructions by mining naturally co-occurring web-page image pairs and generating open-ended instructions via foundation models, training lightweight dual-encoders in a self-supervised setup on 36.7M triplets. It achieves competitive or superior results across eight multimodality-to-image benchmarks and related tasks while being ~50x more parameter-efficient than prior SOTA methods. A large-scale 1.4M open-domain evaluation with human judges demonstrates MagicLens’s ability to satisfy complex and beyond-visual search intents. The work also shows that template-free, diverse instructions and web-derived data significantly boost performance, offering practical, scalable pathways for instruction-guided retrieval and related multimodal tasks.

Abstract

Image retrieval, i.e., finding desired images given a reference image, inherently encompasses rich, multi-faceted search intents that are difficult to capture solely using image-based measures. Recent works leverage text instructions to allow users to more freely express their search intents. However, they primarily focus on image pairs that are visually similar and/or can be characterized by a small set of pre-defined relations. The core thesis of this paper is that text instructions can enable retrieving images with richer relations beyond visual similarity. To show this, we introduce MagicLens, a series of self-supervised image retrieval models that support open-ended instructions. MagicLens is built on a key novel insight: image pairs that naturally occur on the same web pages contain a wide range of implicit relations (e.g., inside view of), and we can bring those implicit relations explicit by synthesizing instructions via foundation models. Trained on 36.7M (query image, instruction, target image) triplets with rich semantic relations mined from the web, MagicLens achieves results comparable with or better than prior best on eight benchmarks of various image retrieval tasks, while maintaining high parameter efficiency with a significantly smaller model size. Additional human analyses on a 1.4M-image unseen corpus further demonstrate the diversity of search intents supported by MagicLens. Code and models are publicly available at https://open-vision-language.github.io/MagicLens/.

Figures (11)

• Figure 1: Top-1 retrieved images using MagicLens and the prior state-of-the-art (SOTA) method Gu2023LinCIR from a retrieval pool with 1.4M images. The prior SOTA method, while accepting text instructions, primarily retrieves images based on visual similarity to the query image, ignoring the nuances of the text instructions. In contrast, MagicLens excels at retrieving both visually similar images and those that align with the deeper meaning and context of the text instructions --- even when the images do not resemble the query. For example, if given a query image of the Burj Al Arab and the instruction "Find other attractions in this country", it can successfully locate images of the Palm Islands in Dubai.
• Figure 2: Data construction overview. We collect naturally occurring image pairs from the same web pages and use PaLI+PaLM2 to generate instructions connecting the two images.
• Figure 3: Data construction pipeline. We mine image pairs from the web via (1) grouping images from the same web page and cleaning them, (2) annotating metadata for each image with LMMs, and (3) scoring and filtering out unqualified image pairs. Eventually, we generate open-ended instructions using LLMs for the remaining image pairs.
• Figure 4: Model architecture and training of MagicLens Encoder (E), which takes the vision and language embeddings and feeds them as a sequence to self-attention layers for modality integration.
• Figure 5: Word distributions of IP2P data and our data.