Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond

TL;DR

This work tackles cross-modal retrieval by enabling multimodal LLMs to memorize images within their parameters and recall them via text-generated identifiers. The proposed GRACE framework uses two training stages—learning to memorize image-identifier mappings and learning to retrieve identifiers from queries—along with constrained, Trie-guided generation during inference. It systematically analyzes multiple identifier types (String, Numeric, Semantic, Structured, Atomic) and shows that Atomic and Structured identifiers offer strong retrieval performance with favorable scalability, even when comparing to two-tower baselines. Beyond retrieval, GRACE enables describing and answering questions about memorized images, signaling a new capability for personalized, memory-rich multimodal interactions. Overall, the work establishes a new generative cross-modal retrieval paradigm that leverages inbuilt visual memory to handle large image catalogs efficiently.

Abstract

The recent advancements in generative language models have demonstrated their ability to memorize knowledge from documents and recall knowledge to respond to user queries effectively. Building upon this capability, we propose to enable multimodal large language models (MLLMs) to memorize and recall images within their parameters. Given a user query for visual content, the MLLM is anticipated to "recall" the relevant image from its parameters as the response. Achieving this target presents notable challenges, including inbuilt visual memory and visual recall schemes within MLLMs. To address these challenges, we introduce a generative cross-modal retrieval framework, which assigns unique identifier strings to represent images and involves two training steps: learning to memorize and learning to retrieve. The first step focuses on training the MLLM to memorize the association between images and their respective identifiers. The latter step teaches the MLLM to generate the corresponding identifier of the target image, given the textual query input. By memorizing images in MLLMs, we introduce a new paradigm to cross-modal retrieval, distinct from previous discriminative approaches. The experiments demonstrate that the generative paradigm performs effectively and efficiently even with large-scale image candidate sets.

Figures (6)

• Figure 1: Real cases from GPT4 illustrate the necessity of visual outputs for LLMs.
• Figure 2: Illustration of our proposed generative cross-modal framework, GRACE, which involves two training steps. (a) Learning to memorize: GRACE trains an MLLM model to memorize images into its parameters. (b) Learning to retrieve: GRACE trains the model to generate the target image's identifiers given queries. (c) Inference: The MLLM directly generates identifiers as the retrieval results.
• Figure 3: (a) depicts an image accompanied by various identifier types. (b) shows the formation of structured identifiers, where each image's identifier is represented as its unique path within a cluster tree.
• Figure 4: The efficiency of CLIP and GRACE varies with image size, measured in terms of queries processed per second. As the image size increases, GRACE demonstrates superior efficiency.
• Figure 5: Cases of interaction with memorized images for an MLLM include retrieving the memorized images, describing them, and answering questions about them, based on specific instructions. It is noted that the MLLM model responds to user instructions without any image input, relying solely on memorized visual information.