OmniRet: Efficient and High-Fidelity Omni Modality Retrieval

TL;DR

The OmniRet model is presented, the first retrieval model capable of handling complex, composed queries spanning three key modalities: text, vision, and audio, and a new Audio-Centric Multimodal Benchmark is curate to more comprehensively evaluate a model's omni-modal embedding capacity.

Abstract

Multimodal retrieval is the task of aggregating information from queries across heterogeneous modalities to retrieve desired targets. State-of-the-art multimodal retrieval models can understand complex queries, yet they are typically limited to two modalities: text and vision. This limitation impedes the development of universal retrieval systems capable of comprehending queries that combine more than two modalities. To advance toward this goal, we present OmniRet, the first retrieval model capable of handling complex, composed queries spanning three key modalities: text, vision, and audio. Our OmniRet model addresses two critical challenges for universal retrieval: computational efficiency and representation fidelity. First, feeding massive token sequences from modality-specific encoders to Large Language Models (LLMs) is computationally inefficient. We therefore introduce an attention-based resampling mechanism to generate compact, fixed-size representations from these sequences. Second, compressing rich omni-modal data into a single embedding vector inevitably causes information loss and discards fine-grained details. We propose Attention Sliced Wasserstein Pooling to preserve these fine-grained details, leading to improved omni-modal representations. OmniRet is trained on an aggregation of approximately 6 million query-target pairs spanning 30 datasets. We benchmark our model on 13 retrieval tasks and a MMEBv2 subset. Our model demonstrates significant improvements on composed query, audio and video retrieval tasks, while achieving on-par performance with state-of-the-art models on others. Furthermore, we curate a new Audio-Centric Multimodal Benchmark (ACM). This new benchmark introduces two critical, previously missing tasks-composed audio retrieval and audio-visual retrieval to more comprehensively evaluate a model's omni-modal embedding capacity.

Figures (5)

• Figure 1: (a) Comparison of retrieval systems. OmniRet is the first to handle composed queries from text, vision, and audio. It introduces a Shared Media Resampler for efficiency and Attention Sliced Wasserstein Pooling to preserve high-fidelity, fine-grained details. (b) Our Audio-Centric MultiModal evaluation suite. We introduce two novel, audio-centric tasks: Visual-Audio Retrieval and Composed Audio-Text Retrieval. These additions fill critical gaps in existing benchmarks, enabling a more comprehensive evaluation of omni-modal retrieval models.
• Figure 2: Overall OmniRet architecture. Our universal retrieval model integrates specialized visual and audio encoders with a Large Language Model acting as a cross-modal composer. Media inputs are processed through a Shared Media Resampler, and the final embedding is derived via our Attention Sliced Wasserstein Pooling. The model is optimized using Contrastive, Triplet and Diversity losses.
• Figure 3: Left: Our Shared Media Resampler condenses the output of media encoders into a compact set of latent vectors before they are fed to the LLM. Right: Our Attention Sliced Wasserstein Pooling (ASWP) aggregates the final LLM output hidden states into a single, high-fidelity embedding vector. Attention Resampler applies to both output of media encoders and LLM universal encoder.
• Figure 4: A composed audio retrieval example from our ACM benchmark where query composes both audio and text modalities.
• Figure 5: The Qualtric survey for verifying the quality of generated texts in our ACM benchmark.