RoarGraph: A Projected Bipartite Graph for Efficient Cross-Modal Approximate Nearest Neighbor Search

TL;DR

Cross-modal ANNS faces a distribution gap where queries from one modality (e.g., text) are Out-of-Distribution relative to base data (e.g., images), causing existing single-modal graph indexes to underperform. RoarGraph addresses this by constructing a query-guided index using a Query-Base Bipartite Graph, followed by Neighborhood-Aware Projection and Connectivity Enhancement to preserve query-derived proximity while keeping routing efficient. The method delivers significant speedups (up to 3.56x) at high recall on multiple cross-modal datasets and shows robust performance on OOD workloads with competitive in-distribution results, plus practical indexing overhead. These results suggest that workload-driven, distribution-aware graph indices can substantially improve cross-modal retrieval tasks in real-world multimodal systems, and RoarGraph has demonstrated state-of-the-art performance in relevant benchmarks.

Abstract

Approximate Nearest Neighbor Search (ANNS) is a fundamental and critical component in many applications, including recommendation systems and large language model-based applications. With the advancement of multimodal neural models, which transform data from different modalities into a shared high-dimensional space as feature vectors, cross-modal ANNS aims to use the data vector from one modality (e.g., texts) as the query to retrieve the most similar items from another (e.g., images or videos). However, there is an inherent distribution gap between embeddings from different modalities, and cross-modal queries become Out-of-Distribution (OOD) to the base data. Consequently, state-of-the-art ANNS approaches suffer poor performance for OOD workloads. In this paper, we quantitatively analyze the properties of the OOD workloads to gain an understanding of their ANNS efficiency. Unlike single-modal workloads, we reveal OOD queries spatially deviate from base data, and the k-nearest neighbors of an OOD query are distant from each other in the embedding space. The property breaks the assumptions of existing ANNS approaches and mismatches their design for efficient search. With insights from the OOD workloads, we propose pRojected bipartite Graph (RoarGraph), an efficient ANNS graph index built under the guidance of query distribution. Extensive experiments show that RoarGraph significantly outperforms state-of-the-art approaches on modern cross-modal datasets, achieving up to 3.56x faster search speed at a 90% recall rate for OOD queries.

Figures (17)

• Figure 1: Mahalanobis distances from OOD/ID queries to the base data.
• Figure 2: Performance evaluation on ID and OOD workloads.
• Figure 3: Evaluation of OOD-DiskANN and DiskANN. The notations ID and OOD in parentheses denote ID and OOD query workloads, respectively.
• Figure 4: Distances between nearest neighbor to ID (visual)/OOD (textual) queries ($10^4$ queries for each category).
• Figure 5: For $10^4$ queries' 100 nearest neighbors, the average distances between one vector and the other 99 vectors are computed. Each count represents the mean value of distances within either OOD or ID query sets.

Theorems & Definitions (1)

• Definition 1