Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities

Woongyeong Yeo, Kangsan Kim, Soyeong Jeong, Jinheon Baek, Sung Ju Hwang

TL;DR

UniversalRAG introduces modality- and granularity-aware retrieval to ground LVLMs over heterogeneous, multi-modal corpora. By employing modality-aware routing and multi-granularity corpora, it avoids the modality gap inherent in unified embedding approaches and enables targeted retrieval across text, images, video, and structured data. The framework supports two router paradigms—training-based and training-free—and demonstrates superior performance on 10 diverse benchmarks, along with improved efficiency and robustness in out-of-domain settings. Ensemble routing further balances in-domain accuracy and OOD generalization. Overall, UniversalRAG advances retrieval-augmented generation toward a flexible, scalable, and practical one-for-all framework for multimodal knowledge grounding.

Abstract

Retrieval-Augmented Generation (RAG) has shown substantial promise in improving factual accuracy by grounding model responses with external knowledge relevant to queries. However, most existing approaches are limited to a text-only corpus, and while recent efforts have extended RAG to other modalities such as images and videos, they typically operate over a single modality-specific corpus. In contrast, real-world queries vary widely in the type of knowledge they require, which a single type of knowledge source cannot address. To address this, we introduce UniversalRAG, designed to retrieve and integrate knowledge from heterogeneous sources with diverse modalities and granularities. Specifically, motivated by the observation that forcing all modalities into a unified representation space derived from a single aggregated corpus causes a modality gap, where the retrieval tends to favor items from the same modality as the query, we propose modality-aware routing, which dynamically identifies the most appropriate modality-specific corpus and performs targeted retrieval within it, and further justify its effectiveness with a theoretical analysis. Moreover, beyond modality, we organize each modality into multiple granularity levels, enabling fine-tuned retrieval tailored to the complexity and scope of the query. We validate UniversalRAG on 10 benchmarks of multiple modalities, showing its superiority over various modality-specific and unified baselines.

UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities

TL;DR

UniversalRAG introduces modality- and granularity-aware retrieval to ground LVLMs over heterogeneous, multi-modal corpora. By employing modality-aware routing and multi-granularity corpora, it avoids the modality gap inherent in unified embedding approaches and enables targeted retrieval across text, images, video, and structured data. The framework supports two router paradigms—training-based and training-free—and demonstrates superior performance on 10 diverse benchmarks, along with improved efficiency and robustness in out-of-domain settings. Ensemble routing further balances in-domain accuracy and OOD generalization. Overall, UniversalRAG advances retrieval-augmented generation toward a flexible, scalable, and practical one-for-all framework for multimodal knowledge grounding.

Abstract

Retrieval-Augmented Generation (RAG) has shown substantial promise in improving factual accuracy by grounding model responses with external knowledge relevant to queries. However, most existing approaches are limited to a text-only corpus, and while recent efforts have extended RAG to other modalities such as images and videos, they typically operate over a single modality-specific corpus. In contrast, real-world queries vary widely in the type of knowledge they require, which a single type of knowledge source cannot address. To address this, we introduce UniversalRAG, designed to retrieve and integrate knowledge from heterogeneous sources with diverse modalities and granularities. Specifically, motivated by the observation that forcing all modalities into a unified representation space derived from a single aggregated corpus causes a modality gap, where the retrieval tends to favor items from the same modality as the query, we propose modality-aware routing, which dynamically identifies the most appropriate modality-specific corpus and performs targeted retrieval within it, and further justify its effectiveness with a theoretical analysis. Moreover, beyond modality, we organize each modality into multiple granularity levels, enabling fine-tuned retrieval tailored to the complexity and scope of the query. We validate UniversalRAG on 10 benchmarks of multiple modalities, showing its superiority over various modality-specific and unified baselines.
Paper Structure (63 sections, 3 theorems, 15 equations, 13 figures, 15 tables)

This paper contains 63 sections, 3 theorems, 15 equations, 13 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Preliminaries
  4. Large Vision Language Models
  5. Retrieval-Augmented Generation
  6. UniversalRAG
  7. Challenges in Multi-Corpus Retrieval
  8. Modality-Aware Retrieval
  9. Granularity-Aware Retrieval
  10. Router Implementation Strategies
  11. Training-based Router
  12. Training-free Router
  13. Experiment
  14. Experimental Setup
  15. Datasets
  16. ...and 48 more sections

Key Result

Proposition 1

Let the similarity score in a unified embedding space $\mathcal{C}_\texttt{unified}$ be defined as where $\alpha>0$ induces modality bias and $r(\cdot, \cdot)$ measures relevance. If $\alpha$ dominates the variability of $r$, modality-aware routing retrieves items from the required modality $m^\ast({\bm{q}})$ with higher probability than unified embedding retrieval.

Figures (13)

  • Figure 1: Conceptual illustration comparing existing RAG strategies with our proposed UniversalRAG.
  • Figure 2: t-SNE plot of the unified embedding space.
  • Figure 2: Performance comparison of uni-modal and cross-modal approaches across different router models. Among models, GPT-5 is the only training-free router.
  • Figure 3: Comparison of averaged evaluation results across different RAG methods and LVLMs.
  • Figure 4: Distribution of the retrieved data modalities.
  • ...and 8 more figures

Theorems & Definitions (7)

  • Proposition 1
  • proof
  • Remark
  • Proposition 2
  • proof
  • Proposition 3
  • proof