RAG-Anything: All-in-One RAG Framework

TL;DR

RAG-Anything tackles the mismatch between retrieval-augmented generation and real-world multimodal knowledge by introducing a unified graph-based framework that treats text, images, tables, and equations as interconnected knowledge entities. It employs a dual-graph construction (cross-modal and text-based) and a cross-modal hybrid retrieval strategy that combines structural navigation with semantic matching, followed by a vision-language synthesis stage to ground responses in both textual and visual content. The approach yields superior performance on long-context multimodal document QA benchmarks (DocBench and MMLongBench), outperforming state-of-the-art baselines and reducing architectural fragmentation across modalities. The framework is open-sourced, offering a scalable foundation for robust multimodal knowledge access in diverse domains.

Abstract

Retrieval-Augmented Generation (RAG) has emerged as a fundamental paradigm for expanding Large Language Models beyond their static training limitations. However, a critical misalignment exists between current RAG capabilities and real-world information environments. Modern knowledge repositories are inherently multimodal, containing rich combinations of textual content, visual elements, structured tables, and mathematical expressions. Yet existing RAG frameworks are limited to textual content, creating fundamental gaps when processing multimodal documents. We present RAG-Anything, a unified framework that enables comprehensive knowledge retrieval across all modalities. Our approach reconceptualizes multimodal content as interconnected knowledge entities rather than isolated data types. The framework introduces dual-graph construction to capture both cross-modal relationships and textual semantics within a unified representation. We develop cross-modal hybrid retrieval that combines structural knowledge navigation with semantic matching. This enables effective reasoning over heterogeneous content where relevant evidence spans multiple modalities. RAG-Anything demonstrates superior performance on challenging multimodal benchmarks, achieving significant improvements over state-of-the-art methods. Performance gains become particularly pronounced on long documents where traditional approaches fail. Our framework establishes a new paradigm for multimodal knowledge access, eliminating the architectural fragmentation that constrains current systems. Our framework is open-sourced at: https://github.com/HKUDS/RAG-Anything.

Figures (12)

• Figure 1: Overview of our proposed universal RAG framework RAG-Anything.
• Figure 2: Performance evaluation across documents of varying lengths.
• Figure 3: Multi-panel figure interpretation case. The query requires identifying cluster separation patterns from the style-space panel, while avoiding confusion from the adjacent content-space panel.
• Figure 4: Financial table navigation case. The query involves locating the specific intersection of "Wages and salaries" row and "2020" column amid similar terminological entries.
• Figure 5: Visual reasoning case. RAG-Anything correctly identifies "-S-A" as the lowest accuracy configuration, while baselines misinterpret spatial relationships.