RobustVisRAG: Causality-Aware Vision-Based Retrieval-Augmented Generation under Visual Degradations

TL;DR

RobustVisRAG is introduced, a causality-guided dual-path framework that improves VisRAG robustness while preserving efficiency and zero-shot generalization, and enforces a clear separation between semantics and degradations, enabling stable retrieval and generation under challenging visual conditions.

Abstract

Vision-based Retrieval-Augmented Generation (VisRAG) leverages vision-language models (VLMs) to jointly retrieve relevant visual documents and generate grounded answers based on multimodal evidence. However, existing VisRAG models degrade in performance when visual inputs suffer from distortions such as blur, noise, low light, or shadow, where semantic and degradation factors become entangled within pretrained visual encoders, leading to errors in both retrieval and generation stages. To address this limitation, we introduce RobustVisRAG, a causality-guided dual-path framework that improves VisRAG robustness while preserving efficiency and zero-shot generalization. RobustVisRAG uses a non-causal path to capture degradation signals through unidirectional attention and a causal path to learn purified semantics guided by these signals. Together with the proposed Non-Causal Distortion Modeling and Causal Semantic Alignment objectives, the framework enforces a clear separation between semantics and degradations, enabling stable retrieval and generation under challenging visual conditions. To evaluate robustness under realistic conditions, we introduce the Distortion-VisRAG dataset, a large-scale benchmark containing both synthetic and real-world degraded documents across seven domains, with 12 synthetic and 5 real distortion types that comprehensively reflect practical visual degradations. Experimental results show that RobustVisRAG improves retrieval, generation, and end-to-end performance by 7.35%, 6.35%, and 12.40%, respectively, on real-world degradations, while maintaining comparable accuracy on clean inputs.

Figures (3)

• Figure 1: Illustration of RobustVisRAG's capabilities. (a) Retrieval performance under clean, synthetic degradation (Degrade-Syn) and real-degradation (Degrade-Real) scenarios. (b) Generation performance using the retrieved documents from RobustVisRAG as input. (c)(d) End-to-end retrieval–generation performance on clean and degraded data, evaluated under the same baselines including VisRAG, its fine-tuned variants (VisRAG-FT) schlarmann2024robust, and a Two-Stage restoration pipeline potlapalli2024promptir. Across all settings, both TextRAG and VisRAG show notable performance drops under degraded inputs, whereas RobustVisRAG preserves clean accuracy and significantly improves robustness in degraded conditions.
• Figure 2: Overview of RobustVisRAG. (a) Structural causal model of VisRAG under visual degradations. (b) Architecture of the vanilla vision-based RAG pipeline with degraded input. (c) The proposed RobustVisRAG, which introduces a causality-guided dual-path encoder to disentangle semantic and degradation factors.
• Figure 3: Comparison of token representations under degradations: Attention visualizations of (a) VisRAG and (b) RobustVisRAG. (c) Clean version corresponding to (a) and (b). t-SNE visualization of $Z_{\text{deg}}$ from (d) RobustVisRAG w/o $\mathcal{L}_{\text{NCDM}}$ & $\mathcal{L}_{\text{CSA}}$ and (e) RobustVisRAG.