BayesRAG: Probabilistic Mutual Evidence Corroboration for Multimodal Retrieval-Augmented Generation

TL;DR

BayesRAG reframes multimodal retrieval for visually rich documents as probabilistic evidence fusion, combining Bayesian inference with Dempster-Shafer theory to jointly evaluate cross-modal evidence. It defines an evidence tuple $(t,v,s)$ and ranks candidates via the posterior $P(E|Q) \propto P(Q|E) \cdot P(E)$, where $P(Q|E)$ is computed from modality similarities using mass functions and DS combination, and $P(E)$ encodes layout- and graph-based consistency priors. The approach preserves native multimodal structure and layout, enabling mutual corroboration across text, images, and document layout, with two priors (Graph-Topology and Layout) to guide coherence. Experiments on DocBench and MMLongBench-Doc show BayesRAG achieves state-of-the-art results, significantly improving recall and generation quality, especially when paired with stronger generators, and demonstrating robust cross-modal reasoning and noise suppression in complex documents. The work lays a foundation for trustworthy multimodal RAG by explicit evidence corroboration and structure-aware priors, with practical code available.

Abstract

Retrieval-Augmented Generation (RAG) has become a pivotal paradigm for Large Language Models (LLMs), yet current approaches struggle with visually rich documents by treating text and images as isolated retrieval targets. Existing methods relying solely on cosine similarity often fail to capture the semantic reinforcement provided by cross-modal alignment and layout-induced coherence. To address these limitations, we propose BayesRAG, a novel multimodal retrieval framework grounded in Bayesian inference and Dempster-Shafer evidence theory. Unlike traditional approaches that rank candidates strictly by similarity, BayesRAG models the intrinsic consistency of retrieved candidates across modalities as probabilistic evidence to refine retrieval confidence. Specifically, our method computes the posterior association probability for combinations of multimodal retrieval results, prioritizing text-image pairs that mutually corroborate each other in terms of both semantics and layout. Extensive experiments demonstrate that BayesRAG significantly outperforms state-of-the-art (SOTA) methods on challenging multimodal benchmarks. This study establishes a new paradigm for multimodal retrieval fusion that effectively resolves the isolation of heterogeneous modalities through an evidence fusion mechanism and enhances the robustness of retrieval outcomes. Our code is available at https://github.com/TioeAre/BayesRAG.

Figures (8)

• Figure 1: The architecture of BayesRAG. We reconceptualize multimodal retrieval as a Bayesian inference process. Our method computes the semantic likelihood $P(Q|E)$ and derives the consistency prior $P(E)$ via knowledge graph topology. The final evidence tuples are re-ranked based on posterior probability $P(E|Q)$, ensuring semantic relevance, mutual corroboration, and logical self-consistency.
• Figure 2: A qualitative comparison illustrating the efficacy of BayesRAG in broadening the scope of visual retrieval and filtering inconsistent evidence. Our statistical analysis reveals that approximately 34% of failure cases in baseline methods are attributed to this retrieval deficiency.
• Figure 3: A sample is for semantic consistency between text and charts to find the correct evidence
• Figure 4: Prompts used by the model to predict answers.
• Figure 5: Prompts used by evaluating the answer in MMLongBench-Doc.