A Decentralized Retrieval Augmented Generation System with Source Reliabilities Secured on Blockchain

TL;DR

Centralized retrieval augmented generation (RAG) poses data management and privacy challenges; this paper proposes dRAG, a decentralized RAG system that banks reliability scores on a blockchain to enable tamper-proof, auditable source evaluation. It combines a reliability scoring framework (using MC-Shapley or information-theoretic rationale) with on-chain governance via smart contracts to dynamically prioritize high-quality sources during retrieval. The authors demonstrate a +10.7% improvement over centralized systems in unreliable data environments and near-centralized performance in ideal conditions, plus approximately 56% cost savings from batched score updates. This approach enables secure, scalable, and transparent RAG across independently managed data sources, addressing trust and privacy concerns in regulated and multi-tenant settings.

Abstract

Existing retrieval-augmented generation (RAG) systems typically use a centralized architecture, causing a high cost of data collection, integration, and management, as well as privacy concerns. There is a great need for a decentralized RAG system that enables foundation models to utilize information directly from data owners who maintain full control over their sources. However, decentralization brings a challenge: the numerous independent data sources vary significantly in reliability, which can diminish retrieval accuracy and response quality. To address this, our decentralized RAG system has a novel reliability scoring mechanism that dynamically evaluates each source based on the quality of responses it contributes to generate and prioritizes high-quality sources during retrieval. To ensure transparency and trust, the scoring process is securely managed through blockchain-based smart contracts, creating verifiable and tamper-proof reliability records without relying on a central authority. We evaluate our decentralized system with two Llama models (3B and 8B) in two simulated environments where six data sources have different levels of reliability. Our system achieves a +10.7\% performance improvement over its centralized counterpart in the real world-like unreliable data environments. Notably, it approaches the upper-bound performance of centralized systems under ideally reliable data environments. The decentralized infrastructure enables secure and trustworthy scoring management, achieving approximately 56\% marginal cost savings through batched update operations. Our code and system are open-sourced at github.com/yining610/Reliable-dRAG.

Figures (6)

• Figure 1: Overview of a centralized RAG system (left) and our dRAG system (right), with their performance comparison (bottom) on Llama-3.2-3B-Instruct. In the unreliable data environments, our dRAG significantly outperforms the centralized system with increasing query exposure, approaching the performance upper-bound from the centralized system achieved in fully reliable data environments.
• Figure 2: System overview of dRAG.
• Figure 3: Detailed Workflow of dRAG, with a feedback log to update the reliability scores. For brevity, state information linking the update to a genuine query is omitted in the example. Ground truth is optional for sentence importance evaluations; if it is unavailable, user feedback (True/False) will be used to update the reliability scores.
• Figure 4: Statistics of two simulated unreliable data environments. Left: Token-level pollution shows the percentage of ground-truth tokens replaced with random tokens across six data sources (A-F), where different sources contain overlapping ground-truth documents. Right: Document-level pollution shows how 3197 ground-truth documents are divided into six sources (A-F) and mixed with polluted documents to create varying reliability levels, with no document overlap between sources.
• Figure 5: (a) Performance comparison between dRAG and centralized RAG systems on Llama-3.1-8B-Instruct across reliable and unreliable data environments; (b) Evolution of reliability score $R_i$ (\ref{['eq: reliability update']}) throughout the query process; (c) Percentage distribution of data sources (A-F) across sequential query bins of 100 queries each.