Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RAGSynth: Synthetic Data for Robust and Faithful RAG Component Optimization

Haiyang Shen, Hang Yan, Zhongshi Xing, Mugeng Liu, Yue Li, Zhiyang Chen, Yuxiang Wang, Jiuzheng Wang, Yun Ma

TL;DR

RagSynth tackles robustness and fidelity gaps in retrieval-augmented generation by generating synthetic data that encodes documents, queries, clues, and evidence mappings. It introduces a unified data model $\mathscr{G}$ and mappings $\mathscr{M}_1$, $\mathscr{M}_2$ to support single- and multi-hop reasoning, plus SynthBench with eight domain corpora and sentence-level citations. A new Criteria-based Score for Generation ($\text{CSG}$) complements traditional metrics to gauge generator faithfulness. Empirical results show consistent improvements in retriever robustness and generator fidelity across multiple RAG paradigms and strong cross-domain generalization, with the work released as open-source.

Abstract

RAG can enhance the performance of LLMs on knowledge-intensive tasks. Various RAG paradigms, including vanilla, planning-based, and iterative RAG, are built upon 2 cores: the retriever, which should robustly select relevant documents across complex queries, and the generator, which should faithfully synthesize responses. However, existing retrievers rely heavily on public knowledge and struggle with queries of varying logical complexity and clue completeness, while generators frequently face fidelity problems. In this work, we introduce RAGSynth, a framework that includes a data construction modeling and a corresponding synthetic data generation implementation, designed to optimize retriever robustness and generator fidelity. Additionally, we present SynthBench, a benchmark encompassing 8 domain-specific documents across 4 domains, featuring diverse query complexities, clue completeness, and fine-grained citation granularity. Leveraging RAGSynth, we generate a large-scale synthetic dataset, including single and multi-hop. Extensive experiments demonstrate that the synthetic data significantly improves the robustness of the retrievers and the fidelity of the generators. Additional evaluations confirm that RAGSynth can also generalize well across different domains. By integrating the optimized retrievers into various RAG paradigms, we consistently observe enhanced RAG system performance. We have open-sourced the implementation on https://github.com/EachSheep/RAGSynth.

RAGSynth: Synthetic Data for Robust and Faithful RAG Component Optimization

TL;DR

RagSynth tackles robustness and fidelity gaps in retrieval-augmented generation by generating synthetic data that encodes documents, queries, clues, and evidence mappings. It introduces a unified data model and mappings , to support single- and multi-hop reasoning, plus SynthBench with eight domain corpora and sentence-level citations. A new Criteria-based Score for Generation () complements traditional metrics to gauge generator faithfulness. Empirical results show consistent improvements in retriever robustness and generator fidelity across multiple RAG paradigms and strong cross-domain generalization, with the work released as open-source.

Abstract

RAG can enhance the performance of LLMs on knowledge-intensive tasks. Various RAG paradigms, including vanilla, planning-based, and iterative RAG, are built upon 2 cores: the retriever, which should robustly select relevant documents across complex queries, and the generator, which should faithfully synthesize responses. However, existing retrievers rely heavily on public knowledge and struggle with queries of varying logical complexity and clue completeness, while generators frequently face fidelity problems. In this work, we introduce RAGSynth, a framework that includes a data construction modeling and a corresponding synthetic data generation implementation, designed to optimize retriever robustness and generator fidelity. Additionally, we present SynthBench, a benchmark encompassing 8 domain-specific documents across 4 domains, featuring diverse query complexities, clue completeness, and fine-grained citation granularity. Leveraging RAGSynth, we generate a large-scale synthetic dataset, including single and multi-hop. Extensive experiments demonstrate that the synthetic data significantly improves the robustness of the retrievers and the fidelity of the generators. Additional evaluations confirm that RAGSynth can also generalize well across different domains. By integrating the optimized retrievers into various RAG paradigms, we consistently observe enhanced RAG system performance. We have open-sourced the implementation on https://github.com/EachSheep/RAGSynth.
Paper Structure (18 sections, 2 equations, 1 figure, 8 tables)

This paper contains 18 sections, 2 equations, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Retrieval-Augmented Generation
  4. Data Synthesis Optimization Methods
  5. RAG Benchmarks
  6. RagSynth: Modeling and Implementation
  7. Modeling
  8. Implementation
  9. Document Corpus and SynthBench
  10. Domain-specific Corpus
  11. SynthBench Construction
  12. Task Metrics
  13. Evaluation
  14. Setup
  15. Main Experiments
  16. ...and 3 more sections

Figures (1)

  • Figure 1: A specific implementation of the RagSynth. For single-hop, mappings among documents, clues, queries, and answers can be directly constructed. For multi-hop, entities and relationships are first extracted from documents, with relationships serving as clues. Subsequently, using these clues as intermediaries, mappings among documents, clues, queries, and answers are established. After constructing the basic dataset, we further generate a large number of variants of the basic queries and their corresponding answers through extensive logical and completeness transformations.