Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Domain-Specific Data Generation Framework for RAG Adaptation

Chris Xing Tian, Weihao Xie, Zhen Chen, Zhengyuan Yi, Hui Liu, Haoliang Li, Shiqi Wang, Siwei Ma

TL;DR

The paper tackles domain-specific RAG adaptation by generating domain-grounded QAC triples. It introduces RAGen, a modular three-stage pipeline that discovers document-level concepts, builds multi-stem question stems, and produces QAC samples with varied difficulty guided by Bloom’s taxonomy, including concept-centered evidence and curated distractors. Empirical results across three domains show that RAGen data improve both retrieval (via embedding customization) and generation (via LLM fine-tuning), with further gains when distractor supervision is used, demonstrating strong cross-domain generalization and scalability to evolving corpora. Overall, RAGen offers a practical, end-to-end data-generation solution to strengthen domain-adapted RAG systems for enterprise knowledge bases and scientific domains.

Abstract

Retrieval-Augmented Generation (RAG) combines the language understanding and reasoning power of large language models (LLMs) with external retrieval to enable domain-grounded responses. Effectively adapting RAG systems to domain-specific settings requires specialized, context-rich training data beyond general-purpose question-answering. Here, we propose RAGen, a scalable and modular framework for generating domain-grounded question-answer-context (QAC) triples tailored to diverse RAG adaptation approaches. RAGen produces these QAC triples by identifying key concepts in documents, generating diverse questions guided by Bloom's Taxonomy-inspired principles, and pairing them with precise answers extracted from relevant contexts. RAGen supports multiple RAG adaptation strategies, including the optimization of key components such as the LLM, retriever, and embedding model, etc. Its modular pipeline features semantic chunking, hierarchical concept extraction, and multi-chunk retrieval, along with the introduction of curated distractor contexts to promote robust reasoning. Designed for scalability, RAGen efficiently handles large and evolving document corpora without redundant processing, making it especially suitable for dynamic evolving domains such as scientific research and enterprise knowledge bases.

Domain-Specific Data Generation Framework for RAG Adaptation

TL;DR

The paper tackles domain-specific RAG adaptation by generating domain-grounded QAC triples. It introduces RAGen, a modular three-stage pipeline that discovers document-level concepts, builds multi-stem question stems, and produces QAC samples with varied difficulty guided by Bloom’s taxonomy, including concept-centered evidence and curated distractors. Empirical results across three domains show that RAGen data improve both retrieval (via embedding customization) and generation (via LLM fine-tuning), with further gains when distractor supervision is used, demonstrating strong cross-domain generalization and scalability to evolving corpora. Overall, RAGen offers a practical, end-to-end data-generation solution to strengthen domain-adapted RAG systems for enterprise knowledge bases and scientific domains.

Abstract

Retrieval-Augmented Generation (RAG) combines the language understanding and reasoning power of large language models (LLMs) with external retrieval to enable domain-grounded responses. Effectively adapting RAG systems to domain-specific settings requires specialized, context-rich training data beyond general-purpose question-answering. Here, we propose RAGen, a scalable and modular framework for generating domain-grounded question-answer-context (QAC) triples tailored to diverse RAG adaptation approaches. RAGen produces these QAC triples by identifying key concepts in documents, generating diverse questions guided by Bloom's Taxonomy-inspired principles, and pairing them with precise answers extracted from relevant contexts. RAGen supports multiple RAG adaptation strategies, including the optimization of key components such as the LLM, retriever, and embedding model, etc. Its modular pipeline features semantic chunking, hierarchical concept extraction, and multi-chunk retrieval, along with the introduction of curated distractor contexts to promote robust reasoning. Designed for scalability, RAGen efficiently handles large and evolving document corpora without redundant processing, making it especially suitable for dynamic evolving domains such as scientific research and enterprise knowledge bases.

Paper Structure

This paper contains 29 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Question Generation
  4. Retrieval-Augmented Generation (RAG)
  5. Methodology
  6. Document Concepts Extraction
  7. Semantic chunking.
  8. Chunk–level concept extraction.
  9. Concept Fusion.
  10. Concept–centered Evidence Assembly
  11. Cross-chunk Retrieval.
  12. Evidence Extraction.
  13. QAC Generation
  14. Bloom's question-type.
  15. Question Generation.
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Overview of RAGen framework, a three-stage process that first extract document concepts and then construct question stems, and finally create Question-Answer-Context datasets.
  • Figure 2: Cognitive level distribution in PPFS domain.
  • Figure 3: Cross-concept question sample: By drawing on 3 non-adjacent evidence sources, cross-concept questions promote deeper, more holistic reasoning, moving beyond localized facts to capture broader operational and financial implications.