DomainRAG: A Chinese Benchmark for Evaluating Domain-specific Retrieval-Augmented Generation

TL;DR

This work introduces DomainRAG, a domain-specific benchmark for evaluating retrieval-augmented generation in the in-domain context of college enrollment in China. It builds HTML and text corpora from official enrollment pages and constructs seven sub-datasets to probe conversational RAG, structural analysis, faithfulness, denoising, time-sensitivity, and multi-document interactions, using seven prominent LLMs under varied retrieval settings. Key findings reveal that closed-book LLMs struggle with domain questions, external knowledge via RAG is essential, and BM25 often generalizes better than dense retrievers, though long-form and multi-document scenarios remain challenging; structural information and data quality significantly influence performance. The dataset and findings aim to guide future RAG research toward improved conversational history understanding, HTML-structural reasoning, robust denoising, reliable multi-document integration, and stronger fidelity to external, domain-specific knowledge, with practical implications for enterprise and expert-domain deployments.

Abstract

Retrieval-Augmented Generation (RAG) offers a promising solution to address various limitations of Large Language Models (LLMs), such as hallucination and difficulties in keeping up with real-time updates. This approach is particularly critical in expert and domain-specific applications where LLMs struggle to cover expert knowledge. Therefore, evaluating RAG models in such scenarios is crucial, yet current studies often rely on general knowledge sources like Wikipedia to assess the models' abilities in solving common-sense problems. In this paper, we evaluated LLMs by RAG settings in a domain-specific context, college enrollment. We identified six required abilities for RAG models, including the ability in conversational RAG, analyzing structural information, faithfulness to external knowledge, denoising, solving time-sensitive problems, and understanding multi-document interactions. Each ability has an associated dataset with shared corpora to evaluate the RAG models' performance. We evaluated popular LLMs such as Llama, Baichuan, ChatGLM, and GPT models. Experimental results indicate that existing closed-book LLMs struggle with domain-specific questions, highlighting the need for RAG models to solve expert problems. Moreover, there is room for RAG models to improve their abilities in comprehending conversational history, analyzing structural information, denoising, processing multi-document interactions, and faithfulness in expert knowledge. We expect future studies could solve these problems better.

Figures (4)

• Figure 1: Important abilities for RAG models.
• Figure 2: The experiments on the structural QA dataset.
• Figure 3: The experiments on the faithful QA dataset.
• Figure 4: Experiments in different noise ratio settings.