R-Eval: A Unified Toolkit for Evaluating Domain Knowledge of Retrieval Augmented Large Language Models

TL;DR

R-Eval tackles the gap in evaluating domain knowledge within Retrieval-Augmented LLMs by offering a modular Python toolkit that unifies RAG workflow evaluation across two domains (Wikipedia and Aminer) and supports template-based data generation. It evaluates 21 feasible RALLM configurations built from four built-in workflows (ReAct, PAL, DFSDT, FC) across three task levels (KS, KU, KA) and finds substantial performance variation by task and domain, with ReAct+GPT-4-1106 often the strongest overall pairing. The analysis introduces a fine-grained error taxonomy and deployment metrics to diagnose failures and guide practical deployment decisions, including insights into how tool-using errors and retrieval quality impact results. The study demonstrates that task- and domain-specific considerations are crucial when selecting RAG workflows and LLMs, and it provides a public toolkit to enable ongoing, fair comparisons for researchers and industry practitioners. Overall, R-Eval offers a scalable framework for rigorous, domain-aware evaluation of RALLMs and aims to accelerate their reliable application in specialized fields.

Abstract

Large language models have achieved remarkable success on general NLP tasks, but they may fall short for domain-specific problems. Recently, various Retrieval-Augmented Large Language Models (RALLMs) are proposed to address this shortcoming. However, existing evaluation tools only provide a few baselines and evaluate them on various domains without mining the depth of domain knowledge. In this paper, we address the challenges of evaluating RALLMs by introducing the R-Eval toolkit, a Python toolkit designed to streamline the evaluation of different RAG workflows in conjunction with LLMs. Our toolkit, which supports popular built-in RAG workflows and allows for the incorporation of customized testing data on the specific domain, is designed to be user-friendly, modular, and extensible. We conduct an evaluation of 21 RALLMs across three task levels and two representative domains, revealing significant variations in the effectiveness of RALLMs across different tasks and domains. Our analysis emphasizes the importance of considering both task and domain requirements when choosing a RAG workflow and LLM combination. We are committed to continuously maintaining our platform at https://github.com/THU-KEG/R-Eval to facilitate both the industry and the researchers.

Figures (9)

• Figure 1: Responses from $4$ RAG workflows, including DFSDT(DT) toolllm, ReAct(RA) yao2022react, PAL gao2023pal and GPT Function Calling (FC) for a domain-specific question.
• Figure 2: The framework of our evaluating and analysing process. We first choose an environment and collect testing data from both existing benchmarks and template-based QA pairs. Then we select the RAG workflows and LLMs to form a RALLM for running the evaluation. After that, we perform a comprehensive analysis on results to get insights.
• Figure 3: Average performance on different levels' tasks and domains for GPT-4 with the PAL and DFSDT workflow.
• Figure 4: Radar map of single system's performance on all tasks for different $4$ workflows.
• Figure 5: Error distribution of GPT-4-preview-1104 with different $4$ workflows on all tasks.