LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversation

TL;DR

LexRAG introduces a dedicated benchmark for evaluating retrieval-augmented generation in multi-turn legal consultations, addressing a gap in domain-specific evaluation. It provides 1,013 dialogues and 17,228 legal articles, defining two tasks—conversational knowledge retrieval and response generation—alongside the LexiT toolkit and an LLM-as-a-judge evaluation pipeline to enable reproducible assessment. Empirical results reveal that dense retrieval and prompt rewriting improve retrieval performance, yet current LLMs still struggle with the nuanced legal reasoning required, highlighting the gap between retrieval quality and generation accuracy. The open-source toolkit and dataset aim to accelerate development of legal-domain RAG systems, with future work extending language coverage and cross-domain evaluation.

Abstract

Retrieval-augmented generation (RAG) has proven highly effective in improving large language models (LLMs) across various domains. However, there is no benchmark specifically designed to assess the effectiveness of RAG in the legal domain, which restricts progress in this area. To fill this gap, we propose LexRAG, the first benchmark to evaluate RAG systems for multi-turn legal consultations. LexRAG consists of 1,013 multi-turn dialogue samples and 17,228 candidate legal articles. Each sample is annotated by legal experts and consists of five rounds of progressive questioning. LexRAG includes two key tasks: (1) Conversational knowledge retrieval, requiring accurate retrieval of relevant legal articles based on multi-turn context. (2) Response generation, focusing on producing legally sound answers. To ensure reliable reproducibility, we develop LexiT, a legal RAG toolkit that provides a comprehensive implementation of RAG system components tailored for the legal domain. Additionally, we introduce an LLM-as-a-judge evaluation pipeline to enable detailed and effective assessment. Through experimental analysis of various LLMs and retrieval methods, we reveal the key limitations of existing RAG systems in handling legal consultation conversations. LexRAG establishes a new benchmark for the practical application of RAG systems in the legal domain, with its code and data available at https://github.com/CSHaitao/LexRAG.

Figures (3)

• Figure 1: An example of a legal consultation in LexRAG.
• Figure 2: The distribution of query types.
• Figure 3: Overview of LexiT Components.