A Reasoning-Focused Legal Retrieval Benchmark

TL;DR

This paper introduces two reasoning-focused open-domain legal retrieval benchmarks, Bar Exam QA and Housing Statute QA, each designed to stress retrieval and downstream QA in realistic legal scenarios. The datasets provide around 10K labeled query–gold passage–answer triples with hand-annotated passages and large legal-text corpora, enabling end-to-end evaluation beyond retrieval alone. Through extensive experiments with lexical retrievers (BM25) and dense models (E5 variants), the authors show that low lexical overlap and multi-hop reasoning challenge standard retrieval pipelines, while query expansion—especially structured legal reasoning—yields statistically significant recall gains and improves downstream QA under certain conditions. The work highlights the need for retrievers that can also perform legal reasoning and offers benchmarks and baselines to guide future development of retrieval-augmented LLMs in law, with implications for practical deployments of legal AI tools.

Abstract

As the legal community increasingly examines the use of large language models (LLMs) for various legal applications, legal AI developers have turned to retrieval-augmented LLMs ("RAG" systems) to improve system performance and robustness. An obstacle to the development of specialized RAG systems is the lack of realistic legal RAG benchmarks which capture the complexity of both legal retrieval and downstream legal question-answering. To address this, we introduce two novel legal RAG benchmarks: Bar Exam QA and Housing Statute QA. Our tasks correspond to real-world legal research tasks, and were produced through annotation processes which resemble legal research. We describe the construction of these benchmarks and the performance of existing retriever pipelines. Our results suggest that legal RAG remains a challenging application, thus motivating future research.

Figures (4)

• Figure 1: Histogram of number of gold passages (statutes) per example in the Housing Statue QA dataset.
• Figure 2: Histograms of the example lexical similarity of (query, gold passage) and (gold passage, answer) over the following datasets: Bar Exam QA and Housing Statute QA (row 1), NQ and HotpotQA (row 2), COLIEE and CLERC (row 3). In Appendix \ref{['sec:lex_sim_stat_test_results']}, we report results for the Kolmogorov-Smirnov test for distributional equivalence between the task similarity distributions, which provide evidence of statistical significant differences between our datasets and other representative datasets at $\alpha = 0.05$.
• Figure 3: Recall of baseline and structured reasoning rollout query expansion retrieval for lexical (BM25) and dense models (E5 family), evaluated on our legal retrieval benchmark tasks. The gain in Recall@10 with the structured reasoning rollout method is labeled above each bar. 95% confidence intervals are estimated with a percentile bootstrap ($n = 1000$). For Housing Statute QA, recall is reported as retrieval of at least one gold passage (upper bound).
• Figure 4: Baseline retrieval performance vs. lexical similarity (query, gold passage) line of best fit over Housing Statute QA task. Recall@10 is averaged over examples bucketed by intervals of lexical similarity scores (bucket sizes of 0.1). 95% confidence intervals are estimated over each bucket. $m$ is the slope of the line of best fit.