Legal Retrieval for Public Defenders

TL;DR

This work addresses the need for trustworthy AI support in public defense by introducing the NJ BriefBank, a retrieval system that surfaces relevant appellate briefs to aid research and drafting. It pairs the BriefBank with a manually annotated Public Defense Dataset (PD) and a taxonomy of defender queries to study realistic information needs, revealing a strong domain shift between existing legal benchmarks and real-world defense work. The authors show that incorporating domain knowledge—through optimized synthetic data, IRAC-based query expansion, and legal-domain pretraining—substantially improves retrieval performance, while fine-tuning on standard legal benchmarks often hurts public-defense results. The work also demonstrates the value of academia–legal-institution collaboration, providing release-ready artifacts and guidance for building practical, reliable retrieval tools for public defenders and for advancing more realistic benchmarks in legal information retrieval.

Abstract

AI tools are increasingly suggested as solutions to assist public agencies with heavy workloads. In public defense, where a constitutional right to counsel meets the complexities of law, overwhelming caseloads and constrained resources, practitioners face especially taxing conditions. Yet, there is little evidence of how AI could meaningfully support defenders' day-to-day work. In partnership with the New Jersey Office of the Public Defender, we develop the NJ BriefBank, a retrieval tool which surfaces relevant appellate briefs to streamline legal research and writing. We show that existing legal retrieval benchmarks fail to transfer to public defense search, however adding domain knowledge improves retrieval quality. This includes query expansion with legal reasoning, domain-specific data and curated synthetic examples. To facilitate further research, we provide a taxonomy of realistic defender search queries and release a manually annotated public defense retrieval dataset. Together, our work offers starting points towards building practical, reliable retrieval AI tools for public defense, and towards more realistic legal retrieval benchmarks.

Figures (7)

• Figure 1: Visualizations of defender queries (pink dots) and other queries in two legal search datasets: LePaRD mahari-etal-2024-lepard and BarExam-QA barexam_qa. We compute low-dimensional projections with t-SNE vandermaaten08a and PCA sklearn. We observe that queries from different datasets are separable in the embedding spaces.
• Figure 2: NJ BriefBank's performance on public defender queries categorized by search intent. Most queries fall into standard/ rule/ doctrine, broad topical search, and legal argument.
• Figure 3: NJ BriefBank's performance on public defender queries categorized by required search strategy. Most queries only require embedding-based retrieval. Queries that require keyword-based retrieval is the most challenging for the current system.
• Figure 4: Average gains or loss in recall @ 5 for four retrieval models. Colored bars indicate model performance after fine-tuning on different retrieval datasets. On the x-axis, we plot resulting model performance on four retrieval datasets (BarExam QA, LePaRD, NJ-OPD and PD). Changes in recall are relative to a zero-shot baseline of the same model. If fine-tuned on BarExam QA, LePaRD or naive synthetic data, performance on public defense datasets decreases. if fine-tuned on carefully tuned synthetic data, performance on public defense datasets increases. Full results are shown in Appendix Table \ref{['tab:recalls_training_updated']}.
• Figure 5: F1 scores of different reranker models.$\Delta$ is the difference from the majority-baseline F1 of 80.3 (most annotated paragraphs are annotated as relevant, so a majority baseline achieves a high F1 score). Most off-the-shelf rerankers perform worse than the majority baseline on detecting good vs. bad search results.