Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models

TL;DR

The paper systematically quantifies legal hallucinations in large language models by testing four mainstream LLMs on a large, jurisdiction-spanning set of legal knowledge queries. It introduces a formal typology of hallucinations, contrasts reference-based ground-truth with reference-free detection, and reveals widespread, task-dependent hallucinations across the U.S. federal judiciary hierarchy. It also examines contra-factual bias and calibration, highlighting that even state-of-the-art models can produce overconfident, incorrect legal information and that some models exhibit fewer contra-factual errors but at the cost of missing valid information. The findings urge cautious, human-centered deployment of LLMs in legal settings and point to mitigation strategies like retrieval augmentation, advanced prompting, and explicit uncertainty signaling to reduce risk and inequality in access to justice.

Abstract

Do large language models (LLMs) know the law? These models are increasingly being used to augment legal practice, education, and research, yet their revolutionary potential is threatened by the presence of hallucinations -- textual output that is not consistent with legal facts. We present the first systematic evidence of these hallucinations, documenting LLMs' varying performance across jurisdictions, courts, time periods, and cases. Our work makes four key contributions. First, we develop a typology of legal hallucinations, providing a conceptual framework for future research in this area. Second, we find that legal hallucinations are alarmingly prevalent, occurring between 58% of the time with ChatGPT 4 and 88% with Llama 2, when these models are asked specific, verifiable questions about random federal court cases. Third, we illustrate that LLMs often fail to correct a user's incorrect legal assumptions in a contra-factual question setup. Fourth, we provide evidence that LLMs cannot always predict, or do not always know, when they are producing legal hallucinations. Taken together, our findings caution against the rapid and unsupervised integration of popular LLMs into legal tasks. Even experienced lawyers must remain wary of legal hallucinations, and the risks are highest for those who stand to benefit from LLMs the most -- pro se litigants or those without access to traditional legal resources.

Figures (31)

• Figure 1: Hallucination rates by LLM, all reference-based tasks pooled. Hallucinations are common across all LLMs when they are asked a direct, verifiable question about a federal court case, but GPT 4 performs best overall.
• Figure 1: Rescaled calibration curves by LLM, all resource-aware tasks pooled.
• Figure 2: Relationship between task complexity and mean hallucination rate. Higher values indicate a greater likelihood of factually incorrect LLM responses. High complexity tasks include several reference-free tasks, so those reported hallucination rates are lower bounds on the true rates. Contra-factual tasks and the doctrinal agreement high complexity task are excluded from this comparison.
• Figure 2: Relationship between USCOA jurisdiction and mean hallucination rate, all resource-aware USCOA tasks and models pooled. (No time cutoff.)
• Figure 3: Relationship between judicial hierarchy and mean hallucination rate, all reference-based tasks pooled. Hallucination rates are higher for lower levels of the federal judiciary.