Bye-bye, Bluebook? Automating Legal Procedure with Large Language Models

TL;DR

This study evaluates whether flagship LLMs can faithfully follow The Bluebook's intricate citation rules, framing citation formatting as a procedural task in legal practice. It introduces a large, expert-annotated dataset of 866 Bluebook tasks (Case law, Enacted law, and Other sources) and tests five major LLMs in zero-shot settings, plus in-context learning on Gemini with a 90k-token Indigo Book rule set. Results show zero-shot accuracies of roughly 69–74%, with in-context learning lifting accuracy only modestly to about 77% for the strongest model, suggesting substantial limitations for automatic procedural compliance. The findings argue for caution in deploying LLMs for routine procedural tasks in law and highlight the need for further research into reliable, rule-based AI systems for legal practice.

Abstract

Legal practice requires careful adherence to procedural rules. In the United States, few are more complex than those found in The Bluebook: A Uniform System of Citation. Compliance with this system's 500+ pages of byzantine formatting instructions is the raison d'etre of thousands of student law review editors and the bete noire of lawyers everywhere. To evaluate whether large language models (LLMs) are able to adhere to the procedures of such a complicated system, we construct an original dataset of 866 Bluebook tasks and test flagship LLMs from OpenAI, Anthropic, Google, Meta, and DeepSeek. We show (1) that these models produce fully compliant Bluebook citations only 69%-74% of the time and (2) that in-context learning on the Bluebook's underlying system of rules raises accuracy only to 77%. These results caution against using off-the-shelf LLMs to automate aspects of the law where fidelity to procedure is paramount.

Figures (12)

• Figure 1: Flagship LLMs achieve between 69% and 74% accuracy on Bluebook tasks in a zero-shot setting, with GPT 4.1 narrowly performing the best. Black bars represent 95% confidence intervals ($n=866$).
• Figure 2: Structure of a Bluebook-compliant citation to case law. The signal indicates the weight that the citation is intended to convey. The case name reflects the names of the parties, which must be consolidated and abbreviated if there are many. The reporter, volume, and page give the physical location where the case is printed. The pincite page provides the specific page to which the citation points. The court and year give the jurisdiction and date of the issuing court. The procedural history notes any subsequent history of the case, such as if it has been vacated by an appellate court. The second reporter, volume, and page give the physical location where the appellate court's opinion is printed. The second year gives the date of that appellate opinion; no court is specified in this particular citation because the reporter for the appellate opinion (U.S.) unambiguously indicates that the appellate court is the Supreme Court.
• Figure 3: Example cloze query from the case reporters task. The model is provided with the caption to a case in natural language and is asked to complete the Bluebook citation. Other tasks mimic this structure, masking out different parts of the citation as appropriate (for cloze tasks) or asking the model to generate the entire citation from scratch (for open tasks). Responses are assessed against ground-truth answers provided by Bluebook experts.
• Figure 4: Zero-shot results by task, pooled across all five models. Flagship LLMs vary greatly in their ability to handle different Bluebook assignments. The models perform the best---but not perfectly---on case law tasks (blue); they perform much worse on enacted law (yellow) and other tasks (green). Black bars represent 95% confidence intervals.
• Figure 5: Memorization analysis. Using an open format for a combined case reporter, court, and date task, figure compares queries about real cases (dark colors) with queries about the same cases, but with a synthetic page number (light colors). All models except Gemini and DeepSeek show evidence of naïve memorization of specific cases rather than the actual rules of the Bluebook. Black bars show 95% confidence intervals ($n=181$).