Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Small Models Reason About Legal Documents? A Comparative Study

Snehit Vaddi

Abstract

Large language models show promise for legal applications, but deploying frontier models raises concerns about cost, latency, and data privacy. We evaluate whether sub-10B parameter models can serve as practical alternatives by testing nine models across three legal benchmarks (ContractNLI, CaseHOLD, and ECtHR) using five prompting strategies (direct, chain-of-thought, few-shot, BM25 RAG, and dense RAG). Across 405 experiments with three random seeds per configuration, we find that a Mixture-of-Experts model activating only 3B parameters matches GPT-4o-mini in mean accuracy while surpassing it on legal holding identification, and that architecture and training quality matter more than raw parameter count. Our largest model (9B parameters) performs worst overall. Chain-of-thought prompting proves sharply task-dependent, improving contract entailment but degrading multiple-choice legal reasoning, while few-shot prompting emerges as the most consistently effective strategy. Comparing BM25 and dense retrieval for RAG, we find near-identical results, suggesting the bottleneck lies in the language model's utilization of retrieved context rather than retrieval quality. All experiments were conducted via cloud inference APIs at a total cost of $62, demonstrating that rigorous LLM evaluation is accessible without dedicated GPU infrastructure.

Can Small Models Reason About Legal Documents? A Comparative Study

Abstract

Large language models show promise for legal applications, but deploying frontier models raises concerns about cost, latency, and data privacy. We evaluate whether sub-10B parameter models can serve as practical alternatives by testing nine models across three legal benchmarks (ContractNLI, CaseHOLD, and ECtHR) using five prompting strategies (direct, chain-of-thought, few-shot, BM25 RAG, and dense RAG). Across 405 experiments with three random seeds per configuration, we find that a Mixture-of-Experts model activating only 3B parameters matches GPT-4o-mini in mean accuracy while surpassing it on legal holding identification, and that architecture and training quality matter more than raw parameter count. Our largest model (9B parameters) performs worst overall. Chain-of-thought prompting proves sharply task-dependent, improving contract entailment but degrading multiple-choice legal reasoning, while few-shot prompting emerges as the most consistently effective strategy. Comparing BM25 and dense retrieval for RAG, we find near-identical results, suggesting the bottleneck lies in the language model's utilization of retrieved context rather than retrieval quality. All experiments were conducted via cloud inference APIs at a total cost of $62, demonstrating that rigorous LLM evaluation is accessible without dedicated GPU infrastructure.

Paper Structure

This paper contains 54 sections, 1 figure, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Legal NLP benchmarks.
  4. Small language models.
  5. Prompting strategies.
  6. Methodology
  7. Models
  8. Datasets
  9. ContractNLI.
  10. CaseHOLD.
  11. ECtHR Task A.
  12. Sampling.
  13. Prompting Strategies
  14. Direct (zero-shot).
  15. Chain-of-Thought (CoT).
  16. ...and 39 more sections

Figures (1)

  • Figure 1: Overview of the five prompting strategies. Each row shows the prompt components assembled for a given strategy. All strategies share the core components (legal text, question, format instruction) but differ in the additional context provided. Output token budgets vary by strategy: 128 tokens (direct, few-shot), 256 tokens (RAG), and 512 tokens (CoT).