Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Attribution analysis of legal language as used by LLM

Richard K. Belew

TL;DR

This work investigates how legal-language exposure and tokenization influence LLM performance on overruling and holding tasks by comparing four public legal LLMs and a generic BERT baseline. Using integrated gradients, it attributes model decisions to input tokens and analyzes how vocabulary and tokenization shape results. The findings show high accuracy on the overruling task across models, while the casehold task remains more challenging and is heavily influenced by the models' legal-language exposure and tokenization, suggesting a path toward ensemble approaches and the development of legal-aware tokenizers. Overall, attribution analyses provide actionable insights for improving reliability and interpretability of legal AI systems in real-world applications.

Abstract

Three publicly-available LLM specifically designed for legal tasks have been implemented and shown that classification accuracy can benefit from training over legal corpora, but why and how? Here we use two publicly-available legal datasets, a simpler binary classification task of ``overruling'' texts, and a more elaborate multiple choice task identifying ``holding'' judicial decisions. We report on experiments contrasting the legal LLM and a generic BERT model for comparison, against both datasets. We use integrated gradient attribution techniques to impute ``causes'' of variation in the models' perfomance, and characterize them in terms of the tokenizations each use. We find that while all models can correctly classify some test examples from the casehold task, other examples can only be identified by only one, model, and attribution can be used to highlight the reasons for this. We find that differential behavior of the models' tokenizers accounts for most of the difference and analyze these differences in terms of the legal language they process. Frequency analysis of tokens generated by dataset texts, combined with use of known ``stop word'' lists, allow identification of tokens that are clear signifiers of legal topics.

Attribution analysis of legal language as used by LLM

TL;DR

This work investigates how legal-language exposure and tokenization influence LLM performance on overruling and holding tasks by comparing four public legal LLMs and a generic BERT baseline. Using integrated gradients, it attributes model decisions to input tokens and analyzes how vocabulary and tokenization shape results. The findings show high accuracy on the overruling task across models, while the casehold task remains more challenging and is heavily influenced by the models' legal-language exposure and tokenization, suggesting a path toward ensemble approaches and the development of legal-aware tokenizers. Overall, attribution analyses provide actionable insights for improving reliability and interpretability of legal AI systems in real-world applications.

Abstract

Three publicly-available LLM specifically designed for legal tasks have been implemented and shown that classification accuracy can benefit from training over legal corpora, but why and how? Here we use two publicly-available legal datasets, a simpler binary classification task of ``overruling'' texts, and a more elaborate multiple choice task identifying ``holding'' judicial decisions. We report on experiments contrasting the legal LLM and a generic BERT model for comparison, against both datasets. We use integrated gradient attribution techniques to impute ``causes'' of variation in the models' perfomance, and characterize them in terms of the tokenizations each use. We find that while all models can correctly classify some test examples from the casehold task, other examples can only be identified by only one, model, and attribution can be used to highlight the reasons for this. We find that differential behavior of the models' tokenizers accounts for most of the difference and analyze these differences in terms of the legal language they process. Frequency analysis of tokens generated by dataset texts, combined with use of known ``stop word'' lists, allow identification of tokens that are clear signifiers of legal topics.

Paper Structure

This paper contains 18 sections, 18 figures.

Table of Contents

  1. Introduction
  2. Methods
  3. LLM models
  4. Datasets
  5. Overrule
  6. Casehold
  7. Attribution across LLM components
  8. Results
  9. Classification performance
  10. Variations in attribution
  11. Meanings in the vocabularies
  12. Variation across LLM vocabularies
  13. Connecting LLM with known legal language
  14. Token frequencies across examples
  15. Discussion
  16. ...and 3 more sections

Figures (18)

  • Figure 1: Positive and negative overruling examples (Table 3 from zhengguha2021)
  • Figure 2: casehold example (Table 1 from zhengguha2021)
  • Figure 3: Sample attributions (overrule, legalBERT)
  • Figure 4: Model performance
  • Figure 5: Prediction, attribution distribution
  • ...and 13 more figures