Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Factuality of Large Language Models in the Legal Domain

Rajaa El Hamdani, Thomas Bonald, Fragkiskos Malliaros, Nils Holzenberger, Fabian Suchanek

Abstract

This paper investigates the factuality of large language models (LLMs) as knowledge bases in the legal domain, in a realistic usage scenario: we allow for acceptable variations in the answer, and let the model abstain from answering when uncertain. First, we design a dataset of diverse factual questions about case law and legislation. We then use the dataset to evaluate several LLMs under different evaluation methods, including exact, alias, and fuzzy matching. Our results show that the performance improves significantly under the alias and fuzzy matching methods. Further, we explore the impact of abstaining and in-context examples, finding that both strategies enhance precision. Finally, we demonstrate that additional pre-training on legal documents, as seen with SaulLM, further improves factual precision from 63% to 81%.

The Factuality of Large Language Models in the Legal Domain

Abstract

This paper investigates the factuality of large language models (LLMs) as knowledge bases in the legal domain, in a realistic usage scenario: we allow for acceptable variations in the answer, and let the model abstain from answering when uncertain. First, we design a dataset of diverse factual questions about case law and legislation. We then use the dataset to evaluate several LLMs under different evaluation methods, including exact, alias, and fuzzy matching. Our results show that the performance improves significantly under the alias and fuzzy matching methods. Further, we explore the impact of abstaining and in-context examples, finding that both strategies enhance precision. Finally, we demonstrate that additional pre-training on legal documents, as seen with SaulLM, further improves factual precision from 63% to 81%.
Paper Structure (12 sections, 3 equations, 2 figures, 2 tables)

This paper contains 12 sections, 3 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Dataset
  4. Models
  5. Prompt strategy
  6. Evaluation methods
  7. Results
  8. Exact matching underestimates the performance of LLM-as-KB
  9. LLMs can be instructed to abstain from generating incorrect answers
  10. In-context examples increase the factuality of LLMs
  11. Training an LLM on legal documents improves its factuality
  12. Conclusions

Figures (2)

  • Figure 1: Example of prompt for the relation majority opinion by. The few shot examples are tailored for the relation majority opinion by and the subject's class legal case.
  • Figure 2: Comparison of precision (FM), with and without the abstain instruction, under the two prompting strategies.