Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Legal Syllogism Prompting: Teaching Large Language Models for Legal Judgment Prediction

Cong Jiang, Xiaolei Yang

TL;DR

This work introduces Legal Syllogism Prompting (LoT), a zero-shot prompting method that directs large language models to perform legal judgment prediction via a structured three-premise syllogism: major premise (law), minor premise (facts), and conclusion (judgment). By prompting the model to produce Major premise, Minor premise, and Conclusion, LoT concentrates reasoning on legally relevant information and enhances explainability, without any learning or fine-tuning. Evaluated on the CAIL2018 dataset with GPT-3, LoT outperforms baseline and zero-shot chain-of-thought prompting, delivering judgments along with supporting law articles. The approach holds promise for explainable AI in law, while acknowledging limitations in practical reasoning and the need for careful, ethically aware deployment in real-world legal settings.

Abstract

Legal syllogism is a form of deductive reasoning commonly used by legal professionals to analyze cases. In this paper, we propose legal syllogism prompting (LoT), a simple prompting method to teach large language models (LLMs) for legal judgment prediction. LoT teaches only that in the legal syllogism the major premise is law, the minor premise is the fact, and the conclusion is judgment. Then the models can produce a syllogism reasoning of the case and give the judgment without any learning, fine-tuning, or examples. On CAIL2018, a Chinese criminal case dataset, we performed zero-shot judgment prediction experiments with GPT-3 models. Our results show that LLMs with LoT achieve better performance than the baseline and chain of thought prompting, the state-of-art prompting method on diverse reasoning tasks. LoT enables the model to concentrate on the key information relevant to the judgment and to correctly understand the legal meaning of acts, as compared to other methods. Our method enables LLMs to predict judgment along with law articles and justification, which significantly enhances the explainability of models.

Legal Syllogism Prompting: Teaching Large Language Models for Legal Judgment Prediction

TL;DR

This work introduces Legal Syllogism Prompting (LoT), a zero-shot prompting method that directs large language models to perform legal judgment prediction via a structured three-premise syllogism: major premise (law), minor premise (facts), and conclusion (judgment). By prompting the model to produce Major premise, Minor premise, and Conclusion, LoT concentrates reasoning on legally relevant information and enhances explainability, without any learning or fine-tuning. Evaluated on the CAIL2018 dataset with GPT-3, LoT outperforms baseline and zero-shot chain-of-thought prompting, delivering judgments along with supporting law articles. The approach holds promise for explainable AI in law, while acknowledging limitations in practical reasoning and the need for careful, ethically aware deployment in real-world legal settings.

Abstract

Legal syllogism is a form of deductive reasoning commonly used by legal professionals to analyze cases. In this paper, we propose legal syllogism prompting (LoT), a simple prompting method to teach large language models (LLMs) for legal judgment prediction. LoT teaches only that in the legal syllogism the major premise is law, the minor premise is the fact, and the conclusion is judgment. Then the models can produce a syllogism reasoning of the case and give the judgment without any learning, fine-tuning, or examples. On CAIL2018, a Chinese criminal case dataset, we performed zero-shot judgment prediction experiments with GPT-3 models. Our results show that LLMs with LoT achieve better performance than the baseline and chain of thought prompting, the state-of-art prompting method on diverse reasoning tasks. LoT enables the model to concentrate on the key information relevant to the judgment and to correctly understand the legal meaning of acts, as compared to other methods. Our method enables LLMs to predict judgment along with law articles and justification, which significantly enhances the explainability of models.
Paper Structure (17 sections, 2 figures, 1 table)

This paper contains 17 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. RELATED WORK
  3. Legal judgment prediction
  4. Large language models and chain of thought
  5. legal syllogism prompting
  6. Legal syllogism
  7. Legal syllogism prompting
  8. EXPERIMENTS
  9. Dataset
  10. Experimental setup
  11. Baselines
  12. Metrics
  13. Results
  14. Error analysis
  15. DISCUSSION
  16. ...and 2 more sections

Figures (2)

  • Figure 1: An overview of three different approaches for LJP: (1) Deep learning text classification models for providing the judgment without any explanations. (2) Chain-of-thought prompting for providing the judgment with intermediate reasoning steps as explanations. (3) Legal syllogism prompting for providing the three deductive reasoning steps: law, fact and judgment.
  • Figure 2: A robbery case as an example to show the difference between three prompting methods to the GPT-3 model: (a) Baseline; (b)Zero-shot CoT; (c) Our method. Our method output the right judgment using legal syllogism. Despite the steps given, Zero-shot CoT misunderstood the legal meaning of A’s two acts and reached the wrong judgment.