Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges

Junyun Cui, Xiaoyu Shen, Feiping Nie, Zheng Wang, Jinglong Wang, Yulong Chen

TL;DR

This survey comprehensively catalogs LJP datasets, metrics, models, and challenges across languages and jurisdictions. It analyzes 31 datasets in 6 languages, introduces a three-attribute taxonomy for LJP tasks, and reviews 12 legal-domain pretrained models. It reports state-of-the-art results across representative court datasets and discusses open challenges such as information adequacy, complex legal reasoning, and interpretability. The paper also offers concrete recommendations for dataset construction and model development to advance practical LJP systems.

Abstract

Legal judgment prediction (LJP) applies Natural Language Processing (NLP) techniques to predict judgment results based on fact descriptions automatically. Recently, large-scale public datasets and advances in NLP research have led to increasing interest in LJP. Despite a clear gap between machine and human performance, impressive results have been achieved in various benchmark datasets. In this paper, to address the current lack of comprehensive survey of existing LJP tasks, datasets, models and evaluations, (1) we analyze 31 LJP datasets in 6 languages, present their construction process and define a classification method of LJP with 3 different attributes; (2) we summarize 14 evaluation metrics under four categories for different outputs of LJP tasks; (3) we review 12 legal-domain pretrained models in 3 languages and highlight 3 major research directions for LJP; (4) we show the state-of-art results for 8 representative datasets from different court cases and discuss the open challenges. This paper can provide up-to-date and comprehensive reviews to help readers understand the status of LJP. We hope to facilitate both NLP researchers and legal professionals for further joint efforts in this problem.

A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges

TL;DR

This survey comprehensively catalogs LJP datasets, metrics, models, and challenges across languages and jurisdictions. It analyzes 31 datasets in 6 languages, introduces a three-attribute taxonomy for LJP tasks, and reviews 12 legal-domain pretrained models. It reports state-of-the-art results across representative court datasets and discusses open challenges such as information adequacy, complex legal reasoning, and interpretability. The paper also offers concrete recommendations for dataset construction and model development to advance practical LJP systems.

Abstract

Legal judgment prediction (LJP) applies Natural Language Processing (NLP) techniques to predict judgment results based on fact descriptions automatically. Recently, large-scale public datasets and advances in NLP research have led to increasing interest in LJP. Despite a clear gap between machine and human performance, impressive results have been achieved in various benchmark datasets. In this paper, to address the current lack of comprehensive survey of existing LJP tasks, datasets, models and evaluations, (1) we analyze 31 LJP datasets in 6 languages, present their construction process and define a classification method of LJP with 3 different attributes; (2) we summarize 14 evaluation metrics under four categories for different outputs of LJP tasks; (3) we review 12 legal-domain pretrained models in 3 languages and highlight 3 major research directions for LJP; (4) we show the state-of-art results for 8 representative datasets from different court cases and discuss the open challenges. This paper can provide up-to-date and comprehensive reviews to help readers understand the status of LJP. We hope to facilitate both NLP researchers and legal professionals for further joint efforts in this problem.
Paper Structure (33 sections, 1 equation, 6 figures, 12 tables)

This paper contains 33 sections, 1 equation, 6 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Tasks Formulations
  3. Background
  4. Taxonomy
  5. Types of LJP Tasks
  6. Datasets
  7. Single-Task Datasets
  8. Article Prediction Datasets
  9. Charge Prediction Datasets
  10. Prison Term Prediction Datasets
  11. Plea Judgment prediction Datasets
  12. Court view Generation Datasets
  13. Multi-Task Datasets
  14. Datasets for Pre-trained Language Model
  15. Evaluation Metrics
  16. ...and 18 more sections

Figures (6)

  • Figure 1: The number of legal-domain papers in major NLP conferences
  • Figure 2: The procedure of court judgment. Firstly, the plaintiff submits his pleas. Secondly, fact verdict is made based on the court debate held between the plaintiff and defendant. Finally, the judge make decisions based on the fact verdict.
  • Figure 3: An outcomes-based judicial framework example interpretation of notations introduced by a law case life-cycle in a real court setting. Within this framework, the LJP tasks are divided into two categories: main LJP tasks (blue) and auxiliary LJP tasks (green).
  • Figure 4: Proposed classification method of legal judgment prediction tasks.
  • Figure 5: The dataset density distribution for specific LJP task.
  • ...and 1 more figures