JuDGE: Benchmarking Judgment Document Generation for Chinese Legal System

TL;DR

JuDGE introduces a benchmark for judgment document generation in the Chinese legal system by pairing factual case descriptions with ground-truth judgments and enriching them with external statutory and case corpora. It presents a multi-dimensional automated evaluation framework covering penalty accuracy, convicting accuracy, referencing accuracy, and semantic similarity, and evaluates baselines including few-shot in-context learning, supervised fine-tuning, and a multi-source retrieval-augmented generation approach across general-purpose and legal-domain LLMs. Experimental results show that direct generation remains challenging, though supervised fine-tuning and MRAG yield measurable improvements, indicating substantial room for further advances at the intersection of retrieval, legal knowledge, and generation. The dataset, evaluation framework, and baseline analyses provide a foundation for scalable assessment and future improvements in automated judgment document drafting with potential to reduce manual workload in judicial settings.

Abstract

This paper introduces JuDGE (Judgment Document Generation Evaluation), a novel benchmark for evaluating the performance of judgment document generation in the Chinese legal system. We define the task as generating a complete legal judgment document from the given factual description of the case. To facilitate this benchmark, we construct a comprehensive dataset consisting of factual descriptions from real legal cases, paired with their corresponding full judgment documents, which serve as the ground truth for evaluating the quality of generated documents. This dataset is further augmented by two external legal corpora that provide additional legal knowledge for the task: one comprising statutes and regulations, and the other consisting of a large collection of past judgment documents. In collaboration with legal professionals, we establish a comprehensive automated evaluation framework to assess the quality of generated judgment documents across various dimensions. We evaluate various baseline approaches, including few-shot in-context learning, fine-tuning, and a multi-source retrieval-augmented generation (RAG) approach, using both general and legal-domain LLMs. The experimental results demonstrate that, while RAG approaches can effectively improve performance in this task, there is still substantial room for further improvement. All the codes and datasets are available at: https://github.com/oneal2000/JuDGE.

Figures (1)

• Figure 1: An illustration of the structure of a judgment document.