Summarisation of German Judgments in conjunction with a Class-based Evaluation

TL;DR

This work addresses automating the generation of guiding principles for German Bundesgerichtshof judgments by fine-tuning a decoder-based large language model, with optional enrichment through legal-entity annotations. It introduces a qualitative evaluation framework consisting of seven content-related classes (language, pertinence, completeness, correctness, intelligibility, and superiority) assessed by multiple legal experts, alongside standard metrics ROUGE and BERTScore. Empirical results show that enriching judgments with legal-entity information helps the model identify relevant content and slightly improves automated scores, though the generated guiding principles still exhibit issues such as incompleteness and occasional hallucinations. The study contributes a structured, human-centered evaluation approach for legal summaries and demonstrates that entity enrichment can meaningfully impact content identification, while highlighting significant work remains before practical deployment in legal practice.

Abstract

The automated summarisation of long legal documents can be a great aid for legal experts in their daily work. We automatically create summaries (guiding principles) of German judgments by fine-tuning a decoder-based large language model. We enrich the judgments with information about legal entities before the training. For the evaluation of the created summaries, we define a set of evaluation classes which allows us to measure their language, pertinence, completeness and correctness. Our results show that employing legal entities helps the generative model to find the relevant content, but the quality of the created summaries is not yet sufficient for a use in practice.