AppellateGen: A Benchmark for Appellate Legal Judgment Generation

TL;DR

AppellateGen targets the neglected second-instance (appellate) stage by introducing a large-scale benchmark of $7{,}351$ paired cases and a fine-grained annotation scheme to evaluate long-context, dialectical legal reasoning. It couples data collection with a SOP-inspired multi-agent workflow (SLMAS) that decomposes appellate generation into issue identification, retrieval, reversal prediction, and drafting, mitigating hallucinations in long-context generation. Empirical results show SLMAS improves logical consistency and reversal accuracy over baselines, with general LLMs and chain-of-thought prompting outperforming many domain-specific models, yet the overall generation quality remains around $2.5$ on a 0–5 scale, signaling substantial room for improvement in causal legal reasoning. The work provides a foundation for modeling causal dependencies across trial stages and offers publicly available data and code to accelerate research in appellate judicial writing and robust reasoning architectures in legal AI.

Abstract

Legal judgment generation is a critical task in legal intelligence. However, existing research in legal judgment generation has predominantly focused on first-instance trials, relying on static fact-to-verdict mappings while neglecting the dialectical nature of appellate (second-instance) review. To address this, we introduce AppellateGen, a benchmark for second-instance legal judgment generation comprising 7,351 case pairs. The task requires models to draft legally binding judgments by reasoning over the initial verdict and evidentiary updates, thereby modeling the causal dependency between trial stages. We further propose a judicial Standard Operating Procedure (SOP)-based Legal Multi-Agent System (SLMAS) to simulate judicial workflows, which decomposes the generation process into discrete stages of issue identification, retrieval, and drafting. Experimental results indicate that while SLMAS improves logical consistency, the complexity of appellate reasoning remains a substantial challenge for current LLMs. The dataset and code are publicly available at: https://anonymous.4open.science/r/AppellateGen-5763.

Figures (4)

• Figure 1: An example from our dataset (translated from Chinese). Facts refers to the key factual findings extracted from the unstructured full text via a Large Language Model (LLM). Reason for Reversal denotes the explanatory rationale synthesized by the LLM, which identifies the critical discrepancies between the first and second instances that led to the judgment change.
• Figure 2: SOP-based Legal Multi-Agent System (SLMAS). With first-instance judgment and new facts provided, the framework mimics the human judicial Standard Operating Procedure (SOP) by orchestrating four specialized agents. These agents sequentially identify disputed issues, retrieve relevant legal statutes, predict the reversal outcome, and finally draft the appellate judgment, ensuring the generation is grounded in explicit legal reasoning.
• Figure 3: Proceedings of the Second Instance: Following the filing of an appeal by the appellant, the admission of new evidence may result in either the reversal of the verdict or the affirmation of the original judgment.
• Figure 4: RE pattern for legal instances matching. This pattern can extract the case identifier of first instance from second-instance judgment document.