AR-BENCH: Benchmarking Legal Reasoning with Judgment Error Detection, Classification and Correction

TL;DR

The paper addresses the problem of auditing finalized legal judgments by detecting, classifying, and correcting errors in appellate review. It introduces AR-Bench, a large-scale benchmark with 8,700 annotated judgments and 34,617 auxiliary documents, and formalizes a three-task framework: error detection, error classification, and error correction. Through evaluation of 14 LLMs across diverse settings, the study reveals key limitations in current models’ legal reasoning, especially for correction and fine-grained error types, highlighting a gap between general language capabilities and reliable legal anomaly detection. The work provides a foundation for improving AI-assisted judicial review and has practical implications for fairness, efficiency, and accountability in legal systems.

Abstract

Legal judgments may contain errors due to the complexity of case circumstances and the abstract nature of legal concepts, while existing appellate review mechanisms face efficiency pressures from a surge in case volumes. Although current legal AI research focuses on tasks like judgment prediction and legal document generation, the task of judgment review differs fundamentally in its objectives and paradigm: it centers on detecting, classifying, and correcting errors after a judgment is issued, constituting anomaly detection rather than prediction or generation. To address this research gap, we introduce a novel task APPELLATE REVIEW, aiming to assess models' diagnostic reasoning and reliability in legal practice. We also construct a novel dataset benchmark AR-BENCH, which comprises 8,700 finely annotated decisions and 34,617 supplementary corpora. By evaluating 14 large language models, we reveal critical limitations in existing models' ability to identify legal application errors, providing empirical evidence for future improvements.

Figures (9)

• Figure 1: Difference between (a) legal judgment prediction, (b) legal document generation, and (c) appellate review.
• Figure 2: Taxonomy of Six Manually Constructed Error Types for AR-Bench. This diagram illustrates the designed errors targeting three adjudication elements: Charges (E1: Erroneous Charge, E2: Omission of Charges), Prison Terms (E3: Sentencing Outside Limits, E4: Failure to Consider Sentencing Factors), and Fines (E5: Fixed-amount Fine Outside Limits, E6: Percentage-based Fine Outside Limits).
• Figure 3: Distribution of AR-Bench, highlighting normal versus anomalous cases and further breaking anomalies down by error type (e.g., omission, erroneous charges, fixed amounts, percentage-based fines) and sentencing factors (None, Mitigated, Aggravated, Mixed).
• Figure 4: Performance comparison across different error types E1–E6 using error detection, classification, and correction metrics, including F1 scores, ImpScore, and correction accuracy at 0.1 tolerance.
• Figure 5: Prompt for Error Detection.