PILOT: Legal Case Outcome Prediction with Case Law

TL;DR

PILOT addresses predicting legal case outcomes in case-law systems by explicitly handling precedent retrieval and temporal pattern shifts. It introduces three components: a precedent retrieval module trained via contrastive learning, an evidence-fusion encoding that combines current-case text with retrieved precedents, and a temporal drift mining module that adjusts predictions over time using a timestamp-driven auxiliary signal. The authors present ECHR2023, a chronologically split dataset derived from the European Court of Human Rights, and show that PILOT achieves state-of-the-art performance across four metrics, outperforming diverse baselines and ablations. The work highlights the importance of temporal awareness and precedent-based evidence in legal AI, while acknowledging limitations and outlining potential improvements such as richer retrieval signals and better interpretability for real-world deployment.

Abstract

Machine learning shows promise in predicting the outcome of legal cases, but most research has concentrated on civil law cases rather than case law systems. We identified two unique challenges in making legal case outcome predictions with case law. First, it is crucial to identify relevant precedent cases that serve as fundamental evidence for judges during decision-making. Second, it is necessary to consider the evolution of legal principles over time, as early cases may adhere to different legal contexts. In this paper, we proposed a new framework named PILOT (PredictIng Legal case OuTcome) for case outcome prediction. It comprises two modules for relevant case retrieval and temporal pattern handling, respectively. To benchmark the performance of existing legal case outcome prediction models, we curated a dataset from a large-scale case law database. We demonstrate the importance of accurately identifying precedent cases and mitigating the temporal shift when making predictions for case law, as our method shows a significant improvement over the prior methods that focus on civil law case outcome predictions.

Figures (5)

• Figure 1: The framework of our proposed model PILOT. PILOT has three modules: Relevant Case Retrieval, Case Encoder with Evidence Fusion, and Temporal Shift Mining. The Relevant Case Retrieval module retrieves relevant cases to use as references for outcome prediction. The Case Encoder with Evidence Fusion module encodes current cases with fact descriptions and relevant cases. The Temporal Shift Mining module adapts directly to temporal drift.
• Figure 2: Hyperparameter analysis of $k$ and $\alpha$ in the relevant case retrieval module. When $k$ equals 5 and $\alpha$ equals 2, the model achieves the best results. When the value of $\alpha$ is 1e10, it indicate an extreme condition that implies the absence of temporal decay in the computation of the similarity score
• Figure 3: Hyperparameter analysis of lambda which is the weight of drift loss. When $\lambda$ equals 0.10, the model achieves the best results.
• Figure 4: The final selected prompt. We also prompt model by telling you are a good judge.
• Figure 5: An example input and output of the LLM about data 001-187931. The original document has 3618 tokens totally. It reduces to 494 tokens after extracting important points of a legal case.