Explainable Statute Prediction via Attention-based Model and LLM Prompting

TL;DR

The paper tackles automatic statute prediction from case descriptions and the accompanying need for explanations. It introduces two complementary approaches: AoS, an attention-over-sentences model that leverages Sentence-BERT embeddings and per-statute attention to predict statutes with explanations, and LLMPrompt, a zero-shot prompting approach that uses LLMs to assess statute applicability and generate explanations, guided by a pre-filter of top-K statutes from AoS. Evaluations on two benchmarks, ILSI and ECtHR_B, show AoS achieving strong performance and interpretable explanations, while LLMPrompt offers a promising but currently weaker alternative that benefits from Chain-of-Thought prompting and larger models; both are evaluated with considerations for label leakage and explanation quality. The work demonstrates practical significance for explainable Legal AI, with insights into model choices, explanation validity via counterfactual metrics, and pathways for future improvements such as ensembles, ranked outputs, and domain-adapted encoders.

Abstract

In this paper, we explore the problem of automatic statute prediction where for a given case description, a subset of relevant statutes are to be predicted. Here, the term "statute" refers to a section, a sub-section, or an article of any specific Act. Addressing this problem would be useful in several applications such as AI-assistant for lawyers and legal question answering system. For better user acceptance of such Legal AI systems, we believe the predictions should also be accompanied by human understandable explanations. We propose two techniques for addressing this problem of statute prediction with explanations -- (i) AoS (Attention-over-Sentences) which uses attention over sentences in a case description to predict statutes relevant for it and (ii) LLMPrompt which prompts an LLM to predict as well as explain relevance of a certain statute. AoS uses smaller language models, specifically sentence transformers and is trained in a supervised manner whereas LLMPrompt uses larger language models in a zero-shot manner and explores both standard as well as Chain-of-Thought (CoT) prompting techniques. Both these models produce explanations for their predictions in human understandable forms. We compare statute prediction performance of both the proposed techniques with each other as well as with a set of competent baselines, across two popular datasets. Also, we evaluate the quality of the generated explanations through an automated counter-factual manner as well as through human evaluation.

Figures (6)

• Figure 1: Architecture of the Attention-over-Sentences (AoS) model
• Figure 2: Computation of attention weights in the Attention-over-Sentences (AoS) model for $h^{th}$ attention head and $i^{th}$ statute
• Figure 3: Histogram showing the number of cases in each dataset as per their lengths measured using number of sentences.
• Figure 4: Statute-wise performance of the AoS model for the ILSI dataset (showing the 10 most frequent statutes).
• Figure 5: Statute-wise performance of the AoS model for the ECtHR_B dataset.