Query-driven Relevant Paragraph Extraction from Legal Judgments

TL;DR

This work tackles the problem of query-driven paragraph-level retrieval from ECtHR judgments by constructing a dedicated dataset from case-law guides and judgments. It benchmarks zero-shot retrieval (MSMARCO-pretrained) against fine-tuning (MSMARCO and LegalBERT initializations) across splits that test generalization to unseen legal concepts and queries, revealing a substantial gap between zero-shot and fine-tuned performance and the impact of distribution shift on both corpus and query sides. The study further evaluates Parameter-Efficient Fine-Tuning (PEFT) methods—Adapter, Prefix-Tuning, and LoRA—on both bi-encoder and cross-encoder architectures, showing that PEFT can approach full fine-tuning performance in some settings but that no single method dominates across all configurations, especially under unseen queries or concepts. Overall, the paper provides a valuable ECtHR-specific dataset and a nuanced analysis of how corpus- and query-focused distribution shifts, model architecture, and PEFT choices shape retrieval effectiveness in legal information retrieval, with practical implications for resource-efficient deployment.

Abstract

Legal professionals often grapple with navigating lengthy legal judgements to pinpoint information that directly address their queries. This paper focus on this task of extracting relevant paragraphs from legal judgements based on the query. We construct a specialized dataset for this task from the European Court of Human Rights (ECtHR) using the case law guides. We assess the performance of current retrieval models in a zero-shot way and also establish fine-tuning benchmarks using various models. The results highlight the significant gap between fine-tuned and zero-shot performance, emphasizing the challenge of handling distribution shift in the legal domain. We notice that the legal pre-training handles distribution shift on the corpus side but still struggles on query side distribution shift, with unseen legal queries. We also explore various Parameter Efficient Fine-Tuning (PEFT) methods to evaluate their practicality within the context of information retrieval, shedding light on the effectiveness of different PEFT methods across diverse configurations with pre-training and model architectures influencing the choice of PEFT method.

Figures (3)

• Figure 1: Query construction process from case law guide. The above table of contents is obtained from 'Rights of LGBTI persons' guide.
• Figure 2: Illustration of pin-pointed paragraph relevance in case law guides.
• Figure 3: Data Analysis