Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LegalLens Shared Task 2024: Legal Violation Identification in Unstructured Text

Ben Hagag, Liav Harpaz, Gil Semo, Dor Bernsohn, Rohit Saha, Pashootan Vaezipoor, Kyryl Truskovskyi, Gerasimos Spanakis

TL;DR

This analysis reveals that while a mix of approaches was used, the top-performing teams in both tasks consistently relied on fine-tuning pre-trained language models, outperforming legal-specific models and few-shot methods.

Abstract

This paper presents the results of the LegalLens Shared Task, focusing on detecting legal violations within text in the wild across two sub-tasks: LegalLens-NER for identifying legal violation entities and LegalLens-NLI for associating these violations with relevant legal contexts and affected individuals. Using an enhanced LegalLens dataset covering labor, privacy, and consumer protection domains, 38 teams participated in the task. Our analysis reveals that while a mix of approaches was used, the top-performing teams in both tasks consistently relied on fine-tuning pre-trained language models, outperforming legal-specific models and few-shot methods. The top-performing team achieved a 7.11% improvement in NER over the baseline, while NLI saw a more marginal improvement of 5.7%. Despite these gains, the complexity of legal texts leaves room for further advancements.

LegalLens Shared Task 2024: Legal Violation Identification in Unstructured Text

TL;DR

This analysis reveals that while a mix of approaches was used, the top-performing teams in both tasks consistently relied on fine-tuning pre-trained language models, outperforming legal-specific models and few-shot methods.

Abstract

This paper presents the results of the LegalLens Shared Task, focusing on detecting legal violations within text in the wild across two sub-tasks: LegalLens-NER for identifying legal violation entities and LegalLens-NLI for associating these violations with relevant legal contexts and affected individuals. Using an enhanced LegalLens dataset covering labor, privacy, and consumer protection domains, 38 teams participated in the task. Our analysis reveals that while a mix of approaches was used, the top-performing teams in both tasks consistently relied on fine-tuning pre-trained language models, outperforming legal-specific models and few-shot methods. The top-performing team achieved a 7.11% improvement in NER over the baseline, while NLI saw a more marginal improvement of 5.7%. Despite these gains, the complexity of legal texts leaves room for further advancements.

Paper Structure

This paper contains 13 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. What is LegalLens
  3. LegalLens-NER
  4. LegalLens-NLI
  5. Dataset Curation for LegalLens Shared Task
  6. System Descriptions and Performance
  7. Baseline Systems
  8. Participating teams
  9. Deeper Analysis
  10. LegalLens-NER Methodologies Overview
  11. LegalLens-NLI Methodologies Overview
  12. Related Work
  13. Conclusion and Future work

Figures (3)

  • Figure 1: NER sub-task example showing highlighted legal violation entities, including Law, Violation, Violation By, and Violation On.
  • Figure 2: An example of the LegalLens NLI task, where the model assesses whether the provided hypothesis (a potential legal violation) is supported, contradicted, or unrelated to the premise (an allegation summary).
  • Figure 3: Example of the NLI sub-task showing how premises like court-filed complaints or articles are used to identify individuals harmed by violations. Both the premise and hypothesis were selected due to matching violation entities identified by the LegalLens-NER model, illustrating the system's ability to link legal grounds to personal experiences and recognize potential victims.