Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LexTime: A Benchmark for Temporal Ordering of Legal Events

Claire Barale, Leslie Barrett, Vikram Sunil Bajaj, Michael Rovatsos

TL;DR

LexTime introduces a dedicated benchmark for temporal event ordering in legal language, built from 512 U.S. federal complaint contexts with explicit and implicit events. The study shows LLMs perform better on legal text than narrative benchmarks and that longer contexts and implicit-explicit event pairs boost accuracy, with a top score of 80.8% for GPT-4 Turbo. CoT prompting generally does not help this task, highlighting the need for domain-specific prompting and modeling strategies to handle legal syntax, paraphrasing, and subordinate clauses. Across dataset construction, linguistic analysis, and benchmarking, the work identifies concrete directions to improve legal temporal reasoning in large language models and provides a resource for evaluating such progress.

Abstract

Understanding temporal relationships and accurately reconstructing the event timeline is important for case law analysis, compliance monitoring, and legal summarization. However, existing benchmarks lack specialized language evaluation, leaving a gap in understanding how LLMs handle event ordering in legal contexts. We introduce LexTime, a dataset designed to evaluate LLMs' event ordering capabilities in legal language, consisting of 512 instances from U.S. Federal Complaints with annotated event pairs and their temporal relations. Our findings show that (1) LLMs are more accurate on legal event ordering than on narrative texts (up to +10.5%); (2) longer input contexts and implicit events boost accuracy, reaching 80.8% for implicit-explicit event pairs; (3) legal linguistic complexities and nested clauses remain a challenge. While performance is promising, specific features of legal texts remain a bottleneck for legal temporal event reasoning, and we propose concrete modeling directions to better address them.

LexTime: A Benchmark for Temporal Ordering of Legal Events

TL;DR

LexTime introduces a dedicated benchmark for temporal event ordering in legal language, built from 512 U.S. federal complaint contexts with explicit and implicit events. The study shows LLMs perform better on legal text than narrative benchmarks and that longer contexts and implicit-explicit event pairs boost accuracy, with a top score of 80.8% for GPT-4 Turbo. CoT prompting generally does not help this task, highlighting the need for domain-specific prompting and modeling strategies to handle legal syntax, paraphrasing, and subordinate clauses. Across dataset construction, linguistic analysis, and benchmarking, the work identifies concrete directions to improve legal temporal reasoning in large language models and provides a resource for evaluating such progress.

Abstract

Understanding temporal relationships and accurately reconstructing the event timeline is important for case law analysis, compliance monitoring, and legal summarization. However, existing benchmarks lack specialized language evaluation, leaving a gap in understanding how LLMs handle event ordering in legal contexts. We introduce LexTime, a dataset designed to evaluate LLMs' event ordering capabilities in legal language, consisting of 512 instances from U.S. Federal Complaints with annotated event pairs and their temporal relations. Our findings show that (1) LLMs are more accurate on legal event ordering than on narrative texts (up to +10.5%); (2) longer input contexts and implicit events boost accuracy, reaching 80.8% for implicit-explicit event pairs; (3) legal linguistic complexities and nested clauses remain a challenge. While performance is promising, specific features of legal texts remain a bottleneck for legal temporal event reasoning, and we propose concrete modeling directions to better address them.

Paper Structure

This paper contains 56 sections, 13 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Datasets
  3. Task Overview and Dataset
  4. Pre-labeling.
  5. Label collection.
  6. Tracie Dataset
  7. Specific Features of Legal Events
  8. Vocabulary
  9. Domain-specific event triggers.
  10. Precise and explicit temporal cues.
  11. Higher density of temporal markers.
  12. Frequent use of modalities.
  13. Frequent use of negation.
  14. Syntax, Discourse, and Tenses
  15. Events in subordinate or conditional clauses.
  16. ...and 41 more sections

Figures (13)

  • Figure 1: Legal event ordering task in : explicit and implicit events (middle) are extracted from a context paragraph (top) and combined into a yes/no question (bottom).
  • Figure 2: 's temporal relations and their correspondence to Allen's interval relations allen1983maintaining.
  • Figure 3: Comparison of linguistic features in and Tracie. Values (y-axis) represent the percentage of sentences that contain each feature, indicating its frequency in the text.
  • Figure 4: Top-10 event triggers in by frequency (%), showing specialized legal vocabulary.
  • Figure 5: Boxplot showing sentence length distributions (in tokens). Median (Med) and quartiles (Q1, Q3) are noted, outliers are excluded for clarity.
  • ...and 8 more figures