Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Timeline-based Sentence Decomposition with In-Context Learning for Temporal Fact Extraction

Jianhao Chen, Haoyuan Ouyang, Junyang Ren, Wentao Ding, Wei Hu, Yuzhong Qu

TL;DR

The paper addresses temporal fact extraction in complex sentences by introducing timeline-based sentence decomposition (TSD) to align events with time expressions. It finds that direct extraction with large language models is insufficient and proposes TSDRE, which fuses LLM-driven decomposition with fine-tuning of smaller PLMs, achieving state-of-the-art results on HyperRED-Temporal and the newly released ComplexTRED dataset. ComplexTRED comprises 19,148 complex sentences, providing a challenging benchmark with multi-time expressions; distant supervision and manual corrections contribute to dataset construction. The results demonstrate that leveraging LLMs for timeline organization, combined with task-focused fine-tuning, yields significant gains for temporal fact extraction with potential impact on knowledge graphs and downstream temporal reasoning tasks.

Abstract

Facts extraction is pivotal for constructing knowledge graphs. Recently, the increasing demand for temporal facts in downstream tasks has led to the emergence of the task of temporal fact extraction. In this paper, we specifically address the extraction of temporal facts from natural language text. Previous studies fail to handle the challenge of establishing time-to-fact correspondences in complex sentences. To overcome this hurdle, we propose a timeline-based sentence decomposition strategy using large language models (LLMs) with in-context learning, ensuring a fine-grained understanding of the timeline associated with various facts. In addition, we evaluate the performance of LLMs for direct temporal fact extraction and get unsatisfactory results. To this end, we introduce TSDRE, a method that incorporates the decomposition capabilities of LLMs into the traditional fine-tuning of smaller pre-trained language models (PLMs). To support the evaluation, we construct ComplexTRED, a complex temporal fact extraction dataset. Our experiments show that TSDRE achieves state-of-the-art results on both HyperRED-Temporal and ComplexTRED datasets.

Timeline-based Sentence Decomposition with In-Context Learning for Temporal Fact Extraction

TL;DR

The paper addresses temporal fact extraction in complex sentences by introducing timeline-based sentence decomposition (TSD) to align events with time expressions. It finds that direct extraction with large language models is insufficient and proposes TSDRE, which fuses LLM-driven decomposition with fine-tuning of smaller PLMs, achieving state-of-the-art results on HyperRED-Temporal and the newly released ComplexTRED dataset. ComplexTRED comprises 19,148 complex sentences, providing a challenging benchmark with multi-time expressions; distant supervision and manual corrections contribute to dataset construction. The results demonstrate that leveraging LLMs for timeline organization, combined with task-focused fine-tuning, yields significant gains for temporal fact extraction with potential impact on knowledge graphs and downstream temporal reasoning tasks.

Abstract

Facts extraction is pivotal for constructing knowledge graphs. Recently, the increasing demand for temporal facts in downstream tasks has led to the emergence of the task of temporal fact extraction. In this paper, we specifically address the extraction of temporal facts from natural language text. Previous studies fail to handle the challenge of establishing time-to-fact correspondences in complex sentences. To overcome this hurdle, we propose a timeline-based sentence decomposition strategy using large language models (LLMs) with in-context learning, ensuring a fine-grained understanding of the timeline associated with various facts. In addition, we evaluate the performance of LLMs for direct temporal fact extraction and get unsatisfactory results. To this end, we introduce TSDRE, a method that incorporates the decomposition capabilities of LLMs into the traditional fine-tuning of smaller pre-trained language models (PLMs). To support the evaluation, we construct ComplexTRED, a complex temporal fact extraction dataset. Our experiments show that TSDRE achieves state-of-the-art results on both HyperRED-Temporal and ComplexTRED datasets.
Paper Structure (43 sections, 3 figures, 7 tables)

This paper contains 43 sections, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Temporal Fact Extraction
  4. Pattern-based methods
  5. Deep learning-based methods
  6. Sentence Decomposition
  7. In-Context Learning
  8. Preliminary
  9. Problem Formulation
  10. Complex Sentence in Temporal Fact Extraction
  11. Method
  12. In-Context Learning with ChatGPT3.5 and Fine-tuning Open-Source LLM
  13. Timeline-based Sentence Decomposition
  14. Human Feedback Enhanced In-Context Learning
  15. Fine-tuning Models with Timeline-based Sentence Decomposition
  16. ...and 28 more sections

Figures (3)

  • Figure 1: A difficult temporal fact extraction example, which contains 12 temporal facts in one sentence.
  • Figure 2: Framework comparison between Flan-T5 and TSDRE. Leveraging Timeline-based sentence decomposition for training can significantly improve the recall of the Flan-T5.
  • Figure 3: Player awards are presented in a timeline.

Theorems & Definitions (2)

  • Example 1
  • Example 2