Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narratives

Xinliang Frederick Zhang, Nick Beauchamp, Lu Wang

TL;DR

A new prompting technique tailored for temporal reasoning, Narrative-of-Thought (NoT), is proposed that first converts the events set to a Python class, then prompts a small model to generate a temporally grounded narrative, guiding the final generation of a temporal graph.

Abstract

Reasoning about time and temporal relations is an integral aspect of human cognition, essential for perceiving the world and navigating our experiences. Though large language models (LLMs) have demonstrated impressive performance in many reasoning tasks, temporal reasoning remains challenging due to its intrinsic complexity. In this work, we first study an essential task of temporal reasoning -- temporal graph generation, to unveil LLMs' inherent, global reasoning capabilities. We show that this task presents great challenges even for the most powerful LLMs, such as GPT-3.5/4. We also notice a significant performance gap by small models (<10B) that lag behind LLMs by 50%. Next, we study how to close this gap with a budget constraint, e.g., not using model finetuning. We propose a new prompting technique tailored for temporal reasoning, Narrative-of-Thought (NoT), that first converts the events set to a Python class, then prompts a small model to generate a temporally grounded narrative, guiding the final generation of a temporal graph. Extensive experiments showcase the efficacy of NoT in improving various metrics. Notably, NoT attains the highest F1 on the Schema-11 evaluation set, while securing an overall F1 on par with GPT-3.5. NoT also achieves the best structural similarity across the board, even compared with GPT-3.5/4. Our code is available at https://github.com/launchnlp/NoT.

Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narratives

TL;DR

A new prompting technique tailored for temporal reasoning, Narrative-of-Thought (NoT), is proposed that first converts the events set to a Python class, then prompts a small model to generate a temporally grounded narrative, guiding the final generation of a temporal graph.

Abstract

Reasoning about time and temporal relations is an integral aspect of human cognition, essential for perceiving the world and navigating our experiences. Though large language models (LLMs) have demonstrated impressive performance in many reasoning tasks, temporal reasoning remains challenging due to its intrinsic complexity. In this work, we first study an essential task of temporal reasoning -- temporal graph generation, to unveil LLMs' inherent, global reasoning capabilities. We show that this task presents great challenges even for the most powerful LLMs, such as GPT-3.5/4. We also notice a significant performance gap by small models (<10B) that lag behind LLMs by 50%. Next, we study how to close this gap with a budget constraint, e.g., not using model finetuning. We propose a new prompting technique tailored for temporal reasoning, Narrative-of-Thought (NoT), that first converts the events set to a Python class, then prompts a small model to generate a temporally grounded narrative, guiding the final generation of a temporal graph. Extensive experiments showcase the efficacy of NoT in improving various metrics. Notably, NoT attains the highest F1 on the Schema-11 evaluation set, while securing an overall F1 on par with GPT-3.5. NoT also achieves the best structural similarity across the board, even compared with GPT-3.5/4. Our code is available at https://github.com/launchnlp/NoT.
Paper Structure (36 sections, 1 equation, 13 figures, 10 tables)

This paper contains 36 sections, 1 equation, 13 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Temporal Reasoning
  4. Chain-of-Thought and its Variants
  5. Method: Narrative-of-Thought
  6. Structural Representation.
  7. Narrative-of-Thought (NoT).
  8. Narrative-aware Demonstrations.
  9. Experiment
  10. Dataset
  11. Setup
  12. Baselines
  13. Evaluation Metrics
  14. Results and Analyses
  15. Main Results
  16. ...and 21 more sections

Figures (13)

  • Figure 1: Task overview of temporal graph generation (TGG), where the input is a goal and a set of unordered events. In this work, to better unleash the pre-training power of LLMs trained with a mixture of text and code, we cast TGG as a code completion task.
  • Figure 2: Overview of Narrative-of-Thought (NoT), a prompting technique tailored for temporal reasoning. NoT improves the temporal graph by recounting a temporally grounded narrative. Also shown are comparisons with existing methods. Full example is in \ref{['fig:full_example_gensort']} and NoT output is in \ref{['fig:output_gensort']}.
  • Figure 3: F1 scores on ProScript and Schema-11 in relation to the number of shots in demonstrations. We identify the instability in the standard prompting, and the performance plateau after 5 shots.
  • Figure 4: F1 scores on ProScript and Schema-11 with different meta prompts. Average performance grouped by prompt type is also shown. Notably, using a Simple Report-style, GPT-4 generated narratives lead to the best score due to its conciseness, simplicity and factuality, which are essential qualities for a high-quality reference narrative.
  • Figure A1: GED scores on ProScript (top) and Schema-11 (bottom) in relation to the number of shots in demonstrations. We identify the instability in the standard prompting, and the performance plateau after 5 shots, along with a slight decline with even more shots.
  • ...and 8 more figures