Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unlocking Temporal Question Answering for Large Language Models with Tailor-Made Reasoning Logic

Xingxuan Li, Liying Cheng, Qingyu Tan, Hwee Tou Ng, Shafiq Joty, Lidong Bing

Abstract

The temporal aspect is a significant dimension of our reality. We notice the challenge that large language models (LLMs) face when engaging in temporal reasoning. Our preliminary experiments show that methods involving the generation of intermediate reasoning steps, such as chain-of-thought and program-aided language models, do not consistently boost the performance of complex temporal question-answering tasks. This limitation can be attributed to the LLMs' inadequate understanding of temporal information. To address this problem, we propose TempLogic, a novel framework designed specifically for temporal question-answering tasks across three levels of reasoning. TempLogic incorporates retrieval-guided context distillation, temporal data extraction, and tailor-made logic reasoning. Extensive experiments and analysis demonstrate the effectiveness of our framework in solving intricate time-bound reasoning tasks.

Unlocking Temporal Question Answering for Large Language Models with Tailor-Made Reasoning Logic

Abstract

The temporal aspect is a significant dimension of our reality. We notice the challenge that large language models (LLMs) face when engaging in temporal reasoning. Our preliminary experiments show that methods involving the generation of intermediate reasoning steps, such as chain-of-thought and program-aided language models, do not consistently boost the performance of complex temporal question-answering tasks. This limitation can be attributed to the LLMs' inadequate understanding of temporal information. To address this problem, we propose TempLogic, a novel framework designed specifically for temporal question-answering tasks across three levels of reasoning. TempLogic incorporates retrieval-guided context distillation, temporal data extraction, and tailor-made logic reasoning. Extensive experiments and analysis demonstrate the effectiveness of our framework in solving intricate time-bound reasoning tasks.
Paper Structure (36 sections, 3 equations, 2 figures, 6 tables)

This paper contains 36 sections, 3 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Temporal Question Answering
  4. Large Language Models
  5. Framework
  6. Retrieval-Guided Context Distillation
  7. Knowledge Retrieval
  8. Context Distillation
  9. Temporal Data Extraction
  10. Task classification
  11. Data extraction
  12. Logic Reasoning
  13. Main Experiments
  14. Datasets
  15. Comprehensive Temporal Reasoning Benchmark (TempReason)
  16. ...and 21 more sections

Figures (2)

  • Figure 1: Comparison between TempLogic and previous methods. TempLogic incorporates three stages: (i@) retrieval-guided context distillation, (ii@) temporal data extraction, and (iii@) logic reasoning.
  • Figure 2: An overview of our proposed framework. (i@) Retrieval-guided context distillation. The question was inputted into a retrieval system to obtain relevant information. The retrieved information is then used to distill the most relevant sentences from the raw context. (ii@) Temporal data extraction. Using the distilled context, an LLM is utilized to extract a temporal dictionary. (iii@) Logic Reasoning. The final answer is obtained by combing and executing the extracted dictionary and a tailor-made reasoning logic.