Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Temporal Referential Consistency: Do LLMs Favor Sequences Over Absolute Time References?

Ashutosh Bajpai, Tanmoy Chakraborty

TL;DR

This work introduces TEMP-ReCon, a multilingual benchmark for temporal referential consistency in LLMs, addressing the gap in evaluating absolute versus chronological temporal references. It formalizes two reasoning pathways and proposes UnTRaP, an alignment-based fine-tuning approach that unifies event- and time-oriented reasoning to improve cross-temporal consistency. Empirical results show that existing models struggle with temporal referential consistency across languages, while UnTRaP yields substantial improvements over baselines, generalizing across languages and base-model families. The study underscores the importance of temporal referential reliability for time-sensitive applications and offers a practical path to enhance trustworthiness in LLMs.

Abstract

The increasing acceptance of large language models (LLMs) as an alternative to knowledge sources marks a significant paradigm shift across various domains, including time-sensitive fields such as law, healthcare, and finance. To fulfill this expanded role, LLMs must not only be factually accurate but also demonstrate consistency across temporal dimensions, necessitating robust temporal reasoning capabilities. Despite this critical requirement, efforts to ensure temporal consistency in LLMs remain scarce including noticeable absence of endeavors aimed at evaluating or augmenting LLMs across temporal references in time-sensitive inquiries. In this paper, we seek to address this gap by introducing a novel benchmark entitled temporal referential consistency, accompanied by a resource TEMP-ReCon designed to benchmark a wide range of both open-source and closed-source LLMs with various linguistic contexts characterized by differing resource richness (including English, French, and Romanian). The findings emphasis that LLMs do exhibit insufficient temporal referent consistency. To address this, we propose \newmodel, a reasoning path alignment-based model that aims to enhance the temporal referential consistency of LLMs. Our empirical experiments substantiate the efficacy of UnTRaP compared to several baseline models.

Temporal Referential Consistency: Do LLMs Favor Sequences Over Absolute Time References?

TL;DR

This work introduces TEMP-ReCon, a multilingual benchmark for temporal referential consistency in LLMs, addressing the gap in evaluating absolute versus chronological temporal references. It formalizes two reasoning pathways and proposes UnTRaP, an alignment-based fine-tuning approach that unifies event- and time-oriented reasoning to improve cross-temporal consistency. Empirical results show that existing models struggle with temporal referential consistency across languages, while UnTRaP yields substantial improvements over baselines, generalizing across languages and base-model families. The study underscores the importance of temporal referential reliability for time-sensitive applications and offers a practical path to enhance trustworthiness in LLMs.

Abstract

The increasing acceptance of large language models (LLMs) as an alternative to knowledge sources marks a significant paradigm shift across various domains, including time-sensitive fields such as law, healthcare, and finance. To fulfill this expanded role, LLMs must not only be factually accurate but also demonstrate consistency across temporal dimensions, necessitating robust temporal reasoning capabilities. Despite this critical requirement, efforts to ensure temporal consistency in LLMs remain scarce including noticeable absence of endeavors aimed at evaluating or augmenting LLMs across temporal references in time-sensitive inquiries. In this paper, we seek to address this gap by introducing a novel benchmark entitled temporal referential consistency, accompanied by a resource TEMP-ReCon designed to benchmark a wide range of both open-source and closed-source LLMs with various linguistic contexts characterized by differing resource richness (including English, French, and Romanian). The findings emphasis that LLMs do exhibit insufficient temporal referent consistency. To address this, we propose \newmodel, a reasoning path alignment-based model that aims to enhance the temporal referential consistency of LLMs. Our empirical experiments substantiate the efficacy of UnTRaP compared to several baseline models.
Paper Structure (39 sections, 10 equations, 6 figures, 11 tables, 1 algorithm)

This paper contains 39 sections, 10 equations, 6 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Consistency.
  3. Temporal Consistency.
  4. Dataset
  5. TEMP-ReCon.
  6. Proposed Methodology
  7. Preliminaries
  8. Unifying Temporal Reasoning Pathways (UnTRaP)
  9. Experiments
  10. Experimental Setting
  11. Baselines.
  12. Evaluation Metrics.
  13. Experimental Results
  14. Temporal Referential Consistency Across LLMs.
  15. Suboptimal Performance Across Languages.
  16. ...and 24 more sections

Figures (6)

  • Figure 1: An illustration of temporal referential consistency.
  • Figure 2: The architectural formulation of Unifying Temporal Reasoning Pathways (UnTRaP). $r^a$: an event-oriented reasoning path that sequences time-event:event-event to address absolute temporal reference queries. $r^c$: a time-oriented reasoning path that follows event-time:time-event for chronological temporal reference queries.
  • Figure 3: A comparison between UnTRaP with unilateral (event-oriented) reasoning path fine-tuning utilizing only absolute temporal reference training instances. M1: Temp-Ref-Fact-Dev[EM], M2:Temp-Ref-Fact-Dev[F1], M3: Temp-Ref-Cons, M4: Temp-Ref-Cons-Fact.
  • Figure 4: Temporally referential consistency across various entity types in the test data for UnTRaP compared to the baseline CoTSeLF.
  • Figure 5: A three-shot prompt that is used to assess temporal referential consistency of LLMs in experiments involving in-context learning (ICL) methods.
  • ...and 1 more figures