Temporal Alignment of LLMs through Cycle Encoding for Long-Range Time Representations

TL;DR

This work tackles temporal misalignment in LLMs across long spans caused by sparse historical data. It introduces Ticktack, which reexpresses years via a 60-term sexagenary cycle, models the cycle with polar coordinates, and injects temporal signals through sine-cosine encodings, followed by a post-training temporal alignment using Elastic Weight Consolidation to bind knowledge to time. The approach yields substantial gains on long-span temporal QA benchmarks (e.g., TempLS) and shows clear clustering of temporal representations in latent space, validating improved long-range temporal reasoning. The results suggest a practical pathway for more durable temporal grounding in LLMs and highlight the need for broader, long-span benchmarks.

Abstract

Large language models (LLMs) suffer from temporal misalignment issues especially across long span of time. The issue arises from knowing that LLMs are trained on large amounts of data where temporal information is rather sparse over long times, such as thousands of years, resulting in insufficient learning or catastrophic forgetting by the LLMs. This paper proposes a methodology named "Ticktack" for addressing the LLM's long-time span misalignment in a yearly setting. Specifically, we first propose to utilize the sexagenary year expression instead of the Gregorian year expression employed by LLMs, achieving a more uniform distribution in yearly granularity. Then, we employ polar coordinates to model the sexagenary cycle of 60 terms and the year order within each term, with additional temporal encoding to ensure LLMs understand them. Finally, we present a temporal representational alignment approach for post-training LLMs that effectively distinguishes time points with relevant knowledge, hence improving performance on time-related tasks, particularly over a long period. We also create a long time span benchmark for evaluation. Experimental results prove the effectiveness of our proposal.

Figures (9)

• Figure 1: The distribution of temporal information in both Wikipedia (English) and Baidu Baike (Chinese), with statistics conducted at intervals of 200 years from BCE to after 2000.
• Figure 2: The correspondence between the sexagenary year (blue) and Gregorian year (black). For instance, both 1864 and 1924 correspond to the "Jiazi" year.
• Figure 3: An overview of Ticktack (a) illustrates the novel way to express the years, leveraging the polar coordinate representation of the sexagenary cycle. (b) adopts sine and cosine functions to encode temporal information based on the sexagenary cycle. (c) describes the temporal alignment process to further transform the original weight space of the LLMs into a temporal re-organized and distinguished weight space.
• Figure 4: The distribution of the years in our constructed TempLS dataset. The above figure summarizes the distribution of Gregorian years. The figure below displays the distribution of sexagenary years, which is apparently more uniform.
• Figure 5: Accuracy of Zero-Shot and Few-shot evaluations on the TempLS for the time-span from years BCE to after 2000.