Temporal Knowledge Question Answering via Abstract Reasoning Induction

TL;DR

The paper tackles the difficulty of temporal knowledge reasoning in LLMs by introducing Abstract Reasoning Induction (ARI), a two-phase framework that separates knowledge integration (knowledge-based) from strategy and method learning (knowledge-agnostic) in a constructivist-inspired approach. ARI leverages fine-grained atomic action templates to interact with temporal knowledge graphs while actively learning abstract reasoning methods from historical errors and successes, stored as clusters. This enables LLMs to perform multi-step temporal reasoning with reduced noise and improved efficiency. Empirical results on MultiTQ and CronQuestions show ARI achieving relative improvements of 29.7% and 9.27% over strong baselines, with ablation analyses confirming the value of abstract guidance, history clustering, and action filtering. The approach highlights the potential of combining structured knowledge interactions with proactive, abstract learning to enhance temporal reasoning in LLMs, and sets the stage for broader applicability beyond temporal QA.

Abstract

In this study, we address the challenge of enhancing temporal knowledge reasoning in Large Language Models (LLMs). LLMs often struggle with this task, leading to the generation of inaccurate or misleading responses. This issue mainly arises from their limited ability to handle evolving factual knowledge and complex temporal logic. To overcome these limitations, we propose Abstract Reasoning Induction (ARI) framework, which divides temporal reasoning into two distinct phases: Knowledge-agnostic and Knowledge-based. This framework offers factual knowledge support to LLMs while minimizing the incorporation of extraneous noisy data. Concurrently, informed by the principles of constructivism, ARI provides LLMs the capability to engage in proactive, self-directed learning from both correct and incorrect historical reasoning samples. By teaching LLMs to actively construct knowledge and methods, it can significantly boosting their temporal reasoning abilities. Our approach achieves remarkable improvements, with relative gains of 29.7% and 9.27% on two temporal QA datasets, underscoring its efficacy in advancing temporal reasoning in LLMs. The code can be found at https://github.com/czy1999/ARI-QA

Figures (9)

• Figure 1: LLMs, when integrated with various levels of information, exhibit varying scopes of applicability; the more abstract and refined the knowledge, the broader its potential application.
• Figure 2: Three levels of information utilisation. Information-Driven Response, which extracts pertinent knowledge to form the basis of answers; Exemplar-Based Learning, offering cases of reasoning for the language model to assimilate and guide current inferences; and Abstract Reasoning Induction, providing step-wise abstract methodological guidance to the present question, distinct from concrete knowledge, thereby steering the language model's inference process.
• Figure 3: Model architecture of ARI. Our framework divides temporal reasoning into two distinct phases: Knowledge-agnostic and Knowledge-based. This division aims to reduce instances of hallucinations and improve LLM's capacity for integrating abstract methodologies derived from historical experience. See the detailed instructions in Appendix \ref{['sec:instruct']}.
• Figure 4: Comparison of average reasoning steps of ARI on MultiTQ.
• Figure 5: A demonstration sample of ARI reasoning on MultiTQ.