Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning Enhanced Multi-hop Reasoning for Temporal Knowledge Question Answering

Wuzhenghong Wen, Chao Xue, Su Pan, Yuwei Sun, Minlong Peng

TL;DR

This work addresses global optimality in multi-hop temporal knowledge graph question answering by introducing the Reinforcement Learning Enhanced Multi-hop Reasoning (MRE) framework. It combines diverse multi-hop trajectory sampling with cold-start supervised fine-tuning and a novel Tree-Group Relative Policy Optimization (T-GRPO) that propagates rewards through a tree-structured search to mitigate sparse rewards. Empirical results on CRONQUESTIONS and TimeQuestions show that MRE achieves state-of-the-art performance, with strong interpretability and robustness to temporal noise. The approach advances trajectory-aware reasoning for LLMs in temporally constrained knowledge graphs, enabling more accurate and explainable temporal QA.»

Abstract

Temporal knowledge graph question answering (TKGQA) involves multi-hop reasoning over temporally constrained entity relationships in the knowledge graph to answer a given question. However, at each hop, large language models (LLMs) retrieve subgraphs with numerous temporally similar and semantically complex relations, increasing the risk of suboptimal decisions and error propagation. To address these challenges, we propose the multi-hop reasoning enhanced (MRE) framework, which enhances both forward and backward reasoning to improve the identification of globally optimal reasoning trajectories. Specifically, MRE begins with prompt engineering to guide the LLM in generating diverse reasoning trajectories for a given question. Valid reasoning trajectories are then selected for supervised fine-tuning, serving as a cold-start strategy. Finally, we introduce Tree-Group Relative Policy Optimization (T-GRPO), a recursive, tree-structured learning-by-exploration approach. At each hop, exploration establishes strong causal dependencies on the previous hop, while evaluation is informed by multi-path exploration feedback from subsequent hops. Experimental results on two TKGQA benchmarks indicate that the proposed MRE-based model consistently surpasses state-of-the-art (SOTA) approaches in handling complex multi-hop queries. Further analysis highlights improved interpretability and robustness to noisy temporal annotations.

Reinforcement Learning Enhanced Multi-hop Reasoning for Temporal Knowledge Question Answering

TL;DR

This work addresses global optimality in multi-hop temporal knowledge graph question answering by introducing the Reinforcement Learning Enhanced Multi-hop Reasoning (MRE) framework. It combines diverse multi-hop trajectory sampling with cold-start supervised fine-tuning and a novel Tree-Group Relative Policy Optimization (T-GRPO) that propagates rewards through a tree-structured search to mitigate sparse rewards. Empirical results on CRONQUESTIONS and TimeQuestions show that MRE achieves state-of-the-art performance, with strong interpretability and robustness to temporal noise. The approach advances trajectory-aware reasoning for LLMs in temporally constrained knowledge graphs, enabling more accurate and explainable temporal QA.»

Abstract

Temporal knowledge graph question answering (TKGQA) involves multi-hop reasoning over temporally constrained entity relationships in the knowledge graph to answer a given question. However, at each hop, large language models (LLMs) retrieve subgraphs with numerous temporally similar and semantically complex relations, increasing the risk of suboptimal decisions and error propagation. To address these challenges, we propose the multi-hop reasoning enhanced (MRE) framework, which enhances both forward and backward reasoning to improve the identification of globally optimal reasoning trajectories. Specifically, MRE begins with prompt engineering to guide the LLM in generating diverse reasoning trajectories for a given question. Valid reasoning trajectories are then selected for supervised fine-tuning, serving as a cold-start strategy. Finally, we introduce Tree-Group Relative Policy Optimization (T-GRPO), a recursive, tree-structured learning-by-exploration approach. At each hop, exploration establishes strong causal dependencies on the previous hop, while evaluation is informed by multi-path exploration feedback from subsequent hops. Experimental results on two TKGQA benchmarks indicate that the proposed MRE-based model consistently surpasses state-of-the-art (SOTA) approaches in handling complex multi-hop queries. Further analysis highlights improved interpretability and robustness to noisy temporal annotations.
Paper Structure (14 sections, 12 equations, 2 figures, 6 tables, 1 algorithm)

This paper contains 14 sections, 12 equations, 2 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. MRE Framework
  4. Multi-Trajectory Sampling
  5. Cold start supervised fine-tuning
  6. T-GRPO
  7. Storage.
  8. Experimental Settings
  9. Experimental Results and Analysis
  10. Overall Performance
  11. Ablation Study
  12. Multi-hop Temporal Reasoning Depth Analysis
  13. RLHF Training Analysis
  14. Conclusion

Figures (2)

  • Figure 1: Retrieval errors under complex temporal facts (b) and the resolution by the MRE framework (c).
  • Figure 2: Overall architecture of the MRE framework. First, prompt engineering is used to guide the LLM in generating diverse multi-hop reasoning trajectories. Second, valid trajectories are selected for SFT, providing a cold-start policy. Finally, tree-structured exploration with T-GRPO recursively optimizes reasoning paths by leveraging forward decisions and backward feedback across each hops.