Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Memory-T1: Reinforcement Learning for Temporal Reasoning in Multi-session Agents

Yiming Du, Baojun Wang, Yifan Xiang, Zhaowei Wang, Wenyu Huang, Boyang Xue, Bin Liang, Xingshan Zeng, Fei Mi, Haoli Bai, Lifeng Shang, Jeff Z. Pan, Yuxin Jiang, Kam-Fai Wong

TL;DR

Memory-T1 tackles temporal reasoning over long, multi-session dialogues by learning a time-aware memory retrieval policy via reinforcement learning. It employs a coarse-to-fine retrieval pipeline (temporal filtering followed by relevance pruning) and a fine-grained RL policy trained with a multi-level reward that combines $R_a$, $R_g$, and $R_t$ to provide dense supervision. On Time-Dialog, Memory-T1 achieves state-of-the-art results, enabling a 3B model to outperform a 14B baseline and showing robustness up to 128k tokens; ablations confirm the critical roles of $R_g$ and $R_t$ in grounding and temporal alignment. The approach also generalizes to LoCoMo and maintains efficient latency, underscoring its potential for reliable, long-term conversational AI. All data and code are planned for public release to support reproducibility and further research.

Abstract

Temporal reasoning over long, multi-session dialogues is a critical capability for conversational agents. However, existing works and our pilot study have shown that as dialogue histories grow in length and accumulate noise, current long-context models struggle to accurately identify temporally pertinent information, significantly impairing reasoning performance. To address this, we introduce Memory-T1, a framework that learns a time-aware memory selection policy using reinforcement learning (RL). It employs a coarse-to-fine strategy, first pruning the dialogue history into a candidate set using temporal and relevance filters, followed by an RL agent that selects the precise evidence sessions. The RL training is guided by a multi-level reward function optimizing (i) answer accuracy, (ii) evidence grounding, and (iii) temporal consistency. In particular, the temporal consistency reward provides a dense signal by evaluating alignment with the query time scope at both the session-level (chronological proximity) and the utterance-level (chronological fidelity), enabling the agent to resolve subtle chronological ambiguities. On the Time-Dialog benchmark, Memory-T1 boosts a 7B model to an overall score of 67.0\%, establishing a new state-of-the-art performance for open-source models and outperforming a 14B baseline by 10.2\%. Ablation studies show temporal consistency and evidence grounding rewards jointly contribute to a 15.0\% performance gain. Moreover, Memory-T1 maintains robustness up to 128k tokens, where baseline models collapse, proving effectiveness against noise in extensive dialogue histories. The code and datasets are publicly available at https://github.com/Elvin-Yiming-Du/Memory-T1/

Memory-T1: Reinforcement Learning for Temporal Reasoning in Multi-session Agents

TL;DR

Memory-T1 tackles temporal reasoning over long, multi-session dialogues by learning a time-aware memory retrieval policy via reinforcement learning. It employs a coarse-to-fine retrieval pipeline (temporal filtering followed by relevance pruning) and a fine-grained RL policy trained with a multi-level reward that combines , , and to provide dense supervision. On Time-Dialog, Memory-T1 achieves state-of-the-art results, enabling a 3B model to outperform a 14B baseline and showing robustness up to 128k tokens; ablations confirm the critical roles of and in grounding and temporal alignment. The approach also generalizes to LoCoMo and maintains efficient latency, underscoring its potential for reliable, long-term conversational AI. All data and code are planned for public release to support reproducibility and further research.

Abstract

Temporal reasoning over long, multi-session dialogues is a critical capability for conversational agents. However, existing works and our pilot study have shown that as dialogue histories grow in length and accumulate noise, current long-context models struggle to accurately identify temporally pertinent information, significantly impairing reasoning performance. To address this, we introduce Memory-T1, a framework that learns a time-aware memory selection policy using reinforcement learning (RL). It employs a coarse-to-fine strategy, first pruning the dialogue history into a candidate set using temporal and relevance filters, followed by an RL agent that selects the precise evidence sessions. The RL training is guided by a multi-level reward function optimizing (i) answer accuracy, (ii) evidence grounding, and (iii) temporal consistency. In particular, the temporal consistency reward provides a dense signal by evaluating alignment with the query time scope at both the session-level (chronological proximity) and the utterance-level (chronological fidelity), enabling the agent to resolve subtle chronological ambiguities. On the Time-Dialog benchmark, Memory-T1 boosts a 7B model to an overall score of 67.0\%, establishing a new state-of-the-art performance for open-source models and outperforming a 14B baseline by 10.2\%. Ablation studies show temporal consistency and evidence grounding rewards jointly contribute to a 15.0\% performance gain. Moreover, Memory-T1 maintains robustness up to 128k tokens, where baseline models collapse, proving effectiveness against noise in extensive dialogue histories. The code and datasets are publicly available at https://github.com/Elvin-Yiming-Du/Memory-T1/
Paper Structure (60 sections, 13 equations, 25 figures, 10 tables, 3 algorithms)

This paper contains 60 sections, 13 equations, 25 figures, 10 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Temporal Reasoning in LLMs.
  4. Reinforcement Learning in Agents:
  5. Memory-T1
  6. Problem Formulation
  7. Memory-T1: Temporal-Aware Memory Retrieval
  8. Phase 2: Fine-grained Selection via Reinforcement Learning.
  9. Reward Design
  10. Accuracy Reward ($R_a$)
  11. Evidence Grounding Reward ($R_g$)
  12. Temporal Consistency Reward ($R_t$):
  13. 1.Chronological Proximity ($R_s$, session-level):
  14. 2. Chronological Fidelity ($R_f$, utterance-level).
  15. Experiments
  16. ...and 45 more sections

Figures (25)

  • Figure 1: Multi-session QA with time-event annotations. Time range marks when an event or query occurs, either a duration or an instantaneous point (start and end coincide). Event span highlights key evidence in the utterance.
  • Figure 2: An overview of Memory-T1. The framework employs a coarse-to-fine cascade to select time-consistent memories for multi-session temporal reasoning.
  • Figure 2: Ablation study on the reward function of Memory-T1 (3B). Relative changes compared to the full model are shown in parentheses.
  • Figure 3: Performance comparison between Memory-T1 (3B) and Qwen2.5-3B (Instruct) under different top-k values (bar charts represent overall F1 scores; line charts represent evidence session recall rate. Comparison conditions: With/without temporal filtering; Top-k refers to the number of sessions retrieved in the candidate generation phase.)
  • Figure 4: Comparison of Qwen2.5 and Memory-T1 models on the test set, where examples are grouped by the length of each test example (tokens) (0k–8k, 8k–16k, 16k–32k, 32k–64k, 64k–128k) to assess performance variation across lengths, along with overall evaluation.
  • ...and 20 more figures