Bi-Mem: Bidirectional Construction of Hierarchical Memory for Personalized LLMs via Inductive-Reflective Agents

TL;DR

The paper tackles the challenge of maintaining accurate, personalized memory in LLMs over long-term interactions, where hierarchical memory can drift from a user’s global persona due to noise and hallucinations. It introduces Bi-Mem, a bidirectional framework that jointly constructs memory bottom-up (fact -> scene -> persona) and calibrates it top-down against a global persona, augmented by an associative retrieval mechanism that spreads activation between facts and scenes. The approach yields a tri-level memory M=(F,S,P) and demonstrates substantial QA improvements on LoCoMo across multiple backbones and question types, with comprehensive ablations validating the value of each component. The work offers a scalable pathway to more coherent and personalized long-term interactions, balancing accuracy gains with retrieval efficiency, and outlines limitations and directions for future refinement.

Abstract

Constructing memory from users' long-term conversations overcomes LLMs' contextual limitations and enables personalized interactions. Recent studies focus on hierarchical memory to model users' multi-granular behavioral patterns via clustering and aggregating historical conversations. However, conversational noise and memory hallucinations can be amplified during clustering, causing locally aggregated memories to misalign with the user's global persona. To mitigate this issue, we propose Bi-Mem, an agentic framework ensuring hierarchical memory fidelity through bidirectional construction. Specifically, we deploy an inductive agent to form the hierarchical memory: it extracts factual information from raw conversations to form fact-level memory, aggregates them into thematic scenes (i.e., local scene-level memory) using graph clustering, and infers users' profiles as global persona-level memory. Simultaneously, a reflective agent is designed to calibrate local scene-level memories using global constraints derived from the persona-level memory, thereby enforcing global-local alignment. For coherent memory recall, we propose an associative retrieval mechanism: beyond initial hierarchical search, a spreading activation process allows facts to evoke contextual scenes, while scene-level matches retrieve salient supporting factual information. Empirical evaluations demonstrate that Bi-Mem achieves significant improvements in question answering performance on long-term personalized conversational tasks.

Figures (9)

• Figure 1: Illustration of local aggregated memory misaligning with the user’s global persona in naive hierarchical memory systems, leading LLMs to generate persona-violating answers.
• Figure 2: The framework of our proposed Bi-Mem, including the bidirectional construction (inductive process and reflective process) of hierarchical memory and the associative retrieval.
• Figure 3: Ablation Study on Reflective Calibration in Hierarchical Memory Construction. "Base" denotes answer generation via long-context LLM backbones without memory, while "w/o Calibration" refers to unidirectional hierarchical memory construction without calibration in the reflective process.
• Figure 4: Sensitivity analysis of Bi-Mem to the hyperparameter $k$, which is the number of initially retrieved memory units in the hierarchical search stage.
• Figure 5: Case study in a cluster of original conversations.