Controllable Memory Usage: Balancing Anchoring and Innovation in Long-Term Human-Agent Interaction

TL;DR

This work addresses controllable memory usage in long-term human-agent interaction by treating memory reliance as a user-defined axis. It introduces SteeM, a framework built on preference-aligned supervised fine-tuning and δ_align-guided reinforcement learning to steer model outputs toward a target memory-dependence level $p(q)$, quantified by the MD-Score $D_{\\mathcal{R}}^{q}(y)$ and alignment error $\\delta_{align}(q, M(q), y)$. A realistic synthetic data pipeline generates $M(q)$ and $(q, M(q))$ pairs, enabling evaluation of memory anchoring and alignment across scenarios, tasks, and unseen subjects. Empirically, SteeM outperforms prompting and memory-masking baselines in achieving target memory dependence while maintaining task quality, and it generalizes with minimal trade-offs, signaling practical potential for personalized, long-horizon AI agents.

Abstract

As LLM-based agents are increasingly used in long-term interactions, cumulative memory is critical for enabling personalization and maintaining stylistic consistency. However, most existing systems adopt an ``all-or-nothing'' approach to memory usage: incorporating all relevant past information can lead to \textit{Memory Anchoring}, where the agent is trapped by past interactions, while excluding memory entirely results in under-utilization and the loss of important interaction history. We show that an agent's reliance on memory can be modeled as an explicit and user-controllable dimension. We first introduce a behavioral metric of memory dependence to quantify the influence of past interactions on current outputs. We then propose \textbf{Stee}rable \textbf{M}emory Agent, \texttt{SteeM}, a framework that allows users to dynamically regulate memory reliance, ranging from a fresh-start mode that promotes innovation to a high-fidelity mode that closely follows interaction history. Experiments across different scenarios demonstrate that our approach consistently outperforms conventional prompting and rigid memory masking strategies, yielding a more nuanced and effective control for personalized human-agent collaboration.

Figures (11)

• Figure 1: Illustration of Memory Anchoring and our solution SteeM, which steers model outputs to align with the user's memory-dependence preference.
• Figure 2: Overview of our approach and findings. (A) We use a rubric-based judge to score a response's memory dependence and compute the alignment error with targeted dependence. (B) We reveal Memory Anchoring in modern LLMs, where outputs default to high memory reliance despite low-dependence user intent. (C) We propose SteeM, built via a preference-aligned data generation pipeline followed by SFT and GRPO, enabling controllable memory usage. (D) SteeM achieves improved alignment to user-specified memory-dependence preferences.
• Figure 3: Human--judge agreement on memory-dependence comparisons (left) and memory-dependence score distributions across models and dependence prompts (right).
• Figure 4: Realized dependence levels $D_{\mathcal{R}}^{\,q}(y)$ conditioned on the target preference $p(q)$. Columns are target levels and rows are realized levels (column-normalized). SteeM concentrates more mass near the diagonal than the baseline.
• Figure 5: Radar plots of the alignment error on unseen subjects settings (Medical and Humanities). Curves closer to the center indicate better alignment.