MemInsight: Autonomous Memory Augmentation for LLM Agents

TL;DR

MemInsight addresses the challenge of scalable, context-rich long-term memory for LLM agents by autonomously augmenting memory with semantically meaningful attributes. The framework comprises attribute mining, annotation, and memory retrieval to produce structured memory representations and enable attribute-guided or embedding-based retrieval. Empirical results on LLM-REDIAL and LoCoMo show improved retrieval quality, higher QA accuracy, and more persuasive recommendations, with priority augmentation often delivering the strongest gains. Importantly, MemInsight complements retrieval-augmented methods like RAG by providing targeted memory signals that improve contextual reasoning without excessive memory usage. The work demonstrates a scalable path toward more coherent, adaptable, and context-aware LLM agents in multi-task settings.

Abstract

Large language model (LLM) agents have evolved to intelligently process information, make decisions, and interact with users or tools. A key capability is the integration of long-term memory capabilities, enabling these agents to draw upon historical interactions and knowledge. However, the growing memory size and need for semantic structuring pose significant challenges. In this work, we propose an autonomous memory augmentation approach, MemInsight, to enhance semantic data representation and retrieval mechanisms. By leveraging autonomous augmentation to historical interactions, LLM agents are shown to deliver more accurate and contextualized responses. We empirically validate the efficacy of our proposed approach in three task scenarios; conversational recommendation, question answering and event summarization. On the LLM-REDIAL dataset, MemInsight boosts persuasiveness of recommendations by up to 14%. Moreover, it outperforms a RAG baseline by 34% in recall for LoCoMo retrieval. Our empirical results show the potential of MemInsight to enhance the contextual performance of LLM agents across multiple tasks.

Figures (10)

• Figure 1: MemInsight framework comprising three core modules: Attribute Mining (including perspective and granularity), Annotation (with attribute prioritization), and Memory Retrieval (including refined and comprehensive retrieval). These components are triggered by various downstream tasks such as Question Answering, Event Summarization, and Conversational Recommendation.
• Figure 2: An example for Turn level and Session level annotations for a sample dialogue conversation from the LoCoMo Dataset.
• Figure 3: Evaluation framework for event summarization with MemInsight, exploring augmentation at Turn and Session levels, considering attributes alone or both attributes and dialogues for richer summaries.
• Figure 4: An example of entity-centric augmentation for the book 'Already Taken', and a conversation-centric augmentation for a sample dialogue from the LLM-REDIAL dataset.
• Figure 5: Top 10 attributes by frequency in the LLM-REDIAL dataset across domains (Movies, Sports Items, Electronics, and Books) using MemInsight Attribute Mining. Frequency indicates how often each attribute was generated to augment different movies.