Scene-Aware Memory Discrimination: Deciding Which Personal Knowledge Stays

TL;DR

The paper tackles the challenge of efficiently storing only valuable personal knowledge by introducing Scene-Aware Memory Discrimination (SAMD). It leverages a frozen LLM with two modules: Gating Unit Module (GUM) to filter non-memorable data using memory-scene salience, and Cluster Prompting Module (CPM) to define adaptive memory standards via an intent–scene affinity matrix and intent–scene joint clustering. Offline memory understanding builds scene-based identifiers and cluster prompts; online memory judging uses these to decide which sentences to remember, enabling fast adaptation to new scenes and intents. Comprehensive direct and indirect evaluations show SAMD improves memory discrimination accuracy, generalizes across dynamic scenarios, reduces computational cost, and enhances memory-driven task performance in personalized applications. The approach provides scalable, robust memory construction that can adapt to evolving user needs without fine-tuning LLMs, enabling more efficient personal-memory systems for intelligent devices.

Abstract

Intelligent devices have become deeply integrated into everyday life, generating vast amounts of user interactions that form valuable personal knowledge. Efficient organization of this knowledge in user memory is essential for enabling personalized applications. However, current research on memory writing, management, and reading using large language models (LLMs) faces challenges in filtering irrelevant information and in dealing with rising computational costs. Inspired by the concept of selective attention in the human brain, we introduce a memory discrimination task. To address large-scale interactions and diverse memory standards in this task, we propose a Scene-Aware Memory Discrimination method (SAMD), which comprises two key components: the Gating Unit Module (GUM) and the Cluster Prompting Module (CPM). GUM enhances processing efficiency by filtering out non-memorable interactions and focusing on the salient content most relevant to application demands. CPM establishes adaptive memory standards, guiding LLMs to discern what information should be remembered or discarded. It also analyzes the relationship between user intents and memory contexts to build effective clustering prompts. Comprehensive direct and indirect evaluations demonstrate the effectiveness and generalization of our approach. We independently assess the performance of memory discrimination, showing that SAMD successfully recalls the majority of memorable data and remains robust in dynamic scenarios. Furthermore, when integrated into personalized applications, SAMD significantly enhances both the efficiency and quality of memory construction, leading to better organization of personal knowledge.

Figures (7)

• Figure 1: The overview of the proposed memory discrimination method via the gating unit module and cluster prompting modules. It has two phases: offline memory understanding and online memory judging. In the memory understanding phase, a memory scene-based identifier and cluster prompts with discrimination rules are created using all memory scenes and user intents. During the memory judging phase, each sentence in the context at time $t$ is evaluated for whether to remember, with memorable sentences being stored in memory $M_t$ for use in time $t+1$.
• Figure 2: Instructions for constructing the memory scene-based identifier in GUM, enabling quick updates for new memory scene demands. Different individuals focus on various aspects of the same memory scene.
• Figure 3: The left side illustrates the proposed Cluster Prompting Module. MSI represents the memory scene-based identifier in GUM. Intents are grouped into different clusters via matrix decomposition. The cluster prompts consist of several positive and negative rules. The rightmost side demonstrates how to design discrimination rules using memory scenes as a bridge to represent the personal information required by real-world applications.
• Figure 4: Comparative examples of various prompts.
• Figure 5: Comparison of different prompt types. The proposed cluster prompt, using only a similar magnitude of rules as the general prompt, performs as well or even better than the specific prompt.