Clustering-driven Memory Compression for On-device Large Language Models

TL;DR

The paper tackles the challenge of personalizing on-device LLMs under tight context budgets by introducing clustering-driven memory compression. Memories from user interactions are first encoded into memory tokens, then grouped with K-Means, and merged within clusters to produce cluster-level representations whose tokens total $K \cdot D_m$. This approach reduces context size while preserving coherence across heterogeneous memories, delivering higher ROUGE-L performance than naive averaging or direct concatenation across multiple on-device models and tasks. The method demonstrates consistent token efficiency and improved personalization quality, highlighting its practical impact for privacy-preserving, edge-based NLP systems.

Abstract

Large language models (LLMs) often rely on user-specific memories distilled from past interactions to enable personalized generation. A common practice is to concatenate these memories with the input prompt, but this approach quickly exhausts the limited context available in on-device LLMs. Compressing memories by averaging can mitigate context growth, yet it frequently harms performance due to semantic conflicts across heterogeneous memories. In this work, we introduce a clustering-based memory compression strategy that balances context efficiency and personalization quality. Our method groups memories by similarity and merges them within clusters prior to concatenation, thereby preserving coherence while reducing redundancy. Experiments demonstrate that our approach substantially lowers the number of memory tokens while outperforming baseline strategies such as naive averaging or direct concatenation. Furthermore, for a fixed context budget, clustering-driven merging yields more compact memory representations and consistently enhances generation quality.

Figures (4)

• Figure 1: Clustering-driven memory compression. Given $N$ supporting documents (e.g., news articles with personalized headlines), we first compress each into a set of memory tokens. These memories are then grouped into $K$ clusters based on similarity. Within each cluster, the memories are merged (e.g., by averaging), and the resulting cluster-level representations are concatenated. The final compressed memories are appended to the new input example, enabling the LLM to generate personalized outputs under a limited context budget.
• Figure 2: Effect of model size on performance. Larger models achieve better performance, with consistent behaviour across the different approaches.
• Figure 3: Effect of the number of memory tokens per document on performance. More memory tokens typically lead to better performance, but good performance can also be achieved with fewer memory tokens. Our clustering solution achieves the best performance in most cases.
• Figure 4: Effect of the number of clusters on performance. Using more clusters in our solution improves performance.