Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Crafting Personalized Agents through Retrieval-Augmented Generation on Editable Memory Graphs

Zheng Wang, Zhongyang Li, Zeren Jiang, Dandan Tu, Wei Shi

TL;DR

This paper tackles the problem of crafting personalized LLM-driven agents that leverage a user's smartphone memories. It proposes EMG-RAG, a Retrieval-Augmented Generation framework built on an Editable Memory Graph (EMG) and optimized by Reinforcement Learning to address data collection, editability, and selectability. It demonstrates benefits on real data across QA, autofill forms, and user services, achieving about 10% improvement and deployment in a real smartphone AI assistant. The work offers a practical path for privacy-conscious on-device personalization of AI assistants with adaptive memory retrieval and editability.

Abstract

In the age of mobile internet, user data, often referred to as memories, is continuously generated on personal devices. Effectively managing and utilizing this data to deliver services to users is a compelling research topic. In this paper, we introduce a novel task of crafting personalized agents powered by large language models (LLMs), which utilize a user's smartphone memories to enhance downstream applications with advanced LLM capabilities. To achieve this goal, we introduce EMG-RAG, a solution that combines Retrieval-Augmented Generation (RAG) techniques with an Editable Memory Graph (EMG). This approach is further optimized using Reinforcement Learning to address three distinct challenges: data collection, editability, and selectability. Extensive experiments on a real-world dataset validate the effectiveness of EMG-RAG, achieving an improvement of approximately 10% over the best existing approach. Additionally, the personalized agents have been transferred into a real smartphone AI assistant, which leads to enhanced usability.

Crafting Personalized Agents through Retrieval-Augmented Generation on Editable Memory Graphs

TL;DR

This paper tackles the problem of crafting personalized LLM-driven agents that leverage a user's smartphone memories. It proposes EMG-RAG, a Retrieval-Augmented Generation framework built on an Editable Memory Graph (EMG) and optimized by Reinforcement Learning to address data collection, editability, and selectability. It demonstrates benefits on real data across QA, autofill forms, and user services, achieving about 10% improvement and deployment in a real smartphone AI assistant. The work offers a practical path for privacy-conscious on-device personalization of AI assistants with adaptive memory retrieval and editability.

Abstract

In the age of mobile internet, user data, often referred to as memories, is continuously generated on personal devices. Effectively managing and utilizing this data to deliver services to users is a compelling research topic. In this paper, we introduce a novel task of crafting personalized agents powered by large language models (LLMs), which utilize a user's smartphone memories to enhance downstream applications with advanced LLM capabilities. To achieve this goal, we introduce EMG-RAG, a solution that combines Retrieval-Augmented Generation (RAG) techniques with an Editable Memory Graph (EMG). This approach is further optimized using Reinforcement Learning to address three distinct challenges: data collection, editability, and selectability. Extensive experiments on a real-world dataset validate the effectiveness of EMG-RAG, achieving an improvement of approximately 10% over the best existing approach. Additionally, the personalized agents have been transferred into a real smartphone AI assistant, which leads to enhanced usability.
Paper Structure (18 sections, 6 equations, 2 figures, 12 tables)

This paper contains 18 sections, 6 equations, 2 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Statement
  4. Methodology
  5. Data Collection
  6. Editable Memory Graphs
  7. MDP for Selecting Memories on EMGs
  8. Discussion on Applications and Cold-start
  9. Experiments
  10. Experimental Setup
  11. Experimental Results
  12. Conclusion
  13. Limitations
  14. Appendix
  15. Data Collection Details
  16. ...and 3 more sections

Figures (2)

  • Figure 1: An example of data collection. Step-1: Raw data is gathered on smartphone AI assistant platforms, e.g., everyday conversations between users and assistants, and the extraction of app screenshot contents through OCR.
  • Figure 2: The architecture of the proposed EMG-RAG, demonstrated with the running example in data collection (Section \ref{['sec:dc']}). It supports three editability operations: insertion (e.g., $M_7$), deletion (e.g., $M_8$), and replacement (e.g., $M_9$), based on the EMG structure (Section \ref{['sec:emg']}). Subsequently, the edited EMG undergoes RAG to select relevant memories (e.g., $M_1,M_2,M_9$) for a given question $Q$ via a MDP (Section \ref{['sec:mdp']}). The generated answers $A$ by a frozen LLM further facilitates three downstream applications (Section \ref{['sec:apps']}).