Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Me-Agent: A Personalized Mobile Agent with Two-Level User Habit Learning for Enhanced Interaction

Shuoxin Wang, Chang Liu, Gowen Loo, Lifan Zheng, Kaiwen Wei, Xinyi Zeng, Jingyuan Zhang, Yu Tian

TL;DR

Me-Agent tackles ambiguity in mobile instruction following by introducing two-level habit learning: prompt-level User Preference Learning with a Personal Reward Model and memory-level Hierarchical Preference Memory, enabling personalization without parameter updates. It leverages a Vision-Language Reward Model to score trajectories and extracts actionable experiences to steer future decisions, stored in a two-tier memory system that retrieves app-specific knowledge on demand. A new User FingerTip benchmark checks implicit preference reasoning under Type I and II ambiguous instructions, while experiments on the E-dataset show strong task performance (e.g., TCR 0.893, TSR 0.600) and superior personalization compared with baselines. Findings suggest Me-Agent provides robust, training-free personalization suitable for API-based deployment and demonstrates strong generalization across backbone models, with limitations around UI dynamics and contextual factors for future work.

Abstract

Large Language Model (LLM)-based mobile agents have made significant performance advancements. However, these agents often follow explicit user instructions while overlooking personalized needs, leading to significant limitations for real users, particularly without personalized context: (1) inability to interpret ambiguous instructions, (2) lack of learning from user interaction history, and (3) failure to handle personalized instructions. To alleviate the above challenges, we propose Me-Agent, a learnable and memorable personalized mobile agent. Specifically, Me-Agent incorporates a two-level user habit learning approach. At the prompt level, we design a user preference learning strategy enhanced with a Personal Reward Model to improve personalization performance. At the memory level, we design a Hierarchical Preference Memory, which stores users' long-term memory and app-specific memory in different level memory. To validate the personalization capabilities of mobile agents, we introduce User FingerTip, a new benchmark featuring numerous ambiguous instructions for daily life. Extensive experiments on User FingerTip and general benchmarks demonstrate that Me-Agent achieves state-of-the-art performance in personalization while maintaining competitive instruction execution performance.

Me-Agent: A Personalized Mobile Agent with Two-Level User Habit Learning for Enhanced Interaction

TL;DR

Me-Agent tackles ambiguity in mobile instruction following by introducing two-level habit learning: prompt-level User Preference Learning with a Personal Reward Model and memory-level Hierarchical Preference Memory, enabling personalization without parameter updates. It leverages a Vision-Language Reward Model to score trajectories and extracts actionable experiences to steer future decisions, stored in a two-tier memory system that retrieves app-specific knowledge on demand. A new User FingerTip benchmark checks implicit preference reasoning under Type I and II ambiguous instructions, while experiments on the E-dataset show strong task performance (e.g., TCR 0.893, TSR 0.600) and superior personalization compared with baselines. Findings suggest Me-Agent provides robust, training-free personalization suitable for API-based deployment and demonstrates strong generalization across backbone models, with limitations around UI dynamics and contextual factors for future work.

Abstract

Large Language Model (LLM)-based mobile agents have made significant performance advancements. However, these agents often follow explicit user instructions while overlooking personalized needs, leading to significant limitations for real users, particularly without personalized context: (1) inability to interpret ambiguous instructions, (2) lack of learning from user interaction history, and (3) failure to handle personalized instructions. To alleviate the above challenges, we propose Me-Agent, a learnable and memorable personalized mobile agent. Specifically, Me-Agent incorporates a two-level user habit learning approach. At the prompt level, we design a user preference learning strategy enhanced with a Personal Reward Model to improve personalization performance. At the memory level, we design a Hierarchical Preference Memory, which stores users' long-term memory and app-specific memory in different level memory. To validate the personalization capabilities of mobile agents, we introduce User FingerTip, a new benchmark featuring numerous ambiguous instructions for daily life. Extensive experiments on User FingerTip and general benchmarks demonstrate that Me-Agent achieves state-of-the-art performance in personalization while maintaining competitive instruction execution performance.
Paper Structure (26 sections, 14 equations, 4 figures, 13 tables)

This paper contains 26 sections, 14 equations, 4 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-based Mobile Agents
  4. Personalization in LLMs
  5. User FingerTip
  6. Data Construction
  7. Data Statistics
  8. Methodology
  9. Problem Formulation
  10. Me-Agent
  11. User Preference Learning
  12. Hierarchical Preference Memory
  13. Application Resolution.
  14. Experiments
  15. Experimental Setup
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Two key challenges for mobile agents in ambiguous instruction.
  • Figure 2: Overview of the proposed Me Agent framework.
  • Figure 3: Ablation Study on the User Fingertip.
  • Figure 4: Example cases of Me-Agent executing personalized mobile instructions