Quick on the Uptake: Eliciting Implicit Intents from Human Demonstrations for Personalized Mobile-Use Agents

Abstract

As multimodal large language models advance rapidly, the automation of mobile tasks has become increasingly feasible through the use of mobile-use agents that mimic human interactions from graphical user interface. To further enhance mobile-use agents, previous studies employ demonstration learning to improve mobile-use agents from human demonstrations. However, these methods focus solely on the explicit intention flows of humans (e.g., step sequences) while neglecting implicit intention flows (e.g., personal preferences), which makes it difficult to construct personalized mobile-use agents. In this work, to evaluate the \textbf{I}ntention \textbf{A}lignment \textbf{R}ate between mobile-use agents and humans, we first collect \textbf{MobileIAR}, a dataset containing human-intent-aligned actions and ground-truth actions. This enables a comprehensive assessment of the agents' understanding of human intent. Then we propose \textbf{IFRAgent}, a framework built upon \textbf{I}ntention \textbf{F}low \textbf{R}ecognition from human demonstrations. IFRAgent analyzes explicit intention flows from human demonstrations to construct a query-level vector library of standard operating procedures (SOP), and analyzes implicit intention flows to build a user-level habit repository. IFRAgent then leverages a SOP extractor combined with retrieval-augmented generation and a query rewriter to generate personalized query and SOP from a raw ambiguous query, enhancing the alignment between mobile-use agents and human intent. Experimental results demonstrate that IFRAgent outperforms baselines by an average of 6.79\% (32.06\% relative improvement) in human intention alignment rate and improves step completion rates by an average of 5.30\% (26.34\% relative improvement). The codes are available at https://github.com/MadeAgents/Quick-on-the-Uptake.

Figures (8)

• Figure 1: Comparing IFRAgent with normal mobile-use agents and existing demonstration learning methods for mobile-use agents. IFRAgent considers both explicit intention flow and implicit intention flow, enabling it to capture more personalized information, such as taste preferences.
• Figure 2: Examples of human-intent-aligned actions and SR ground truth lists. Even for the same query, the human-intent-aligned actions of user 1, user 2, and user 3 are different. However, these actions, along with some other actions that are also beneficial for completing the query, all belong to the SR ground truth list.
• Figure 3: Construction of the MobileIAR dataset. In Round 1, annotators independently label human intent-aligned actions. In Round 2, groups of three participants are formed to jointly label the SR (step-wise success rate) ground truth lists based on the trajectories obtained from Round 1.
• Figure 4: The workflow pipeline of IFRAgent. The phase above the dashed line represents the intention flow extraction phase, while the phase below the dashed line represents the deployment phase. In the intention flow extraction phase, IFRAgent extracts SOPs and user profiles based on users' trajectories. In the deployment phase, IFRAgent uses this information to rewrite personalized queries and generate personalized SOPs.
• Figure 5: Experimental results of ablation study.