Curiosity Driven Knowledge Retrieval for Mobile Agents

TL;DR

The paper tackles the challenge of incomplete knowledge and poor generalization in mobile agents by introducing a curiosity-driven framework that triggers external knowledge retrieval when epistemic uncertainty exceeds a threshold. Retrieved information is organized into AppCards, modular knowledge units that encode functional semantics, IO constraints, and interface mappings, and are selectively injected into the agent's reasoning to improve planning reliability. Empirical evaluation on the AndroidWorld DroidRun benchmark demonstrates consistent gains across backbones, achieving a new state-of-the-art SR of 88.8% with GPT-5; analyses show AppCards are especially beneficial for multi-step and cross-application tasks, though gains depend on the backbone model. The work provides a practical pathway toward knowledge-aware mobile agents and highlights important interactions between external knowledge and model capabilities, with future directions including adaptive triggering and AppCard versioning.

Abstract

Mobile agents have made progress toward reliable smartphone automation, yet performance in complex applications remains limited by incomplete knowledge and weak generalization to unseen environments. We introduce a curiosity driven knowledge retrieval framework that formalizes uncertainty during execution as a curiosity score. When this score exceeds a threshold, the system retrieves external information from documentation, code repositories, and historical trajectories. Retrieved content is organized into structured AppCards, which encode functional semantics, parameter conventions, interface mappings, and interaction patterns. During execution, an enhanced agent selectively integrates relevant AppCards into its reasoning process, thereby compensating for knowledge blind spots and improving planning reliability. Evaluation on the AndroidWorld benchmark shows consistent improvements across backbones, with an average gain of six percentage points and a new state of the art success rate of 88.8\% when combined with GPT-5. Analysis indicates that AppCards are particularly effective for multi step and cross application tasks, while improvements depend on the backbone model. Case studies further confirm that AppCards reduce ambiguity, shorten exploration, and support stable execution trajectories. Task trajectories are publicly available at https://lisalsj.github.io/Droidrun-appcard/.

Figures (7)

• Figure 1: Overall framework of the curiosity driven knowledge retrieval system for mobile agents. Task execution is guided by AppCards. Uncertainty estimation produces curiosity signals that activate external retrieval. Retrieved knowledge is consolidated to update AppCards and the updated AppCards are reintegrated into the execution pipeline.
• Figure 2: JS-divergence based curiosity estimation. The agent predicts the next interface state from the current state and action as a prior distribution, then observes the next state as a posterior distribution. The divergence between these distributions is measured with a tail adjusted Jensen Shannon divergence, yielding an information gain signal quantifying curiosity.
• Figure 3: An example of aggregated global semantic distributions with Top-19 tokens and the residual OTHER category
• Figure 4: An example of token-level contributions to the adjusted JS divergence, showing the top 19 tokens and the residual OTHER category.
• Figure 5: Case study of the task Expense Add Multiple From Gallery. The baseline path on the left fails due to application name ambiguity, while the AppCard enhanced path on the right leverages structured knowledge to enable stable and successful task execution.