Agent-Initiated Interaction in Phone UI Automation

TL;DR

This work formalizes agent-initiated interaction in phone UI automation by defining the need-for-interaction task and message generation, and introduces AndroidInteraction, a diverse dataset built from AndroidControl episodes. It evaluates text-based and multimodal baselines, revealing that current large language models struggle to accurately detect when user input is required and to craft appropriate prompts. The study analyzes data quality, annotation subjectivity, and interaction typologies, underscoring the importance of timely, context-aware user engagement for aligning agent behavior with user preferences. The results offer a foundation for developing personalized, user-centered phone automation and suggest future work on multi-turn interactions, memory integration, and larger-scale data collection.

Abstract

Phone automation agents aim to autonomously perform a given natural-language user request, such as scheduling appointments or booking a hotel. While much research effort has been devoted to screen understanding and action planning, complex tasks often necessitate user interaction for successful completion. Aligning the agent with the user's expectations is crucial for building trust and enabling personalized experiences. This requires the agent to proactively engage the user when necessary, avoiding actions that violate their preferences while refraining from unnecessary questions where a default action is expected. We argue that such subtle agent-initiated interaction with the user deserves focused research attention. To promote such research, this paper introduces a task formulation for detecting the need for user interaction and generating appropriate messages. We thoroughly define the task, including aspects like interaction timing and the scope of the agent's autonomy. Using this definition, we derived annotation guidelines and created AndroidInteraction, a diverse dataset for the task, leveraging an existing UI automation dataset. We tested several text-based and multimodal baseline models for the task, finding that it is very challenging for current LLMs. We suggest that our task formulation, dataset, baseline models and analysis will be valuable for future UI automation research, specifically in addressing this crucial yet often overlooked aspect of agent-initiated interaction. This work provides a needed foundation to allow personalized agents to properly engage the user when needed, within the context of phone UI automation.

Figures (6)

• Figure 1: An illustrating example of an instruction requiring user interaction in phone UI automation. The user requests to cancel their meeting tomorrow, unaware that there are two meetings scheduled for the next day. See figure \ref{['fig:scratch_episode']} and appendix \ref{['app:additional_examples']} for additional examples.
• Figure 2: First steps of an example episode from the AndroidControl dataset. The user instruction is to book a train from London to Brighton on a specific date using the Omio app. The 4th step was labelled as requiring user interaction in our dataset, with the message: "From which London station would you like to depart?".
• Figure 3: Distribution of (a) the episodes length and (b) the interactions necessity scores in AndroidInteraction. (c) shows the distribution of the domains for the various apps in the dataset
• Figure 4: Example episode from the AndroidControl dataset, with user instruction "I want biking directions as I am getting late to reach my destination". The last step was annotated as requiring user interaction in AndroidInteraction, with the message: "Seems like Bike route is not available for this journey".
• Figure 5: Example episode from the AndroidControl dataset, with user instruction "As I have a short circuit, I'd want to search for copper wire Round in the Flipkart app.". The last step was annotated as requiring user interaction in AndroidInteraction, with the message: "What type of Copper Wire you are looking for?".