DigiData: Training and Evaluating General-Purpose Mobile Control Agents

TL;DR

DigiData presents a large-scale, multi-modal dataset (DigiData) for training general-purpose mobile control agents, constructed via exhaustive feature exploration to yield deep, diverse goals. Accompanied by DigiData-Bench, a real-world benchmark, the work argues that traditional step-accuracy assessments are insufficient and introduces dynamic human- and AI-assisted evaluation protocols, including LLM judges and Chain-of-Thought annotations. Through extensive experiments, the authors show that training with DigiData improves agent performance, especially when combined with CoT data, and that AI judges can closely track human judgments, enabling scalable evaluation. The dataset and benchmark collectively aim to enable more capable, explainable, and scalable mobile control agents with broad practical impact.

Abstract

AI agents capable of controlling user interfaces have the potential to transform human interaction with digital devices. To accelerate this transformation, two fundamental building blocks are essential: high-quality datasets that enable agents to achieve complex and human-relevant goals, and robust evaluation methods that allow researchers and practitioners to rapidly enhance agent performance. In this paper, we introduce DigiData, a large-scale, high-quality, diverse, multi-modal dataset designed for training mobile control agents. Unlike existing datasets, which derive goals from unstructured interactions, DigiData is meticulously constructed through comprehensive exploration of app features, resulting in greater diversity and higher goal complexity. Additionally, we present DigiData-Bench, a benchmark for evaluating mobile control agents on real-world complex tasks. We demonstrate that the commonly used step-accuracy metric falls short in reliably assessing mobile control agents and, to address this, we propose dynamic evaluation protocols and AI-powered evaluations as rigorous alternatives for agent assessment. Our contributions aim to significantly advance the development of mobile control agents, paving the way for more intuitive and effective human-device interactions.

Figures (10)

• Figure 1: Visualization of different features of existing mobile control datasets. DigiData constitutes a step change in terms of goal depth, being the first large-scale dataset obtained by comprehensive exploration of the functionalities of mobile device apps.
• Figure 2: A representation of our data collection pipeline. For each app, our pipeline includes three phases. In the first phase (goal curation), human workers exhaustively explore the app and curate a list of goals that attempts to cover all of its features. In the second phase (demonstrations collection), human annotators create a set of demonstrations, generating trajectories that achieve the specified goals. In the third phase (trajectory verification), trajectories that do not achieve their corresponding goal are filtered out of the dataset, by a verification system based on a combination of LLMs and humans. Overall, this pipeline allows to collect in-depth and high-quality mobile control data.
• Figure 3: (a) Percentage of data distribution for DigiData's top apps. The dataset presents no major imbalance towards specific apps. (b) Comparison of distribution of pairwise cosine distances across datasets. DigiData exhibits the largest degree of goal diversity, especially compared to AitW.
• Figure 4: A visual representation of the evaluations on DigiData-Bench. Following a goal-specific protocol, a human worker initializes the app for meaningfully achieve the goal, monitors the agent executing actions and detects whether the agent was successful in achieving the goal.
• Figure 5: (Left) Comparison of evaluation metrics across a number of models. Rankings provided by step accuracy are generally not reliable. (Right) Success rate by app category. Training agents with more data significantly improves performance on seen and familiar apps, but not on novel apps.