Learning with Challenges: Adaptive Difficulty-Aware Data Generation for Mobile GUI Agent Training

TL;DR

This paper addresses the challenge of obtaining high-quality GUI interaction data for mobile agents by proposing MobileGen, an adaptive data generation framework that aligns training difficulty with the agent’s current capabilities. It decouples trajectory difficulty into structural and semantic dimensions, profiles the agent’s abilities on a curated prior dataset, and uses an alpha-guided strategy to generate difficulty distributions that steer trajectory synthesis. A multi-agent controllable generator (MCG), comprising explorer, supervisor, and synthesizer components, creates high-quality, diverse trajectories with inverse synthesis to recover reasoning traces and instructions. Through extensive experiments on AndroidWorld, AndroidControl-Curated, and GUIOdyssey, MobileGen achieves consistent improvements over zero-shot baselines and existing data-synthesis methods, illustrating the value of capability-aligned data generation for training robust mobile GUI agents.

Abstract

Large-scale, high-quality interaction trajectories are essential for advancing mobile Graphical User Interface (GUI) agents. While existing methods typically rely on labor-intensive human demonstrations or automated model exploration to generate GUI trajectories, they lack fine-grained control over task difficulty. This fundamentally restricts learning effectiveness due to the mismatch between the training difficulty and the agent's capabilities. Inspired by how humans acquire skills through progressively challenging tasks, we propose MobileGen, a novel data generation framework that adaptively aligns training difficulty with the GUI agent's capability frontier. Specifically, MobileGen explicitly decouples task difficulty into structural (e.g., trajectory length) and semantic (e.g., task goal) dimensions. It then iteratively evaluates the agent on a curated prior dataset to construct a systematic profile of its capability frontier across these two dimensions. With this profile, the probability distribution of task difficulty is adaptively computed, from which the target difficulty for the next round of training can be sampled. Guided by the sampled difficulty, a multi-agent controllable generator is finally used to synthesize high-quality interaction trajectories along with corresponding task instructions. Extensive experiments show that MobileGen consistently outperforms existing data generation methods by improving the average performance of GUI agents by 1.57 times across multiple challenging benchmarks. This highlights the importance of capability-aligned data generation for effective mobile GUI agent training.

Figures (21)

• Figure 1: Overview of MobileGen. Our pipeline consists of three key stages: (1) Agent Capability Profiling: The student agent $\mathcal{M}_\text{student}$ is evaluated on a prior dataset to derive structural and semantic capability profiles. (2) Difficulty Distribution Generation: Based on the profiles, the challenge point is set to form the desired difficulty distribution. (3) Difficulty-Aware Trajectory Generation: Guided by the sampled difficulty parameters, two agents $\mathcal{M}_\text{explorer}$ and $\mathcal{M}_\text{supervisor}$ collaborate to generate interaction trajectories. The reasoning traces and instructions will then be reconstructed via inverse synthesis.
• Figure 2: Detailed workflow of MCG. Concretely, MCG generates difficulty-aware training trajectories controlled by sampled difficulty parameters through multi-agent collaboration. During Interaction Trajectory Generation, the supervisor $\mathcal{M}_{\text{supervisor}}$ allocates exploration step budgets for each application before the exploration begins and dynamically manages the explorer $\mathcal{M}_{\text{explorer}}$’s step usage across applications while performing rollbacks to correct interaction errors. Following exploration, the synthesizer $\mathcal{M}_{\text{synthesizer}}$ reconstructs step-level thoughts and trajectory-level instructions.
• Figure 3: Effectiveness of critical components of MobileGen. AndroidWorld is short for "AW", and AndroidControl-Curated is short for "AC".
• Figure 4: Evaluation of training under different challenge point setting. Success rate and training loss distribution are reported.
• Figure 5: Case study of semantic difficulty controllability. By visualizing trajectories generated in Simple Calendar Pro, we demonstrate the effectiveness of adjusting ICD and IUD parameters to control semantic complexity: (a) illustrates a setting modification task with low operational complexity and an ambiguous instruction; (b) presents a standard event creation task with medium difficulty in both linguistic logic and interaction; and (c) showcases a more complex event creation task that demands intricate operational requirements but straightforward instruction understanding.