M$^2$-Miner: Multi-Agent Enhanced MCTS for Mobile GUI Agent Data Mining

TL;DR

The paper tackles the scarcity and cost of high-quality mobile GUI intent-trajectory data essential for training GUI agents. It introduces M$^2$-Miner, a Monte Carlo Tree Search–based framework with a collaborative InferAgent, OrchestraAgent, and JudgeAgent, augmented by an intent recycling mechanism and a progressive model-in-the-loop training strategy to produce diverse, high-quality trajectories at reduced cost. Key contributions include (1) a formal intent-trajectory tree representation, (2) a multi-agent expansion/simulation pipeline that improves mining efficiency, (3) an intent recycling method to harvest additional data from non-primary paths, and (4) staged model-in-the-loop training that enhances performance in unseen environments. Empirical results across multiple mobile GUI benchmarks demonstrate state-of-the-art performance and substantial data-production cost savings, underscoring the framework’s practical impact for GUI research and development.

Abstract

Graphical User Interface (GUI) agent is pivotal to advancing intelligent human-computer interaction paradigms. Constructing powerful GUI agents necessitates the large-scale annotation of high-quality user-behavior trajectory data (i.e., intent-trajectory pairs) for training. However, manual annotation methods and current GUI agent data mining approaches typically face three critical challenges: high construction cost, poor data quality, and low data richness. To address these issues, we propose M$^2$-Miner, the first low-cost and automated mobile GUI agent data-mining framework based on Monte Carlo Tree Search (MCTS). For better data mining efficiency and quality, we present a collaborative multi-agent framework, comprising InferAgent, OrchestraAgent, and JudgeAgent for guidance, acceleration, and evaluation. To further enhance the efficiency of mining and enrich intent diversity, we design an intent recycling strategy to extract extra valuable interaction trajectories. Additionally, a progressive model-in-the-loop training strategy is introduced to improve the success rate of data mining. Extensive experiments have demonstrated that the GUI agent fine-tuned using our mined data achieves state-of-the-art performance on several commonly used mobile GUI benchmarks. Our work will be released to facilitate the community research.

Figures (24)

• Figure 1: Left: Different GUI data structure. (a) Flat human-annotated GUI data, which stores only the single trajectory corresponding to each intent. (b) Vanilla tree-structured GUI agent data, where each tree is associated with a single intent. (c) Our multi-intent tree structure, where each tree contains multiple intents and their corresponding trajectories. Right: t-SNE Visualization of intent distributions. Compared to existing methods, our intent recycling strategy produces a more diverse set of intents from the same initial intent.
• Figure 1: Statistics of different datasets. Size: number of images; Su.RL: suitability for reinforcement learning algorithms; Auto: auto-annotated data; AvgStp: average step length; Trajs: number of trajectories; Cost: estimated data production cost (USD). Cost per image: estimated cost of producing one image (USD).
• Figure 2: Overview. (a) The mining process of M$^2$-Miner consists of four phases: Selection, Expansion, Simulation, and Backpropagation. During these phases, InferAgent, OrchestraAgent, and JudgeAgent are employed for exploration guidance, acceleration, and state evaluation. (b) Intent recycling process. For a selected node, its corresponding trajectory is first passed through a dedicated intent-recycling filter, then a novel intent is generated via MLLM. If the JudgeAgent verifies that the generated intent aligns with the trajectory, the recycling is considered complete.
• Figure 3: Intent Generation. Our intent generation method consists of three stages: basic intents generation, complex intents generation, and intent recycling.
• Figure 4: Infrastructure Framework. A layered framework for GUI agent data mining with model-in-the-loop training, comprising data, engine, algorithm, agent, execution, and environment layers.