AndroidControl-Curated: Revealing the True Potential of GUI Agents through Benchmark Purification

TL;DR

This work identifies systemic flaws in the AndroidControl GUI benchmark that artificially understate on-device GUI agents. It introduces AndroidControl-Curated, a two-stage purification pipeline that shifts grounding evaluation to bounding-box-based intent alignment and applies LLM-assisted, human-verified task corrections. Leveraging this purified benchmark, the authors train Magma-R1, a compact 3B model, with GRPO using only ~2.4k curated samples, achieving performance rivaling substantially larger models on challenging GUI tasks. The results demonstrate that on-device GUI agents are closer to practical deployment than previously believed, and the authors provide open-source access to the purified benchmark and Magma-R1 to accelerate progress in robust, offline GUI automation.

Abstract

On-device virtual assistants like Siri and Google Assistant are increasingly pivotal, yet their capabilities are hamstrung by a reliance on rigid, developer-dependent APIs. GUI agents offer a powerful, API-independent alternative, but their adoption is hindered by the perception of poor performance, as even the best models (e.g. Qwen3-VL-235B) scores are capped at around 60% on benchmarks like AndroidControl, far from viability for real-world use. Our research reveals that issue lies not only with the models but with the benchmarks themselves. We identified notable shortcomings in AndroidControl, including ambiguities and factual errors, which systematically underrates agent capabilities. To address this critical oversight, we enhanced AndroidControl into AndroidControl-Curated, a refined version of the benchmark improved through a rigorous purification pipeline. On this enhanced benchmark, state-of-the-art models achieve success rates nearing 75% on complex tasks (15% improvement), reflecting that on-device GUI agents are actually closer to practical deployment than previously thought. We introduce our new SOTA model, Magma-R1- 3B, post-trained on just 2.4k curated samples using 60 hours of an H20 GPU (approximately $60). Despite being 200 times smaller in parameters, this model delivers performance comparable to Qwen3- VL-235B. We release both AndroidControl-Curated benchmark and Magma-R1 model to the research community, encouraging adoption of this enhanced benchmark to better reflect model capabilities and accelerate the development of robust, on-device virtual assistants.

Figures (4)

• Figure 1: Benchmark Quality Reveals True Model Capabilities. Performance comparison on AndroidControl (original) vs AndroidControl-Curated (purified). Our benchmark purification dramatically improves evaluation accuracy, revealing that compact 3B models (Magma-R1: 75.3%) achieve performance remarkably close to large 235B models (Qwen3-VL-235B: 76.5%) on challenging GUI tasks, up from an upper limit of around 60% of even the world best models. This demonstrates that previous low performance scores were largely due to benchmark deficiencies rather than fundamental model limitations.
• Figure 2: Overview of our integrated pipeline. The upper part shows the Magma-R1 training process with GRPO training, featuring 2D Gaussian reward optimization and ratio optimization for balanced learning. The lower part illustrates the AndroidControl benchmark refinement process, including grounding optimization, multi-model filtering, LLM review and rewrite, leading to the creation of AndroidControl-Curated.
• Figure 3: Illustration of our grounding optimization approach. We transform the evaluation from exact point matching to bounding box-based intent alignment. The example shows a GUI task where the agent needs to interact with a UI element, and our method evaluates whether the predicted point falls within the target element's bounding box rather than matching exact coordinates.
• Figure 4: Case Study: Three Major Categories of Benchmark Deficiencies. We show representative examples of the systematic errors found in the original AndroidControl benchmark: (Top) Unclear Task - vague task descriptions that lack clear navigation instructions; (Middle) Multiple Valid Actions - multiple valid actions for achieving the same goal; (Bottom) Wrong Ground Truth - incorrect labels that contradict the current UI state. Our AndroidControl-Curated corrections address each category by providing clearer instructions, accepting multiple valid actions, and fixing factual errors.