Adaptive Milestone Reward for GUI Agents

TL;DR

ADMIRE tackles the temporal credit assignment problem in long-horizon mobile GUI tasks by introducing Adaptive Milestone Reward, which jointly learns adaptive milestone checkpoints and applies asymmetric credit to positive and negative trajectories. Milestones are generated from successful explorations via a generative abstraction function and updated as policies improve, enabling dense yet verifiable rewards through semantic matching of actions to milestones. Empirical results on AndroidWorld and cross-domain benchmarks show consistent improvements over outcome- and process-based rewards and strong generalization across RL algorithms (GRPO, RLOO, DAPO) and domains (ALFWorld, WebShop). The approach is lightweight in computation, with robust milestone quality and coverage, offering a scalable, algorithm-agnostic paradigm for training long-horizon GUI agents.

Abstract

Reinforcement Learning (RL) has emerged as a mainstream paradigm for training Mobile GUI Agents, yet it struggles with the temporal credit assignment problem inherent in long-horizon tasks. A primary challenge lies in the trade-off between reward fidelity and density: outcome reward offers high fidelity but suffers from signal sparsity, while process reward provides dense supervision but remains prone to bias and reward hacking. To resolve this conflict, we propose the Adaptive Milestone Reward (ADMIRE) mechanism. ADMIRE constructs a verifiable, adaptive reward system by anchoring trajectory to milestones, which are dynamically distilled from successful explorations. Crucially, ADMIRE integrates an asymmetric credit assignment strategy that denoises successful trajectories and scaffolds failed trajectories. Extensive experiments demonstrate that ADMIRE consistently yields over 10% absolute improvement in success rate across different base models on AndroidWorld. Moreover, the method exhibits robust generalizability, achieving strong performance across diverse RL algorithms and heterogeneous environments such as web navigation and embodied tasks.

Figures (19)

• Figure 1: Comparison of different reward mechanisms. Milestones are identified as key state transitions to enable verifiable and interpretable reward triggering.
• Figure 2: The overall framework of ADMIRE. Circles $s_i$ represent trajectory steps, while diamond markers $m_i$ denote task milestones.
• Figure 3: Comparison of success rates across different task difficulties on AndroidWorld for the base model and 7B variants trained with Outcome Reward, Process Reward, and ADMIRE. Results for the 3B model are presented in Figure \ref{['fig:difficulty_3B']}.
• Figure 4: Ablation results comparing different ADMIRE-based reward designs and their effects on model performance. Detailed definitions of each ablation component can be found in the Appendix \ref{['app:ablation_definitions']}.
• Figure 5: Comparison of success rates across different task difficulties on AndroidWorld for the base model and models trained with outcome, process reward, and ADMIRE.