Hi-Agent: Hierarchical Vision-Language Agents for Mobile Device Control

TL;DR

Hi-Agent addresses the generalization limitations of flat state-to-action mappings in mobile UI control by introducing a trainable hierarchical vision-language agent that jointly optimizes a high-level subgoal planner and a low-level action executor. It reformulates long-horizon decision-making as a sequence of single-step subgoals and employs a foresight advantage to propagate low-level execution feedback to high-level learning, reducing the GRPO path explosion from $G^n$ to $n \cdot G$. The approach achieves state-of-the-art task success on Android-in-the-Wild with $87.9\%$ on the General subset and demonstrates strong generalization on ScreenSpot-v2 and scalability on AndroidWorld with larger backbones. This work provides a practical, data-efficient framework for robust mobile-device automation with hierarchical reasoning and end-to-end adaptation.

Abstract

Building agents that autonomously operate mobile devices has attracted increasing attention. While Vision-Language Models (VLMs) show promise, most existing approaches rely on direct state-to-action mappings, which lack structured reasoning and planning, and thus generalize poorly to novel tasks or unseen UI layouts. We introduce Hi-Agent, a trainable hierarchical vision-language agent for mobile control, featuring a high-level reasoning model and a low-level action model that are jointly optimized. For efficient training, we reformulate multi-step decision-making as a sequence of single-step subgoals and propose a foresight advantage function, which leverages execution feedback from the low-level model to guide high-level optimization. This design alleviates the path explosion issue encountered by Group Relative Policy Optimization (GRPO) in long-horizon tasks and enables stable, critic-free joint training. Hi-Agent achieves a new State-Of-The-Art (SOTA) 87.9% task success rate on the Android-in-the-Wild (AitW) benchmark, significantly outperforming prior methods across three paradigms: prompt-based (AppAgent: 17.7%), supervised (Filtered BC: 54.5%), and reinforcement learning-based (DigiRL: 71.9%). It also demonstrates competitive zero-shot generalization on the ScreenSpot-v2 benchmark. On the more challenging AndroidWorld benchmark, Hi-Agent also scales effectively with larger backbones, showing strong adaptability in high-complexity mobile control scenarios.

Figures (5)

• Figure 1: Different paradigms for mobile control agents. (a) Flat Agents use a single trainable model for state-to-(reason-)action mapping. (b) Hierarchical Agents use a planner to improve reasoning, but it is typically a frozen black-box. (c) Hi-Agent (Ours) enables joint optimization where the high-level reasoning and low-level action models are co-adapted and fully trainable.
• Figure 2: Illustration of recursive task decomposition under a hierarchical policy.
• Figure 3: Hierarchical Policy Optimization.(a) Standard GRPO incurs exponential sample complexity ($G^n$) and lacks intermediate reward signals in long-horizon tasks. (b) Hi-Agent reduces complexity to $n \cdot G$ by decoupling subgoal generation from execution, and enables efficient joint training through foresight-guided subgoal evaluation.
• Figure 4: Robustness analysis on AitW. (a) Over 70% of General tasks are search-based, causing prior RL methods to overfit. (b) Layout shift from home screen to all-apps view alters app positions. (c) Hi-Agent remains robust (87.9% $\rightarrow$ 83.2%), while DigiRL drops sharply (71.9% $\rightarrow$ 27.6%).
• Figure 5: Effectiveness and efficiency of Hi-Agent. (a) Task success across model scales and training algorithms, showing consistent gains from hierarchical modeling and post-training. (b) Training curves under different group sizes $G$; larger $G$ improves stability and speeds up convergence.