MobileDreamer: Generative Sketch World Model for GUI Agent

TL;DR

MobileDreamer tackles long-horizon mobile GUI automation by introducing a lightweight Textual Sketch World Model that forecasts action-conditioned future GUI states as structured text. A rollout imagination module then performs multi-step lookahead via a tree of predicted trajectories, feeding back forecasts to guide the agent's decisions. Empirical results on Android World show that the sketch-based forecasting outperforms text- and image-based baselines and yields consistent task-success improvements across diverse backbones. The approach demonstrates that efficient, geometry-aware world models can significantly enhance proactive planning in mobile GUI automation, with practical implications for automation, testing, and user-device interactions.

Abstract

Mobile GUI agents have shown strong potential in real-world automation and practical applications. However, most existing agents remain reactive, making decisions mainly from current screen, which limits their performance on long-horizon tasks. Building a world model from repeated interactions enables forecasting action outcomes and supports better decision making for mobile GUI agents. This is challenging because the model must predict post-action states with spatial awareness while remaining efficient enough for practical deployment. In this paper, we propose MobileDreamer, an efficient world-model-based lookahead framework to equip the GUI agents based on the future imagination provided by the world model. It consists of textual sketch world model and rollout imagination for GUI agent. Textual sketch world model forecasts post-action states through a learning process to transform digital images into key task-related sketches, and designs a novel order-invariant learning strategy to preserve the spatial information of GUI elements. The rollout imagination strategy for GUI agent optimizes the action-selection process by leveraging the prediction capability of world model. Experiments on Android World show that MobileDreamer achieves state-of-the-art performance and improves task success by 5.25%. World model evaluations further verify that our textual sketch modeling accurately forecasts key GUI elements.

Figures (4)

• Figure 1: MobileDreamer trains a world model to predict future GUI states and performs rollout imagination to select actions from candidates. It consistently improves multiple LLM backbones on Android World.
• Figure 2: Overview of the MobileDreamer framework. In stage 1, we build a textual sketch world model to predict future states through order-invariant learning. In stage 2, we design a rollout imagination strategy that recursively feeds the tree-of-prediction back to the GUI agent to optimize the action-selection processes.
• Figure 3: Comparison between the real GUI screenshot and the sketch predicted by our world model. The predicted sketch captures key UI elements with texts and positions. For clarity, we display the element text outside the corresponding element boxes.
• Figure 4: Case study of MobileDreamer on a timer-creation task.