ViMo: A Generative Visual GUI World Model for App Agents

TL;DR

ViMo introduces a visual GUI world model that predicts future App observations as images by decoupling graphic and text content through Symbolic Text Representation (STR). A diffusion-based STR Predictor generates the next GUI structure while a GUI-text Predictor (LLM) fills in the corresponding text, enabling more accurate long-horizon planning for App agents. Across world-model quality and agent-performance metrics, ViMo achieves substantial gains, including automatic metric improvements (~29%), higher user-study scores (~183%), and improved online navigation performance, while demonstrating zero-shot generalization to unseen Apps. The approach offers a practical, deployable solution for realistic GUI prediction and decision-making in mobile environments.

Abstract

App agents, which autonomously operate mobile Apps through Graphical User Interfaces (GUIs), have gained significant interest in real-world applications. Yet, they often struggle with long-horizon planning, failing to find the optimal actions for complex tasks with longer steps. To address this, world models are used to predict the next GUI observation based on user actions, enabling more effective agent planning. However, existing world models primarily focus on generating only textual descriptions, lacking essential visual details. To fill this gap, we propose ViMo, the first visual world model designed to generate future App observations as images. For the challenge of generating text in image patches, where even minor pixel errors can distort readability, we decompose GUI generation into graphic and text content generation. We propose a novel data representation, the Symbolic Text Representation~(STR) to overlay text content with symbolic placeholders while preserving graphics. With this design, ViMo employs a STR Predictor to predict future GUIs' graphics and a GUI-text Predictor for generating the corresponding text. Moreover, we deploy ViMo to enhance agent-focused tasks by predicting the outcome of different action options. Experiments show ViMo's ability to generate visually plausible and functionally effective GUIs that enable App agents to make more informed decisions.

Figures (11)

• Figure 1: GUIs generated by image-based methods (UI-Diffuser ai-diffuser, TextDiffuser-2 text-diffuser2, and IP2P ip2p fine-tuned on GUI dataset, denoted as IP2P*). Images are cropped for space efficiency.
• Figure 2: Framework of our ViMo. We first detect text content (actual words) in the current GUI ($x_k$) and overlay it with text symbols (rectangle-shaped placeholders with a black border and white fill), to create $\text{STR}_{x_k}$. Then $\text{STR}_{x_k}$ and the user action ($a$) are input to the STR predictor to generate the STR of the next GUI ($\text{STR}_{{x^a_{k+1}}}$). Next, text symbols within $\text{STR}_{{x^a_{k+1}}}$ are located and assigned unique ID token. Then the LLM predicts the text content corresponding to each token. Finally, the next GUI image is constructed by overlaying the predicted text into the STR.
• Figure 3: GUI generation comparison in graphic generation (Top) and text generation (Bottom).
• Figure 4: Qualitative ablation studies. Left: Static text generation. (a) Generating static text via an LLM; (b) Preserving the original text in the image by rendering it as image pixels. Right: STR generation under two input formats—(a) action command and (b) action instruction.
• Figure 5: GUI generation conditioned on a novel combination of current GUI observation and user action.