GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior

TL;DR

GUI-Reflection presents a comprehensive framework to endow multimodal GUI agents with self-reflection and error-correction capabilities. It integrates reflection-oriented training across GUI-specific pre-training, offline SFT, and online reflection tuning, and introduces a dedicated GUI-Reflection Task Suite to explicitly cultivate these abilities. The approach employs automatic data generation, an online mobile training environment, and an iterative tuning algorithm to continually improve verification, undo, and informed retry behaviors. Empirical results show notable gains from incorporating reflection data and online tuning, with strong performance on the GUI-Reflection Task Suite and competitive results on real-world Android benchmarks. This work advances robust, adaptable GUI automation by enabling agents to recognize mistakes, backtrack, and learn from errors in a self-guided manner.

Abstract

Multimodal Large Language Models (MLLMs) have shown great potential in revolutionizing Graphical User Interface (GUI) automation. However, existing GUI models mostly rely on learning from nearly error-free offline trajectories, thus lacking reflection and error recovery capabilities. To bridge this gap, we propose GUI-Reflection, a novel framework that explicitly integrates self-reflection and error correction capabilities into end-to-end multimodal GUI models throughout dedicated training stages: GUI-specific pre-training, offline supervised fine-tuning (SFT), and online reflection tuning. GUI-reflection enables self-reflection behavior emergence with fully automated data generation and learning processes without requiring any human annotation. Specifically, 1) we first propose scalable data pipelines to automatically construct reflection and error correction data from existing successful trajectories. While existing GUI models mainly focus on grounding and UI understanding ability, we propose the GUI-Reflection Task Suite to learn and evaluate reflection-oriented abilities explicitly. 2) Furthermore, we built a diverse and efficient environment for online training and data collection of GUI models on mobile devices. 3) We also present an iterative online reflection tuning algorithm leveraging the proposed environment, enabling the model to continuously enhance its reflection and error correction abilities. Our framework equips GUI agents with self-reflection and correction capabilities, paving the way for more robust, adaptable, and intelligent GUI automation, with all data, models, environments, and tools to be released publicly.

Figures (10)

• Figure 1: Illustrative comparison of typical GUI models versus our proposed GUI model with self-reflection behaviors. While current models fail to recognize and recover from errors (left), our model (right) demonstrates the ability to: 1. Recognize its mistake; 2. Undo the incorrect action and get back on track; 3. Summarize the mistake and make another try, ultimately succeeding.
• Figure 2: The GUI-Reflection framework includes (1) Learning basic reflection-oriented skills from GUI-Reflection Task Suite in the GUI pre-training stage; (2) Learning reflection and correction behaviours from automatically generated error scenarios in the offline SFT stage; (3) Continuously enhancing reflection and correction capabilities via reflection tuning in the online learning stage.
• Figure 2: Ablation study on reflection data in SFT and online training.
• Figure 3: Examples of Action Verification (left), Action Reversal (middle), and Mistake-informed Reattempt (right) tasks from the GUI-Reflection Task Suite.
• Figure 3: Comparison of our model against other baselines on AndroidWorld, showing Success Rates (SR). Acc. Tree denotes Accessibility Tree. The best number in 8B-scale end-to-end models is marked in bold.