Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation

TL;DR

The paper addresses safety and efficiency challenges in online GUI automation by introducing a pre-operative critic that evaluates proposed actions before execution. It presents GUI-Critic-R1, trained with a novel Suggestion-aware Group Relative Policy Optimization (S-GRPO) and a reasoning-bootstrapping data pipeline to generate high-quality critiques and remedial suggestions. Empirical results show strong static critic accuracy across mobile and web domains and improved dynamic performance on the AndroidWorld benchmark, including higher success rates and reduced steps. The work provides a concrete framework and datasets (GUI-Critic-Train/Test) that advance reliable, real-time GUI reasoning and action selection, with potential for lighter models and trajectory-level critiques in future work.

Abstract

In recent years, Multimodal Large Language Models (MLLMs) have been extensively utilized for multimodal reasoning tasks, including Graphical User Interface (GUI) automation. Unlike general offline multimodal tasks, GUI automation is executed in online interactive environments, necessitating step-by-step decision-making based on real-time status of the environment. This task has a lower tolerance for decision-making errors at each step, as any mistakes may cumulatively disrupt the process and potentially lead to irreversible outcomes like deletions or payments. To address these issues, we introduce a pre-operative critic mechanism that provides effective feedback prior to the actual execution, by reasoning about the potential outcome and correctness of actions. Specifically, we propose a Suggestion-aware Gradient Relative Policy Optimization (S-GRPO) strategy to construct our pre-operative critic model GUI-Critic-R1, incorporating a novel suggestion reward to enhance the reliability of the model's feedback. Furthermore, we develop a reasoning-bootstrapping based data collection pipeline to create a GUI-Critic-Train and a GUI-Critic-Test, filling existing gaps in GUI critic data. Static experiments on the GUI-Critic-Test across both mobile and web domains reveal that our GUI-Critic-R1 offers significant advantages in critic accuracy compared to current MLLMs. Dynamic evaluation on GUI automation benchmark further highlights the effectiveness and superiority of our model, as evidenced by improved success rates and operational efficiency.

Figures (8)

• Figure 1: (a) shows an example of GUI automation. Case studies in (b-c) demonstrate how pre-critic prevents erroneous and redundant actions in GUI automation. (d) illustrates the quantitative performance comparison between pre-critic methods and baseline on AndroidWorld rawles2024androidworld.
• Figure 2: The left shows our reasoning-bootstrapping based data collection pipeline, including the GUI operations collection and GUI critiques generation. Specifically, a progressive CoT paradigm and a reasoning-bootstrapping strategy are employed to ensure the quality of critiques. The right illustrates the training strategy for our GUI-Critic-R1 model. The process begins with a RFT cold-start on the GUI-Critic-Train dataset, and followed by the implementation of our proposed S-GRPO. Besides, a novel suggestion reward is employed to constrain the correctness of suggestions.
• Figure 3: Analysis of suggestion reward weight $\lambda_{s}$ and Group Size on the GUI-I setting.
• Figure 4: A case of pre-critic on GUI automation.
• Figure 5: An instance of GUI-Critic-R1 avoiding a redundant action within GUI automation through a pre-operative critique approach.