Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MagicGUI-RMS: A Multi-Agent Reward Model System for Self-Evolving GUI Agents via Automated Feedback Reflux

Zecheng Li, Zhihui Cao, Wenke Huang, Yudong Zhang, Keying Qi, Rui Wang, Zeyu Zheng, Jian Zhao, Hao Zhu, Hengxin Wu, Yuran Wang, Guitao Fan, Guokun Wu, Yicong Liu, Zhilin Gao, Haikun Xu, He Yang, Minqi Xiang, Xingyu Liu, Zuojian Wang

TL;DR

GUI agents face challenges in automated trajectory evaluation and scalable data generation for continual improvement. MagicGUI-RMS introduces a dual reward system consisting of a Domain-Specific Reward Model and a General-Purpose Reward Model, coupled with a structured reward-data pipeline and a reward-guided reflux mechanism to drive self-evolution. Key contributions include a hierarchical reward evaluation architecture, a scalable data synthesis procedure with rule-based validation and structured perturbations, intention-centric grounding corrections, and a closed-loop data reflux that co-evolves rewards and policies. Empirical results across offline benchmarks and real-world tasks demonstrate improved task accuracy, robustness, and convergence, with strong generalization to out-of-domain GUI tasks. The framework offers a scalable foundation for autonomous, generalizable GUI agents that continuously improve through reward-driven adaptation.

Abstract

Graphical user interface (GUI) agents are rapidly progressing toward autonomous interaction and reliable task execution across diverse applications. However, two central challenges remain unresolved: automating the evaluation of agent trajectories and generating high-quality training data at scale to enable continual improvement. Existing approaches often depend on manual annotation or static rule-based verification, which restricts scalability and limits adaptability in dynamic environments. We present MagicGUI-RMS, a multi-agent reward model system that delivers adaptive trajectory evaluation, corrective feedback, and self-evolving learning capabilities. MagicGUI-RMS integrates a Domain-Specific Reward Model (DS-RM) with a General-Purpose Reward Model (GP-RM), enabling fine-grained action assessment and robust generalization across heterogeneous GUI tasks. To support reward learning at scale, we design a structured data construction pipeline that automatically produces balanced and diverse reward datasets, effectively reducing annotation costs while maintaining sample fidelity. During execution, the reward model system identifies erroneous actions, proposes refined alternatives, and continuously enhances agent behavior through an automated data-reflux mechanism. Extensive experiments demonstrate that MagicGUI-RMS yields substantial gains in task accuracy, behavioral robustness. These results establish MagicGUI-RMS as a principled and effective foundation for building self-improving GUI agents driven by reward-based adaptation.

MagicGUI-RMS: A Multi-Agent Reward Model System for Self-Evolving GUI Agents via Automated Feedback Reflux

TL;DR

GUI agents face challenges in automated trajectory evaluation and scalable data generation for continual improvement. MagicGUI-RMS introduces a dual reward system consisting of a Domain-Specific Reward Model and a General-Purpose Reward Model, coupled with a structured reward-data pipeline and a reward-guided reflux mechanism to drive self-evolution. Key contributions include a hierarchical reward evaluation architecture, a scalable data synthesis procedure with rule-based validation and structured perturbations, intention-centric grounding corrections, and a closed-loop data reflux that co-evolves rewards and policies. Empirical results across offline benchmarks and real-world tasks demonstrate improved task accuracy, robustness, and convergence, with strong generalization to out-of-domain GUI tasks. The framework offers a scalable foundation for autonomous, generalizable GUI agents that continuously improve through reward-driven adaptation.

Abstract

Graphical user interface (GUI) agents are rapidly progressing toward autonomous interaction and reliable task execution across diverse applications. However, two central challenges remain unresolved: automating the evaluation of agent trajectories and generating high-quality training data at scale to enable continual improvement. Existing approaches often depend on manual annotation or static rule-based verification, which restricts scalability and limits adaptability in dynamic environments. We present MagicGUI-RMS, a multi-agent reward model system that delivers adaptive trajectory evaluation, corrective feedback, and self-evolving learning capabilities. MagicGUI-RMS integrates a Domain-Specific Reward Model (DS-RM) with a General-Purpose Reward Model (GP-RM), enabling fine-grained action assessment and robust generalization across heterogeneous GUI tasks. To support reward learning at scale, we design a structured data construction pipeline that automatically produces balanced and diverse reward datasets, effectively reducing annotation costs while maintaining sample fidelity. During execution, the reward model system identifies erroneous actions, proposes refined alternatives, and continuously enhances agent behavior through an automated data-reflux mechanism. Extensive experiments demonstrate that MagicGUI-RMS yields substantial gains in task accuracy, behavioral robustness. These results establish MagicGUI-RMS as a principled and effective foundation for building self-improving GUI agents driven by reward-based adaptation.
Paper Structure (26 sections, 17 equations, 5 figures, 6 tables)

This paper contains 26 sections, 17 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Prompt-Driven GUI Agents
  4. Supervised Fine-Tuning-Based GUI Agents
  5. Rule-Driven Reinforcement Fine-Tuning-Based GUI Agents
  6. Reward-Model-Driven Reinforcement Fine-Tuning-Based GUI Agents
  7. Methodology
  8. MagicGUI-RMS: A Multi-Agent Reward Model System
  9. Domain-Specific Reward Model
  10. General-Purpose Reward Model
  11. Data Construction for Reward Modeling
  12. Rule-Based Verification
  13. Structured Perturbation
  14. Intention-Centric Grounding Correction
  15. Reward Model Dataset
  16. ...and 11 more sections

Figures (5)

  • Figure 1: MagicGUI-RMS architecture. The system operates in a three-stage pipeline: (1) the UI Agent proposes a step-level action conditioned on the task instruction and screen state; (2) the action undergoes hierarchical assessment by DS-RM and GP-RM within the Reward Model System; and (3) two complementary data-reflux loops iteratively improve the reward models and the UI Agent.
  • Figure 2: Overview of the reward data construction pipeline. (1) Structured perturbations, including instruction substitution and trajectory stitching, introduce controllable inconsistencies that yield easy negatives. (2) Rule-based verification evaluates MagicGUI actions under standard instructions to produce positive and hard-negative samples. (3) Intention-centric grounding correction refines actions from open-source UI Agents, generating positive samples from intention-aligned behaviors and moderate negatives.
  • Figure 3: Statistics of the MagicGUI-RMS-72k reward dataset. (a) Distribution of application categories. (b) Distribution of difficulty levels. (c) Distribution of positive and negative samples.
  • Figure 4: Step-level success rates (Step SR) of MagicGUI-Agent and DS-RM across iterative self-improvement rounds. Results are reported on the ALL, IDD (In-Domain Distribution), and OOD (Out-of-Domain) subsets.
  • Figure 5: A representative failure case under the DS-RM–only setting. While DS-RM judges the "click [Confirm Seat Selection]" to be acceptable based on immediate interface cues, GP-RM correctly identifies the semantic mismatch between the current screening date and the user’s instruction.