Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Co-EPG: A Framework for Co-Evolution of Planning and Grounding in Autonomous GUI Agents

Yuan Zhao, Hualei Zhu, Tingyu Jiang, Shen Li, Xiaohang Xu, Hao Henry Wang

TL;DR

Co-EPG introduces a self-iterative, co-evolutionary framework that tightly couples planning and grounding for autonomous GUI agents. By employing a P-G dual-model with a GRPO-based optimization loop and a confidence-based dynamic reward ensemble (C-DREM), the method iteratively improves both modules while distilling high-quality data from self-generated experiences. It delivers state-of-the-art results on Multimodal-Mind2Web and AndroidControl with minimal data, demonstrating strong data efficiency and generalization across cross-task and cross-domain settings. The approach highlights the value of integrated planning-grounding synergies and robust reward signaling for scalable GUI intelligence.

Abstract

Graphical User Interface (GUI) task automation constitutes a critical frontier in artificial intelligence research. While effective GUI agents synergistically integrate planning and grounding capabilities, current methodologies exhibit two fundamental limitations: (1) insufficient exploitation of cross-model synergies, and (2) over-reliance on synthetic data generation without sufficient utilization. To address these challenges, we propose Co-EPG, a self-iterative training framework for Co-Evolution of Planning and Grounding. Co-EPG establishes an iterative positive feedback loop: through this loop, the planning model explores superior strategies under grounding-based reward guidance via Group Relative Policy Optimization (GRPO), generating diverse data to optimize the grounding model. Concurrently, the optimized Grounding model provides more effective rewards for subsequent GRPO training of the planning model, fostering continuous improvement. Co-EPG thus enables iterative enhancement of agent capabilities through self-play optimization and training data distillation. On the Multimodal-Mind2Web and AndroidControl benchmarks, our framework outperforms existing state-of-the-art methods after just three iterations without requiring external data. The agent consistently improves with each iteration, demonstrating robust self-enhancement capabilities. This work establishes a novel training paradigm for GUI agents, shifting from isolated optimization to an integrated, self-driven co-evolution approach.

Co-EPG: A Framework for Co-Evolution of Planning and Grounding in Autonomous GUI Agents

TL;DR

Co-EPG introduces a self-iterative, co-evolutionary framework that tightly couples planning and grounding for autonomous GUI agents. By employing a P-G dual-model with a GRPO-based optimization loop and a confidence-based dynamic reward ensemble (C-DREM), the method iteratively improves both modules while distilling high-quality data from self-generated experiences. It delivers state-of-the-art results on Multimodal-Mind2Web and AndroidControl with minimal data, demonstrating strong data efficiency and generalization across cross-task and cross-domain settings. The approach highlights the value of integrated planning-grounding synergies and robust reward signaling for scalable GUI intelligence.

Abstract

Graphical User Interface (GUI) task automation constitutes a critical frontier in artificial intelligence research. While effective GUI agents synergistically integrate planning and grounding capabilities, current methodologies exhibit two fundamental limitations: (1) insufficient exploitation of cross-model synergies, and (2) over-reliance on synthetic data generation without sufficient utilization. To address these challenges, we propose Co-EPG, a self-iterative training framework for Co-Evolution of Planning and Grounding. Co-EPG establishes an iterative positive feedback loop: through this loop, the planning model explores superior strategies under grounding-based reward guidance via Group Relative Policy Optimization (GRPO), generating diverse data to optimize the grounding model. Concurrently, the optimized Grounding model provides more effective rewards for subsequent GRPO training of the planning model, fostering continuous improvement. Co-EPG thus enables iterative enhancement of agent capabilities through self-play optimization and training data distillation. On the Multimodal-Mind2Web and AndroidControl benchmarks, our framework outperforms existing state-of-the-art methods after just three iterations without requiring external data. The agent consistently improves with each iteration, demonstrating robust self-enhancement capabilities. This work establishes a novel training paradigm for GUI agents, shifting from isolated optimization to an integrated, self-driven co-evolution approach.

Paper Structure

This paper contains 26 sections, 10 equations, 9 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. GUI Agent Architecture Design
  4. GUI Agent Capability Enhancement
  5. Preliminaries
  6. Methodology
  7. P-G Dual-Model
  8. Co-Evolving Optimization Loop
  9. C-DREM
  10. Experiments
  11. Datasets and Metrics
  12. Implementation Details
  13. Main Results
  14. Ablation Study
  15. Impact of P-G Dual-Model.
  16. ...and 11 more sections

Figures (9)

  • Figure 1: Overview of our proposed Co-EPG framework. The framework drives the co-evolution of the planning model ($\pi$) and the grounding model ($\phi$) through an optimization loop. This loop alternates between Iterative Training, which employs C-DREM to drive the co-evolution of both models, and Data Enhancement, which refines the dataset for the next iteration.
  • Figure 2: Effectiveness of the self-iterative evolution mechanism in Co-EPG on Multimodal-Mind2Web.
  • Figure 3: An Evolution Example of Co-EPG.
  • Figure 4: Performance and data efficiency on the Multimodal-Mind2Web benchmark.
  • Figure 5: The performance curves of different reward mechanisms evaluated on Multimodal-Mind2Web.
  • ...and 4 more figures

Theorems & Definitions (1)

  • Definition 1: POMDP