Compress to Focus: Efficient Coordinate Compression for Policy Optimization in Multi-Turn GUI Agents

TL;DR

This paper tackles context inflation in multi-turn GUI agents by proposing Coordinate Compression Policy Optimization (CCPO), a framework that jointly optimizes visual compression and coordinate-based policy learning. The core innovations are Coordinate-Aware Spatial Compression (CASC), which adaptively constructs ROI boundaries from aggregated interaction coordinates and crops visual history to focus attention, and a Distance-Based Advantage that provides smooth, distance-aware supervision for coordinate actions. CCPO leverages a Progressive Rollout Trajectory to share and refine coordinate history across rollouts, enabling cross-rollout learning and better ROI estimation, alongside a correlation of coordinate signals with policy updates. Empirical results across four GUI benchmarks show state-of-the-art performance with up to 55% token compression and up to 3.8x training speedups, while maintaining or improving grounding accuracy and task success, proving its practicality for efficient multi-turn GUI agents.

Abstract

Multi-turn GUI agents enable complex task completion through sequential decision-making, but suffer from severe context inflation as interaction history accumulates. Existing strategies either sacrifice long-term context via truncation or compromise spatial structure through token pruning. In this paper, we propose Coordinate Compression Policy Optimization (CCPO), an efficient policy optimization framework that couples visual compression with policy optimization for multi-turn GUI agents. CCPO introduces Coordinate-Aware Spatial Compression (CASC), which aggregates coordinates from multiple rollouts to capture target-relevant regions and progressively narrow historical attention around key visual areas. From interactions across rollouts, CASC adaptively constructs attention boundaries that concentrate computation on the most informative regions of the scene. We further design a Distance-Based Advantage that provides fine-grained learning signals based on distance rather than binary correctness, improving both grounding accuracy and compression quality. Extensive experiments demonstrate that CCPO achieves SOTA performance across four benchmarks with up to 55% token compression and 3.8$\times$ training speedup.

Figures (17)

• Figure 1: Top: Existing multi-turn methods tend to truncate the visual history due to the limited context length. Bottom: CCPO preserves the key visual history to maintain the longer trajectory visibility.
• Figure 2: Overview of CCPO framework. The training phase (top) optimizes policies via multi-turn rollouts evaluated by the Distance-Based Advantage. The Coordinate-Aware Spatial Compression module (bottom) tracks $n$ actions and aggregates coordinates to predict ROI of each step, then crops the task-relevant region as a focused visual history $h_{t+1}$.
• Figure 3: Performance comparison for different AO on AITW dataset.
• Figure 4: Attention maps between SFT and CCPO. CCPO accurately predicts actions and localizes coordinates, with stronger focus on detailed historical context and key elements.
• Figure 5: Actions Distribution for Android Control, GUI Odyssey and Android in the Wild dataset