LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization

TL;DR

This work introduces Location Preference Optimization (LPO) to address the persistent challenge of precise spatial localization in GUI agents. LPO combines a window-based information-density reward, grounded in information entropy, with a dynamic location reward that uses distance to guide actions, all integrated through Group Relative Preference Optimization (GRPO) to enable broad exploration of GUI spaces. Across offline benchmarks (Mind2Web, VisualWebBench, Screenspot) and online evaluations (WebVoyager), LPO achieves state-of-the-art performance for GUI interaction and grounding, outperforming prior preference optimization approaches. The study demonstrates that focusing on information-rich zones and spatially accurate actions markedly improves interaction fidelity, suggesting practical impact for autonomous GUI agents while acknowledging data and compute requirements. The work points to future directions in diversifying high-precision datasets, reducing computational overhead for real-time use, and ensuring ethical, responsible deployment.

Abstract

The advent of autonomous agents is transforming interactions with Graphical User Interfaces (GUIs) by employing natural language as a powerful intermediary. Despite the predominance of Supervised Fine-Tuning (SFT) methods in current GUI agents for achieving spatial localization, these methods face substantial challenges due to their limited capacity to accurately perceive positional data. Existing strategies, such as reinforcement learning, often fail to assess positional accuracy effectively, thereby restricting their utility. In response, we introduce Location Preference Optimization (LPO), a novel approach that leverages locational data to optimize interaction preferences. LPO uses information entropy to predict interaction positions by focusing on zones rich in information. Besides, it further introduces a dynamic location reward function based on physical distance, reflecting the varying importance of interaction positions. Supported by Group Relative Preference Optimization (GRPO), LPO facilitates an extensive exploration of GUI environments and significantly enhances interaction precision. Comprehensive experiments demonstrate LPO's superior performance, achieving SOTA results across both offline benchmarks and real-world online evaluations. Our code will be made publicly available soon, at https://github.com/AIDC-AI/LPO.

Figures (3)

• Figure 1: Motivation of dynamic location reward. (a) UITARS qin2025uitarspioneeringautomatedgui uses direct text-level matching; (b) UI-R1 lu2025ui, InfiGUI-R1 liu2025infiguir1advancingmultimodalgui and RUIG zhang2023reinforced employ bounding boxes for interaction preferences; (c) GUI-R1 xia2025gui relies on fixed positional boundaries. (d) Our dynamic location reward offers a more precise positional representation, addressing the limitations of previous methods.
• Figure 2: Example of $r_w$. Green zones indicate high interaction likelihood due to rich information, earning greater rewards. In contrast, red zones, like blank areas, have lower interaction probability and rewards. Key interactive areas, such as login, search, and editing zones, align with user interaction tendencies.
• Figure 3: Example of $r_d$. When users need to interact at a point located on the search button, the reward increases as the generated interaction point gets closer to this target point, while it decreases as the point moves further away. This highlights the importance of precision in interaction positioning.