History-Aware Reasoning for GUI Agents

TL;DR

This work tackles the challenge of history-aware reasoning for GUI agents by introducing the History-Aware Reasoning (HAR) framework, which injects GUI-domain knowledge and error-guided corrections to cultivate stable short-term memory for long-horizon tasks. HAR combines a reflective learning setup, tailored correction guidelines, and a hybrid RL reward (including a Memory-Aware component) across two training rounds, culminating in HAR-GUI-3B, a native GUI model. Across extensive benchmarks for GUI episodic reasoning, grounding, and understanding, HAR-GUI-3B consistently outperforms state-of-the-art methods with similar or smaller parameter budgets and shows strong out-of-distribution generalization. The approach advances GUI automation by enabling explicit historical-context perception and robust error-driven improvement, with practical impact for accessibility, automated testing, and complex device interactions.

Abstract

Advances in Multimodal Large Language Models have significantly enhanced Graphical User Interface (GUI) automation. Equipping GUI agents with reliable episodic reasoning capabilities is essential for bridging the gap between users' concise task descriptions and the complexities of real-world execution. Current methods integrate Reinforcement Learning (RL) with System-2 Chain-of-Thought, yielding notable gains in reasoning enhancement. For long-horizon GUI tasks, historical interactions connect each screen to the goal-oriented episode chain, and effectively leveraging these clues is crucial for the current decision. However, existing native GUI agents exhibit weak short-term memory in their explicit reasoning, interpreting the chained interactions as discrete screen understanding, i.e., unawareness of the historical interactions within the episode. This history-agnostic reasoning challenges their performance in GUI automation. To alleviate this weakness, we propose a History-Aware Reasoning (HAR) framework, which encourages an agent to reflect on its own errors and acquire episodic reasoning knowledge from them via tailored strategies that enhance short-term memory in long-horizon interaction. The framework mainly comprises constructing a reflective learning scenario, synthesizing tailored correction guidelines, and designing a hybrid RL reward function. Using the HAR framework, we develop a native end-to-end model, HAR-GUI-3B, which alters the inherent reasoning mode from history-agnostic to history-aware, equipping the GUI agent with stable short-term memory and reliable perception of screen details. Comprehensive evaluations across a range of GUI-related benchmarks demonstrate the effectiveness and generalization of our method.

Figures (15)

• Figure 1: Overview of the Histoty-Aware Reasoning (HAR) framework. HAR framework is an error-centric learning approach designed to enhance the reasoning capability of the GUI agent by performing error-aware cognitive correction within a tailored reflection scenario. The framework consists of two critical training stages: (i)GUI Scenario Warm-up Stage. During this phase, comprehensive domain-specific knowledge is injected into the agent via GUI-related data collection and synthesis for knowledge distillation. (ii)Learning From Failure Stage. In this stage, the agent's short-term memory is enhanced. It involves a round of RL within the reflection scenario to perform error-aware cognitive corrections that boost episodic reasoning, followed by another round of RL employing a task-mixing training strategy to assist the GUI agent perceive screen visual details.
• Figure 2: Short-term memory emergence and reasoning enhancement through HAR framework.
• Figure 3: Effectiveness of HAR framework. $\mathcal{M}_{\mathcal{Q}}$ is Qwen2.5-VL-3B zero-shot results. $\mathcal{M}_{\mathcal{\text{GRPO}}}$ refers to the method keeping the same settings as $\mathcal{M}_{\mathcal{\text{HAR-GUI}}}$, but excluding the reflection scenario (using the inference-format instruction), MAR, and TMTS. $\mathcal{M}_{\mathcal{\text{Q}}}^{PT}$, $\mathcal{M}_{\mathcal{\text{GRPO}}}^{PT}$ and $\mathcal{M}_{\mathcal{\text{HAR-GUI}}}^{PT}$ are the post-training results on each benchmark for $\mathcal{M}_{\mathcal{\text{Q}}}$, $\mathcal{M}_{\mathcal{\text{GRPO}}}$ and $\mathcal{M}_{\mathcal{\text{HAR-GUI}}}$, respectively. Compared with $\mathcal{M}_{\mathcal{\text{GRPO}}}$, $\mathcal{M}_{\mathcal{\text{GRPO}}}^*$ mandates that the agent focuses on historical interaction context of the episode in the instructions.
• Figure 4: Case of guidance synthesis for error correction. The results are generated by the teacher model Qwen2.5-VL-72B-Instruct.
• Figure 5: Case of guidance synthesis for error correction. The results are generated by the teacher model Qwen2.5-VL-72B-Instruct.