Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hybrid Self-evolving Structured Memory for GUI Agents

Sibo Zhu, Wenyi Wu, Kun Zhou, Stephen Wang, Biwei Huang

TL;DR

Inspired by the brain, HyMEM is proposed, a graph-based memory that couples discrete high-level symbolic nodes with continuous trajectory embeddings that consistently improves open-source GUI agents, enabling 7B/8B backbones to match or surpass strong closed-source models.

Abstract

The remarkable progress of vision-language models (VLMs) has enabled GUI agents to interact with computers in a human-like manner. Yet real-world computer-use tasks remain difficult due to long-horizon workflows, diverse interfaces, and frequent intermediate errors. Prior work equips agents with external memory built from large collections of trajectories, but relies on flat retrieval over discrete summaries or continuous embeddings, falling short of the structured organization and self-evolving characteristics of human memory. Inspired by the brain, we propose Hybrid Self-evolving Structured Memory (HyMEM), a graph-based memory that couples discrete high-level symbolic nodes with continuous trajectory embeddings. HyMEM maintains a graph structure to support multi-hop retrieval, self-evolution via node update operations, and on-the-fly working-memory refreshing during inference. Extensive experiments show that HyMEM consistently improves open-source GUI agents, enabling 7B/8B backbones to match or surpass strong closed-source models; notably, it boosts Qwen2.5-VL-7B by +22.5% and outperforms Gemini2.5-Pro-Vision and GPT-4o.

Hybrid Self-evolving Structured Memory for GUI Agents

TL;DR

Inspired by the brain, HyMEM is proposed, a graph-based memory that couples discrete high-level symbolic nodes with continuous trajectory embeddings that consistently improves open-source GUI agents, enabling 7B/8B backbones to match or surpass strong closed-source models.

Abstract

The remarkable progress of vision-language models (VLMs) has enabled GUI agents to interact with computers in a human-like manner. Yet real-world computer-use tasks remain difficult due to long-horizon workflows, diverse interfaces, and frequent intermediate errors. Prior work equips agents with external memory built from large collections of trajectories, but relies on flat retrieval over discrete summaries or continuous embeddings, falling short of the structured organization and self-evolving characteristics of human memory. Inspired by the brain, we propose Hybrid Self-evolving Structured Memory (HyMEM), a graph-based memory that couples discrete high-level symbolic nodes with continuous trajectory embeddings. HyMEM maintains a graph structure to support multi-hop retrieval, self-evolution via node update operations, and on-the-fly working-memory refreshing during inference. Extensive experiments show that HyMEM consistently improves open-source GUI agents, enabling 7B/8B backbones to match or surpass strong closed-source models; notably, it boosts Qwen2.5-VL-7B by +22.5% and outperforms Gemini2.5-Pro-Vision and GPT-4o.
Paper Structure (46 sections, 1 equation, 12 figures, 4 tables)

This paper contains 46 sections, 1 equation, 12 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. GUI Agents.
  4. Language Model Memory.
  5. Methodology
  6. Hybrid Structured Memory
  7. Graph Schema.
  8. Node Encoding.
  9. Memory Construction via Self-Evolving
  10. Relevant Nodes Retrieval.
  11. Redundancy Checking.
  12. Memory Update.
  13. Memory Utilization
  14. Structured Retrieval.
  15. Working Memory Initialization.
  16. ...and 31 more sections

Figures (12)

  • Figure 1: Overview of the Hybrid Self-Evolving Memory (HyMEM) system. Top: memory construction via graph evolution, where new trajectories are integrated using retrieval and VLM-based redundancy checks. Bottom: four inference phases—(1) Query & Retrieve via structured graph expansion, (2) Working Memory Initialization using hybrid encoding, (3) Agent Execution, and (4) On-the-fly Working Memory Refresh based on new observations.
  • Figure 2: Success rate comparison between static memory and self-evolving HyMEM. (a): Global evolution. (b): Local evolution via working memory refresh.
  • Figure 3: (a) Scaling Behavior of Memory Size in Coursera domain and (b) Graph Compression Analysis.
  • Figure 4: Visualization of Discrete Memory Update Flow.
  • Figure 5: t-SNE visualization of continuous memory embeddings across domains. Each point represents a stored trajectory embedding, color-coded by domain.
  • ...and 7 more figures