Auto-scaling Continuous Memory for GUI Agent

TL;DR

The paper tackles the challenge of generalizing GUI agents to unfamiliar interfaces and long-horizon tasks by introducing a continuous memory that encodes trajectories into compact, fixed-length embeddings via a VLM-based memory encoder. This memory is injected into the model without architectural changes, enabling scalable context and capturing fine-grained visual cues. To scale memory at low cost, the authors design a four-phase auto-scaling data flywheel that autonomously discovers environments, synthesizes tasks, rolls out trajectories, and verifies success, yielding over 100k trajectories at about $4k. Empirically, memory-augmented agents show monotonic gains as memory size and retrieval depth increase and achieve competitive performance with leading closed-source models, with notable strength on WebVoyager and robust generalization to out-of-domain GUI environments.

Abstract

We study how to endow GUI agents with scalable memory that help generalize across unfamiliar interfaces and long-horizon tasks. Prior GUI agents compress past trajectories into text tokens, which balloons context length and misses decisive visual cues (e.g., exact widget size and position). We propose a continuous memory that encodes each GUI trajectory into a fixed-length sequence of continuous embeddings using the VLM itself as an encoder; these embeddings are plugged directly into the backbone's input layer, sharply reducing context cost while preserving fine-grained visual information. As memory size and retrieval depth increase, performance improves monotonically, unlike text memories that degrade with long prompts. To grow memory at low cost, we introduce an auto-scaling data flywheel that (i) discovers new environments via search, (ii) synthesizes tasks with an open-source VLM, (iii) rolls out trajectories with the agent, and (iv) verifies success with the same VLM. Using this pipeline, we collect 100k+ trajectories for about \$4000 and fine-tune only the memory encoder (LoRA on a Q-Former, 1.2\% parameters) with 1,500 samples. On real-world GUI benchmarks, our memory-augmented agent consistently improves success rates under long horizons and distribution shifts. Notably, Qwen-2.5-VL-7B + continuous memory achieves performance comparable to state-of-the-art closed-source models (e.g., GPT-4o, Claude-4).

Figures (6)

• Figure 1: Experimental results to verify the effectiveness of our continuous memory in GUI agent setting. (a) Performance scaling law with the memory size; (b) Performance scaling law with retrieved sample number; (c) CoMEM outperforms closed-source state-of-the-art models across benchmarks.
• Figure 2: Overview of our memory-augmented VLM agent framework. We devise (1) a four-phase memory auto-scaling data flywheel; (2) a memory storing diverse environments and successful trajectories; and (3) a VLM-based memory encoder that converts retrieved trajectories from the memory into compact embeddings to guide the VLM during inference.
• Figure 3: Successful case 1 by Qwen2.5-VL-7B + CoMEM in shopping scenario. The model accurately understands the goal, retrieves relevant memory, and selects the correct product.
• Figure 4: Failure case of the base model Qwen2.5-VL-7B without memory. The model selects an incorrect product that exceeds the price limit, highlighting the importance of contextual memory.
• Figure 5: Successful case 2 by Qwen2.5-VL-7B + CoMEM in information scenario. The model accurately understands the goal, retrieves relevant memory, and finds the answer.