MAGNET: Towards Adaptive GUI Agents with Memory-Driven Knowledge Evolution

TL;DR

This work tackles the challenge of GUI agents failing to generalize under appearance and workflow drift in evolving mobile apps. It introduces MAGNET, a dual-memory framework combining procedural memory for stable task intent and stationary memory for stable UI grounding, augmented by a dynamic memory-evolution mechanism that prioritizes frequently accessed knowledge. The authors provide memory-construction pipelines, a UI-40K multimodal dataset, and a retention-based memory ranking scheme $R_i = \exp(-g_i / n_i)$ to support continual adaptation. Comprehensive offline and online experiments show MAGNET outperforms memory-free baselines and remains competitive with specialized models, demonstrating improved adaptability and generalization in dynamic software environments. The results suggest practical deployment potential for memory-driven GUI agents capable of adapting to interface evolution and restructured workflows.

Abstract

Mobile GUI agents powered by large foundation models enable autonomous task execution, but frequent updates altering UI appearance and reorganizing workflows cause agents trained on historical data to fail. Despite surface changes, functional semantics and task intents remain fundamentally stable. Building on this insight, we introduce MAGNET, a memory-driven adaptive agent framework with dual-level memory: stationary memory linking diverse visual features to stable functional semantics for robust action grounding and procedural memory capturing stable task intents across varying workflows. We propose a dynamic memory evolution mechanism that continuously refines both memories by prioritizing frequently accessed knowledge. Online benchmark AndroidWorld evaluations show substantial improvements over baselines, while offline benchmarks confirm consistent gains under distribution shifts. These results validate that leveraging stable structures across interface changes improves agent performance and generalization in evolving software environments.

Figures (6)

• Figure 1: Challenges and opportunities in evolving mobile interfaces. (Left) Two types of drifts and their underlying stability: Appearance Drift vs. Semantic Stability, Workflow Drift vs. Intent Stability. (Right) MAGNET exploits these stable aspects to maintain effectiveness, while memory-less agents relying on frozen knowledge struggle to adapt.
• Figure 2: MAGNET framework. The planner leverages procedural memory to decompose user requests into subtasks, while the actor grounds each subtask with stationary memory of UI elements.
• Figure 3: Construction pipeline of stationary memory. Each newly generated entry $\langle d_i, v_i \rangle$ is checked against the stationary memory by retrieving similar functional descriptions and corresponding UI element patches. If a duplicate entry is detected, it is discarded to avoid redundancy.
• Figure 4: Results on the template, app, and domain shifted subsets. The MLLM serves as both the planner and the actor. The dashed lines in the figure indicate the baseline performance.
• Figure 5: Domain/App distribution of the Amex dataset.