PIRA-Bench: A Transition from Reactive GUI Agents to GUI-based Proactive Intent Recommendation Agents

TL;DR

PIRA-Bench (Proactive Intent Recommendation Agent Benchmark), a novel benchmark for evaluating multimodal large language models (MLLMs) on continuous, weakly-supervised visual inputs, and the PIRF baseline, a memory-aware, state-tracking framework that empowers general MLLMs to manage multiple task threads and handle misleading visual inputs are proposed.

Abstract

Current Graphical User Interface (GUI) agents operate primarily under a reactive paradigm: a user must provide an explicit instruction for the agent to execute a task. However, an intelligent AI assistant should be proactive, which is capable of anticipating user intentions directly from continuous visual inputs, such as mobile or desktop screenshots, and offering timely recommendations without explicit user prompting. Transitioning to this proactive paradigm presents significant challenges. Real-world screen activity is rarely linear; it consists of long-horizon trajectories fraught with noisy browsing, meaningless actions, and multithreaded task-switching. To address this gap, we introduce PIRA-Bench (Proactive Intent Recommendation Agent Benchmark), a novel benchmark for evaluating multimodal large language models (MLLMs) on continuous, weakly-supervised visual inputs. Unlike reactive datasets, PIRA-Bench features complex trajectories with multiple interleaved intents and noisy segments with various user profile contexts, challenging agents to detect actionable events while fitting to user preferences. Furthermore, we propose the PIRF baseline, a memory-aware, state-tracking framework that empowers general MLLMs to manage multiple task threads and handle misleading visual inputs. PIRA-Bench serves as an initial step toward robust and proactive GUI-based personal assistants.

Figures (2)

• Figure 1: A comparison between the traditional reactive GUI agent paradigm and the proposed proactive intent recommendation agent. Reactive agents (left) function as passive executors, remaining idle until they receive instructions to initiate device control. In contrast, a proactive agent (right) continuously monitors the visual context to autonomously infer the user's latent goals before the user even formulates a command.
• Figure 2: The overall architecture of the Proactive Intent Recommendation Framework (PIRF). The system processes continuous visual inputs sequentially by integrating a sliding window of historical context with a dynamic Memory Module that maintains both static user profiles and a continuously updated intent bank. Driven by a Multimodal Large Language Model, the framework evaluates the current state against a structured action space (i.e., Create, Resume, Update, or IDLE) to autonomously refine its memory and generate a final set of actionable, proactive intents.