ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response

TL;DR

This work tackles the challenge of proactive robotic agents by introducing ProAct-75, a large, graph-structured vision benchmark across assistance, maintenance, and safety domains. It proposes ProAct-Helper, a multimodal baseline built on an MLLM with a Hierarchical Binding Module and an entropy-driven planner that selects graph-feasible actions to exploit parallel threads. Empirical results show improved trigger, task, and step detection and more efficient proactive action selection compared to strong baselines, with demonstrated cross-view generalization and reduced planning latency. The findings highlight the value of explicit task-graph grounding and cross-level alignment for scalable, structure-aware proactive decision making in real-world settings.

Abstract

While passive agents merely follow instructions, proactive agents align with higher-level objectives, such as assistance and safety by continuously monitoring the environment to determine when and how to act. However, developing proactive agents is hindered by the lack of specialized resources. To address this, we introduce ProAct-75, a benchmark designed to train and evaluate proactive agents across diverse domains, including assistance, maintenance, and safety monitoring. Spanning 75 tasks, our dataset features 91,581 step-level annotations enriched with explicit task graphs. These graphs encode step dependencies and parallel execution possibilities, providing the structural grounding necessary for complex decision-making. Building on this benchmark, we propose ProAct-Helper, a reference baseline powered by a Multimodal Large Language Model (MLLM) that grounds decision-making in state detection, and leveraging task graphs to enable entropy-driven heuristic search for action selection, allowing agents to execute parallel threads independently rather than mirroring the human's next step. Extensive experiments demonstrate that ProAct-Helper outperforms strong closed-source models, improving trigger detection mF1 by 6.21%, saving 0.25 more steps in online one-step decision, and increasing the rate of parallel actions by 15.58%.

Figures (11)

• Figure 1: Overview of proactive response tasks. ProAct-75 supports five vision-based tasks with step-level annotations, hierarchical labels, and task graphs. Traditional intent-following approaches predict human-intended actions (e.g., tie the bag) and execute them, inadvertently blocking workflows. Our benchmark enables evaluation of strategies where robots pursue independent parallel threads to reduce disruptions.
• Figure 2: Qualitative examples of ProAct-75 across three application scenarios. We visualize the previous-current-future observation window and structured annotations for our proactive visual response tasks. Assistance and Maintenance examples are from self-collected exocentric videos. Safety examples are from UCF-Crime. Safety videos omit future action prediction and proactive action selection due to the absence of human-robot collaboration.
• Figure 3: ProAct-75 data collection and annotation pipeline. We combine videos from public datasets and self-collected recordings, then annotate step spans/names, triggers, and best views. Each task is equipped with a task-graph annotation.
• Figure 4: Overview of ProAct-Helper framework. Given keyframes and prompts, it detect trigger, task, step, and human's future actions, trained with hierarchical binding losses for cross-level consistency under long-tail data. It then selects the next robot action on the task DAG via an entropy-driven heuristic search to favor feasible, low thread-mixing progress.
• Figure 5: Failure case analysis. (a) We compare hallucination rates across trigger, step, and future prediction actions for different mllm. (b) We analyze models' parallel execution tendencies, waiting behaviors, and task graph constraint comprehension. For clarity, we omit non-parallel actions from the visualization.