When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents

TL;DR

The paper addresses misaligned actions in computer-use agents (CUAs) caused by external prompts or internal limitations. It introduces MisActBench, a benchmark with 2264 action-level labels across 558 trajectories, and DeAction, a two-stage runtime guardrail that detects misaligned actions before execution and iteratively corrects them via structured feedback using a compact narrative history. In offline and online experiments, DeAction achieves over 15 percentage points improvement in F1 on MisActBench, reduces attack success rate by over 90% under adversarial conditions, and preserves or improves task success in benign settings with moderate overhead. This work delivers a practical, plug-and-play defense for real-world CUA deployment, enhancing safety, reliability, and user trust.

Abstract

Computer-use agents (CUAs) have made tremendous progress in the past year, yet they still frequently produce misaligned actions that deviate from the user's original intent. Such misaligned actions may arise from external attacks (e.g., indirect prompt injection) or from internal limitations (e.g., erroneous reasoning). They not only expose CUAs to safety risks, but also degrade task efficiency and reliability. This work makes the first effort to define and study misaligned action detection in CUAs, with comprehensive coverage of both externally induced and internally arising misaligned actions. We further identify three common categories in real-world CUA deployment and construct MisActBench, a benchmark of realistic trajectories with human-annotated, action-level alignment labels. Moreover, we propose DeAction, a practical and universal guardrail that detects misaligned actions before execution and iteratively corrects them through structured feedback. DeAction outperforms all existing baselines across offline and online evaluations with moderate latency overhead: (1) On MisActBench, it outperforms baselines by over 15% absolute in F1 score; (2) In online evaluation, it reduces attack success rate by over 90% under adversarial settings while preserving or even improving task success rate in benign environments.

Figures (12)

• Figure 1: Examples of the three categories of misaligned actions. (a) Malicious Instruction Following: the action complies with external malicious instructions; (b) Harmful Unintended Behavior: the action causes harm due to inherent limitations rather than adversarial attacks; (c) Other Task-Irrelevant Behavior: the action does not cause harm but is irrelevant to the task.
• Figure 2: Trajectory Collection Workflow for MisActBench. (a) Collect trajectories with misaligned actions induced by external attacks by running diverse CUAs on existing benchmarks. (b) Synthesize trajectories with unintended behaviors in benign settings.
• Figure 3: Overview of DeAction. (a) Misaligned action detection. At each step, the guardrail intercepts the proposed action before execution through a two-stage analysis, with a compact summary of interaction history. (b) Iterative correction. When an action is flagged, the guardrail provides structured feedback to the agent, prompting action revision through a closed-loop interaction.
• Figure 4: Performance under different history representations.
• Figure 5: Comparison of attack success rate (ASR) versus per-step guardrail latency on RedTeamCUA. DeAction is Pareto-optimal, achieving the lowest ASR while incurring moderate latency.