Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

"Are We Done Yet?": A Vision-Based Judge for Autonomous Task Completion of Computer Use Agents

Marta Sumyk, Oleksandr Kosovan

TL;DR

This work tackles the challenge of verifying task completion for autonomous computer-use agents by introducing a vision-language evaluation framework that judges success from final desktop screenshots and task descriptions. By collecting a diverse dataset of 1,260 tasks across 42 macOS apps and evaluating multiple vision-language models, the authors demonstrate that zero-shot evaluators can reliably distinguish completed tasks and provide actionable feedback. The feedback loop enables CUAs to replan and retry from the current state, yielding up to 73% task-done accuracy and about a 27% average improvement in success rates. The approach highlights the potential of vision-grounded evaluation to improve reliability and self-correction in CUAs, with future work extending to other operating systems, step-level evaluation, and RL/multi-agent integrations.

Abstract

Computer Use Agents (CUAs) are designed to autonomously operate digital interfaces, yet they often fail to reliably determine whether a given task has been completed. We present an autonomous evaluation and feedback framework that uses vision-language models to assess task completion directly from screenshots and task descriptions. Our dataset covers 42 built-in macOS applications and 1,260 human-labeled tasks across a wide range of scenarios. Our framework achieves up to 73 percent accuracy in task success detection and yields an average relative improvement of 27 percent in overall task success when evaluator feedback is applied. These results show that vision-based evaluation can serve as an effective feedback mechanism that improves the reliability and self-correction of autonomous computer-use agents.

"Are We Done Yet?": A Vision-Based Judge for Autonomous Task Completion of Computer Use Agents

TL;DR

This work tackles the challenge of verifying task completion for autonomous computer-use agents by introducing a vision-language evaluation framework that judges success from final desktop screenshots and task descriptions. By collecting a diverse dataset of 1,260 tasks across 42 macOS apps and evaluating multiple vision-language models, the authors demonstrate that zero-shot evaluators can reliably distinguish completed tasks and provide actionable feedback. The feedback loop enables CUAs to replan and retry from the current state, yielding up to 73% task-done accuracy and about a 27% average improvement in success rates. The approach highlights the potential of vision-grounded evaluation to improve reliability and self-correction in CUAs, with future work extending to other operating systems, step-level evaluation, and RL/multi-agent integrations.

Abstract

Computer Use Agents (CUAs) are designed to autonomously operate digital interfaces, yet they often fail to reliably determine whether a given task has been completed. We present an autonomous evaluation and feedback framework that uses vision-language models to assess task completion directly from screenshots and task descriptions. Our dataset covers 42 built-in macOS applications and 1,260 human-labeled tasks across a wide range of scenarios. Our framework achieves up to 73 percent accuracy in task success detection and yields an average relative improvement of 27 percent in overall task success when evaluator feedback is applied. These results show that vision-based evaluation can serve as an effective feedback mechanism that improves the reliability and self-correction of autonomous computer-use agents.

Paper Structure

This paper contains 15 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Computer Use Agents
  4. Autonomous Evaluation of Task Completion
  5. Methodology
  6. Dataset
  7. Autonomous Evaluation
  8. Task Execution.
  9. Outcome Evaluation.
  10. Feedback and Retry.
  11. Results
  12. Evaluator Accuracy Across CUAs
  13. Effect of Evaluator Feedback on Success Rate
  14. Future Work
  15. Conclusion

Figures (2)

  • Figure 1: Overview of the proposed evaluation-feedback pipeline. CUA executes a user-defined task (e.g., “Change appearance on macOS to dark mode”) and produces a final screenshot of the desktop state. The VLM receives the screenshot and task description, then judges whether the task was successfully completed. If the task is deemed incomplete, the VLM provides reasoning that is passed back to the CUA, which reattempts the task based on this feedback.
  • Figure 2: Task success rates before and after evaluator feedback across three CUAs. Gray bars represent baseline success rates before feedback, while colored bars indicate post-feedback (after only one retry) success rate for five VLM evaluators. Proprietary evaluators (GPT-4o and Claude 3.5 Sonnet) achieve the largest relative gains, whereas open-source models (LLaVA-v1.5-7B, InternVL 2-8B, and Qwen2-VL-7B) provide consistent improvements across all agents.