Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities

Xiaoxue Ren, Penghao Jiang, Kaixin Li, Zhiyong Huang, Xiaoning Du, Jiaojiao Jiang, Zhenchang Xing, Jiamou Sun, Terry Yue Zhuo

TL;DR

HackWorld addresses the gap in evaluating computer-use agents for cybersecurity by exposing CUAs to realistic web vulnerabilities through a visual-interaction interface and a CKF-based evaluation. The framework combines a Kali-Linux tool ecosystem, Dockerized web challenges across diverse tech stacks, and an end-to-end interaction pipeline to test perception, reasoning, tool use, and exploitation capabilities. Experimental results show strikingly low exploitation rates (below 12%), and reveal that improvements in perception alone do not translate to better performance; the dominant bottleneck lies in strategic planning and tool orchestration within web security contexts. These findings highlight the need for security-aware agent designs and provide a foundation for future work to develop CUAs capable of robust vulnerability discovery and exploitation in real-world web environments.

Abstract

Web applications are prime targets for cyberattacks as gateways to critical services and sensitive data. Traditional penetration testing is costly and expertise-intensive, making it difficult to scale with the growing web ecosystem. While language model agents show promise in cybersecurity, modern web applications demand visual understanding, dynamic content handling, and multi-step interactions that only computer-use agents (CUAs) can perform. Yet, their ability to discover and exploit vulnerabilities through graphical interfaces remains largely unexplored. We present HackWorld, the first framework for systematically evaluating CUAs' capabilities to exploit web application vulnerabilities via visual interaction. Unlike sanitized benchmarks, HackWorld includes 36 real-world applications across 11 frameworks and 7 languages, featuring realistic flaws such as injection vulnerabilities, authentication bypasses, and unsafe input handling. Using a Capture-the-Flag (CTF) setup, it tests CUAs' capacity to identify and exploit these weaknesses while navigating complex web interfaces. Evaluation of state-of-the-art CUAs reveals concerning trends: exploitation rates below 12% and low cybersecurity awareness. CUAs often fail at multi-step attack planning and misuse security tools. These results expose the current limitations of CUAs in web security contexts and highlight opportunities for developing more security-aware agents capable of effective vulnerability detection and exploitation.

HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilities

TL;DR

HackWorld addresses the gap in evaluating computer-use agents for cybersecurity by exposing CUAs to realistic web vulnerabilities through a visual-interaction interface and a CKF-based evaluation. The framework combines a Kali-Linux tool ecosystem, Dockerized web challenges across diverse tech stacks, and an end-to-end interaction pipeline to test perception, reasoning, tool use, and exploitation capabilities. Experimental results show strikingly low exploitation rates (below 12%), and reveal that improvements in perception alone do not translate to better performance; the dominant bottleneck lies in strategic planning and tool orchestration within web security contexts. These findings highlight the need for security-aware agent designs and provide a foundation for future work to develop CUAs capable of robust vulnerability discovery and exploitation in real-world web environments.

Abstract

Web applications are prime targets for cyberattacks as gateways to critical services and sensitive data. Traditional penetration testing is costly and expertise-intensive, making it difficult to scale with the growing web ecosystem. While language model agents show promise in cybersecurity, modern web applications demand visual understanding, dynamic content handling, and multi-step interactions that only computer-use agents (CUAs) can perform. Yet, their ability to discover and exploit vulnerabilities through graphical interfaces remains largely unexplored. We present HackWorld, the first framework for systematically evaluating CUAs' capabilities to exploit web application vulnerabilities via visual interaction. Unlike sanitized benchmarks, HackWorld includes 36 real-world applications across 11 frameworks and 7 languages, featuring realistic flaws such as injection vulnerabilities, authentication bypasses, and unsafe input handling. Using a Capture-the-Flag (CTF) setup, it tests CUAs' capacity to identify and exploit these weaknesses while navigating complex web interfaces. Evaluation of state-of-the-art CUAs reveals concerning trends: exploitation rates below 12% and low cybersecurity awareness. CUAs often fail at multi-step attack planning and misuse security tools. These results expose the current limitations of CUAs in web security contexts and highlight opportunities for developing more security-aware agents capable of effective vulnerability detection and exploitation.
Paper Structure (38 sections, 4 figures, 10 tables)

This paper contains 38 sections, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. HackWorld Environment
  3. Preliminaries and Task Definition
  4. Web Security Evaluation Framework
  5. System Architecture and Environment Setup
  6. Challenge Deployment Process
  7. Agent Interaction Pipeline
  8. Comprehensive Tool Integration
  9. Evaluation and Logging Infrastructure
  10. HackWorld Benchmark
  11. Statistics of HackWorld benchmark
  12. Challenge Collection
  13. Technology Stacks
  14. Criteria for Challenges Selection
  15. Experiments
  16. ...and 23 more sections

Figures (4)

  • Figure 1: Motivating Example of HackWorld. The agent explored the environment autonomously and successfully captured a Local File Inclusion vulnerability in the website. Then it exploited the defect and extracted the secret flag. The full trajectory of this task can be found in Appendix \ref{['apd:case_study']}.
  • Figure 1: Representative security tools integrated in HackWorld. Full tools are listed in \ref{['apd:tool_usage']}.
  • Figure 2: Workflow of HackWorld
  • Figure 3: Distribution of technology stacks in HackWorld. Full Challenges are listed in \ref{['app:benchmark']}.