Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond BeautifulSoup: Benchmarking LLM-Powered Web Scraping for Everyday Users

Arth Bhardwaj, Nirav Diwan, Gang Wang

TL;DR

This study benchmarks two LLM-enabled web-scraping workflows—LLM-assisted Scripting (LAS) and End-to-End LLM Agent (ELA)—against traditional tools across 35 websites with five defense tiers. It finds that end-to-end agents enable novice users to autonomously navigate authentication, anti-bot, and CAPTCHA barriers, albeit with higher runtimes, while traditional scripting remains fastest for static pages. The results illuminate a practical accessibility-reliability trade-off and suggest hybrid approaches that leverage the strengths of both paradigms. The work has implications for defenders and attackers alike, showing that off-the-shelf LLM tools can democratize automated data extraction and prompting further research into robust defenses and safer deployment practices.

Abstract

Web scraping has historically required technical expertise in HTML parsing, session management, and authentication circumvention, which limited large-scale data extraction to skilled developers. We argue that large language models (LLMs) have democratized web scraping, enabling low-skill users to execute sophisticated operations through simple natural language prompts. While extensive benchmarks evaluate these tools under optimal expert conditions, we show that without extensive manual effort, current LLM-based workflows allow novice users to scrape complex websites that would otherwise be inaccessible. We systematically benchmark what everyday users can do with off-the-shelf LLM tools across 35 sites spanning five security tiers, including authentication, anti-bot, and CAPTCHA controls. We devise and evaluate two distinct workflows: (a) LLM-assisted scripting, where users prompt LLMs to generate traditional scraping code but maintain manual execution control, and (b) end-to-end LLM agents, which autonomously navigate and extract data through integrated tool use. Our results demonstrate that end-to-end agents have made complex scraping accessible - requiring as little as a single prompt with minimal refinement (less than 5 changes) to complete workflows. We also highlight scenarios where LLM-assisted scripting may be simpler and faster for static sites. In light of these findings, we provide simple procedures for novices to use these workflows and gauge what adversaries could achieve using these.

Beyond BeautifulSoup: Benchmarking LLM-Powered Web Scraping for Everyday Users

TL;DR

This study benchmarks two LLM-enabled web-scraping workflows—LLM-assisted Scripting (LAS) and End-to-End LLM Agent (ELA)—against traditional tools across 35 websites with five defense tiers. It finds that end-to-end agents enable novice users to autonomously navigate authentication, anti-bot, and CAPTCHA barriers, albeit with higher runtimes, while traditional scripting remains fastest for static pages. The results illuminate a practical accessibility-reliability trade-off and suggest hybrid approaches that leverage the strengths of both paradigms. The work has implications for defenders and attackers alike, showing that off-the-shelf LLM tools can democratize automated data extraction and prompting further research into robust defenses and safer deployment practices.

Abstract

Web scraping has historically required technical expertise in HTML parsing, session management, and authentication circumvention, which limited large-scale data extraction to skilled developers. We argue that large language models (LLMs) have democratized web scraping, enabling low-skill users to execute sophisticated operations through simple natural language prompts. While extensive benchmarks evaluate these tools under optimal expert conditions, we show that without extensive manual effort, current LLM-based workflows allow novice users to scrape complex websites that would otherwise be inaccessible. We systematically benchmark what everyday users can do with off-the-shelf LLM tools across 35 sites spanning five security tiers, including authentication, anti-bot, and CAPTCHA controls. We devise and evaluate two distinct workflows: (a) LLM-assisted scripting, where users prompt LLMs to generate traditional scraping code but maintain manual execution control, and (b) end-to-end LLM agents, which autonomously navigate and extract data through integrated tool use. Our results demonstrate that end-to-end agents have made complex scraping accessible - requiring as little as a single prompt with minimal refinement (less than 5 changes) to complete workflows. We also highlight scenarios where LLM-assisted scripting may be simpler and faster for static sites. In light of these findings, we provide simple procedures for novices to use these workflows and gauge what adversaries could achieve using these.
Paper Structure (14 sections, 2 figures, 3 tables)

This paper contains 14 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Threat Model
  4. Methodology
  5. Website Selection Criteria
  6. Scraping Workflows
  7. Standardization and Control
  8. Evaluation and Results
  9. Setup
  10. Results
  11. Discussion
  12. Conclusion and Future Work
  13. Acknowledgment
  14. Appendix

Figures (2)

  • Figure 1: Benchmark for non-expert web scraping. We introduce a benchmark that evaluates what non-expert users can achieve with off-the-shelf tools, modeling two workflows: (i) LLM-assisted Scripting (LAS) and (ii) End-to-End LLM Agent (ELA). LAS: the LLM drafts code that the user executes and manually refines; ELA: a production agent plans and acts via internal tools.
  • Figure 2: Average execution time per category. Note that the y-axis is on a log scale (in seconds).