Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WebDS: An End-to-End Benchmark for Web-based Data Science

Ethan Hsu, Hong Meng Yam, Ines Bouissou, Aaron Murali John, Raj Thota, Josh Koe, Vivek Sarath Putta, G K Dharesan, Alexander Spangher, Shikhar Murty, Tenghao Huang, Christopher D. Manning

TL;DR

WebDS is introduced, the first end-to-end web-based data science benchmark, challenging agents to perform complex, multi-step, tool-based operations, across heterogeneous data formats, to better reflect the realities of modern data analytics.

Abstract

Many real-world data science tasks involve complex web-based interactions: finding appropriate data available on the internet, synthesizing multimodal data from different locations, and producing summarized analyses. Existing web benchmarks often focus on simplistic interactions and often do not require diverse tool-using capabilities. Conversely, traditional data science benchmarks typically concentrate on static, highly structured datasets and do not assess end-to-end workflows that encompass data acquisition, cleaning, analysis, and insight generation. In response, we introduce WebDS, the first end-to-end web-based data science benchmark. It comprises 870 web-based data science tasks across 29 diverse websites from structured government data portals to unstructured news media, challenging agents to perform complex, multi-step, tool-based operations, across heterogeneous data formats, to better reflect the realities of modern data analytics. Evaluations of current SOTA LLM agents indicate significant performance gaps in accomplishing these tasks. For instance, Browser Use, which accomplishes $80\%$ of tasks on WebVoyager, completes only 15% of tasks in WebDS, which our analysis suggests is due to new failure modes, such as poor information grounding, repetitive behavior and shortcut-taking that agents performing WebDS's tasks display. By contrast, humans achieve around 90% accuracy, highlighting a substantial gap between current agents and human performance. By providing a more robust and realistic testing ground, WebDS sets the stage for significant advances in the development of practically useful LLM-based data science.

WebDS: An End-to-End Benchmark for Web-based Data Science

TL;DR

WebDS is introduced, the first end-to-end web-based data science benchmark, challenging agents to perform complex, multi-step, tool-based operations, across heterogeneous data formats, to better reflect the realities of modern data analytics.

Abstract

Many real-world data science tasks involve complex web-based interactions: finding appropriate data available on the internet, synthesizing multimodal data from different locations, and producing summarized analyses. Existing web benchmarks often focus on simplistic interactions and often do not require diverse tool-using capabilities. Conversely, traditional data science benchmarks typically concentrate on static, highly structured datasets and do not assess end-to-end workflows that encompass data acquisition, cleaning, analysis, and insight generation. In response, we introduce WebDS, the first end-to-end web-based data science benchmark. It comprises 870 web-based data science tasks across 29 diverse websites from structured government data portals to unstructured news media, challenging agents to perform complex, multi-step, tool-based operations, across heterogeneous data formats, to better reflect the realities of modern data analytics. Evaluations of current SOTA LLM agents indicate significant performance gaps in accomplishing these tasks. For instance, Browser Use, which accomplishes of tasks on WebVoyager, completes only 15% of tasks in WebDS, which our analysis suggests is due to new failure modes, such as poor information grounding, repetitive behavior and shortcut-taking that agents performing WebDS's tasks display. By contrast, humans achieve around 90% accuracy, highlighting a substantial gap between current agents and human performance. By providing a more robust and realistic testing ground, WebDS sets the stage for significant advances in the development of practically useful LLM-based data science.

Paper Structure

This paper contains 59 sections, 6 equations, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data Analysis Benchmarks
  4. Web Agent Benchmarks
  5. Benchmark Design
  6. Subject Interviews
  7. Domains
  8. Action Space
  9. State and Observation Space
  10. Reproducibility and Realism
  11. Evaluation Granularity
  12. Release Strategy and Benchmark Integrity
  13. Benchmark Tasks
  14. Task Formulation
  15. Task Generation
  16. ...and 44 more sections

Figures (6)

  • Figure 1: Example of the task, "Analyze the total enrollment numbers by racial/ethnic category for undergraduates (both degree- and non-degree-seeking) as of October 19, 2022. Cross-reference these numbers with national demographic trends and discuss the potential impact on the university's diversity initiatives. Write a report for the university's strategic planning committee on these trends and recommendations." Note: task was selected to save space while still illustrating the multisite attribute.
  • Figure 2: Distribution of tasks with respect to domain (rounded to nearest decimal), where each domain is a grouping of websites according to \ref{['tab:domain_mapping']}. Our domain distribution is chosen based on subject interviews \ref{['par:subject_interviews']}.
  • Figure 3: Agent Performance on Various Benchmarks. Models perform much worse on our benchmark (e.g., BrowserUse + GPT-4o achieves 81.9% on WebVoyager but 12.9% on WebDS). Note: WebVoyager Performance is based on GPT-4, as no verified GPT-4o results exist he2024webvoyagerbuildingendtoendweb.
  • Figure 4: Counts of tasks per attribute
  • Figure 5: Counts of Task Difficulty
  • ...and 1 more figures