Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DCA-Bench: A Benchmark for Dataset Curation Agents

Benhao Huang, Yingzhuo Yu, Jin Huang, Xingjian Zhang, Jiaqi Ma

TL;DR

DCA-Bench introduces a benchmark and an automatic GPT-4o-based evaluation framework to assess whether LLM agents can discover hidden data quality issues in community-contributed datasets. It assembles 221 real-world test cases from eight platforms, packaged with multi-level hints and ground-truth annotations, to simulate autonomous dataset curation. The study validates the Evaluator against human judgments and benchmarks multiple baseline Curators, finding that current state-of-the-art models struggle to reveal all issues without hints and still face limitations even with rich guidance. The work demonstrates both the potential and the challenges of using LLM agents for dataset curation at scale, and highlights avenues for improvement such as better knowledge integration and multi-agent collaboration. Overall, DCA-Bench provides a practical testbed and an evaluative framework to push toward reliable autonomous dataset curation in open AI platforms.

Abstract

The quality of datasets plays an increasingly crucial role in the research and development of modern artificial intelligence (AI). Despite the proliferation of open dataset platforms nowadays, data quality issues, such as incomplete documentation, inaccurate labels, ethical concerns, and outdated information, remain common in widely used datasets. Furthermore, these issues are often subtle and difficult to be detected by rule-based scripts, therefore requiring identification and verification by dataset users or maintainers--a process that is both time-consuming and prone to human mistakes. With the surging ability of large language models (LLM), it's promising to streamline the discovery of hidden dataset issues with LLM agents. To achieve this, one significant challenge is enabling LLM agents to detect issues in the wild rather than simply fixing known ones. In this work, we establish a benchmark to measure LLM agent's ability to tackle this challenge. We carefully curate 221 real-world test cases from eight popular dataset platforms and propose an automatic evaluation framework using GPT-4o. Our proposed framework shows strong empirical alignment with expert evaluations, validated through extensive comparisons with human annotations. Without any hints, most competitive Curator agent can only reveal $\sim$30\% of the data quality issues in the proposed dataset, highlighting the complexity of this task and indicating that applying LLM agents to real-world dataset curation still requires further in-depth exploration and innovation. The data and code are available at \href{https://github.com/TRAIS-Lab/dca-bench}{https://github.com/TRAIS-Lab/dca-bench}.

DCA-Bench: A Benchmark for Dataset Curation Agents

TL;DR

DCA-Bench introduces a benchmark and an automatic GPT-4o-based evaluation framework to assess whether LLM agents can discover hidden data quality issues in community-contributed datasets. It assembles 221 real-world test cases from eight platforms, packaged with multi-level hints and ground-truth annotations, to simulate autonomous dataset curation. The study validates the Evaluator against human judgments and benchmarks multiple baseline Curators, finding that current state-of-the-art models struggle to reveal all issues without hints and still face limitations even with rich guidance. The work demonstrates both the potential and the challenges of using LLM agents for dataset curation at scale, and highlights avenues for improvement such as better knowledge integration and multi-agent collaboration. Overall, DCA-Bench provides a practical testbed and an evaluative framework to push toward reliable autonomous dataset curation in open AI platforms.

Abstract

The quality of datasets plays an increasingly crucial role in the research and development of modern artificial intelligence (AI). Despite the proliferation of open dataset platforms nowadays, data quality issues, such as incomplete documentation, inaccurate labels, ethical concerns, and outdated information, remain common in widely used datasets. Furthermore, these issues are often subtle and difficult to be detected by rule-based scripts, therefore requiring identification and verification by dataset users or maintainers--a process that is both time-consuming and prone to human mistakes. With the surging ability of large language models (LLM), it's promising to streamline the discovery of hidden dataset issues with LLM agents. To achieve this, one significant challenge is enabling LLM agents to detect issues in the wild rather than simply fixing known ones. In this work, we establish a benchmark to measure LLM agent's ability to tackle this challenge. We carefully curate 221 real-world test cases from eight popular dataset platforms and propose an automatic evaluation framework using GPT-4o. Our proposed framework shows strong empirical alignment with expert evaluations, validated through extensive comparisons with human annotations. Without any hints, most competitive Curator agent can only reveal 30\% of the data quality issues in the proposed dataset, highlighting the complexity of this task and indicating that applying LLM agents to real-world dataset curation still requires further in-depth exploration and innovation. The data and code are available at \href{https://github.com/TRAIS-Lab/dca-bench}{https://github.com/TRAIS-Lab/dca-bench}.
Paper Structure (44 sections, 1 equation, 3 figures, 17 tables)

This paper contains 44 sections, 1 equation, 3 figures, 17 tables.

Table of Contents

  1. Introduction
  2. The Dataset Curation Agent Benchmark
  3. Dataset Overview
  4. Test Case Construction
  5. Issue Collection and Preprocessing
  6. Hints Generation
  7. Benchmark API
  8. Evaluation Pipeline
  9. Experiments
  10. Validation of the Evaluator
  11. High Alignment with Human
  12. Consistent Evaluation
  13. Negligible Self-Preference and Length Bias
  14. Benchmarking the Baseline Curators
  15. Experimental Setup
  16. ...and 29 more sections

Figures (3)

  • Figure 1: Illustration of instances in DCA-Bench. The Issue Card displays a specific test case, including metadata such as Issue ID, source platform (e.g., BIG-Bench), type, associated tags, and hints. Relevant dataset files can be found using the Issue ID. Furthermore, DCA-Bench incorporates documentation from the dataset platform along with additional reference materials related to dataset curation and quality. We provide a convenient API to access data of each test case as well as reference materials. The Curator is asked to detect the issues in files by describing the issue context and pinpointing the location in the file where issues occur. The elements labeled title, content, and involved serve as ground truth for evaluating the Curator’s performance and are hidden from the Curator during testing.
  • Figure 2: The Task I/O of DCA-Bench. For each test case, the input for the Curator includes dataset files and hints, with reference materials and platform documentation being optional. The Curator is then required to provide a description of the issue and corresponding contextual evidence. The label of the test case includes the issue title, content, the involved file names, and corresponding contextual evidence. Given the output from the Curator and the label, the Evaluator is then asked to rate the performance of the Curator.
  • Figure 3: Number of success & failure and success rate versus different content length of test cases.