Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets

Ann Yuan, Daphne Ippolito, Vitaly Nikolaev, Chris Callison-Burch, Andy Coenen, Sebastian Gehrmann

TL;DR

This work tackles the scarcity and bias in high-quality NLP benchmarks by proposing a human–AI collaborative workflow for dataset curation. It demonstrates SynthBio, a synthetic WikiBio benchmark created via seed LM generations edited by human raters, yielding a dataset with reduced noise and more balanced demographics. Evaluations show SynthBio achieves greater faithfulness and coverage than WikiBio, while remaining fluent, and reveal that models trained on WikiBio struggle more on SynthBio, highlighting the value of grounded evaluation. The study argues for controlled, human-guided synthetic data as a practical tool for robust benchmarking and broader distributional testing in NLG tasks.

Abstract

NLP researchers need more, higher-quality text datasets. Human-labeled datasets are expensive to collect, while datasets collected via automatic retrieval from the web such as WikiBio are noisy and can include undesired biases. Moreover, data sourced from the web is often included in datasets used to pretrain models, leading to inadvertent cross-contamination of training and test sets. In this work we introduce a novel method for efficient dataset curation: we use a large language model to provide seed generations to human raters, thereby changing dataset authoring from a writing task to an editing task. We use our method to curate SynthBio - a new evaluation set for WikiBio - composed of structured attribute lists describing fictional individuals, mapped to natural language biographies. We show that our dataset of fictional biographies is less noisy than WikiBio, and also more balanced with respect to gender and nationality.

SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets

TL;DR

This work tackles the scarcity and bias in high-quality NLP benchmarks by proposing a human–AI collaborative workflow for dataset curation. It demonstrates SynthBio, a synthetic WikiBio benchmark created via seed LM generations edited by human raters, yielding a dataset with reduced noise and more balanced demographics. Evaluations show SynthBio achieves greater faithfulness and coverage than WikiBio, while remaining fluent, and reveal that models trained on WikiBio struggle more on SynthBio, highlighting the value of grounded evaluation. The study argues for controlled, human-guided synthetic data as a practical tool for robust benchmarking and broader distributional testing in NLG tasks.

Abstract

NLP researchers need more, higher-quality text datasets. Human-labeled datasets are expensive to collect, while datasets collected via automatic retrieval from the web such as WikiBio are noisy and can include undesired biases. Moreover, data sourced from the web is often included in datasets used to pretrain models, leading to inadvertent cross-contamination of training and test sets. In this work we introduce a novel method for efficient dataset curation: we use a large language model to provide seed generations to human raters, thereby changing dataset authoring from a writing task to an editing task. We use our method to curate SynthBio - a new evaluation set for WikiBio - composed of structured attribute lists describing fictional individuals, mapped to natural language biographies. We show that our dataset of fictional biographies is less noisy than WikiBio, and also more balanced with respect to gender and nationality.

Paper Structure

This paper contains 54 sections, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. SynthBio: A Synthesized Benchmark for WikiBio
  3. WikiBio
  4. SynthBio
  5. Dataset curation methodology
  6. Step 1: Synthesizing Attribute Lists
  7. Programmatically selected attributes
  8. LM Generation of additional attributes
  9. Human revision of infoboxes
  10. Human revision of infoboxes
  11. Step 2: Synthesizing Biographies
  12. LM generation of biographies
  13. Human revision of biographies
  14. Dataset evaluation
  15. Automated evaluation
  16. ...and 39 more sections

Figures (3)

  • Figure 1: On the left, we show an overview of the steps involved in SynthBio's construction. On the right, we show examples of the prompts used for generation and the instructions given to raters. are programmatic steps--selecting a notability type then randomly sampling basic attributes from a schema. are steps that query the Llm-Dialog. are steps that involve human raters. are steps that query the Llm. Text that is generated by a language model is bolded.
  • Figure 2: Types of individuals included in SynthBio, along with their respective representations in WikiBio and SynthBio.
  • Figure 3: The distribution of continents for the "birth place" attribute compared to the distribution of world population world-population. For 23% of WikiBio and 8% of SynthBio infoboxes, continent information was unavailable either because the attribute was missing or the Geocoding API failed to parse it.