Examining the Expanding Role of Synthetic Data Throughout the AI Development Pipeline

TL;DR

The paper investigates the expanding use of synthetic data across the AI development pipeline, revealing that auxiliary generative models increasingly generate data, labels, and evaluation signals used to train, test, and debug primary models. Through 29 interviews with AI practitioners and responsible AI experts, the authors map motivations, practices, and challenges, highlighting benefits in scalability and controllability but also significant concerns about output control, representational fairness, and validation bottlenecks. The study identifies four top-level themes: pervasive integration of synthetic data, generation and validation challenges, ethical and governance considerations, and organizational pressures that prioritize speed over rigor. The authors conclude with concrete considerations for improving responsible use, including better validation, documentation, stakeholder engagement, and policy reporting to balance efficiency with accountability in the AI supply chain.

Abstract

Alongside the growth of generative AI, we are witnessing a surge in the use of synthetic data across all stages of the AI development pipeline. It is now common practice for researchers and practitioners to use one large generative model (which we refer to as an auxiliary model) to generate synthetic data that is used to train or evaluate another, reconfiguring AI workflows and reshaping the very nature of data. While scholars have raised concerns over the risks of synthetic data, policy guidance and best practices for its responsible use have not kept up with these rapidly evolving industry trends, in part because we lack a clear picture of current practices and challenges. Our work aims to address this gap. Through 29 interviews with AI practitioners and responsible AI experts, we examine the expanding role of synthetic data in AI development. Our findings reveal how auxiliary models are now widely used across the AI development pipeline. Practitioners describe synthetic data as crucial for addressing data scarcity and providing a competitive edge, noting that evaluation of generative AI systems at scale would be infeasible without auxiliary models. However, they face challenges controlling the outputs of auxiliary models, generating data that accurately depict underrepresented groups, and scaling data validation practices that are based primarily on manual inspection. We detail general limitations of and ethical considerations for synthetic data and conclude with a proposal of concrete steps towards the development of best practices for its responsible use.