Real Risks of Fake Data: Synthetic Data, Diversity-Washing and Consent Circumvention

TL;DR

The paper argues that synthetic data, while useful for addressing data collection challenges, introduces two high-risk dynamics: diversity-washing, where synthetic augmentation superficially diversifies data without improving representational fidelity, and consent circumvention, which weakens regulatory and participatory safeguards around data usage. It grounds these claims in a real-world facial recognition evaluation and a critique of FTC enforcement, highlighting how synthetic data can obscure data provenance and complicate model deletion and accountability. The authors connect technical challenges with governance concerns, calling for participatory data stewardship and transparency to prevent power consolidation away from data subjects. Overall, the work emphasizes careful provenance, lineage, and governance frameworks to harness synthetic data’s potential without reinforcing harm or undermining consent.

Abstract

Machine learning systems require representations of the real world for training and testing - they require data, and lots of it. Collecting data at scale has logistical and ethical challenges, and synthetic data promises a solution to these challenges. Instead of needing to collect photos of real people's faces to train a facial recognition system, a model creator could create and use photo-realistic, synthetic faces. The comparative ease of generating this synthetic data rather than relying on collecting data has made it a common practice. We present two key risks of using synthetic data in model development. First, we detail the high risk of false confidence when using synthetic data to increase dataset diversity and representation. We base this in the examination of a real world use-case of synthetic data, where synthetic datasets were generated for an evaluation of facial recognition technology. Second, we examine how using synthetic data risks circumventing consent for data usage. We illustrate this by considering the importance of consent to the U.S. Federal Trade Commission's regulation of data collection and affected models. Finally, we discuss how these two risks exemplify how synthetic data complicates existing governance and ethical practice; by decoupling data from those it impacts, synthetic data is prone to consolidating power away those most impacted by algorithmically-mediated harm.