The illusion of artificial inclusion

TL;DR

The paper interrogates whether substituting human participants with LLMs is a viable path in AI development and social-behavioral research. Through a scoping review of sixteen sources, it identifies four primary motivations—speed/scale, cost reduction, diversity augmentation, and harms protection—while noting substantial practical and intrinsic challenges. Technical issues (hallucinations, value lock-in, bias) and epistemic-social tensions (representation, inclusion, intersubjectivity) undermine substitution and clash with core research values. The authors argue for reimagining AI's role to genuinely center and empower participants, proposing governance and participatory frameworks that sustain authentic inclusion and accountability throughout development and research processes.

Abstract

Human participants play a central role in the development of modern artificial intelligence (AI) technology, in psychological science, and in user research. Recent advances in generative AI have attracted growing interest to the possibility of replacing human participants in these domains with AI surrogates. We survey several such "substitution proposals" to better understand the arguments for and against substituting human participants with modern generative AI. Our scoping review indicates that the recent wave of these proposals is motivated by goals such as reducing the costs of research and development work and increasing the diversity of collected data. However, these proposals ignore and ultimately conflict with foundational values of work with human participants: representation, inclusion, and understanding. This paper critically examines the principles and goals underlying human participation to help chart out paths for future work that truly centers and empowers participants.