When Machines Get It Wrong: Large Language Models Perpetuate Autism Myths More Than Humans Do

TL;DR

This study compares autism knowledge between three leading LLMs (GPT-4, Claude, Gemini) and a human sample (n=178) using a 30-item instrument spanning myths and facts. Contrary to expectations, humans outperformed LLMs on the majority of items, though AI models correctly rejected well-documented biomedical misconceptions while retaining social-emotional myths. The findings reveal a critical blind spot in current AI systems for neurodiversity and have implications for human-AI interaction design, AI governance, and the need to center autistic perspectives in development. The work supports domain-specific validation and cautions against treating LLMs as universal authorities on stigmatized health information.

Abstract

As Large Language Models become ubiquitous sources of health information, understanding their capacity to accurately represent stigmatized conditions is crucial for responsible deployment. This study examines whether leading AI systems perpetuate or challenge misconceptions about Autism Spectrum Disorder, a condition particularly vulnerable to harmful myths. We administered a 30-item instrument measuring autism knowledge to 178 participants and three state-of-the-art LLMs including GPT-4, Claude, and Gemini. Contrary to expectations that AI systems would leverage their vast training data to outperform humans, we found the opposite pattern: human participants endorsed significantly fewer myths than LLMs (36.2% vs. 44.8% error rate; z = -2.59, p = .0048). In 18 of the 30 evaluated items, humans significantly outperformed AI systems. These findings reveal a critical blind spot in current AI systems and have important implications for human-AI interaction design, the epistemology of machine knowledge, and the need to center neurodivergent perspectives in AI development.