Detecting the Use of Generative AI in Crowdsourced Surveys: Implications for Data Integrity

TL;DR

Generative AI tools enable automated responses in crowdsourced surveys, threatening data integrity and potentially biasing findings. The authors compare two detection strategies—LLM-based zero-shot detection and signature-based detection that uses LLM-generated signatures and SBERT similarity—to identify AI-generated survey responses across seven studies spanning pre- and post-ChatGPT release. They demonstrate that GenAI usage increases after 2022 and that signature-based detection not only detects AI-generated inputs but also helps flag irrelevant or low-quality responses, providing a practical safeguard for data quality. The work highlights the need for adaptive methodologies to safeguard empirical research in sensitive domains such as health, politics, and social behavior.

Abstract

The widespread adoption of generative AI (GenAI) has introduced new challenges in crowdsourced data collection, particularly in survey-based research. While GenAI offers powerful capabilities, its unintended use in crowdsourcing, such as generating automated survey responses, threatens the integrity of empirical research and complicates efforts to understand public opinion and behavior. In this study, we investigate and evaluate two approaches for detecting AI-generated responses in online surveys: LLM-based detection and signature-based detection. We conducted experiments across seven survey studies, comparing responses collected before 2022 with those collected after the release of ChatGPT. Our findings reveal a significant increase in AI-generated responses in the post-2022 studies, highlighting how GenAI may silently distort crowdsourced data. This work raises broader concerns about evolving landscape of data integrity, where GenAI can compromise data quality, mislead researchers, and influence downstream findings in fields such as health, politics, and social behavior. By surfacing detection strategies and empirical evidence of GenAI's impact, we aim to contribute to ongoing conversation about safeguarding research integrity and supporting scholars navigating these methodological and ethical challenges.

Figures (3)

• Figure 1: Overview of Proposed Approaches for Detecting the Use of GenAI in Crowdsourcing.
• Figure 2: Distribution of Similarity between Collected Responses and Signatures for Pre-2022 and Post-2022 Studies.
• Figure 3: Examples of AI-generated Responses and Signatures with High and Low Similarity Scores.