Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Surveys Considered Harmful? Reflecting on the Use of Surveys in AI Research, Development, and Governance

Mohammmad Tahaei, Daricia Wilkinson, Alisa Frik, Michael Muller, Ruba Abu-Salma, Lauren Wilcox

TL;DR

Public surveys are increasingly used to guide AI research and governance, but risk amplifying Western-centric views and marginalizing non-Western perspectives. The authors pair a reflexive six-country pilot with a systematic review of 44 AI-survey studies to expose representational, ethical, and transparency shortcomings, and to propose heuristic questions for responsible survey design. Their findings reveal inconsistent reporting of ethics approvals, uneven geographic author representation, and limited community input, suggesting surveys alone cannot achieve inclusive engagement. The work aims to shift the field toward co-design, transparency, and critical reflexivity in survey methods to better align AI development with diverse societal needs.

Abstract

Calls for engagement with the public in Artificial Intelligence (AI) research, development, and governance are increasing, leading to the use of surveys to capture people's values, perceptions, and experiences related to AI. In this paper, we critically examine the state of human participant surveys associated with these topics. Through both a reflexive analysis of a survey pilot spanning six countries and a systematic literature review of 44 papers featuring public surveys related to AI, we explore prominent perspectives and methodological nuances associated with surveys to date. We find that public surveys on AI topics are vulnerable to specific Western knowledge, values, and assumptions in their design, including in their positioning of ethical concepts and societal values, lack sufficient critical discourse surrounding deployment strategies, and demonstrate inconsistent forms of transparency in their reporting. Based on our findings, we distill provocations and heuristic questions for our community, to recognize the limitations of surveys for meeting the goals of engagement, and to cultivate shared principles to design, deploy, and interpret surveys cautiously and responsibly.

Surveys Considered Harmful? Reflecting on the Use of Surveys in AI Research, Development, and Governance

TL;DR

Public surveys are increasingly used to guide AI research and governance, but risk amplifying Western-centric views and marginalizing non-Western perspectives. The authors pair a reflexive six-country pilot with a systematic review of 44 AI-survey studies to expose representational, ethical, and transparency shortcomings, and to propose heuristic questions for responsible survey design. Their findings reveal inconsistent reporting of ethics approvals, uneven geographic author representation, and limited community input, suggesting surveys alone cannot achieve inclusive engagement. The work aims to shift the field toward co-design, transparency, and critical reflexivity in survey methods to better align AI development with diverse societal needs.

Abstract

Calls for engagement with the public in Artificial Intelligence (AI) research, development, and governance are increasing, leading to the use of surveys to capture people's values, perceptions, and experiences related to AI. In this paper, we critically examine the state of human participant surveys associated with these topics. Through both a reflexive analysis of a survey pilot spanning six countries and a systematic literature review of 44 papers featuring public surveys related to AI, we explore prominent perspectives and methodological nuances associated with surveys to date. We find that public surveys on AI topics are vulnerable to specific Western knowledge, values, and assumptions in their design, including in their positioning of ethical concepts and societal values, lack sufficient critical discourse surrounding deployment strategies, and demonstrate inconsistent forms of transparency in their reporting. Based on our findings, we distill provocations and heuristic questions for our community, to recognize the limitations of surveys for meeting the goals of engagement, and to cultivate shared principles to design, deploy, and interpret surveys cautiously and responsibly.
Paper Structure (33 sections, 1 figure, 1 table)

This paper contains 33 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. A Brief History of Surveys: From Agriculture and Military to Computing and AI
  4. Rapid Expansion of Survey Use in AI Research, Development, and Governance
  5. Critically Reflexive Practices in AI Research
  6. Methods
  7. Pilot Survey of Perceived Benefits and Risks of AI
  8. Systematic Literature Review
  9. Large-Scale Surveys of AI in the Literature
  10. Discussion
  11. Pitfalls of Surveys as Enablers of "Participation"
  12. Representation Issues in Surveys
  13. True Representation or an Illusion of Representation?
  14. Cross-Cultural Surveys Considered Helpful or Harmful?
  15. Value Tensions in Surveys: Heuristic Questions
  16. ...and 18 more sections

Figures (1)

  • Figure 1: Prisma diagram for the systematic literature review. See Sections \ref{['sec:lit-review']} and \ref{['app:prisma']} for details.