Locating Risk: Task Designers and the Challenge of Risk Disclosure in RAI Content Work

TL;DR

This study foregrounds task designers as the pivotal yet under-supported actors in the risk landscape of crowdwork for Responsible AI. Through 23 semi-structured interviews across academia and industry, it reveals that risk disclosure is uneven, context-dependent, and often occurs without shared standards or formal accountability. The findings show risk communication happens across three design stages—conceptualization, specification, and evaluation—and is shaped by personal ethics, platform affordances, and organizational norms, frequently trading off against data quality. The authors argue for reimagining disclosure as an integrated design and infrastructural challenge, with better tools, norms, and accountability to protect crowdworkers and bolster ethical AI development.

Abstract

As AI systems are increasingly tested and deployed in open-ended and high-stakes domains, crowd workers are often tasked with responsible AI (RAI) content work. These tasks include labeling violent content, moderating disturbing text, or simulating harmful behavior for red teaming exercises to shape AI system behaviors. While prior efforts have highlighted the risks to worker well-being associated with RAI content work, far less attention has been paid to how these risks are communicated to workers. Existing transparency frameworks and guidelines such as model cards, datasheets, and crowdworksheets focus on documenting model information and dataset collection processes, but they overlook an important aspect of disclosing well-being risks to workers. In the absence of standard workflows or clear guidance, the consistent application of content warnings, consent flows, or other forms of well-being risk disclosure remain unclear. This study investigates how task designers approach risk disclosure in crowdsourced RAI tasks. Drawing on interviews with 23 task designers across academic and industry sectors, we examine how well-being risk is recognized, interpreted, and communicated in practice. Our findings surface a need to support task designers in identifying and communicating well-being risk not only to support crowdworker well-being but also to strengthen the ethical integrity and technical efficacy of AI development pipelines.

Figures (2)

• Figure 1: Prolific's warning for sensitive content in tasks from the task designer interface.
• Figure 2: Risk-related decision-making across stages of AI task development.