Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Disability-First AI Dataset Annotation: Co-designing Stuttered Speech Annotation Guidelines with People Who Stutter

Xinru Tang, Jingjin Li, Shaomei Wu

TL;DR

This paper addresses label noise and misalignment in AI accessibility data by introducing a disability-first approach to annotating stuttered speech. It employs a three-phase co-design process with people who stutter (PWS) and speech-language pathologists to develop and refine PWS-centered annotation guidelines, integrating embodied knowledge and addressing the intrinsic subjectivity of stuttering. Key contributions include the firstPWS-centered stuttered-speech annotation guidelines, an analysis of challenges in embedding embodied cues into labeled data, and a governance framework for continuous, community-driven annotation stewardship. The work demonstrates that disability expertise can improve dataset interpretation and calls for multiplicity-aware practices across the AI development pipeline to better represent diverse experiences and downstream applications.

Abstract

Despite efforts to increase the representation of disabled people in AI datasets, accessibility datasets are often annotated by crowdworkers without disability-specific expertise, leading to inconsistent or inaccurate labels. This paper examines these annotation challenges through a case study of annotating speech data from people who stutter (PWS). Given the variability of stuttering and differing views on how it manifests, annotating and transcribing stuttered speech remains difficult, even for trained professionals. Through interviews and co-design workshops with PWS and domain experts, we identify challenges in stuttered speech annotation and develop practices that integrate the lived experiences of PWS into the annotation process. Our findings highlight the value of embodied knowledge in improving dataset quality, while revealing tensions between the complexity of disability experiences and the rigidity of static labels. We conclude with implications for disability-first and multiplicity-aware approaches to data interpretation across the AI pipeline.

Disability-First AI Dataset Annotation: Co-designing Stuttered Speech Annotation Guidelines with People Who Stutter

TL;DR

This paper addresses label noise and misalignment in AI accessibility data by introducing a disability-first approach to annotating stuttered speech. It employs a three-phase co-design process with people who stutter (PWS) and speech-language pathologists to develop and refine PWS-centered annotation guidelines, integrating embodied knowledge and addressing the intrinsic subjectivity of stuttering. Key contributions include the firstPWS-centered stuttered-speech annotation guidelines, an analysis of challenges in embedding embodied cues into labeled data, and a governance framework for continuous, community-driven annotation stewardship. The work demonstrates that disability expertise can improve dataset interpretation and calls for multiplicity-aware practices across the AI development pipeline to better represent diverse experiences and downstream applications.

Abstract

Despite efforts to increase the representation of disabled people in AI datasets, accessibility datasets are often annotated by crowdworkers without disability-specific expertise, leading to inconsistent or inaccurate labels. This paper examines these annotation challenges through a case study of annotating speech data from people who stutter (PWS). Given the variability of stuttering and differing views on how it manifests, annotating and transcribing stuttered speech remains difficult, even for trained professionals. Through interviews and co-design workshops with PWS and domain experts, we identify challenges in stuttered speech annotation and develop practices that integrate the lived experiences of PWS into the annotation process. Our findings highlight the value of embodied knowledge in improving dataset quality, while revealing tensions between the complexity of disability experiences and the rigidity of static labels. We conclude with implications for disability-first and multiplicity-aware approaches to data interpretation across the AI pipeline.
Paper Structure (54 sections, 2 figures, 7 tables)

This paper contains 54 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Positionality
  3. Related Work
  4. Stuttering as a Contested Disability
  5. Disability-First AI Dataset Creation
  6. Stuttered Speech Datasets
  7. Study Overview
  8. Formative Studies
  9. Methods
  10. Datasets Review
  11. Interviews
  12. Findings
  13. Lack of Embodied Understandings of Stuttering
  14. Challenges in Transcribing Disfluent Speech Verbatim
  15. Inherent Subjectivity in Speech Perception
  16. ...and 39 more sections

Figures (2)

  • Figure 1: Study flow with main takeaways at each stage.
  • Figure 2: Screenshot of Praat interface, with a selected audio segment (in red) and corresponding editable text panel. The example shows how a prolongation might easily be overlooked due to selective listening and editing. The audio waveform corresponding to the prolongation labeled in I/ps occurs outside the selected segment.