Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AI Failure Loops in Devalued Work: The Confluence of Overconfidence in AI and Underconfidence in Worker Expertise

Anna Kawakami, Jordan Taylor, Sarah Fox, Haiyi Zhu, Kenneth Holstein

TL;DR

This paper investigates why AI deployments in feminized labor—teaching, social work, and home health care—frequently fail to augment frontline workers. It introduces AI Failure Loops, a framework of six interrelated failure modes that arise when worker expertise is undervalued and AI capabilities are overestimated, spanning design, evaluation, and governance. Through three case studies, the authors show how misjudgments about workers’ tacit knowledge, coupled with power imbalances and gatekeeping, produce deployments that erode worker visibility and value. The work advocates for worker-centered, pro-worker AI practices, participatory design, and policy measures to ensure AI technologies enhance rather than undermine dignity and expertise in feminized occupations.

Abstract

A growing body of literature has focused on understanding and addressing workplace AI design failures. However, past work has largely overlooked the role of the devaluation of worker expertise in shaping the dynamics of AI development and deployment. In this paper, we examine the case of feminized labor: a class of devalued occupations historically misnomered as ``women's work,'' such as social work, K-12 teaching, and home healthcare. Drawing on literature on AI deployments in feminized labor contexts, we conceptualize AI Failure Loops: a set of interwoven, socio-technical failure modes that help explain how the systemic devaluation of workers' expertise negatively impacts, and is impacted by, AI design, evaluation, and governance practices. These failures demonstrate how misjudgments on the automatability of workers' skills can lead to AI deployments that fail to bring value to workers and, instead, further diminish the visibility of workers' expertise. We discuss research and design implications for workplace AI, especially for devalued occupations.

AI Failure Loops in Devalued Work: The Confluence of Overconfidence in AI and Underconfidence in Worker Expertise

TL;DR

This paper investigates why AI deployments in feminized labor—teaching, social work, and home health care—frequently fail to augment frontline workers. It introduces AI Failure Loops, a framework of six interrelated failure modes that arise when worker expertise is undervalued and AI capabilities are overestimated, spanning design, evaluation, and governance. Through three case studies, the authors show how misjudgments about workers’ tacit knowledge, coupled with power imbalances and gatekeeping, produce deployments that erode worker visibility and value. The work advocates for worker-centered, pro-worker AI practices, participatory design, and policy measures to ensure AI technologies enhance rather than undermine dignity and expertise in feminized occupations.

Abstract

A growing body of literature has focused on understanding and addressing workplace AI design failures. However, past work has largely overlooked the role of the devaluation of worker expertise in shaping the dynamics of AI development and deployment. In this paper, we examine the case of feminized labor: a class of devalued occupations historically misnomered as ``women's work,'' such as social work, K-12 teaching, and home healthcare. Drawing on literature on AI deployments in feminized labor contexts, we conceptualize AI Failure Loops: a set of interwoven, socio-technical failure modes that help explain how the systemic devaluation of workers' expertise negatively impacts, and is impacted by, AI design, evaluation, and governance practices. These failures demonstrate how misjudgments on the automatability of workers' skills can lead to AI deployments that fail to bring value to workers and, instead, further diminish the visibility of workers' expertise. We discuss research and design implications for workplace AI, especially for devalued occupations.

Paper Structure

This paper contains 32 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Addressing Challenges in the Deployment of Workplace Technologies
  3. Devaluation of Feminized Labor
  4. Methods
  5. Team Experience and Case Selection
  6. Literature Review and Analysis
  7. AI Failure Loops
  8. Case Studies
  9. AI-Based Risk Assessment Tools Have Amplified Misconceptions of Social Workers' Expertise
  10. Context
  11. AI Failures in Social Work
  12. AI tool design fundamentally misunderstands social workers' expertise and overlooks their needs
  13. Developers interpret worker complaints around AI as signals of worker deficits
  14. Social workers are forced to use AI, even when they disagree
  15. "AI for Healthcare" Has Left Behind Devalued Healthcare Experts
  16. ...and 17 more sections

Figures (1)

  • Figure 1: A visual overview of AI Failure Loops. The six failure modes (in circles) that contribute to the dynamics of the AI Failure Loops exist within a web to illustrate the inter-connected relationship amongst the failure modes.