Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AI Mismatches: Identifying Potential Algorithmic Harms Before AI Development

Devansh Saxena, Ji-Youn Jung, Jodi Forlizzi, Kenneth Holstein, John Zimmerman

TL;DR

This paper defines AI Mismatches as the gap between the required performance to deliver value and minimize harm and the realistic model performance achievable under real-world constraints. It introduces seven AI Mismatch Matrices (three core performance-focused and four supporting) derived from 774 real-world cases, visualized as 3×3 grids to illuminate risk regions. Through three comparative case studies (AFST vs Hello Baby, GizmodoBot vs DuolingoLLM, and CLEAR vs Fraud Detect), the authors demonstrate how early-stage analysis can reveal feasibility, equity, and harm issues and guide safer, more worker-centered AI ideation. The study argues that moderate AI performance, combined with robust error detection and human-in-the-loop processes, can reduce risk and still create meaningful value, offering a practical framework for responsible AI design and future research directions.

Abstract

AI systems are often introduced with high expectations, yet many fail to deliver, resulting in unintended harm and missed opportunities for benefit. We frequently observe significant "AI Mismatches", where the system's actual performance falls short of what is needed to ensure safety and co-create value. These mismatches are particularly difficult to address once development is underway, highlighting the need for early-stage intervention. Navigating complex, multi-dimensional risk factors that contribute to AI Mismatches is a persistent challenge. To address it, we propose an AI Mismatch approach to anticipate and mitigate risks early on, focusing on the gap between realistic model performance and required task performance. Through an analysis of 774 AI cases, we extracted a set of critical factors, which informed the development of seven matrices that map the relationships between these factors and highlight high-risk areas. Through case studies, we demonstrate how our approach can help reduce risks in AI development.

AI Mismatches: Identifying Potential Algorithmic Harms Before AI Development

TL;DR

This paper defines AI Mismatches as the gap between the required performance to deliver value and minimize harm and the realistic model performance achievable under real-world constraints. It introduces seven AI Mismatch Matrices (three core performance-focused and four supporting) derived from 774 real-world cases, visualized as 3×3 grids to illuminate risk regions. Through three comparative case studies (AFST vs Hello Baby, GizmodoBot vs DuolingoLLM, and CLEAR vs Fraud Detect), the authors demonstrate how early-stage analysis can reveal feasibility, equity, and harm issues and guide safer, more worker-centered AI ideation. The study argues that moderate AI performance, combined with robust error detection and human-in-the-loop processes, can reduce risk and still create meaningful value, offering a practical framework for responsible AI design and future research directions.

Abstract

AI systems are often introduced with high expectations, yet many fail to deliver, resulting in unintended harm and missed opportunities for benefit. We frequently observe significant "AI Mismatches", where the system's actual performance falls short of what is needed to ensure safety and co-create value. These mismatches are particularly difficult to address once development is underway, highlighting the need for early-stage intervention. Navigating complex, multi-dimensional risk factors that contribute to AI Mismatches is a persistent challenge. To address it, we propose an AI Mismatch approach to anticipate and mitigate risks early on, focusing on the gap between realistic model performance and required task performance. Through an analysis of 774 AI cases, we extracted a set of critical factors, which informed the development of seven matrices that map the relationships between these factors and highlight high-risk areas. Through case studies, we demonstrate how our approach can help reduce risks in AI development.

Paper Structure

This paper contains 54 sections, 15 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. AI as a Design Material
  4. Responsible AI and FATE Research
  5. Method
  6. Requirements
  7. Overview of the Process
  8. Collecting AI Examples
  9. AI Examples Analysis and Extracting Risk Factors
  10. Synthesizing Risk Factors
  11. Overview of the Matrices
  12. Performance Mismatch Matrices
  13. Examples of Lower-Level, Supporting Matrices
  14. Scaffolding Reflection on Expected Performance
  15. Scaffolding Reflection on the Severity of Error Consequences
  16. ...and 39 more sections

Figures (15)

  • Figure 1: Motivations and Interconnections of the Seven AI Mismatch Matrices
  • Figure 2: The Required Performance matrix
  • Figure 3: The Disparate Performance matrix.
  • Figure 4: The Cost of Errors matrix.
  • Figure 5: The Data Quality matrix.
  • ...and 10 more figures