Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can AI Model the Complexities of Human Moral Decision-Making? A Qualitative Study of Kidney Allocation Decisions

Vijay Keswani, Vincent Conitzer, Walter Sinnott-Armstrong, Breanna K. Nguyen, Hoda Heidari, Jana Schaich Borg

TL;DR

The paper investigates whether AI can accurately model the nuanced and dynamic moral judgments humans employ in kidney allocation decisions. Using semi-structured interviews with twenty lay participants, it reveals substantial variability in which patient features are deemed relevant, how those features are valued, and the decision rules people apply, including thresholds and feature hierarchies. It also finds that opinions can change with deliberation and that while many participants see potential for AI assistance, they advocate for substantial human oversight and caution due to concerns about bias and empathy. The authors argue that current AI approaches—often assuming stable, linear preferences—fall short of capturing the non-linear, interacting, and evolving nature of moral decision-making, and they propose more individualized, deliberation-aware elicitation and hybrid modeling strategies, including implications for governance and preference aggregation.

Abstract

A growing body of work in Ethical AI attempts to capture human moral judgments through simple computational models. The key question we address in this work is whether such simple AI models capture {the critical} nuances of moral decision-making by focusing on the use case of kidney allocation. We conducted twenty interviews where participants explained their rationale for their judgments about who should receive a kidney. We observe participants: (a) value patients' morally-relevant attributes to different degrees; (b) use diverse decision-making processes, citing heuristics to reduce decision complexity; (c) can change their opinions; (d) sometimes lack confidence in their decisions (e.g., due to incomplete information); and (e) express enthusiasm and concern regarding AI assisting humans in kidney allocation decisions. Based on these findings, we discuss challenges of computationally modeling moral judgments {as a stand-in for human input}, highlight drawbacks of current approaches, and suggest future directions to address these issues.

Can AI Model the Complexities of Human Moral Decision-Making? A Qualitative Study of Kidney Allocation Decisions

TL;DR

The paper investigates whether AI can accurately model the nuanced and dynamic moral judgments humans employ in kidney allocation decisions. Using semi-structured interviews with twenty lay participants, it reveals substantial variability in which patient features are deemed relevant, how those features are valued, and the decision rules people apply, including thresholds and feature hierarchies. It also finds that opinions can change with deliberation and that while many participants see potential for AI assistance, they advocate for substantial human oversight and caution due to concerns about bias and empathy. The authors argue that current AI approaches—often assuming stable, linear preferences—fall short of capturing the non-linear, interacting, and evolving nature of moral decision-making, and they propose more individualized, deliberation-aware elicitation and hybrid modeling strategies, including implications for governance and preference aggregation.

Abstract

A growing body of work in Ethical AI attempts to capture human moral judgments through simple computational models. The key question we address in this work is whether such simple AI models capture {the critical} nuances of moral decision-making by focusing on the use case of kidney allocation. We conducted twenty interviews where participants explained their rationale for their judgments about who should receive a kidney. We observe participants: (a) value patients' morally-relevant attributes to different degrees; (b) use diverse decision-making processes, citing heuristics to reduce decision complexity; (c) can change their opinions; (d) sometimes lack confidence in their decisions (e.g., due to incomplete information); and (e) express enthusiasm and concern regarding AI assisting humans in kidney allocation decisions. Based on these findings, we discuss challenges of computationally modeling moral judgments {as a stand-in for human input}, highlight drawbacks of current approaches, and suggest future directions to address these issues.

Paper Structure

This paper contains 49 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Moral Cognition and Decision-Making
  4. AI for Moral Decision-Making
  5. Prior Works on Limitations of Moral Preference Elicitation
  6. AI Explainability and Interpretability
  7. Methodology
  8. Background on Kidney Allocation
  9. Recruitment
  10. Interview Format and Data Collection
  11. (A) Feature relevance.
  12. (B) Decision-making for pairwise comparisons.
  13. (C) AI attitudes.
  14. Analysis and Coding
  15. Researcher Positionality
  16. ...and 34 more sections

Figures (4)

  • Figure 1: A high-level overview of participants' moral decision model for pairwise comparisons in the kidney allocation domain. Note that this only represents participants whose strategies could be clearly represented using decision rules. Not all presented components are used by all participants and this chart doesn't capture cases where participants changed their previously expressed opinions.
  • Figure 2: Pairwise Comparison #2. The second pairwise comparison presented to all interview participants.
  • Figure 3: Pairwise Comparison #3. The third pairwise comparison presented to all interview participants.
  • Figure 4: Aggregated alignment scores for all five strategies discussed with the participants.