Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AI Chatbots for Mental Health: Values and Harms from Lived Experiences of Depression

Dong Whi Yoo, Jiayue Melissa Shi, Violeta J. Rodriguez, Koustuv Saha

TL;DR

The paper addresses the risks and values associated with AI chatbots in mental health, focusing on depression self-management. It introduces Zenny, a GPT-4o-based technology probe, and conducts semi-structured interviews with 17 people with lived depression experiences to identify prioritized values and potential harms. Through reflexive thematic analysis, five core values emerge—informational and emotional support, personalization, privacy, and crisis management—and are mapped to concrete design recommendations and a harm-mitigation checklist. The work advances value-sensitive AI design for mental health by grounding harms in user lived experiences and proposing governance, transparency, and clinical-compatibility considerations to guide responsible deployment. Its findings offer a practical pathway to develop AI chatbots that complement human care while minimizing risks for a highly sensitive population.

Abstract

Recent advancements in LLMs enable chatbots to interact with individuals on a range of queries, including sensitive mental health contexts. Despite uncertainties about their effectiveness and reliability, the development of LLMs in these areas is growing, potentially leading to harms. To better identify and mitigate these harms, it is critical to understand how the values of people with lived experiences relate to the harms. In this study, we developed a technology probe, a GPT-4o based chatbot called Zenny, enabling participants to engage with depression self-management scenarios informed by previous research. We used Zenny to interview 17 individuals with lived experiences of depression. Our thematic analysis revealed key values: informational support, emotional support, personalization, privacy, and crisis management. This work explores the relationship between lived experience values, potential harms, and design recommendations for mental health AI chatbots, aiming to enhance self-management support while minimizing risks.

AI Chatbots for Mental Health: Values and Harms from Lived Experiences of Depression

TL;DR

The paper addresses the risks and values associated with AI chatbots in mental health, focusing on depression self-management. It introduces Zenny, a GPT-4o-based technology probe, and conducts semi-structured interviews with 17 people with lived depression experiences to identify prioritized values and potential harms. Through reflexive thematic analysis, five core values emerge—informational and emotional support, personalization, privacy, and crisis management—and are mapped to concrete design recommendations and a harm-mitigation checklist. The work advances value-sensitive AI design for mental health by grounding harms in user lived experiences and proposing governance, transparency, and clinical-compatibility considerations to guide responsible deployment. Its findings offer a practical pathway to develop AI chatbots that complement human care while minimizing risks for a highly sensitive population.

Abstract

Recent advancements in LLMs enable chatbots to interact with individuals on a range of queries, including sensitive mental health contexts. Despite uncertainties about their effectiveness and reliability, the development of LLMs in these areas is growing, potentially leading to harms. To better identify and mitigate these harms, it is critical to understand how the values of people with lived experiences relate to the harms. In this study, we developed a technology probe, a GPT-4o based chatbot called Zenny, enabling participants to engage with depression self-management scenarios informed by previous research. We used Zenny to interview 17 individuals with lived experiences of depression. Our thematic analysis revealed key values: informational support, emotional support, personalization, privacy, and crisis management. This work explores the relationship between lived experience values, potential harms, and design recommendations for mental health AI chatbots, aiming to enhance self-management support while minimizing risks.
Paper Structure (31 sections, 1 figure, 4 tables)

This paper contains 31 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. AI Ethics and Values: Anticipating Harms
  4. Chatbots and LLMs for Mental Health
  5. Depression Research in HCI
  6. Interview Study using Technology Probe
  7. Zenny: Technology Probe for Depression Self-Management
  8. Scenario Design
  9. Development of Zenny
  10. Participant Recruitment
  11. Safety Measures
  12. Interview Procedure
  13. Data Analysis
  14. Positionality Statement
  15. Findings
  16. ...and 16 more sections

Figures (1)

  • Figure 1: Screenshots of the technology probe used in our interview study. The technology probe consists of a chatbot called Zenny for depression self-management, which is built with GPT-4. (a) On the chat page, participants see a scenario borrowed from prior work van2015patients, and (b) participants can ask their own questions related to depression self-management.