Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dark and Bright Side of Participatory Red-Teaming with Targets of Stereotyping for Eliciting Harmful Behaviors from Large Language Models

Sieun Kim, Yeeun Jo, Sungmin Na, Hyunseung Lim, Eunchae Lee, Yu Min Choi, Soohyun Cho, Hwajung Hong

Abstract

Red-teaming, where adversarial prompts are crafted to expose harmful behaviors and assess risks, offers a dynamic approach to surfacing underlying stereotypical bias in large language models. Because such subtle harms are best recognized by those with lived experience, involving targets of stereotyping as red-teamers is essential. However, critical challenges remain in leveraging their lived experience for red-teaming while safeguarding psychological well-being. We conducted an empirical study of participatory red-teaming with 20 individuals stigmatized by stereotypes against nonprestigious college graduates in South Korea. Through mixed methods analysis, we found participants transformed experienced discrimination into strategic expertise for identifying biases, while facing psychological costs such as stress and negative reflections on group identity. Notably, red-team participation enhanced their sense of agency and empowerment through their role as guardians of the AI ecosystem. We discuss implications for designing participatory red-teaming that prioritizes both the ethical treatment and empowerment of stigmatized groups.

Dark and Bright Side of Participatory Red-Teaming with Targets of Stereotyping for Eliciting Harmful Behaviors from Large Language Models

Abstract

Red-teaming, where adversarial prompts are crafted to expose harmful behaviors and assess risks, offers a dynamic approach to surfacing underlying stereotypical bias in large language models. Because such subtle harms are best recognized by those with lived experience, involving targets of stereotyping as red-teamers is essential. However, critical challenges remain in leveraging their lived experience for red-teaming while safeguarding psychological well-being. We conducted an empirical study of participatory red-teaming with 20 individuals stigmatized by stereotypes against nonprestigious college graduates in South Korea. Through mixed methods analysis, we found participants transformed experienced discrimination into strategic expertise for identifying biases, while facing psychological costs such as stress and negative reflections on group identity. Notably, red-team participation enhanced their sense of agency and empowerment through their role as guardians of the AI ecosystem. We discuss implications for designing participatory red-teaming that prioritizes both the ethical treatment and empowerment of stigmatized groups.
Paper Structure (55 sections, 3 figures, 4 tables)

This paper contains 55 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Stereotypical Bias in LLMs as Representational Harm
  4. Participatory Red-Teaming with Targets of Stereotypes
  5. The Stereotype Targeted in Our Study: Jibangdae in South Korea
  6. Method
  7. Recruitment
  8. Psychological Measures
  9. Study Design
  10. Overall Protocol
  11. Introductory Session
  12. Session 1 : Red-Teaming Task & Pre-Post Psychological Survey
  13. Meditation Break
  14. Session 2 : Interview & Debriefing
  15. Analysis
  16. ...and 40 more sections

Figures (3)

  • Figure 1: Study Procedure. After surveys and the red-teaming task (Session 1), participants completed a meditation break before Session 2 (interview and debriefing). Ethical safeguards—such as guided decompression, continuous distress monitoring, meditation break, and structured debriefing—were integrated throughout.
  • Figure 2: Red-teaming Documentation. Participants generated attacks on AI, judged harmfulness, and reflected on stereotypes by explaining their judgments with conversation excerpts.
  • Figure 3: After participating in the red-teaming task, psychological distress (SUDS), negative affect (NA), and stigma consciousness (SCQ) significantly increased, while collective self-esteem (CSES) significantly decreased. Individual self-esteem (RSES) and positive affect (PA) remained unchanged. ns; $p > 0.05$, * $p \leq 0.05$; ** $p \leq 0.01$; *** $p \leq 0.001$.