"They are uncultured": Unveiling Covert Harms and Social Threats in LLM Generated Conversations

TL;DR

The paper introduces CHAST, a seven-metric framework to uncover covert harms and social threats in LLM-generated conversations within hiring scenarios, addressing a gap in cross-cultural bias research by focusing on race and caste. It combines a 1,920-dialogue dataset across eight LLMs with expert-annotated gold standards and GPT-4-based scaling to validate CHAST, and further provides a fine-tuned open-source Vicuna-13b-16K to promote reproducibility. Findings show that seven of eight LLMs produce CHAST content, with caste-based conversations exhibiting stronger harms than race-based ones, while mainstream toxicity detectors fail to flag many of these covert harms. The work emphasizes the readiness gap of LLM-powered recruitment tools, urging broader auditing across Global South contexts and the development of methods capable of detecting subtle social threats beyond overt toxicity. It also advances scientific practice by releasing open-source tools and prompting strategies to align evaluation models with human judgments."

Abstract

Large language models (LLMs) have emerged as an integral part of modern societies, powering user-facing applications such as personal assistants and enterprise applications like recruitment tools. Despite their utility, research indicates that LLMs perpetuate systemic biases. Yet, prior works on LLM harms predominantly focus on Western concepts like race and gender, often overlooking cultural concepts from other parts of the world. Additionally, these studies typically investigate "harm" as a singular dimension, ignoring the various and subtle forms in which harms manifest. To address this gap, we introduce the Covert Harms and Social Threats (CHAST), a set of seven metrics grounded in social science literature. We utilize evaluation models aligned with human assessments to examine the presence of covert harms in LLM-generated conversations, particularly in the context of recruitment. Our experiments reveal that seven out of the eight LLMs included in this study generated conversations riddled with CHAST, characterized by malign views expressed in seemingly neutral language unlikely to be detected by existing methods. Notably, these LLMs manifested more extreme views and opinions when dealing with non-Western concepts like caste, compared to Western ones such as race.

Figures (21)

• Figure 1: Pipeline Overview. We prompt LLMs with a dialogue between two colleagues (depicted as icons) in various hiring scenarios, varying based on race and caste attributes. The LLMs generate the remaining conversation about an applicant for a job. Using a human-validated LLM, we measure Chast metrics in the generated conversations, detecting (subtle) harms regarding group identity that Perspective API and other baseline models often miss.
• Figure 2: Heatmaps of Chast scores by occupation for caste (left) and race (right) on 1,920 LLM-generated conversations. Scores for caste are significantly higher in all LLMs, except for GPT-4-Turbo, where both concepts exhibit safe scores. The heatmaps are ordered based on the LLMs that generated least (top) to most (bottom) Chast in the conversations.
• Figure 3: Heatmaps of mean Chast scores by LLM for caste (left) and race (right). Scores for caste are significantly higher in all LLMs, except for GPT-4-Turbo, where both race and caste concepts exhibit safe scores.
• Figure 4: Heatmaps of mean Chast scores by occupation and LLM for caste (left) and race (right). 5 out of 8 LLMs generate higher Chast mean scores for older occupations that date back centuries (e.g. doctor, nurse, teacher) in the caste context compared to relatively modern occupations, such as software developer.
• Figure 5: Bar plots illustrating the comparison of binarized Chast scores for 1,920 conversations generated from eight LLMs for caste and race. Scores computed by GPT-4-Turbo.