First-Person Fairness in Chatbots

TL;DR

The study tackles the problem of evaluating fairness in open-ended chatbots from a first-person user perspective by introducing a scalable Language Model Research Assistant (LMRA) that analyzes name-based biases across 6 models, 66 tasks in 9 domains, and 2 genders with 4 races. It defines a counterfactual fairness framework using names to generate forward and reverse harm probabilities, culminating in a net harmfulness metric $H(A,B)=\mathbb{E}_{x\sim \phi}[h(x, A, B)]$ and related axes of difference to describe task-specific biases. Empirically, the approach reveals minimal quality disparities but detectable biases that vary by task and domain, with larger harmful stereotypes in open-ended tasks; importantly, post-training reinforcement learning significantly mitigates such biases, supporting a practical, scalable bias-monitoring pipeline. The work also demonstrates partial alignment between LMRA judgments and human ratings for gender bias, provides a reproducible methodology (with privacy-preserving steps), and outlines future directions for broader demographic scopes and multimodal settings. Overall, the paper offers a concrete, scalable framework for ongoing detection and mitigation of first-person biases in deployed chatbots, enabling more equitable user interactions in real-world usage.

Abstract

Evaluating chatbot fairness is crucial given their rapid proliferation, yet typical chatbot tasks (e.g., resume writing, entertainment) diverge from the institutional decision-making tasks (e.g., resume screening) which have traditionally been central to discussion of algorithmic fairness. The open-ended nature and diverse use-cases of chatbots necessitate novel methods for bias assessment. This paper addresses these challenges by introducing a scalable counterfactual approach to evaluate "first-person fairness," meaning fairness toward chatbot users based on demographic characteristics. Our method employs a Language Model as a Research Assistant (LMRA) to yield quantitative measures of harmful stereotypes and qualitative analyses of demographic differences in chatbot responses. We apply this approach to assess biases in six of our language models across millions of interactions, covering sixty-six tasks in nine domains and spanning two genders and four races. Independent human annotations corroborate the LMRA-generated bias evaluations. This study represents the first large-scale fairness evaluation based on real-world chat data. We highlight that post-training reinforcement learning techniques significantly mitigate these biases. This evaluation provides a practical methodology for ongoing bias monitoring and mitigation.

Figures (27)

• Figure 1: Some chatbots store names. Left: ChatGPT stores a user name for use in the current and future chats, when names are stated explicitly (top) or implicitly (bottom) by different users. Right: Inflection's Pi chatbot explicitly asks for every user's first name for use in chats.
• Figure 2: Top: Based on a query from the public LMSYS dataset, ChatGPT generally responds with either educational or engineering projects. ChatGPT's distribution of responses vary statistically as we artificially vary the name. Bottom: Response distributions vary unpredictably---changing "5" to "some" entirely shifts the response distribution to be the same for both names. Since chatbot responses are stochastic, biases are statistical in nature.
• Figure 3: Our LMRA template for identifying harmful stereotypes for two groups, e.g., is woman and is man for gender stereotypes. For any pair of responses, the above template is instantiated twice, with the responses swapped, and the results averaged to avoid order bias. The decision to hide which response was generated for which group was found to improve performance.
• Figure 4: Abbreviated version of the feature brainstorming template, full version in \ref{['fig:brainstorm']} of \ref{['sec:enum-details']}.
• Figure 5: Top: harmful gender bias ratings for some of the most biased tasks across domains and models, using CI. The write a story task exhibited the greatest rate of harms, and the early model GPT-3.5-turbo exhibited the greatest harm rate. Bottom: harmful gender bias ratings for an unweighted random sample of 20K chats for both CI and Memory (except for ChatGPT-3.5-turbo which predated Memory). In both plots, error bars represent 95% confidence intervals calculated using the t-distribution.