Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Wisdom from Diversity: Bias Mitigation Through Hybrid Human-LLM Crowds

Axel Abels, Tom Lenaerts

TL;DR

This work evaluates social biases in large language models (LLMs) by comparing their responses to bias-eliciting headlines with human responses. It shows that naive averaging of multiple LLMs amplifies biases due to low diversity within LLM crowds, while locally weighted aggregation (via ExpertiseTrees) can both mitigate bias and improve accuracy. The study demonstrates that hybrid crowds combining human diversity with LLM accuracy yield the strongest performance and fairness gains, outperforming LLM-only and human-only groups. These findings highlight the promise of hybrid human-LLM ensembles for responsible decision-making in applications like content moderation and hiring, and they underscore the importance of context-sensitive aggregation in leveraging collective intelligence. The results also identify limitations, including reliance on a single dataset and the need for strategies to engineer diversity within LLM ensembles to further reduce biases.

Abstract

Despite their performance, large language models (LLMs) can inadvertently perpetuate biases found in the data they are trained on. By analyzing LLM responses to bias-eliciting headlines, we find that these models often mirror human biases. To address this, we explore crowd-based strategies for mitigating bias through response aggregation. We first demonstrate that simply averaging responses from multiple LLMs, intended to leverage the "wisdom of the crowd", can exacerbate existing biases due to the limited diversity within LLM crowds. In contrast, we show that locally weighted aggregation methods more effectively leverage the wisdom of the LLM crowd, achieving both bias mitigation and improved accuracy. Finally, recognizing the complementary strengths of LLMs (accuracy) and humans (diversity), we demonstrate that hybrid crowds containing both significantly enhance performance and further reduce biases across ethnic and gender-related contexts.

Wisdom from Diversity: Bias Mitigation Through Hybrid Human-LLM Crowds

TL;DR

This work evaluates social biases in large language models (LLMs) by comparing their responses to bias-eliciting headlines with human responses. It shows that naive averaging of multiple LLMs amplifies biases due to low diversity within LLM crowds, while locally weighted aggregation (via ExpertiseTrees) can both mitigate bias and improve accuracy. The study demonstrates that hybrid crowds combining human diversity with LLM accuracy yield the strongest performance and fairness gains, outperforming LLM-only and human-only groups. These findings highlight the promise of hybrid human-LLM ensembles for responsible decision-making in applications like content moderation and hiring, and they underscore the importance of context-sensitive aggregation in leveraging collective intelligence. The results also identify limitations, including reliance on a single dataset and the need for strategies to engineer diversity within LLM ensembles to further reduce biases.

Abstract

Despite their performance, large language models (LLMs) can inadvertently perpetuate biases found in the data they are trained on. By analyzing LLM responses to bias-eliciting headlines, we find that these models often mirror human biases. To address this, we explore crowd-based strategies for mitigating bias through response aggregation. We first demonstrate that simply averaging responses from multiple LLMs, intended to leverage the "wisdom of the crowd", can exacerbate existing biases due to the limited diversity within LLM crowds. In contrast, we show that locally weighted aggregation methods more effectively leverage the wisdom of the LLM crowd, achieving both bias mitigation and improved accuracy. Finally, recognizing the complementary strengths of LLMs (accuracy) and humans (diversity), we demonstrate that hybrid crowds containing both significantly enhance performance and further reduce biases across ethnic and gender-related contexts.
Paper Structure (36 sections, 4 equations, 5 figures, 2 tables)

This paper contains 36 sections, 4 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Social Biases in LLMs
  4. Collective Intelligence
  5. Wisdom of LLM Crowds
  6. Bias Mitigation in Human Crowds
  7. The Headline Experiment
  8. Restoring Crowd Wisdom in Human Groups
  9. Methods
  10. Experimental Procedure
  11. Metrics
  12. Accuracy
  13. Counterfactual Bias
  14. Framing Effects
  15. Diversity
  16. ...and 21 more sections

Figures (5)

  • Figure 1: A. Model accuracy across prompt variations, with each dot representing the accuracy of a model for a specific prompt ending. B. t-SNE visualization of responder diversity. C. Q-statistics (see Equation \ref{['eq:qstatistic']}) matrix. Each cell gives the Q-statistic between responder pairs.
  • Figure 2: Accuracy for different group sizes and aggregators. Shaded areas show $95\%$ confidence intervals. LLMs are either sampled randomly (A) or based on their MMLU scores (B).
  • Figure S3: The prompt presented to the LLMs.
  • Figure S4: Distribution of framing effects for headlines reporting positive or negative outcomes across three demographic groups. Asterisks indicate statistical significance of the framing effects (Wilcoxon test, *: $p<0.10$, **: $p<0.05$, ***: $p<0.01$).
  • Figure S5: Distribution of framing effects for headlines reporting positive or negative outcomes across three demographic groups. Asterisks indicate statistical significance of the framing effects (Wilcoxon test, *: $p<0.10$, **: $p<0.05$, ***: $p<0.01$).