Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception

Luyang Lin, Lingzhi Wang, Jinsong Guo, Kam-Fai Wong

TL;DR

This work tackles the problem of biases embedded in LLMs when used for media bias detection, distinguishing system biases from content bias and outlining four research questions. It introduces two evaluation perspectives—LLM-based political bias prediction and article continuation—and leverages FlipBias and ABP datasets to quantify biases, including a Bias Tendency Index (BTI). Through prompt-based debiasing and selective fine-tuning, the study shows that LLMs exhibit an overall left-leaning tendency with topic-dependent variation, and that prompt-based strategies can reduce bias with modest trade-offs in accuracy, while finetuning can reduce bias but risk adding new biases. The findings underscore the need for robust debiasing in LLM-powered bias-detection pipelines and highlight cross-model variation, informing the design of fairer AI systems for media analysis.

Abstract

The pervasive spread of misinformation and disinformation in social media underscores the critical importance of detecting media bias. While robust Large Language Models (LLMs) have emerged as foundational tools for bias prediction, concerns about inherent biases within these models persist. In this work, we investigate the presence and nature of bias within LLMs and its consequential impact on media bias detection. Departing from conventional approaches that focus solely on bias detection in media content, we delve into biases within the LLM systems themselves. Through meticulous examination, we probe whether LLMs exhibit biases, particularly in political bias prediction and text continuation tasks. Additionally, we explore bias across diverse topics, aiming to uncover nuanced variations in bias expression within the LLM framework. Importantly, we propose debiasing strategies, including prompt engineering and model fine-tuning. Extensive analysis of bias tendencies across different LLMs sheds light on the broader landscape of bias propagation in language models. This study advances our understanding of LLM bias, offering critical insights into its implications for bias detection tasks and paving the way for more robust and equitable AI systems

Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception

TL;DR

This work tackles the problem of biases embedded in LLMs when used for media bias detection, distinguishing system biases from content bias and outlining four research questions. It introduces two evaluation perspectives—LLM-based political bias prediction and article continuation—and leverages FlipBias and ABP datasets to quantify biases, including a Bias Tendency Index (BTI). Through prompt-based debiasing and selective fine-tuning, the study shows that LLMs exhibit an overall left-leaning tendency with topic-dependent variation, and that prompt-based strategies can reduce bias with modest trade-offs in accuracy, while finetuning can reduce bias but risk adding new biases. The findings underscore the need for robust debiasing in LLM-powered bias-detection pipelines and highlight cross-model variation, informing the design of fairer AI systems for media analysis.

Abstract

The pervasive spread of misinformation and disinformation in social media underscores the critical importance of detecting media bias. While robust Large Language Models (LLMs) have emerged as foundational tools for bias prediction, concerns about inherent biases within these models persist. In this work, we investigate the presence and nature of bias within LLMs and its consequential impact on media bias detection. Departing from conventional approaches that focus solely on bias detection in media content, we delve into biases within the LLM systems themselves. Through meticulous examination, we probe whether LLMs exhibit biases, particularly in political bias prediction and text continuation tasks. Additionally, we explore bias across diverse topics, aiming to uncover nuanced variations in bias expression within the LLM framework. Importantly, we propose debiasing strategies, including prompt engineering and model fine-tuning. Extensive analysis of bias tendencies across different LLMs sheds light on the broader landscape of bias propagation in language models. This study advances our understanding of LLM bias, offering critical insights into its implications for bias detection tasks and paving the way for more robust and equitable AI systems
Paper Structure (42 sections, 3 equations, 12 figures, 13 tables)

This paper contains 42 sections, 3 equations, 12 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Bias of LMs.
  4. Bias Mitigation.
  5. RQ1: Do LLMs exhibit political bias?
  6. LLM-based Bias Prediction
  7. LLM-based Article Continuation
  8. Article Continuation.
  9. Embedding-Based Similarity Matching.
  10. Left and Right Vocabulary-Based Matching.
  11. Discussion about Results of Classifier
  12. RQ2: Do LLMs demonstrate consistent bias across all topics?
  13. Visualization Based on Bias Tendency Index.
  14. Case Study of Biased Topics.
  15. RQ3: How to debias LLMs and further improve performance?
  16. ...and 27 more sections

Figures (12)

  • Figure 1: Interpretation of Biased Systems.
  • Figure 2: FlipBias Len.
  • Figure 3: LLM's prediction on FlipBias and ABP.
  • Figure 4: Article Continuation Results on FlipBias: The inner pie chart presents the outcomes of embedding-based similarity matching, while the outer doughnut illustrates the results of vocabulary-based matching.
  • Figure 5: Joint plot displaying kernel density estimates.
  • ...and 7 more figures