Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Covert Bias: The Severity of Social Views' Unalignment in Language Models Towards Implicit and Explicit Opinion

Abeer Aldayel, Areej Alokaili, Rehab Alahmadi

TL;DR

The paper investigates how implicit vs explicit social views contribute to bias in large language models by employing edge-case stress tests and downstream tasks (hate speech and stance detection) focused on misogyny and religious bigotry. It compares two open-source LLMs under three biasing conditions (persona, fine-tuned, and zero-shot) and analyzes linguistic calibration through uncertainty markers. Key findings show a discrepancy between implicit and explicit opinions, with explicit opinions often driving stronger signals, while bias-aligned outputs tend to be more cautious through increased uncertainty phrases. The work highlights the need for improved decisiveness and calibrated uncertainty in outputs on socially nuanced topics, offering a framework for stress-testing bias and guiding future bias mitigation and reliability improvements.

Abstract

While various approaches have recently been studied for bias identification, little is known about how implicit language that does not explicitly convey a viewpoint affects bias amplification in large language models. To examine the severity of bias toward a view, we evaluated the performance of two downstream tasks where the implicit and explicit knowledge of social groups were used. First, we present a stress test evaluation by using a biased model in edge cases of excessive bias scenarios. Then, we evaluate how LLMs calibrate linguistically in response to both implicit and explicit opinions when they are aligned with conflicting viewpoints. Our findings reveal a discrepancy in LLM performance in identifying implicit and explicit opinions, with a general tendency of bias toward explicit opinions of opposing stances. Moreover, the bias-aligned models generate more cautious responses using uncertainty phrases compared to the unaligned (zero-shot) base models. The direct, incautious responses of the unaligned models suggest a need for further refinement of decisiveness by incorporating uncertainty markers to enhance their reliability, especially on socially nuanced topics with high subjectivity.

Covert Bias: The Severity of Social Views' Unalignment in Language Models Towards Implicit and Explicit Opinion

TL;DR

The paper investigates how implicit vs explicit social views contribute to bias in large language models by employing edge-case stress tests and downstream tasks (hate speech and stance detection) focused on misogyny and religious bigotry. It compares two open-source LLMs under three biasing conditions (persona, fine-tuned, and zero-shot) and analyzes linguistic calibration through uncertainty markers. Key findings show a discrepancy between implicit and explicit opinions, with explicit opinions often driving stronger signals, while bias-aligned outputs tend to be more cautious through increased uncertainty phrases. The work highlights the need for improved decisiveness and calibrated uncertainty in outputs on socially nuanced topics, offering a framework for stress-testing bias and guiding future bias mitigation and reliability improvements.

Abstract

While various approaches have recently been studied for bias identification, little is known about how implicit language that does not explicitly convey a viewpoint affects bias amplification in large language models. To examine the severity of bias toward a view, we evaluated the performance of two downstream tasks where the implicit and explicit knowledge of social groups were used. First, we present a stress test evaluation by using a biased model in edge cases of excessive bias scenarios. Then, we evaluate how LLMs calibrate linguistically in response to both implicit and explicit opinions when they are aligned with conflicting viewpoints. Our findings reveal a discrepancy in LLM performance in identifying implicit and explicit opinions, with a general tendency of bias toward explicit opinions of opposing stances. Moreover, the bias-aligned models generate more cautious responses using uncertainty phrases compared to the unaligned (zero-shot) base models. The direct, incautious responses of the unaligned models suggest a need for further refinement of decisiveness by incorporating uncertainty markers to enhance their reliability, especially on socially nuanced topics with high subjectivity.
Paper Structure (20 sections, 3 figures, 9 tables)

This paper contains 20 sections, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Experimental Setup
  4. Datasets
  5. Bias-based models
  6. Persona Bias
  7. Fine-tuned Bias
  8. Expressions of Uncertainty
  9. Results
  10. Bias Amplification Between Implicit and Explicit Opinion
  11. Impact of Bias on the Expressions of Uncertainty
  12. Discussion
  13. Conclusion and Future Work
  14. Dataset preprocesing
  15. Training and testing
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Variation of bias and baseline models' responses (%) that are high confidence, low confidence, uncertain, direct, or refusal corresponds to the expressed opinion (explicit and implicit) for hateful or opposing stance comments.
  • Figure 2: Uncertainty scores per topic with explicit and implicit expressions of opinion, with the median for each model. Two-tailed t-significant test illustrated between the explicit and implicit as * (p <= .01), ** (p < .0001).
  • Figure 3: Distribution of uncertainty between Implicit and Explicit opinions for two tasks stance and hate