Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement

Junyu Lu, Kai Ma, Kaichun Wang, Kelaiti Xiao, Roy Ka-Wei Lee, Bo Xu, Liang Yang, Hongfei Lin

TL;DR

Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement investigates how large language models handle subjective offensive language with annotation disagreement, leveraging the MD-Agreement dataset. The study evaluates multiple closed- and open-source LLMs in zero-shot, few-shot, and instruction-fine-tuning regimes, focusing on both binary accuracy and confidence alignment with human judgments. It finds that models perform well on unanimously labeled samples but struggle with low-agreement cases and tend to be overconfident in ambiguous inputs; disagreement-aware training improves both accuracy and alignment with human judgments. The findings highlight the need for uncertainty-aware moderation and suggest ensemble or human-in-the-loop approaches to mitigate overconfidence and better reflect human subjectivity in moderation tasks.

Abstract

Large Language Models (LLMs) have become essential for offensive language detection, yet their ability to handle annotation disagreement remains underexplored. Disagreement samples, which arise from subjective interpretations, pose a unique challenge due to their ambiguous nature. Understanding how LLMs process these cases, particularly their confidence levels, can offer insight into their alignment with human annotators. This study systematically evaluates the performance of multiple LLMs in detecting offensive language at varying levels of annotation agreement. We analyze binary classification accuracy, examine the relationship between model confidence and human disagreement, and explore how disagreement samples influence model decision-making during few-shot learning and instruction fine-tuning. Our findings reveal that LLMs struggle with low-agreement samples, often exhibiting overconfidence in these ambiguous cases. However, utilizing disagreement samples in training improves both detection accuracy and model alignment with human judgment. These insights provide a foundation for enhancing LLM-based offensive language detection in real-world moderation tasks.

Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement

TL;DR

Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement investigates how large language models handle subjective offensive language with annotation disagreement, leveraging the MD-Agreement dataset. The study evaluates multiple closed- and open-source LLMs in zero-shot, few-shot, and instruction-fine-tuning regimes, focusing on both binary accuracy and confidence alignment with human judgments. It finds that models perform well on unanimously labeled samples but struggle with low-agreement cases and tend to be overconfident in ambiguous inputs; disagreement-aware training improves both accuracy and alignment with human judgments. The findings highlight the need for uncertainty-aware moderation and suggest ensemble or human-in-the-loop approaches to mitigate overconfidence and better reflect human subjectivity in moderation tasks.

Abstract

Large Language Models (LLMs) have become essential for offensive language detection, yet their ability to handle annotation disagreement remains underexplored. Disagreement samples, which arise from subjective interpretations, pose a unique challenge due to their ambiguous nature. Understanding how LLMs process these cases, particularly their confidence levels, can offer insight into their alignment with human annotators. This study systematically evaluates the performance of multiple LLMs in detecting offensive language at varying levels of annotation agreement. We analyze binary classification accuracy, examine the relationship between model confidence and human disagreement, and explore how disagreement samples influence model decision-making during few-shot learning and instruction fine-tuning. Our findings reveal that LLMs struggle with low-agreement samples, often exhibiting overconfidence in these ambiguous cases. However, utilizing disagreement samples in training improves both detection accuracy and model alignment with human judgment. These insights provide a foundation for enhancing LLM-based offensive language detection in real-world moderation tasks.

Paper Structure

This paper contains 45 sections, 5 equations, 6 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Motivation.
  3. Research Objectives.
  4. Contributions.
  5. Preliminary
  6. Dataset
  7. Models
  8. RQ1: Evaluating LLMs on Offensive Language with Annotation Disagreement
  9. Evaluation of Binary Classification Performance
  10. (1) LLMs achieve high accuracy for unanimous agreement ($A^{++}$) samples.
  11. (2) LLM performance declines sharply for ambiguous cases.
  12. (3) Larger models improve accuracy but do not resolve annotation disagreement.
  13. (4) LLMs are biased toward classifying uncertain cases as offensive.
  14. Evaluation of Relationship between Agreement Degree and LLM Confidence
  15. (1) As annotation agreement decreases, the alignment between model confidence and human agreement weakens.
  16. ...and 30 more sections

Figures (6)

  • Figure 1: Accuracy of LLMs on detecting offensive and non-offensive language with different degrees of annotation agreement.
  • Figure 2: Self-consistency of several LLMs across varying degrees of annotation agreement with Cohen's Kappa ($\kappa$) as the metric.
  • Figure 3: Confusion matrix (raw counts and percentage) between confidence scores of GPT-4o (x-axis) and soft labels (y-axis).
  • Figure B1: Accuracy of LLMs on detecting offensive language with different degrees of annotation agreement under different temperature sampling settings.
  • Figure B2: Consistency of outputs from different LLMs across varying degrees of annotation agreement with Cohen's Kappa as the metric. The color scale represents different Kappa values.
  • ...and 1 more figures