Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Accuracy: An Explainability-Driven Analysis of Harmful Content Detection

Trishita Dhara, Siddhesh Sheth

Abstract

Although automated harmful content detection systems are frequently used to monitor online platforms, moderators and end users frequently cannot understand the logic underlying their predictions. While recent studies have focused on increasing classification accuracy, little focus has been placed on comprehending why neural models identify content as harmful, especially when it comes to borderline, contextual, and politically sensitive situations. In this work, a neural harmful content detection model trained on the Civil Comments dataset is analyzed explainability-drivenly. Two popular post-hoc explanation methods, Shapley Additive Explanations and Integrated Gradients, are used to analyze the behavior of a RoBERTa-based classifier in both correct predictions and systematic failure cases. Despite strong overall performance, with an area under the curve of 0.93 and an accuracy of 0.94, the analysis reveals limitations that are not observable from aggregate evaluation metrics alone. Integrated Gradients appear to extract more diffuse contextual attributions while Shapley Additive Explanations extract more focused attributions on explicit lexical cues. The consequent divergence in their outputs manifests in both false negatives and false positives. Qualitative case studies reveal recurring failure modes such as indirect toxicity, lexical over-attribution, or political discourse. The results suggest that explainable AI can foster human-in-the-loop moderation by exposing model uncertainty and increasing the interpretable rationale behind automated decisions. Most importantly, this work highlights the role of explainability as a transparency and diagnostic resource for online harmful content detection systems rather than as a performance-enhancing lever.

Beyond Accuracy: An Explainability-Driven Analysis of Harmful Content Detection

Abstract

Although automated harmful content detection systems are frequently used to monitor online platforms, moderators and end users frequently cannot understand the logic underlying their predictions. While recent studies have focused on increasing classification accuracy, little focus has been placed on comprehending why neural models identify content as harmful, especially when it comes to borderline, contextual, and politically sensitive situations. In this work, a neural harmful content detection model trained on the Civil Comments dataset is analyzed explainability-drivenly. Two popular post-hoc explanation methods, Shapley Additive Explanations and Integrated Gradients, are used to analyze the behavior of a RoBERTa-based classifier in both correct predictions and systematic failure cases. Despite strong overall performance, with an area under the curve of 0.93 and an accuracy of 0.94, the analysis reveals limitations that are not observable from aggregate evaluation metrics alone. Integrated Gradients appear to extract more diffuse contextual attributions while Shapley Additive Explanations extract more focused attributions on explicit lexical cues. The consequent divergence in their outputs manifests in both false negatives and false positives. Qualitative case studies reveal recurring failure modes such as indirect toxicity, lexical over-attribution, or political discourse. The results suggest that explainable AI can foster human-in-the-loop moderation by exposing model uncertainty and increasing the interpretable rationale behind automated decisions. Most importantly, this work highlights the role of explainability as a transparency and diagnostic resource for online harmful content detection systems rather than as a performance-enhancing lever.
Paper Structure (26 sections, 4 figures, 3 tables)

This paper contains 26 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Harmful Content and Toxic Language Detection
  4. Explainable Artificial Intelligence for Text Classification
  5. Explainability in Harmful Content Moderation
  6. Methodolody
  7. Task Description
  8. Dataset
  9. Model Architecture
  10. Pipeline Overview
  11. Training Configuration
  12. Explainability Methods
  13. Shapley Additive Explanations.
  14. Integrated Gradients.
  15. Explainability Evaluation Protocol
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Overview of the harmful content detection and explainability analysis pipeline.
  • Figure 2: Confusion matrix for binary harmful content classification on the Civil Comments test set.
  • Figure 3: Shapley Additive Explanations for a correctly classified toxic comment.
  • Figure 4: Shapley Additive Explanations for an incorrect prediction (False Positive)