Explaining News Bias Detection: A Comparative SHAP Analysis of Transformer Model Decision Mechanisms

TL;DR

This paper uses SHAP-based interpretability to compare two transformer-based bias-detection systems on the BABE news bias dataset. It finds that the Bias-Detector model exhibits strong attribution signals for many false positives, leading to over-flagging neutral content, while the DA-RoBERTa-BABE-FT model achieves fewer false positives and attribution patterns that better align with actual predictions. The study introduces a rigorous word-level attribution pipeline, stratified sampling, and McNemar-based significance testing to characterize how architectural choices shape interpretability and error modes. The findings highlight the importance of interpretability-aware evaluation for deploying bias-detection systems in journalism, and suggest that domain-adaptive pretraining and contextual framing cues improve reliability. Overall, the work provides actionable insights into designing more trustworthy, deployment-ready bias-detection tools for media analysis.

Abstract

Automated bias detection in news text is heavily used to support journalistic analysis and media accountability, yet little is known about how bias detection models arrive at their decisions or why they fail. In this work, we present a comparative interpretability study of two transformer-based bias detection models: a bias detector fine-tuned on the BABE dataset and a domain-adapted pre-trained RoBERTa model fine-tuned on the BABE dataset, using SHAP-based explanations. We analyze word-level attributions across correct and incorrect predictions to characterize how different model architectures operationalize linguistic bias. Our results show that although both models attend to similar categories of evaluative language, they differ substantially in how these signals are integrated into predictions. The bias detector model assigns stronger internal evidence to false positives than to true positives, indicating a misalignment between attribution strength and prediction correctness and contributing to systematic over-flagging of neutral journalistic content. In contrast, the domain-adaptive model exhibits attribution patterns that better align with prediction outcomes and produces 63\% fewer false positives. We further demonstrate that model errors arise from distinct linguistic mechanisms, with false positives driven by discourse-level ambiguity rather than explicit bias cues. These findings highlight the importance of interpretability-aware evaluation for bias detection systems and suggest that architectural and training choices critically affect both model reliability and deployment suitability in journalistic contexts.

Figures (8)

• Figure 1: SHAP Reveals the Word Attributions for the model's decision. Red words increase bias, Blue words decrease bias in the sentence. This reveals, how the two models look differently into the sentence to make the decision.
• Figure 2: Word-level SHAP aggregation pipeline. Token-level SHAP values produced by a transformer model are grouped into words based on tokenizer boundaries and text alignment, then aggregated to produce interpretable word-level attributions.
• Figure 3: SHAP Magnitude Distributions: False positives in bias-detector exhibit higher attribution magnitude than true positives, while DA-RoBERTa-BABE-FT shows the opposite, indicating better alignment between explanations and predictions.
• Figure 4: Top Bias Indicators for the bias-detector model.
• Figure 5: Word category composition of the top 100 SHAP-attributed features for each model.