To Bias or Not to Bias: Detecting bias in News with bias-detector

TL;DR

The paper tackles the challenge of detecting bias in news at the sentence level by fine-tuning a RoBERTa-base encoder on the expert-annotated BABE dataset and integrating a bias-type classifier into a complete bias-analysis pipeline. It shows that bias-detector achieves a macro-F1 of 0.852 on BABE, outperforming the domain-adapted DA-RoBERTa-FT baseline (0.823), with statistical significance established via McNemar's test ($p=0.0167$) and a 5×2 cross-validation t-test ($p=3.59\times 10^{-7}$). Attention analyses indicate that the model focuses on contextually meaningful tokens rather than relying on politically charged terms, improving interpretability. The work also discusses limitations due to dataset size and sentence-level scope, and proposes future directions such as discourse-level modeling, multilingual adaptation, and bias-neutralization through generation to enhance robustness and transparency in media bias analysis.

Abstract

Media bias detection is a critical task in ensuring fair and balanced information dissemination, yet it remains challenging due to the subjectivity of bias and the scarcity of high-quality annotated data. In this work, we perform sentence-level bias classification by fine-tuning a RoBERTa-based model on the expert-annotated BABE dataset. Using McNemar's test and the 5x2 cross-validation paired t-test, we show statistically significant improvements in performance when comparing our model to a domain-adaptively pre-trained DA-RoBERTa baseline. Furthermore, attention-based analysis shows that our model avoids common pitfalls like oversensitivity to politically charged terms and instead attends more meaningfully to contextually relevant tokens. For a comprehensive examination of media bias, we present a pipeline that combines our model with an already-existing bias-type classifier. Our method exhibits good generalization and interpretability, despite being constrained by sentence-level analysis and dataset size because of a lack of larger and more advanced bias corpora. We talk about context-aware modeling, bias neutralization, and advanced bias type classification as potential future directions. Our findings contribute to building more robust, explainable, and socially responsible NLP systems for media bias detection.

Figures (5)

• Figure 1: Bias Detection and Classification Scheme which shows how the pipeline detects sentence-level biases.
• Figure 2: Architecture of the Bias Detector Model. The output vector of the CLS token serves as input to the classifier head
• Figure 3: 5x2 CV t-test Results: Statistically significant difference on Macro-F1 score across 5 folds of the CV t-test between bias-detector and da-roberta-ft baseline
• Figure 4: Attention weights heatmap for False Negatives
• Figure 5: Attention weights heatmap for False Positives