BiasLab: Toward Explainable Political Bias Detection with Dual-Axis Annotations and Rationale Indicators
Kma Solaiman
TL;DR
BiasLab addresses the subjectivity of perceived political bias by collecting article-level dual-axis annotations (toward the Democratic and Republican parties) complemented by rationale indicators and a human-in-the-loop annotation workflow. The approach combines crowdsourced labels with schema-constrained GPT-4o comparisons to study perception drift and rationale classification, and provides a gold standard (270 articles) for supervised evaluation. The paper reports baseline models for perception drift and rationale-type classification, revealing the difficulty of explainable bias detection and the persistence of human–model misalignment on subtle right-leaning framing. By releasing the dataset, annotation schema, and modeling code, BiasLab offers a transparent benchmark for interpretable, perception-based bias modeling and aligns NLP research with real-world needs for explainability in politically sensitive domains. The results underscore the importance of incorporating rationale diversity and human feedback to improve alignment between reader perceptions and automated systems, informing future interactive, explainable NLP models.
Abstract
We present BiasLab, a dataset of 300 political news articles annotated for perceived ideological bias. These articles were selected from a curated 900-document pool covering diverse political events and source biases. Each article is labeled by crowdworkers along two independent scales, assessing sentiment toward the Democratic and Republican parties, and enriched with rationale indicators. The annotation pipeline incorporates targeted worker qualification and was refined through pilot-phase analysis. We quantify inter-annotator agreement, analyze misalignment with source-level outlet bias, and organize the resulting labels into interpretable subsets. Additionally, we simulate annotation using schema-constrained GPT-4o, enabling direct comparison to human labels and revealing mirrored asymmetries, especially in misclassifying subtly right-leaning content. We define two modeling tasks: perception drift prediction and rationale type classification, and report baseline performance to illustrate the challenge of explainable bias detection. BiasLab's rich rationale annotations provide actionable interpretations that facilitate explainable modeling of political bias, supporting the development of transparent, socially aware NLP systems. We release the dataset, annotation schema, and modeling code to encourage research on human-in-the-loop interpretability and the evaluation of explanation effectiveness in real-world settings.