Bridging Human and Model Perspectives: A Comparative Analysis of Political Bias Detection in News Media Using Large Language Models
Shreya Adrita Banik, Niaz Nafi Rahman, Tahsina Moiukh, Farig Sadeque
TL;DR
This paper introduces a comparative framework to evaluate how human judgments of political bias align with multiple language model families, across zero-shot, guideline-conditioned, and few-shot prompting settings. It combines a manually annotated FIGNEWS-based bias corpus with a unified evaluation pipeline that contrasts traditional supervised transformers (e.g., RoBERTa, BERT) against generative LLMs (e.g., GPT-5) in both classification accuracy and alignment with human labels. Key findings show RoBERTa offers the strongest supervised performance while GPT-5 exhibits stronger zero-shot alignment with human judgments; however, all models struggle with implicit, context-dependent, or multi-actor bias, revealing a gap between pattern-matching and interpretive reasoning. The study advocates hybrid evaluation and discourse-aware modeling to bridge human and model perspectives, enabling more trustworthy and explainable automated media bias detection systems.
Abstract
Detecting political bias in news media is a complex task that requires interpreting subtle linguistic and contextual cues. Although recent advances in Natural Language Processing (NLP) have enabled automatic bias classification, the extent to which large language models (LLMs) align with human judgment still remains relatively underexplored and not yet well understood. This study aims to present a comparative framework for evaluating the detection of political bias across human annotations and multiple LLMs, including GPT, BERT, RoBERTa, and FLAN. We construct a manually annotated dataset of news articles and assess annotation consistency, bias polarity, and inter-model agreement to quantify divergence between human and model perceptions of bias. Experimental results show that among traditional transformer-based models, RoBERTa achieves the highest alignment with human labels, whereas generative models such as GPT demonstrate the strongest overall agreement with human annotations in a zero-shot setting. Among all transformer-based baselines, our fine-tuned RoBERTa model acquired the highest accuracy and the strongest alignment with human-annotated labels. Our findings highlight systematic differences in how humans and LLMs perceive political slant, underscoring the need for hybrid evaluation frameworks that combine human interpretability with model scalability in automated media bias detection.