An evaluation of LLMs for political bias in Western media: Israel-Hamas and Ukraine-Russia wars

TL;DR

This study investigates political bias in Western media (BBC and The Guardian) during the Russia–Ukraine and Israel–Hamas wars by applying three LLM-based classifiers (BERT, Gemini, and DeepSeek) to large article corpora. It combines n-gram topical checks and DistilRoBERTa-based sentiment analysis to characterize bias shifts over time and across outlets, revealing outlet-specific baselines and event-driven deviations. Key findings show DeepSeek’s stable left-leaning tendency, while BERT and Gemini trend toward centrism, with Guardian coverage exhibiting larger bias swings, especially in the Israel–Hamas period. The work highlights model-dependent biases and emphasizes the need for careful interpretation, human calibration, and broader, multimodal datasets for robust media-bias auditing in real-world settings.

Abstract

Political bias in media plays a critical role in shaping public opinion, voter behaviour, and broader democratic discourse. Subjective opinions and political bias can be found in media sources, such as newspapers, depending on their funding mechanisms and alliances with political parties. Automating the detection of political biases in media content can limit biases in elections. The impact of large language models (LLMs) in politics and media studies is becoming prominent. In this study, we utilise LLMs to compare the left-wing, right-wing, and neutral political opinions expressed in the Guardian and BBC. We review newspaper reporting that includes significant events such as the Russia-Ukraine war and the Hamas-Israel conflict. We analyse the proportion for each opinion to find the bias under different LLMs, including BERT, Gemini, and DeepSeek. Our results show that after the outbreak of the wars, the political bias of Western media shifts towards the left-wing and each LLM gives a different result. DeepSeek consistently showed a stable Left-leaning tendency, while BERT and Gemini remained closer to the Centre. The BBC and The Guardian showed distinct reporting behaviours across the two conflicts. In the Russia-Ukraine war, both outlets maintained relatively stable positions; however, in the Israel-Hamas conflict, we identified larger political bias shifts, particularly in Guardian coverage, suggesting a more event-driven pattern of reporting bias. These variations suggest that LLMs are shaped not only by their training data and architecture, but also by underlying worldviews with associated political biases.

Figures (7)

• Figure 1: Overall framework of analysis of political biases in Western media for this study.
• Figure 2: All Models – Russia–Ukraine (Pre vs During). Proportion of articles by Category (Left/Centre/Right) for BBC and The Guardian; hatch fills distinguish models (BERT, DeepSeek, Gemini).
• Figure 3: All Models – Israel–Hamas (Pre vs During). Proportion of articles by Category (Left/Centre/Right) for BBC and The Guardian; hatch fills distinguish models (BERT, DeepSeek, Gemini).
• Figure 4: The political leaning bias (score) variation across time for BBC and the Guardian under the Israel–Hamas and Russia–Ukraine conflicts.
• Figure 5: Comparison of Left and Right difference between BBC and The Guardian by three models (BERT, DeepSeek, Gemini). Each panel shows monthly trends across the two conflicts. Right -- Left = Difference, hence negative territory indicates Left-leaning, and positive territory indicates Right.