Independent fact-checking organizations exhibit a departure from political neutrality

TL;DR

Independent fact-checking organizations in the USA and India exhibit departures from political neutrality, selectively framing misinformation to shape reader perception. The authors implement a longitudinal neutrality measure on 2018–2023 data from six organizations using GPT-3.5 prompted 5W1H summaries and transformer-based topical embeddings to quantify What, Why, and Who of debunked content and to estimate how entities are portrayed via an entity polarity score PS defined as $PS(X,e_i) = (N_p - N_n)/N_t$, where $N_p$, $N_n$, and $N_t$ are counts of positive, negative, and total tags. They find average neutrality scores around $-0.17$ to $-0.24$, with organization- and region-specific patterns, such as stronger negative coverage of opposition leaders in India and Trump-negative portrayals in the USA. The work highlights a subtle but robust bias signal in ostensibly objective fact-checking and calls for transparency and cross-geography evaluation to safeguard public understanding of news.

Abstract

Independent fact-checking organizations have emerged as the crusaders to debunk fake news. However, they may not always remain neutral, as they can be selective in the false news they choose to expose and in how they present the information. They can deviate from neutrality by being selective in what false news they debunk and how the information is presented. Prompting the now popular large language model, GPT-3.5, with journalistic frameworks, we establish a longitudinal measure (2018-2023) for political neutrality that looks beyond the left-right spectrum. Specified on a range of -1 to 1 (with zero being absolute neutrality), we establish the extent of negative portrayal of political entities that makes a difference in the readers' perception in the USA and India. Here, we observe an average score of -0.17 and -0.24 in the USA and India, respectively. The findings indicate how seemingly objective fact-checking can still carry distorted political views, indirectly and subtly impacting the perception of consumers of the news.

Figures (2)

• Figure 1: The data from six independent fact-checking organizations curated over five years ($2018$-$2023$) -- PolitiFact (P), Snopes (S), and Check Your Fact (C) from the USA, and Alt News (A), OpIndia (O), and Boom (B) from India. (a) Year-wise breakdown of the number of instances per organization in English. (b) Inter-organization (X-Y) maximum topical similarity (TS) comparing samples of organization X with organization Y within a timeframe of $\pm15$ days and similarity threshold $0.75$. The max TS values are recorded across the "Claim", "What" and "Why" tags, respectively. (c) Inter-organization (X-Y) Jaccard similarity comparing samples of organization X with organization Y over the $top_k=100$ political entities within a timeframe of $\pm15$ days. Note: In (b) and (c), we record the similarity only among organizations within a geography.
• Figure 2: The extent of neutrality ($-1\leq PS \leq 1$) for the $top_k=5$ entities per fact-checking organization. PolitiFact, Snopes, and Check Your Fact are in the USA. Alt News, OpIndia, and Boom are based in India. $PS$ establishes "how" the coverage of the political entities in the fake news leads to positive, negative, or neutral image for the entity, impacting the reader's perception. A higher neutrality is observed if $PS\approx0$. Meanwhile, a score closer to -1 (1) highlights a more pessimistic/critical (positive/promoting) tone in terms of portraying the entities. Note: The log error bars account for the uncertainty in the prediction of PS.