Media Bias Detector: Designing and Implementing a Tool for Real-Time Selection and Framing Bias Analysis in News Coverage

TL;DR

The paper tackles the challenge of detecting subtle media bias embedded in newsroom practices, specifically selection and framing, by introducing the Media Bias Detector, an LLM-driven tool that annotates individual articles for topics, subtopics, tone, and political lean and aggregates the results at the publisher level for near-real-time analysis. It combines a two-dashboard interface (Coverage and Events) with a human-in-the-loop validation process to deliver multidimensional insights while preserving transparency. Through a mixed-methods evaluation with $n=13$ experts and $n=150$ everyday news consumers, the study demonstrates usability, potential educational and research value, and nuanced trust dynamics around AI-powered bias detection. The work contributes a scalable, open approach to understanding editorial choices and empowering critical engagement with news, with implications for media literacy, journalism, and political science research.

Abstract

Mainstream media, through their decisions on what to cover and how to frame the stories they cover, can mislead readers without using outright falsehoods. Therefore, it is crucial to have tools that expose these editorial choices underlying media bias. In this paper, we introduce the Media Bias Detector, a tool for researchers, journalists, and news consumers. By integrating large language models, we provide near real-time granular insights into the topics, tone, political lean, and facts of news articles aggregated to the publisher level. We assessed the tool's impact by interviewing 13 experts from journalism, communications, and political science, revealing key insights into usability and functionality, practical applications, and AI's role in powering media bias tools. We explored this in more depth with a follow-up survey of 150 news consumers. This work highlights opportunities for AI-driven tools that empower users to critically engage with media content, particularly in politically charged environments.

Figures (16)

• Figure 1: The default view of the Coverage dashboard which allows broad exploration of the date (D1). Every user lands at a screen showing the category-wise coverage of the news publishers (A). Each publisher is represented with a stacked bar where a segment represents the number of articles published by them in a given category, allowing them to directly compare the proportion of attention they give to those topics (D3). Users are provided with a variety of controls to adjust the chart type and color, and to filter on publishers, article type, and date range (D1). Users can also toggle normalization off to allow direct comparisons of absolute numbers instead of the proportion of coverage. Coloring by Lean (B) shows the distribution of articles that are more aligned with a given political viewpoint, and coloring by Tone (C) shows the variation in sentiment across the same articles. Hovering on a segment displays a tooltip explaining what it represents and presenting the count and proportion of articles that fall within it.
• Figure 2: The grid view of the Coverage dashboard presents an alternative visualization to the stacked bar in Figure \ref{['fig:coverage_bar']} by giving each bar segment its own cell in a grid. This allows similarly broad exploration (D1) by enabling more direct comparisons between different categories and publishers (D3) without the need to hover or click on a news category to highlight it. Each cell shows how many articles were published on a given topic by a particular publisher, with the highest-publishing publisher highlighted by its article count. The news category color map for this view matches that of the stacked bar chart. Similar to that view, the bars can be colored by Lean (B) or Tone (C), where the cell color represents the average political lean or tone of the articles in that category. Hovering on a cell displays a tooltip explaining what it represents and presenting the count of articles that fall within it.
• Figure 3: To enable deep exploration of specific data (D2), we allow users to click through a hierarchy of news topics and subtopics to zoom into news of interest to them and be able to compare the volume, tone, and lean across publishers and date ranges. In this sequence of images, we show the user interacting with the dashboard to focus on the 'Presidential Horse Race' subtopic (D) after starting with the default all-category view (A) and then clicking on the 'Politics' category (B), the '2024 Election' topic within that, and finally the horse race subtopic within it (D). At each level of interaction, the user is provided the same controls to color by Category, Lean, or Tone, to filter by publishers and article type, and to select a date range to focus on (D1). Hovering on a segment displays a tooltip explaining what it represents and presenting the count and proportion of articles in it.
• Figure 4: Complementary to the Coverage view, the Events dashboard shown here presents an event-level view of the news that focuses on the fast pace of the news cycle and caters to both broad (D1) and deep (D2) exploration of the news. It offers a quick overview of happenings from the past three days but allows users to dig deeper into each event and its major facts and compare selection and framing bias across publishers (D3). Each row is a news event and each cell represents whether it was covered or not by the respective publisher. The events are sorted by importance (measured as their amount of coverage) and we provide a summarized title for the event and a count of the number of articles about it across all publishers. Clicking on an event using the button on the right displays a detailed view (B) which contains the full event description as well as a summary of its sentence-level composition in terms of facts, quotes, and opinions. This view also lists the top facts about the event and shows which publishers mentioned or omitted certain statements. Clicking on any one of these top facts shows different variations of it (C) as they were written in the original news articles allowing users to compare their framing. Clicking on any one of these variations takes the user to the original article on the publisher's website.
• Figure 5: Overall results on the NASA-Task Load Index (NASA-TLX)'s measures of subjective workload for the Baseline and Media Bias Detector task evaluations in the user study with experts.