Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NewsUnfold: Creating a News-Reading Application That Indicates Linguistic Media Bias and Collects Feedback

Smi Hinterreiter, Martin Wessel, Fabian Schliski, Isao Echizen, Marc Erich Latoschik, Timo Spinde

TL;DR

This work introduces NewsUnfold, a news-reading platform that visually highlights linguistic bias and collects reader feedback to build a high-quality bias dataset (NUDA) and improve automatic bias detection. By testing two HITL feedback mechanisms, the study demonstrates that user feedback can significantly raise inter-annotator agreement and, when merged with an existing bias dataset (BABE), yield measurable gains in F1 performance. NUDA, created from crowd-augmented annotations with spam filtering and majority voting, achieves notable data-quality gains and supports training improvements across classifiers. The approach highlights the viability of human-in-the-loop data collection for dynamic bias contexts and points to scalable, user-centered methods for ongoing bias detection and reader awareness with broad potential across platforms.

Abstract

Media bias is a multifaceted problem, leading to one-sided views and impacting decision-making. A way to address digital media bias is to detect and indicate it automatically through machine-learning methods. However, such detection is limited due to the difficulty of obtaining reliable training data. Human-in-the-loop-based feedback mechanisms have proven an effective way to facilitate the data-gathering process. Therefore, we introduce and test feedback mechanisms for the media bias domain, which we then implement on NewsUnfold, a news-reading web application to collect reader feedback on machine-generated bias highlights within online news articles. Our approach augments dataset quality by significantly increasing inter-annotator agreement by 26.31% and improving classifier performance by 2.49%. As the first human-in-the-loop application for media bias, the feedback mechanism shows that a user-centric approach to media bias data collection can return reliable data while being scalable and evaluated as easy to use. NewsUnfold demonstrates that feedback mechanisms are a promising strategy to reduce data collection expenses and continuously update datasets to changes in context.

NewsUnfold: Creating a News-Reading Application That Indicates Linguistic Media Bias and Collects Feedback

TL;DR

This work introduces NewsUnfold, a news-reading platform that visually highlights linguistic bias and collects reader feedback to build a high-quality bias dataset (NUDA) and improve automatic bias detection. By testing two HITL feedback mechanisms, the study demonstrates that user feedback can significantly raise inter-annotator agreement and, when merged with an existing bias dataset (BABE), yield measurable gains in F1 performance. NUDA, created from crowd-augmented annotations with spam filtering and majority voting, achieves notable data-quality gains and supports training improvements across classifiers. The approach highlights the viability of human-in-the-loop data collection for dynamic bias contexts and points to scalable, user-centered methods for ongoing bias detection and reader awareness with broad potential across platforms.

Abstract

Media bias is a multifaceted problem, leading to one-sided views and impacting decision-making. A way to address digital media bias is to detect and indicate it automatically through machine-learning methods. However, such detection is limited due to the difficulty of obtaining reliable training data. Human-in-the-loop-based feedback mechanisms have proven an effective way to facilitate the data-gathering process. Therefore, we introduce and test feedback mechanisms for the media bias domain, which we then implement on NewsUnfold, a news-reading web application to collect reader feedback on machine-generated bias highlights within online news articles. Our approach augments dataset quality by significantly increasing inter-annotator agreement by 26.31% and improving classifier performance by 2.49%. As the first human-in-the-loop application for media bias, the feedback mechanism shows that a user-centric approach to media bias data collection can return reliable data while being scalable and evaluated as easy to use. NewsUnfold demonstrates that feedback mechanisms are a promising strategy to reduce data collection expenses and continuously update datasets to changes in context.
Paper Structure (42 sections, 17 figures, 5 tables)

This paper contains 42 sections, 17 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Media Bias
  4. Automatic Media Bias Detection
  5. Media Bias Awareness
  6. HITL Platforms For Crowdsourcing Annotations
  7. Feedback Mechanisms
  8. Study Design
  9. Results
  10. The NewsUnfold Platform
  11. Application Design
  12. Study Design
  13. Dataset Creation and Evaluation
  14. Results
  15. Data Quality
  16. ...and 27 more sections

Figures (17)

  • Figure 1: Three-step process of the NewsUnfold Development and Evaluation.
  • Figure 2: The feedback mechanism Highlights uses the BABE classifier to highlight biased sentences in yellow and not biased sentences in grey. Readers can agree or disagree with this classification through the feedback module on the right.
  • Figure 3: The feedback mechanism Comparison operates on sentence pairs and uses the BABE classifier to highlight the first sentence as biased in yellow. Readers can agree or disagree with this classification through the feedback module on the right. The next sentence is merely outlined. Here, the feedback module asks for a bias rating without the classifier anchor.
  • Figure 4: The classifier shows the highlights in yellow (biased) and grey (not biased) on NewsUnfold. The feedback module on the right allows readers to agree or disagree and leave optional feedback. The Sparkles draw attention to controversial sentences or sentences that need more feedback. \ref{['table:key-elements']} explains the elements with yellow numbers.
  • Figure 5: Comparison of the expert-generated dataset with the NUDA dataset. The non-overlapping confidence intervals indicate a significant increase.
  • ...and 12 more figures