Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding Fine-grained Distortions in Reports of Scientific Findings

Amelie Wührl, Dustin Wright, Roman Klinger, Isabelle Augenstein

TL;DR

This work annotates 1,600 instances of scientific findings from academic papers paired with corresponding findings as reported in news articles and tweets, establishing baselines for automatically detecting four characteristics: causality, certainty, generality and sensationalism.

Abstract

Distorted science communication harms individuals and society as it can lead to unhealthy behavior change and decrease trust in scientific institutions. Given the rapidly increasing volume of science communication in recent years, a fine-grained understanding of how findings from scientific publications are reported to the general public, and methods to detect distortions from the original work automatically, are crucial. Prior work focused on individual aspects of distortions or worked with unpaired data. In this work, we make three foundational contributions towards addressing this problem: (1) annotating 1,600 instances of scientific findings from academic papers paired with corresponding findings as reported in news articles and tweets wrt. four characteristics: causality, certainty, generality and sensationalism; (2) establishing baselines for automatically detecting these characteristics; and (3) analyzing the prevalence of changes in these characteristics in both human-annotated and large-scale unlabeled data. Our results show that scientific findings frequently undergo subtle distortions when reported. Tweets distort findings more often than science news reports. Detecting fine-grained distortions automatically poses a challenging task. In our experiments, fine-tuned task-specific models consistently outperform few-shot LLM prompting.

Understanding Fine-grained Distortions in Reports of Scientific Findings

TL;DR

This work annotates 1,600 instances of scientific findings from academic papers paired with corresponding findings as reported in news articles and tweets, establishing baselines for automatically detecting four characteristics: causality, certainty, generality and sensationalism.

Abstract

Distorted science communication harms individuals and society as it can lead to unhealthy behavior change and decrease trust in scientific institutions. Given the rapidly increasing volume of science communication in recent years, a fine-grained understanding of how findings from scientific publications are reported to the general public, and methods to detect distortions from the original work automatically, are crucial. Prior work focused on individual aspects of distortions or worked with unpaired data. In this work, we make three foundational contributions towards addressing this problem: (1) annotating 1,600 instances of scientific findings from academic papers paired with corresponding findings as reported in news articles and tweets wrt. four characteristics: causality, certainty, generality and sensationalism; (2) establishing baselines for automatically detecting these characteristics; and (3) analyzing the prevalence of changes in these characteristics in both human-annotated and large-scale unlabeled data. Our results show that scientific findings frequently undergo subtle distortions when reported. Tweets distort findings more often than science news reports. Detecting fine-grained distortions automatically poses a challenging task. In our experiments, fine-tuned task-specific models consistently outperform few-shot LLM prompting.
Paper Structure (56 sections, 8 figures, 4 tables)

This paper contains 56 sections, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Science Communication.
  4. Misinformation.
  5. Dimensions of Information Changes
  6. Change Dimensions
  7. Dataset Construction
  8. Source Data
  9. Annotation Tasks
  10. Causality.
  11. Certainty.
  12. Generalization.
  13. Sensationalism.
  14. Annotation Environment
  15. Label Aggregation
  16. ...and 41 more sections

Figures (8)

  • Figure 1: Pair of scientific finding and reported finding with fine-grained labels of distortions.
  • Figure 2: Sankey diagrams visualize changes in causality and certainty. The left side of the charts depict the labels for the paper finding, the right side depicts the reported finding. Bar plot visualizes the distribution of generalization labels. Density plot visualizes the difference in sensationalism scores across reporting source.
  • Figure 3: Co-occurrence matrix of critical distortions. Diagonals represent the number of paired findings affected by a particular distortion. All other counts represent distortion co-occurrences.
  • Figure 4: Density plot visualizing distribution of sensationalism scores across 1,655,570 paper, 422,626 news, and 356,275 tweet findings. Differences in the degree of sensationalism across different findings sources are statistically significant (see Fig. \ref{['fig:sensationalism-significance']} in Appendix \ref{['sec:additional_figs']}).
  • Figure 5: Prompt template. We provide the instantiated prompts along with the LLM-specific system prompts and markup in the supplementary material.
  • ...and 3 more figures