Computational Fact-Checking of Online Discourse: Scoring scientific accuracy in climate change related news articles

TL;DR

The paper tackles the challenge of misinformation in climate-related online media by proposing a neurosymbolic fact-checking pipeline that combines LLM-based statement extraction with knowledge-graph verification against climate-ground-truth graphs to produce a scientific accuracy score. It reviews related work and climate KG initiatives, then details a modular implementation with media processing, triple extraction, alignment, KG extension, veracity checks, and scoring. Two evaluations—expert feedback and a public user survey—demonstrate strong interest and practical relevance, while identifying bottlenecks in extraction, context preservation, and the availability of high-quality ground truth. The work highlights critical needs for FAIR ground-truth data, scalable semantification, and energy-conscious AI reuse, outlining a path toward more transparent, interoperable, and socially acceptable fact-checking tools for public discourse.

Abstract

Democratic societies need reliable information. Misinformation in popular media, such as news articles or videos, threatens to impair civic discourse. Citizens are, unfortunately, not equipped to verify the flood of content consumed daily at increasing rates. This work aims to quantify the scientific accuracy of online media semi-automatically. We investigate the state of the art of climate-related ground truth knowledge representation. By semantifying media content of unknown veracity, their statements can be compared against these ground truth knowledge graphs. We implemented a workflow using LLM-based statement extraction and knowledge graph analysis. Our implementation can streamline content processing towards state-of-the-art knowledge representation and veracity quantification. Developed and evaluated with the help of 27 experts and detailed interviews with 10, the tool evidently provides a beneficial veracity indication. These findings are supported by 43 anonymous participants from a parallel user survey. This initial step, however, is unable to annotate public media at the required granularity and scale. Additionally, the identified state of climate change knowledge graphs is vastly insufficient to support this neurosymbolic fact-checking approach. Further work towards a FAIR (Findable, Accessible, Interoperable, Reusable) ground truth and complementary metrics is required to support civic discourse scientifically.

Figures (4)

• Figure 1: Proposed scoring pipeline consisting of (i) textualizing media from different file types, (ii) LLM-based statement extraction, verification, and alignment. (iii.a) Trusted statements extend the ground truth knowledge graph, (iii.b) untrusted statements are checked for veracity using graph analysis on the ground truth, concluding in (iv) a final score calculation.
• Figure 2: Triple extraction workflow example from text body to aligned triples. An LLM was used to handle initial extraction, base forms and synonyms. A prototypical ontology mapping was implemented to a placeholder example ontology.
• Figure 3: User interface mock-up showing the statement veracity score with color coding and ground truth reference.
• Figure 4: User survey results (N=43). They show the demand for a scientific accuracy score, approve the tool in its current state, and indicate suitability for various media types.