Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MisSpans: Fine-Grained False Span Identification in Cross-Domain Fake News

Zhiwei Liu, Paul Thompson, Jiaqi Rong, Baojie Qu, Runteng Guo, Min Peng, Qianqian Xie, Sophia Ananiadou

TL;DR

MisSpans introduces the first cross-domain, human-annotated benchmark for span-level misinformation, comprising three tasks— pinpointing false spans, classifying their misinformation type, and generating grounded explanations—based on paired real and fake news from FakeNewsAMT. The dataset enables fine-grained localization and interpretability beyond binary veracity, with expert-driven annotation ensuring reliability. An extensive evaluation of 15 LLMs under zero-shot and one-shot prompts reveals the inherent difficulty of fine-grained span identification and the nuanced, domain-dependent benefits of reasoning capabilities and prompting strategies. The work highlights notable cross-domain performance differences and underscores the need for domain adaptation in misinformation detection, providing a valuable resource for developing more accurate and transparent systems. The MisSpans dataset and findings offer a foundation for future research on span-level misinformation analysis across multiple domains.

Abstract

Online misinformation is increasingly pervasive, yet most existing benchmarks and methods evaluate veracity at the level of whole claims or paragraphs using coarse binary labels, obscuring how true and false details often co-exist within single sentences. These simplifications also limit interpretability: global explanations cannot identify which specific segments are misleading or differentiate how a detail is false (e.g., distorted vs. fabricated). To address these gaps, we introduce MisSpans, the first multi-domain, human-annotated benchmark for span-level misinformation detection and analysis, consisting of paired real and fake news stories. MisSpans defines three complementary tasks: MisSpansIdentity for pinpointing false spans within sentences, MisSpansType for categorising false spans by misinformation type, and MisSpansExplanation for providing rationales grounded in identified spans. Together, these tasks enable fine-grained localisation, nuanced characterisation beyond true/false and actionable explanations. Expert annotators were guided by standardised guidelines and consistency checks, leading to high inter-annotator agreement. We evaluate 15 representative LLMs, including reasoning-enhanced and non-reasoning variants, under zero-shot and one-shot settings. Results reveal the challenging nature of fine-grained misinformation identification and analysis, and highlight the need for a deeper understanding of how performance may be influenced by multiple interacting factors, including model size and reasoning capabilities, along with domain-specific textual features. This project will be available at https://github.com/lzw108/MisSpans.

MisSpans: Fine-Grained False Span Identification in Cross-Domain Fake News

TL;DR

MisSpans introduces the first cross-domain, human-annotated benchmark for span-level misinformation, comprising three tasks— pinpointing false spans, classifying their misinformation type, and generating grounded explanations—based on paired real and fake news from FakeNewsAMT. The dataset enables fine-grained localization and interpretability beyond binary veracity, with expert-driven annotation ensuring reliability. An extensive evaluation of 15 LLMs under zero-shot and one-shot prompts reveals the inherent difficulty of fine-grained span identification and the nuanced, domain-dependent benefits of reasoning capabilities and prompting strategies. The work highlights notable cross-domain performance differences and underscores the need for domain adaptation in misinformation detection, providing a valuable resource for developing more accurate and transparent systems. The MisSpans dataset and findings offer a foundation for future research on span-level misinformation analysis across multiple domains.

Abstract

Online misinformation is increasingly pervasive, yet most existing benchmarks and methods evaluate veracity at the level of whole claims or paragraphs using coarse binary labels, obscuring how true and false details often co-exist within single sentences. These simplifications also limit interpretability: global explanations cannot identify which specific segments are misleading or differentiate how a detail is false (e.g., distorted vs. fabricated). To address these gaps, we introduce MisSpans, the first multi-domain, human-annotated benchmark for span-level misinformation detection and analysis, consisting of paired real and fake news stories. MisSpans defines three complementary tasks: MisSpansIdentity for pinpointing false spans within sentences, MisSpansType for categorising false spans by misinformation type, and MisSpansExplanation for providing rationales grounded in identified spans. Together, these tasks enable fine-grained localisation, nuanced characterisation beyond true/false and actionable explanations. Expert annotators were guided by standardised guidelines and consistency checks, leading to high inter-annotator agreement. We evaluate 15 representative LLMs, including reasoning-enhanced and non-reasoning variants, under zero-shot and one-shot settings. Results reveal the challenging nature of fine-grained misinformation identification and analysis, and highlight the need for a deeper understanding of how performance may be influenced by multiple interacting factors, including model size and reasoning capabilities, along with domain-specific textual features. This project will be available at https://github.com/lzw108/MisSpans.
Paper Structure (35 sections, 4 equations, 3 figures, 7 tables)

This paper contains 35 sections, 4 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. MisSpans Benchmark
  3. Task Formulation
  4. Data Collection
  5. Task Design
  6. Task 1: MisSpansIdentity
  7. Task 2: MisSpansType
  8. Task 3: MisSpansExplanation
  9. Annotation
  10. Data Quality Validation
  11. Experimental Setup
  12. Experimental LLMs
  13. Experimental settings and metrics
  14. Experimental Results
  15. Main Findings
  16. ...and 20 more sections

Figures (3)

  • Figure 1: Example from the MisSpans. In each fake news sentence, false spans are identified, through comparison with the real news story, and assigned a misinformation type label. Finally, a reason why the span is considered to represent misinformation is provided.
  • Figure 2: Overview of the process for labelling the MisSpans dataset. ① Annotation stage: each annotator labels half the dataset. ② Review stage: Each annotator labels the other half of the dataset and compares with the other annotator's labels to create a consolidated set.
  • Figure 3: Radar charts comparing model performance across domains in zero-shot setting. Task 1: exact score, Task 2: F1 score. Task 3: similarity score. The complete results can be found in Tables \ref{['tab:task1_domsins']}, \ref{['tab:task2_domsins']},\ref{['tab:task3_domsins']}.