Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NewsScope: Schema-Grounded Cross-Domain News Claim Extraction with Open Models

Nidhi Pandya

TL;DR

NewsScope tackles automated, schema-constrained extraction of news claims and cross-domain generalization for verification. It contributes a 455-article dataset across politics, health, science/environment, and business, a public benchmark, and an open-weight LLaMA 3.1 8B model fine-tuned with LoRA that achieves 98.8% schema validity and competitive accuracy against GPT-4o-mini. A numeric grounding filter further improves accuracy to 91.6%, narrowing the GPT-4o-mini gap to about 2.1 percentage points, with human evaluation showing high reliability (IAA ≈ 94.6%). The approach enables offline deployment at roughly $15 on-demand compute, and the authors release code and benchmark for reproducibility and practical use in newsrooms.

Abstract

Automated news verification requires structured claim extraction, but existing approaches either lack schema compliance or generalize poorly across domains. This paper presents NewsScope, a cross-domain dataset, benchmark, and fine-tuned model for schema-grounded news claim extraction. The dataset contains 455 articles across politics, health, science/environment, and business, consisting of 395 in-domain articles and 60 out-of-source articles for generalization testing. LLaMA 3.1 8B was fine-tuned using LoRA on 315 training examples and evaluated on held-out in-domain (80 articles) and out-of-source (60 articles) test sets. Human evaluation on 400 claims shows NewsScope achieves 89.4% human-evaluated accuracy compared to GPT-4o-mini's 93.7% (p=0.07). NewsScope outperforms GPT-4o-mini on political claims (94.3% vs. 87.8%). A numeric grounding filter further improves accuracy to 91.6%, narrowing the gap to 2.1 percentage points. Inter-annotator agreement studies (160 claims) confirm labeling reliability (94.6% positive agreement on SUPPORTED judgments). The open-weight model enables offline deployment at approximately $15 on-demand compute (or $0 on free tiers). Code and benchmark are publicly released.

NewsScope: Schema-Grounded Cross-Domain News Claim Extraction with Open Models

TL;DR

NewsScope tackles automated, schema-constrained extraction of news claims and cross-domain generalization for verification. It contributes a 455-article dataset across politics, health, science/environment, and business, a public benchmark, and an open-weight LLaMA 3.1 8B model fine-tuned with LoRA that achieves 98.8% schema validity and competitive accuracy against GPT-4o-mini. A numeric grounding filter further improves accuracy to 91.6%, narrowing the GPT-4o-mini gap to about 2.1 percentage points, with human evaluation showing high reliability (IAA ≈ 94.6%). The approach enables offline deployment at roughly $15 on-demand compute, and the authors release code and benchmark for reproducibility and practical use in newsrooms.

Abstract

Automated news verification requires structured claim extraction, but existing approaches either lack schema compliance or generalize poorly across domains. This paper presents NewsScope, a cross-domain dataset, benchmark, and fine-tuned model for schema-grounded news claim extraction. The dataset contains 455 articles across politics, health, science/environment, and business, consisting of 395 in-domain articles and 60 out-of-source articles for generalization testing. LLaMA 3.1 8B was fine-tuned using LoRA on 315 training examples and evaluated on held-out in-domain (80 articles) and out-of-source (60 articles) test sets. Human evaluation on 400 claims shows NewsScope achieves 89.4% human-evaluated accuracy compared to GPT-4o-mini's 93.7% (p=0.07). NewsScope outperforms GPT-4o-mini on political claims (94.3% vs. 87.8%). A numeric grounding filter further improves accuracy to 91.6%, narrowing the gap to 2.1 percentage points. Inter-annotator agreement studies (160 claims) confirm labeling reliability (94.6% positive agreement on SUPPORTED judgments). The open-weight model enables offline deployment at approximately 0 on free tiers). Code and benchmark are publicly released.
Paper Structure (25 sections, 3 tables)

This paper contains 25 sections, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset and Annotation
  4. Article Collection
  5. Schema Design
  6. Annotation Pipeline
  7. Data Splits
  8. Model
  9. Base Model
  10. Fine-tuning
  11. Numeric Grounding Filter
  12. Evaluation
  13. Metrics
  14. Human Evaluation
  15. Inter-Annotator Agreement
  16. ...and 10 more sections