CrediBench: Building Web-Scale Network Datasets for Information Integrity

TL;DR

CrediBench addresses the problem of misinformation detection by building temporal, text-attributed web graphs that fuse content and hyperlink structure at web scale. It introduces a PySpark-based pipeline to construct a one-month graph from Common Crawl (Dec 2024) with about $45{,}000{,}000$ nodes and $1{,}010{,}000{,}000$ edges, annotated with credibility labels MBFC and $PC1$. Experiments show that both textual content and graph topology provide predictive signals for domain credibility, with text-only learning achieving $MAE$ improvements over baselines and graph-based models (notably $GAT$) delivering further gains, demonstrating the value of joint content-structure modeling. By releasing the pipeline and dataset, the work offers a scalable resource for misinformation research and a foundation for studying temporal dynamics in credibility signals.

Abstract

Online misinformation poses an escalating threat, amplified by the Internet's open nature and increasingly capable LLMs that generate persuasive yet deceptive content. Existing misinformation detection methods typically focus on either textual content or network structure in isolation, failing to leverage the rich, dynamic interplay between website content and hyperlink relationships that characterizes real-world misinformation ecosystems. We introduce CrediBench: a large-scale data processing pipeline for constructing temporal web graphs that jointly model textual content and hyperlink structure for misinformation detection. Unlike prior work, our approach captures the dynamic evolution of general misinformation domains, including changes in both content and inter-site references over time. Our processed one-month snapshot extracted from the Common Crawl archive in December 2024 contains 45 million nodes and 1 billion edges, representing the largest web graph dataset made publicly available for misinformation research to date. From our experiments on this graph snapshot, we demonstrate the strength of both structural and webpage content signals for learning credibility scores, which measure source reliability. The pipeline and experimentation code are all available here, and the dataset is in this folder.

Figures (4)

• Figure 1: Overview of the CrediBench pipeline. Common Crawl data is extracted and processed to produce temporal web graphs, where nodes are web domains and edges hyperlinks. These temporal graphs, annotated with human-evaluated credibility scores, are passed to Graph Neural Networks (GNNs) for them to learn the regression task of assigning credibility scores.
• Figure 2: The temporal graph construction pipeline is implemented on a PySpark cluster to process Common Crawl domains collected across multiple months. WAT file segments are used to construct the domain-level web graph, with nodes of degree $\leq 3$ excluded. Domain-level textual content is extracted from WET files, aggregated, and embedded with an off-the-shelf large language model to initialize node features. Edge timestamps are assigned based on the first day of the week of the crawl (usually, the first week of the month).
• Figure 3: Domain text provides the MLP with a clear signal for predicting the PC1 and MBFC scores.
• Figure 4: The relative sizes of CC-December 2024 webgraph, the processed subgraph, and the DQR.

Theorems & Definitions (2)

• Definition 1: Temporal text-Attributed Graphs
• Definition 2: Temporal Node Regression Task