Integrating Large Citation Datasets

TL;DR

The paper addresses the limitations of relying on single citation datasets for measuring scientific impact by proposing the construction of a comprehensive citation graph through merging large citation datasets. An agnostic, parallelizable merging method is introduced that avoids proprietary database systems and uses iterative record matching, reference matching, data slicing, and parallel processing to produce a unified dataset with MUIDs. Applied to two real-world datasets, the approach yields a merged graph with over 119 million nodes and about 1.45 billion edges, with high cross-dataset matching accuracy and improved citation and reference coverage compared with using either source alone. This work demonstrates that large-scale data integration can produce richer representations of scholarly influence and provides a scalable blueprint for incorporating additional datasets to further improve reliability.

Abstract

This paper explores methods for building a comprehensive citation graph using big data techniques to evaluate scientific impact more accurately. Traditional citation metrics have limitations, and this work investigates merging large citation datasets to create a more accurate picture. Challenges of big data, like inconsistent data formats and lack of unique identifiers, are addressed through deduplication efforts, resulting in a streamlined and reliable merged dataset with over 119 million records and 1.4 billion citations. We demonstrate that merging large citation datasets builds a more accurate citation graph facilitating a more robust evaluation of scientific impact.

Figures (3)

• Figure 1: Merging two datasets
• Figure 2: Results of merging datasets
• Figure 3: Comparison of citation and reference counts of matching records