Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PST-Bench: Tracing and Benchmarking the Source of Publications

Fanjin Zhang, Kun Cao, Yukuo Cen, Jifan Yu, Da Yin, Jie Tang

TL;DR

PST-Bench defines and tackles paper source tracing (PST) by constructing a professionally annotated, scalable dataset for identifying direct ref-sources of publications in computer science. It introduces a reading-group workflow to build 1,576 labeled papers (plus 4,800 rule-generated pairs) and analyzes topic- and venue-specific evolution patterns to reveal the hardness of automatic PST. The study surveys statistical, graph-based, and PLM-based methods, finding that pre-trained language models, especially SciBERT, show the most promise though current accuracy leaves room for improvement. Overall, PST-Bench enables rigorous analysis of scientific evolution and provides a benchmark to advance automatic PST research and related analytics.

Abstract

Tracing the source of research papers is a fundamental yet challenging task for researchers. The billion-scale citation relations between papers hinder researchers from understanding the evolution of science efficiently. To date, there is still a lack of an accurate and scalable dataset constructed by professional researchers to identify the direct source of their studied papers, based on which automatic algorithms can be developed to expand the evolutionary knowledge of science. In this paper, we study the problem of paper source tracing (PST) and construct a high-quality and ever-increasing dataset PST-Bench in computer science. Based on PST-Bench, we reveal several intriguing discoveries, such as the differing evolution patterns across various topics. An exploration of various methods underscores the hardness of PST-Bench, pinpointing potential directions on this topic. The dataset and codes have been available at https://github.com/THUDM/paper-source-trace.

PST-Bench: Tracing and Benchmarking the Source of Publications

TL;DR

PST-Bench defines and tackles paper source tracing (PST) by constructing a professionally annotated, scalable dataset for identifying direct ref-sources of publications in computer science. It introduces a reading-group workflow to build 1,576 labeled papers (plus 4,800 rule-generated pairs) and analyzes topic- and venue-specific evolution patterns to reveal the hardness of automatic PST. The study surveys statistical, graph-based, and PLM-based methods, finding that pre-trained language models, especially SciBERT, show the most promise though current accuracy leaves room for improvement. Overall, PST-Bench enables rigorous analysis of scientific evolution and provides a benchmark to advance automatic PST research and related analytics.

Abstract

Tracing the source of research papers is a fundamental yet challenging task for researchers. The billion-scale citation relations between papers hinder researchers from understanding the evolution of science efficiently. To date, there is still a lack of an accurate and scalable dataset constructed by professional researchers to identify the direct source of their studied papers, based on which automatic algorithms can be developed to expand the evolutionary knowledge of science. In this paper, we study the problem of paper source tracing (PST) and construct a high-quality and ever-increasing dataset PST-Bench in computer science. Based on PST-Bench, we reveal several intriguing discoveries, such as the differing evolution patterns across various topics. An exploration of various methods underscores the hardness of PST-Bench, pinpointing potential directions on this topic. The dataset and codes have been available at https://github.com/THUDM/paper-source-trace.
Paper Structure (23 sections, 2 equations, 10 figures, 3 tables)

This paper contains 23 sections, 2 equations, 10 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Problem Definition
  3. Building the PST-Bench
  4. Preliminary Study
  5. Overall Analysis of PST-Bench
  6. Distribution Analysis of ref-sources
  7. Analysis of Different Topics
  8. PST Approach
  9. Statistical Methods
  10. Graph-based Methods
  11. PLM-based Methods
  12. Experiments
  13. Experimental Setup
  14. Main Results
  15. Feature Analysis
  16. ...and 8 more sections

Figures (10)

  • Figure 1: The cumulative distribution function (CDF) of the references' citation ranks for source papers.
  • Figure 2: A filling example. Multiple items are separated by "###" in the fields of ref-sources and keywords.
  • Figure 3: Paper topic distribution. DB and DM: Database and Data Mining, AI: Artificial Intelligence and Pattern Recognition, HPC: High Performance Computing, Graphics and MM: Computer Graphics and Multimedia.
  • Figure 4: Visualization of the simplified PST graph and the simplified citation graph.
  • Figure 5: Analysis of the distribution of ref-sources.
  • ...and 5 more figures