Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CausalCite: A Causal Formulation of Paper Citations

Ishan Kumar, Zhijing Jin, Ehsan Mokhtarian, Siyuan Guo, Yuen Chen, Mrinmaya Sachan, Bernhard Schölkopf

TL;DR

CausalCite addresses the limitations of citation counts by reframing paper impact as a counterfactual causal effect. It introduces TextMatch, which leverages LLM-based text embeddings to match high-dimensional textual confounders and synthesize stable counterfactuals for each treated follow-up paper. Empirical results show that CausalCite better correlates with expert judgments and test-of-time awards, and it exhibits greater topic invariance across AI subfields. The work provides practical guidance and case studies for applying a causal-textual metric to identify high-quality papers that may be undervalued by traditional citations, while noting substantial computational cost and data-dependence. Future directions include efficiency improvements and extending the framework to broader scholarly domains.

Abstract

Citation count of a paper is a commonly used proxy for evaluating the significance of a paper in the scientific community. Yet citation measures are widely criticized for failing to accurately reflect the true impact of a paper. Thus, we propose CausalCite, a new way to measure the significance of a paper by assessing the causal impact of the paper on its follow-up papers. CausalCite is based on a novel causal inference method, TextMatch, which adapts the traditional matching framework to high-dimensional text embeddings. TextMatch encodes each paper using text embeddings from large language models (LLMs), extracts similar samples by cosine similarity, and synthesizes a counterfactual sample as the weighted average of similar papers according to their similarity values. We demonstrate the effectiveness of CausalCite on various criteria, such as high correlation with paper impact as reported by scientific experts on a previous dataset of 1K papers, (test-of-time) awards for past papers, and its stability across various subfields of AI. We also provide a set of findings that can serve as suggested ways for future researchers to use our metric for a better understanding of the quality of a paper. Our code is available at https://github.com/causalNLP/causal-cite.

CausalCite: A Causal Formulation of Paper Citations

TL;DR

CausalCite addresses the limitations of citation counts by reframing paper impact as a counterfactual causal effect. It introduces TextMatch, which leverages LLM-based text embeddings to match high-dimensional textual confounders and synthesize stable counterfactuals for each treated follow-up paper. Empirical results show that CausalCite better correlates with expert judgments and test-of-time awards, and it exhibits greater topic invariance across AI subfields. The work provides practical guidance and case studies for applying a causal-textual metric to identify high-quality papers that may be undervalued by traditional citations, while noting substantial computational cost and data-dependence. Future directions include efficiency improvements and extending the framework to broader scholarly domains.

Abstract

Citation count of a paper is a commonly used proxy for evaluating the significance of a paper in the scientific community. Yet citation measures are widely criticized for failing to accurately reflect the true impact of a paper. Thus, we propose CausalCite, a new way to measure the significance of a paper by assessing the causal impact of the paper on its follow-up papers. CausalCite is based on a novel causal inference method, TextMatch, which adapts the traditional matching framework to high-dimensional text embeddings. TextMatch encodes each paper using text embeddings from large language models (LLMs), extracts similar samples by cosine similarity, and synthesizes a counterfactual sample as the weighted average of similar papers according to their similarity values. We demonstrate the effectiveness of CausalCite on various criteria, such as high correlation with paper impact as reported by scientific experts on a previous dataset of 1K papers, (test-of-time) awards for past papers, and its stability across various subfields of AI. We also provide a set of findings that can serve as suggested ways for future researchers to use our metric for a better understanding of the quality of a paper. Our code is available at https://github.com/causalNLP/causal-cite.
Paper Structure (39 sections, 6 equations, 11 figures, 7 tables, 1 algorithm)

This paper contains 39 sections, 6 equations, 11 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Citation Graph
  4. Causal Graph.
  5. CausalCite Indices
  6. Two-Paper Interaction: Pairwise Causal Impact (PCI).
  7. Single-Paper Quality Metrics: Total Causal Impact (TCI) and Average Causal Impact (ACI).
  8. The TextMatch Method
  9. What Does Causal Inference Tell Us about What Variables to Control for, and What Not?
  10. Can Existing Causal Inference Methods Handle This Control?
  11. Randomized Control Trials (RCTs) Assumes Intervenability.
  12. Ratio Matching Iterates over Discretized Confounder Values.
  13. One-to-One Matching Is Susceptible to Variance.
  14. How Do We Extending Causal Inference to Text Variables?
  15. Theoretical Formulation of TextMatch: Stabilizing Text Matching by Synthesis
  16. ...and 24 more sections

Figures (11)

  • Figure 1: An overview of our research question.
  • Figure 1: Accuracy of all three citation metrics.
  • Figure 2: The causal graph of our study.
  • Figure 3: Distributions of ToT (mean: 142) and non-ToT papers (mean: 1,623).
  • Figure 4: The CausalCite values of three example ToT papers from general AI, NLP, and CV.
  • ...and 6 more figures