Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Auditing citation polarization during the early COVID-19 pandemic

Taekho You, Jinseo Park, June Young Lee, Jinhyuk Yun

TL;DR

The paper addresses how the early COVID-19 publication surge reshaped citation dynamics and journal impact-factor (IF) distributions, revealing a pronounced polarization that favors high-IF journals. Using a Web of Science audit (2017–2022), it analyzes tail behavior with a power-law framework, shows a heavy-tailed citation pattern for COVID-19 work (tail exponent $\alpha$; tail slope $k$ in $y \sim x^k$), and demonstrates that COVID-19 publications contributed substantially to IF inflation, especially for prestigious journals. A superlinear relationship $y \sim x^{1.7}$ links baseline IF to surplus IF, while per-publication gains diminish with more COVID-19 items; most highly cited COVID-19 papers appeared in top-tier journals early in the pandemic, amplifying inequality. The findings challenge the use of IF as a proxy for individual paper significance and highlight the need for robust, qualitative evaluation alongside alternative metrics, given the risk of transient shocks and retractions in a rapidly evolving research landscape.

Abstract

The recent pandemic stimulated scientists to publish a significant amount of research that created a surge of citations of COVID-19-related publications in a short time, leading to an abrupt inflation of the journal impact factor (IF). By auditing the complete set of COVID-19-related publications in the Web of Science, we reveal here that COVID-19-related research worsened the polarization of academic journals: the IF before the pandemic was proportional to the increment of IF, which had the effect of increasing inequality while retaining the journal rankings. We also found that the most highly cited studies related to COVID-19 were published in prestigious journals at the onset of the epidemic. Through the present quantitative investigation, our findings caution against the belief that quantitative metrics, particularly IF, can indicate the significance of individual papers. Rather, such metrics reflect the social attention given to a particular study.

Auditing citation polarization during the early COVID-19 pandemic

TL;DR

The paper addresses how the early COVID-19 publication surge reshaped citation dynamics and journal impact-factor (IF) distributions, revealing a pronounced polarization that favors high-IF journals. Using a Web of Science audit (2017–2022), it analyzes tail behavior with a power-law framework, shows a heavy-tailed citation pattern for COVID-19 work (tail exponent ; tail slope in ), and demonstrates that COVID-19 publications contributed substantially to IF inflation, especially for prestigious journals. A superlinear relationship links baseline IF to surplus IF, while per-publication gains diminish with more COVID-19 items; most highly cited COVID-19 papers appeared in top-tier journals early in the pandemic, amplifying inequality. The findings challenge the use of IF as a proxy for individual paper significance and highlight the need for robust, qualitative evaluation alongside alternative metrics, given the risk of transient shocks and retractions in a rapidly evolving research landscape.

Abstract

The recent pandemic stimulated scientists to publish a significant amount of research that created a surge of citations of COVID-19-related publications in a short time, leading to an abrupt inflation of the journal impact factor (IF). By auditing the complete set of COVID-19-related publications in the Web of Science, we reveal here that COVID-19-related research worsened the polarization of academic journals: the IF before the pandemic was proportional to the increment of IF, which had the effect of increasing inequality while retaining the journal rankings. We also found that the most highly cited studies related to COVID-19 were published in prestigious journals at the onset of the epidemic. Through the present quantitative investigation, our findings caution against the belief that quantitative metrics, particularly IF, can indicate the significance of individual papers. Rather, such metrics reflect the social attention given to a particular study.
Paper Structure (12 sections, 2 equations, 14 figures, 1 table)

This paper contains 12 sections, 2 equations, 14 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Data
  4. COVID-19-related publications
  5. Estimation of the power-law exponent
  6. Reproduction of the journal impact factors
  7. Keyword co-occurrence analysis
  8. Results
  9. Citation exchange between COVID-19-related publications
  10. Contribution of COVID-19-related research to IF inflation
  11. The Matthew effect of IF polarization during the pandemic
  12. Discussion

Figures (14)

  • Figure 1: Difference in citation distribution between COVID-19-related and non-COVID-19-related publications.A Citation distribution of COVID-19-related and non-COVID-19-related publications that contribute to the annual IF calculation. B Citation origin of COVID-19-related publications. We display both the percentage of citations received from other COVID-19-related publications and references citing other COVID-19 publications.
  • Figure 2: Surplus impact factor (IF) by COVID-19-related publications.A Journal impact factor increase by publishing COVID-19-related publications, where the simple superlinear growth $y \sim x^{1.7}$ can characterize the growth pattern (dotted line). B Increase in IF per COVID-19-related publication in proportion to the number of COVID-19-related publications published in journals. In both A and B, the red dots represent the average values in the log-scale (i.e. geometric means) of surplus IF, and the error bars show the standard deviation in the log-scale.
  • Figure 3: Relative ratio of surplus IF from publishing COVID-19-related publications by the 2021 journal rankings for JCR categories. The ratio was calculated by dividing the IF including COVID-19-related publications by the IF excluding COVID-19-related publications for 2021. All journals that published at least one COVID-19-related publication are accounted for, regardless of whether the IF is gained or dropped by publishing COVID-19-related publications. The dotted line indicates the average ratio from publishing COVID-19-related research (15.2%). Here, the boxes represent the quartiles of the dataset except for points determined to be outliers.
  • Figure 4: Distribution of COVID-19-related publications and their disparities.A Distribution of COVID-19-related research by journal ranking. The journal ranking is computed within the research category. B Distribution of COVID-19-related publications by publication date. C Plot of the Gini coefficient of the IF distribution by JCR category. Each dot represents a JCR category. The Gini coefficient is computed using the IF distribution of journals in a particular category including and excluding COVID-19-related publications. Blue (orange) dots indicate an increase (decrease) in the Gini coefficient by publishing COVID-19-related research. The size of the dots is proportional to the number of COVID-19-related studies published in the category.
  • Figure S1: Correlation between the IFs provided by JCR and those calculated in this paper. The Pearson correlation is 0.997 and 0.998 for 2020 and 2021. The Spearman correlation is 0.992 and 0.994 for 2020 and 2021. Insets show the lower part that ranges from 0 to 50.
  • ...and 9 more figures