The origin, consequence, and visibility of criticism in science

TL;DR

The study investigates the origins, consequences, and visibility of peer critique in science by analyzing over 3,000 explicit critical letters across elite journals. Using a data-driven framework, it links criticism to target paper features (impact, interdisciplinarity, novelty) and employs quasi-experimental matching to compare cited trajectories and author outcomes with comparable papers. The main finding is that criticized papers are highly cited and often interdisciplinary or novel, yet receiving a critical letter does not measurably alter their citation growth or authors' careers, suggesting criticism has limited influence in practice. Limited visibility of critical letters—only a fraction of post-criticism citations co-occur with the letter—emerges as a key explanation, with stronger visibility within generalist journals than within APS fields, highlighting challenges for post-publication critique to shape scientific discourse at scale.

Abstract

Critique between peers plays a vital role in the production of scientific knowledge. Yet, there is limited empirical evidence on the origins of criticism, its effects on the papers and individuals involved, and its visibility within the scientific literature. Here, we address these gaps through a data-driven analysis of papers that received substantiated and explicit written criticisms. Our analysis draws on data representing over 3,000 ``critical letters'' -- papers explicitly published to critique another -- from four high profile journals, with each letter linked to its target paper. We find that the papers receiving critical letters are disproportionately among the most highly-cited in their respective journal and, to a lesser extent, among the most interdisciplinary and novel. However, despite the theoretical importance of criticism in scientific progress, we observe no evidence that receiving a critical letter affects a paper's citation trajectory or the productivity and citation impact of its authors. One explanation for the limited consequence of critical letters is that they often go unnoticed. Indeed, we find that critical letters attract only a small fraction of the citations received by their targets, even years after publication. An analysis of topical similarity between criticized papers and their citing papers indicates that critical letters are primarily cited by researchers actively engaged in a similar field of study, whereas they are overlooked by more distant communities. Although criticism is celebrated as a cornerstone to science, our findings reveal that it is concentrated on high-impact papers, has minimal measurable consequences, and suffers from limited visibility. These results raise important questions about the role and value of critique in scientific practice.

Figures (13)

• Figure 1: Characterizing the population of papers targeted by critical letters. Shown is the distribution of percentile ranks for publications targeted by critical letters across four metrics and journals Percentile ranks are calculated within each journal, over a four-year time period, and within a high-level field. The metrics include (A.1-5) citations received by papers within 3 years of publication; (B.1-5) the diversity of referenced papers, measured using the Simpson's Index of their high-level field categories; (C.1-5) the diversity of citing papers published within five years of the target, calculated using the same index; (D.1-5) a bibliometric indicator measuring the atypicality of a paper’s cited references. For detailed definitions, see Materials & Methods. For each journal and metric, $\mu$ (referred to in-text as $mu_{rank}$ denotes the average percentile rank of papers targeted by criticism. To examine whether differences in diversity and novelty are mediated by citation impact, a second population of matched papers (black dashed line) is included. Using propensity score matching, we identify for each targeted paper the nearest match from the same journal, in the same high-level field, published within a three-year window, and within a 5% tolerance of logarithmic three-year impact (see Materials & Methods for details). To guide interpretation, we employ two Kolmogorov–Smirnov (KS) tests. ("1s KS") A one-sample, one-sided KS test compares the distribution of percentile ranks for targeted papers against a uniform distribution. A low p-value indicates that targeted papers are concentrated among higher ranks, rather than being randomly distributed across the journal. ("2s KS") A two-sample, one-sided KS test compares the distribution of percentile ranks for targeted papers against the matched population. A low p-value suggests that targeted papers are concentrated among higher ranks compared to their matched counterparts. P-values for each test are shown in each panel, and results are shown in greater detail in Tables \ref{['si:table:paper-features-1sKS']}-\ref{['si:table:paper-features-2sKS']}.
• Figure 2: The consequences of criticism. The x-axis shows, from left to right: (A) Comparison between the cumulative impact growth of papers targeted by a critical letter compared to a matched population of control papers from the same journal, within one year of publication, the same field, and with similar impact. Here, change in "paper impact" ($\Delta$Treatment and $\Delta$Control) is defined as the ratio between the citations received by a paper after receipt of a critical letter compared to those received prior; in the case of the control population we use an equivalent time lag. (B-C) Comparison of the change in the average yearly fractional productivity before and after the critical letter for first and last authors of targeted papers compared against a control population of authors sampled from the same journal and with similar prior performance. (D-E) Comparison of the change in average impact prior for first and last authors of targeted papers compared against the control population. Here, author impact refers to the ratio in average field-normalized 3-year citation impact of papers published in the five years before and after receipt of the critical letter. For matching papers, citation tolerance is set to 5%. For matching authors, both citation and productivity tolerance is set to 10%. Error bars correspond to 95% confidence intervals. For each comparison we conduct paired t-tests comparing the treatment and control groups, with asterisks included to indicate the significance level ("*" when $p < 0.05$); these tests are to guide interpretation, and not for confirmatory analysis.
• Figure 3: Critical letters have low impact despite continued citation to criticized papers. The x-axis shows time normalized as the number of years since receipt of a critical letter. The y-axis shows: (A) the pooled average number of citations received each year by papers that were targeted by a critical letter; (B) the pooled average percentage of citations to the targeted paper which also cite the critical letter. Author self-citations, defined as any overlap between authors of the cited and citing paper, are excluded in all plots. Replies to critical letters are also excluded.
• Figure 4: Papers similar to the criticized study co-cite the critical letter. Shown are the distribution of similarity ranks of studies that cite both the study which received a critical letter along with the letter itself. Similarity ranks are computed for each criticized paper and based on the population of all citing papers; for example, a rank of 0.9 indicates that the citing paper is more similar to the criticized study than 90% of its citations. For each journal, $\mu$ refers to the average percentile rank of co-citing paper similarity. We conduct two Kolmogorov–Smirnov tests to guide interpretation. “1s KS” refers to a one-sample, one-sided Kolmogorov–Smirnov test comparing the distribution of percentile ranks of co-citing papers against a uniform distribution. low p-value (by convention, $p < 0.05$) suggests that co-citing papers are concentrated among the most similar studies. Results are shown in greater detail in Table \ref{['si:table:embedding-tests']}.
• Figure S1: Receipt of critical letters is associated with impact. Shown is the relationship between the 3-year citation impact of papers and the likelihood of receipt of a critical letter, $r(k)$. The likelihood, $r(k)$, is estimated by first diving all publications in each journal into discrete bins based on their 3-year citation impact. Bins are logarithmic scaled such that they grow progressively wider. Then, within each bin, $r(k)$ is estimated as the ratio between the proportion of all papers targeted by a critical letter appearing in that bin and the proportion of all non-targeted papers within the same bin. Each point corresponds to an estimated $r(k)$ within a particular bin and journal. A loess regression is shown for each journal to aid interpretation. The legend shows Pearson's $\rho$ to summarize the linear correlation between impact and $r(k)$ for each journal; to compute this we use the logarithmically-scaled right-hand edge of each bin. This graph illustrates that the likelihood of receiving a critical letter grows roughly linearly with papers' log-impact.