Science of science -- Citation models and research evaluation
V. A. Traag
TL;DR
This chapter tackles how citations can inform research evaluation when scientific quality is unobservable. It surveys citation distributions, aging dynamics, and a spectrum of models, then links this to the literature on peer review and metrics. A core message is that indicators are potentially biased by confounders and that normalization and explicit uncertainty modelling are essential. By combining citation models with causal reasoning, the chapter advocates an integrated approach to progress in the science of science and policy-relevant evaluation.
Abstract
Citations in science are being studied from several perspectives, among which approaches such as scientometrics and science of science. In this chapter I briefly review some of the literature on citations, citation distributions and models of citations. These citations feature prominently in another part of the literature which is dealing with research evaluation and the role of metrics and indicators in that process. Here I briefly review part of the discussion in research evaluation. This also touches on the subject of how citations relate to peer review. Finally, I conclude by trying to integrate the two literatures. The fundamental problem in research evaluation is that research quality is unobservable. This has consequences for conclusions that we can draw from quantitative studies of citations and citation models. The term ``indicators'' is a relevant concept in this context, which I try to clarify. Causality is important for properly understanding indicators, especially when indicators are used in practice: when we act on indicators, we enter causal territory. Even when an indicator might have been valid, through its very use, the consequences of its use may invalidate it. By combining citation models with proper causal reasoning and acknowledging the fundamental problem about unobservable research quality, we may hope to make progress.