Could Bibliometrics Reveal Top Science and Technology Achievements and Researchers? The Case for Evaluatology-based Science and Technology Evaluation

TL;DR

Bibliometric indicators inadequately capture science and technology achievements due to confounding factors, potential manipulation, and field biases. The authors propose evaluatology-based S&T evaluation built around an extended evaluation condition (EC), a real-world evaluation system (ES), a perfect evaluation model (EM), and a pragmatic EM, plus four relationships to connect achievements. They operationalize Top N @X @Y to extract a field-specific Top N set by pruning non-significant items via four rounds, and compute a simple composite score $V = \sum_{i} \lg(X_i)$ to aggregate multi-dimensional impacts. A Chip100 case study demonstrates that this approach identifies top achievements and contributors beyond bibliometric rankings like CSRankings and Highly Cited Researchers, highlighting practical impact and diversity of researchers and institutions.

Abstract

By utilizing statistical methods to analyze bibliographic data, bibliometrics faces inherent limitations in identifying the most significant science and technology achievements and researchers. To overcome this challenge, we present an evaluatology-based science and technology evaluation methodology. At the heart of this approach lies the concept of an extended evaluation condition, encompassing eight crucial components derived from a field. We define four relationships that illustrate the connections among various achievements based on their mapped extended EC components, as well as their temporal and citation links. Within a relationship under an extended evaluation condition, evaluators can effectively compare these achievements by carefully addressing the influence of confounding variables. We establish a real-world evaluation system encompassing an entire collection of achievements, each of which is mapped to several components of an extended EC. Within a specific field like chip technology or open source, we construct a perfect evaluation model that can accurately trace the evolution and development of all achievements in terms of four relationships based on the real-world evaluation system. Building upon the foundation of the perfect evaluation model, we put forth four-round rules to eliminate non-significant achievements by utilizing four relationships. This process allows us to establish a pragmatic evaluation model that effectively captures the essential achievements, serving as a curated collection of the top N achievements within a specific field during a specific timeframe. We present a case study on the top 100 Chip achievements which highlights its practical application and efficacy in identifying significant achievements and researchers that otherwise can not be identified by using bibliometrics.

Figures (8)

• Figure 1: Fundamental differences between bibliometrics and S&T Evaluatology.
• Figure 2: Comparison of the number of scientists selected for the global top 2% in different disciplines.
• Figure 3: Comparison of accepted papers by top conferences in the fields of computer vision and computer architecture.
• Figure 4: The overview of an extended EC.
• Figure 5: Illustrating S&T Evaluatology with an example.