Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating the originality of scientific papers across time and domain: A quantitative analysis

Jack H. Culbert, Yoed N. Kenett, Philipp Mayr

TL;DR

This study introduces Divergent Semantic Integration (DSI) as a text-based metric of scientific originality and applies it to 51,200 WoS abstracts/titles using both BERT and SciBERT embeddings. It demonstrates that DSI, particularly with BERT, modestly predicts five-year citation counts after controlling for field, year, and author count, with adjusted R^2 around 0.10, while SciBERT-DSI shows weaker or non-significant predictive power. The results reveal cross-field variability and a general decline in predictive strength for more recent papers, highlighting the value and limits of textual originality measures in scientometrics. The work provides a foundation for integrating text-derived originality into future evaluations of scientific impact and suggests avenues for refining embedding models and modelling approaches.

Abstract

The study of creativity in science has long sought quantitative metrics capable of capturing the originality of the scientific insights contained within articles and other scientific works. In recent years, the field has witnessed a substantial expansion of research activity, enabled by advances in natural language processing and network analysis, and has utilised both macro- and micro-scale approaches with success. However, they often do not examine the text itself for evidence of originality. In this paper, we apply a computational measure correlating with originality from creativity science, Divergent Semantic Integration (DSI), to a set of 51,200 scientific abstracts and titles sourced from the Web of Science. To adapt DSI for application to scientific texts, we advance the original BERT method by incorporating SciBERT (a model trained on scientific corpora) into the computation of DSI. In our study, we observe that DSI plays a more pronounced role in the accrual of early citations for papers with fewer authors, varies substantially across subjects and research fields, and exhibits a declining correlation with citation counts over time. Furthermore, by modelling SciBERT- and BERT-DSI as predictors of the logarithm of 5-year citation counts alongside field, publication year, and the logarithm of author count, we find statistically significant relationships, with adjusted R-squared of 0.103 and 0.101 for BERT-DSI and SciBERT-DSI. Because existing scientometric measures rarely assess the originality expressed in textual content, DSI provides a valuable means of directly quantifying the conceptual originality embedded in scientific writing.

Investigating the originality of scientific papers across time and domain: A quantitative analysis

TL;DR

This study introduces Divergent Semantic Integration (DSI) as a text-based metric of scientific originality and applies it to 51,200 WoS abstracts/titles using both BERT and SciBERT embeddings. It demonstrates that DSI, particularly with BERT, modestly predicts five-year citation counts after controlling for field, year, and author count, with adjusted R^2 around 0.10, while SciBERT-DSI shows weaker or non-significant predictive power. The results reveal cross-field variability and a general decline in predictive strength for more recent papers, highlighting the value and limits of textual originality measures in scientometrics. The work provides a foundation for integrating text-derived originality into future evaluations of scientific impact and suggests avenues for refining embedding models and modelling approaches.

Abstract

The study of creativity in science has long sought quantitative metrics capable of capturing the originality of the scientific insights contained within articles and other scientific works. In recent years, the field has witnessed a substantial expansion of research activity, enabled by advances in natural language processing and network analysis, and has utilised both macro- and micro-scale approaches with success. However, they often do not examine the text itself for evidence of originality. In this paper, we apply a computational measure correlating with originality from creativity science, Divergent Semantic Integration (DSI), to a set of 51,200 scientific abstracts and titles sourced from the Web of Science. To adapt DSI for application to scientific texts, we advance the original BERT method by incorporating SciBERT (a model trained on scientific corpora) into the computation of DSI. In our study, we observe that DSI plays a more pronounced role in the accrual of early citations for papers with fewer authors, varies substantially across subjects and research fields, and exhibits a declining correlation with citation counts over time. Furthermore, by modelling SciBERT- and BERT-DSI as predictors of the logarithm of 5-year citation counts alongside field, publication year, and the logarithm of author count, we find statistically significant relationships, with adjusted R-squared of 0.103 and 0.101 for BERT-DSI and SciBERT-DSI. Because existing scientometric measures rarely assess the originality expressed in textual content, DSI provides a valuable means of directly quantifying the conceptual originality embedded in scientific writing.

Paper Structure

This paper contains 31 sections, 3 equations, 10 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Divergent Semantic Integration
  3. Creativity in Science
  4. The Significance of Embedding Models
  5. The Current Study
  6. Methodology
  7. Data
  8. Measuring Abstract Originality with DSI
  9. DSI Computation Illustration
  10. Analysis Approach
  11. Results
  12. BERT- and SciBERT-DSI Distributions
  13. DSI and Author Count Sensitivity Analysis
  14. DSI and Publishing Year Sensitivity Analysis
  15. Modelling Scientific Papers' Citation Count Based on DSI
  16. ...and 16 more sections

Figures (10)

  • Figure 1: Violin plots of the DSI for each field, ordered by mean BERT-DSI
  • Figure 2: Line plots of DSI by publication year with 95% confidence interval, by field
  • Figure 3: Boxplot of DSI scores per subject, grouped by field of research, ordered by mean BERT-DSI including outliers within each field
  • Figure 4: Correlation of DSI with citation counts over binned number of authors for both models
  • Figure 5: Correlation of DSI with citation counts over publication year ranges for both models
  • ...and 5 more figures