Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the performativity of SDG classifications in large bibliometric databases

Matteo Ottaviani, Stephan Stahlschmidt

TL;DR

This paper investigates how SDG classifications in large bibliometric databases shape scientific visibility and impact metrics. It uses a fine-tuned DistilGPT-2 language model for each database and SDG to interrogate biases by analyzing a jointly indexed dataset of roughly 15 million publications across five SDGs. The study finds very limited agreement across Web of Science, OpenAlex, and Scopus in SDG assignments, with cross-database overlap as low as 1.3–7.2% and substantial DB-specific noun phrases, and shows that model choice and decoding strategy strongly influence outputs. The results highlight the performative risk of SDG classifications in research practice and policy and demonstrate how LLM-based analyses can reveal biases, underscoring the need for careful methodology and standardization in SDG bibliometrics.

Abstract

Large bibliometric databases, such as Web of Science, Scopus, and OpenAlex, facilitate bibliometric analyses, but are performative, affecting the visibility of scientific outputs and the impact measurement of participating entities. Recently, these databases have taken up the UN's Sustainable Development Goals (SDGs) in their respective classifications, which have been criticised for their diverging nature. This work proposes using the feature of large language models (LLMs) to learn about the "data bias" injected by diverse SDG classifications into bibliometric data by exploring five SDGs. We build a LLM that is fine-tuned in parallel by the diverse SDG classifications inscribed into the databases' SDG classifications. Our results show high sensitivity in model architecture, classified publications, fine-tuning process, and natural language generation. The wide arbitrariness at different levels raises concerns about using LLM in research practice.

On the performativity of SDG classifications in large bibliometric databases

TL;DR

This paper investigates how SDG classifications in large bibliometric databases shape scientific visibility and impact metrics. It uses a fine-tuned DistilGPT-2 language model for each database and SDG to interrogate biases by analyzing a jointly indexed dataset of roughly 15 million publications across five SDGs. The study finds very limited agreement across Web of Science, OpenAlex, and Scopus in SDG assignments, with cross-database overlap as low as 1.3–7.2% and substantial DB-specific noun phrases, and shows that model choice and decoding strategy strongly influence outputs. The results highlight the performative risk of SDG classifications in research practice and policy and demonstrate how LLM-based analyses can reveal biases, underscoring the need for careful methodology and standardization in SDG bibliometrics.

Abstract

Large bibliometric databases, such as Web of Science, Scopus, and OpenAlex, facilitate bibliometric analyses, but are performative, affecting the visibility of scientific outputs and the impact measurement of participating entities. Recently, these databases have taken up the UN's Sustainable Development Goals (SDGs) in their respective classifications, which have been criticised for their diverging nature. This work proposes using the feature of large language models (LLMs) to learn about the "data bias" injected by diverse SDG classifications into bibliometric data by exploring five SDGs. We build a LLM that is fine-tuned in parallel by the diverse SDG classifications inscribed into the databases' SDG classifications. Our results show high sensitivity in model architecture, classified publications, fine-tuning process, and natural language generation. The wide arbitrariness at different levels raises concerns about using LLM in research practice.
Paper Structure (6 sections, 5 figures)

This paper contains 6 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Data Collection
  3. Methods
  4. Results
  5. Discussion of the results
  6. Conclusions

Figures (5)

  • Figure 1: The jointly indexed publication dataset of Web of Science, OpenAlex and Scopus, counts 15 471 336 publications. It is obtained on an exact DOI match, when publications are published between 2015 and July 2023, and where the DOI is unique. Moreover, only article or review in journal are accounted.
  • Figure 2: Venn Diagrams of SDGs 4, 5, 8, 9, and 10, for Web of Science, OpenAlex, and Scopus.
  • Figure 3: Schematic illustration of the research design followed in this paper. We fine-tune a blank large language model based on the architecture DistilGPT-2 to the subset of publication abstracts classified to a given SDG by a given bibliometric DB. Once obtained the fine-tuned LLM, we administrate to it a set of prompts (tailored on the SDG) through three different decoding strategies. Then, we collect into the same set the noun phrases extracted from the three response sets according to a minimum threshold. For each SDG, once obtained the latter sets for all the DBs involved, we filter out the common words, gathering them into another set.
  • Figure 4: For each SDG, the common set is the collection of those noun phrases which emerge from LLM responses in all the three databases.
  • Figure 5: For each SDG and bibliometric DB, frequency bar charts of the "unique" sets; i.e., noun phrases sets except those items that are in common among the three databases.