Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adapting to LLMs: How Insiders and Outsiders Reshape Scientific Knowledge Production

Huimin Xu, Houjiang Liu, Yan Leng, Ying Ding

TL;DR

This study investigates how scientists adapt to large language models (LLMs) and introduces an insider–outsider evaluation workflow anchored in a five-dimension knowledge-production framework to quantify shifts across disciplinary and institutional boundaries. It combines OpenAlex-based large-scale data with few-shot LLM classification and bibliometric validation to compare pre- and post-LLM publications, revealing that outsiders push more toward application-oriented, transdisciplinary, and socially accountable research, while insiders broaden collaboration networks. The findings highlight opportunities for CSCW to design domain-specific tools and mediator roles that support outsider-led AI integration and cross-domain collaboration in AI-driven science. The work advances understanding of how AI reshapes knowledge production, offering practical guidance for governance, collaboration, and infrastructure in diverse research communities.

Abstract

CSCW has long examined how emerging technologies reshape the ways researchers collaborate and produce knowledge, with scientific knowledge production as a central area of focus. As AI becomes increasingly integrated into scientific research, understanding how researchers adapt to it reveals timely opportunities for CSCW research -- particularly in supporting new forms of collaboration, knowledge practices, and infrastructure in AI-driven science. This study quantifies LLM impacts on scientific knowledge production based on an evaluation workflow that combines an insider-outsider perspective with a knowledge production framework. Our findings reveal how LLMs catalyze both innovation and reorganization in scientific communities, offering insights into the broader transformation of knowledge production in the age of generative AI and sheds light on new research opportunities in CSCW.

Adapting to LLMs: How Insiders and Outsiders Reshape Scientific Knowledge Production

TL;DR

This study investigates how scientists adapt to large language models (LLMs) and introduces an insider–outsider evaluation workflow anchored in a five-dimension knowledge-production framework to quantify shifts across disciplinary and institutional boundaries. It combines OpenAlex-based large-scale data with few-shot LLM classification and bibliometric validation to compare pre- and post-LLM publications, revealing that outsiders push more toward application-oriented, transdisciplinary, and socially accountable research, while insiders broaden collaboration networks. The findings highlight opportunities for CSCW to design domain-specific tools and mediator roles that support outsider-led AI integration and cross-domain collaboration in AI-driven science. The work advances understanding of how AI reshapes knowledge production, offering practical guidance for governance, collaboration, and infrastructure in diverse research communities.

Abstract

CSCW has long examined how emerging technologies reshape the ways researchers collaborate and produce knowledge, with scientific knowledge production as a central area of focus. As AI becomes increasingly integrated into scientific research, understanding how researchers adapt to it reveals timely opportunities for CSCW research -- particularly in supporting new forms of collaboration, knowledge practices, and infrastructure in AI-driven science. This study quantifies LLM impacts on scientific knowledge production based on an evaluation workflow that combines an insider-outsider perspective with a knowledge production framework. Our findings reveal how LLMs catalyze both innovation and reorganization in scientific communities, offering insights into the broader transformation of knowledge production in the age of generative AI and sheds light on new research opportunities in CSCW.
Paper Structure (33 sections, 7 figures, 1 table)

This paper contains 33 sections, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Role of AI in Scientific Knowledge Production
  4. AI as a core infrastructure that mediates and transforms scientific collaborative work
  5. AI itself as an object of research investigation
  6. Evaluating AI Impact in Scientific Knowledge Production
  7. Micro-level evaluations
  8. Macro-level evaluations
  9. Insider vs. Outsider Adaptive Changes
  10. Benefits of outsider-within for scientific knowledge production
  11. Challenges of outsider-within for scientific knowledge production
  12. Method
  13. Adapting Insider-Outsider Perspective on Evaluating LLM's Scientific Impact
  14. Dataset
  15. Dataset description and search scope
  16. ...and 18 more sections

Figures (7)

  • Figure 1: The computational process of evaluating the impact of LLM in scientific knowledge production
  • Figure 2: The definition of insider and outsider
  • Figure 3: Comparison of application focus between pre-LLM and LLM papers for both insiders and outsiders. (a) Probability of being classified as application-oriented based on the model. (b) Distribution of application-related keywords across different groups.
  • Figure 4: Comparison of disciplinary orientation between pre-LLM and LLM papers: (a) Model-based probability of disciplinary orientation. (b) Keyword distribution across groups.
  • Figure 5: Comparison of collaboration patterns between pre-LLM and LLM papers for both insiders and outsiders. (a) Probability of being classified as having heterogeneous collaboration based on the model. (b) Probability of involving heterogeneous institution types (e.g., universities, industry, healthcare, government) across different groups.
  • ...and 2 more figures