Structural shifts in institutional participation and collaboration within the AI arXiv preprint research ecosystem

TL;DR

This paper analyzes how the AI research ecosystem, as reflected in cs.AI arXiv preprints from 2021–2025, reorganized after the advent of large language models, notably ChatGPT. It uses a two-stage pipeline—data enrichment and LLM-based institution labeling—to quantify publication growth, team-size expansion, and cross-sector collaboration via the Normalized Collaboration Index (NCI). The results show a massive post-2022 surge in output, dominated by academic institutions, while actual academic–industry collaboration remains consistently below random-mixing expectations (NCI < 1) across subfields, suggesting a compute divide that reinforces institutional silos. The findings highlight the structural implications of generative AI on the science of science and point to policy avenues to bridge theory and industry capabilities in an era of increasingly compute-intensive AI research.

Abstract

The emergence of large language models (LLMs) represents a significant technological shift within the scientific ecosystem, particularly within the field of artificial intelligence (AI). This paper examines structural changes in the AI research landscape using a dataset of arXiv preprints (cs.AI) from 2021 through 2025. Given the rapid pace of AI development, the preprint ecosystem has become a critical barometer for real-time scientific shifts, often preceding formal peer-reviewed publication by months or years. By employing a multi-stage data collection and enrichment pipeline in conjunction with LLM-based institution classification, we analyze the evolution of publication volumes, author team sizes, and academic--industry collaboration patterns. Our results reveal an unprecedented surge in publication output following the introduction of ChatGPT, with academic institutions continuing to provide the largest volume of research. However, we observe that academic--industry collaboration is still suppressed, as measured by a Normalized Collaboration Index (NCI) that remains significantly below the random-mixing baseline across all major subfields. These findings highlight a continuing institutional divide and suggest that the capital-intensive nature of generative AI research may be reshaping the boundaries of scientific collaboration.

Figures (5)

• Figure 1: Overview of the two-stage data collection, enrichment, and institution-classification pipeline. Papers are collected from arXiv (cs.AI), enriched using OpenAlex and arXiv (Stage 1), classified via a two-pass LLM-based approach (Stage 2), and analyzed using collaboration metrics addressing RQ1--RQ3 (bottom).
• Figure 2: Example of arXiv metadata before and after enrichment for a representative paper (arXiv ID: 2501.00692v1). Step 1 corresponds to raw arXiv metadata, while Step 2 includes additional affiliation and email information extracted during enrichment.
• Figure 3: Illustration of the metadata enrichment process. (a) The original arXiv abstract page contains only basic bibliographic information without structured affiliations or contact details. (b) The corresponding ar5iv HTML mirror exposes author affiliations and email addresses, which are harvested during the enrichment stage of our pipeline.
• Figure 4: Trends in AI publications by affiliation type from 2021–2025, showing (a) observed counts and (b) adjusted counts after redistributing unknown affiliations using year-specific validation samples.
• Figure 5: Normalized Collaboration Index (NCI) and its components. Panel A shows monthly NCI values relative to the random-mixing baseline ($\mathrm{NCI}=1$). Panel B displays publication volume and observed mixed-collaboration proportion.