Collaboration or Corporate Capture? Quantifying NLP's Reliance on Industry Artifacts and Contributions

TL;DR

The paper investigates whether NLP research increasingly relies on industry artifacts and whether this reflects collaboration or corporate capture. It analyzes 100 EMNLP 2022 papers, classifying affiliations, citation types (Datasets, PTMs, Prior Bests, Full Setting), and SOTA claims, and applies a Gaussian smoothing to track industry-citation trends over time. The findings show industry-affiliated citations far exceed expectations, with PTMs and datasets predominantly industry-originating, and with industry papers delivering larger relative SOTA gains. The authors discuss implications for transparency, funding disclosures, and compute equity, offering concrete recommendations to mitigate biases while fostering progress. Overall, the work provides a baseline for measuring industry influence in NLP and motivates strategies to ensure diverse and independent research directions.

Abstract

Impressive performance of pre-trained models has garnered public attention and made news headlines in recent years. Almost always, these models are produced by or in collaboration with industry. Using them is critical for competing on natural language processing (NLP) benchmarks and correspondingly to stay relevant in NLP research. We surveyed 100 papers published at EMNLP 2022 to determine the degree to which researchers rely on industry models, other artifacts, and contributions to publish in prestigious NLP venues and found that the ratio of their citation is at least three times greater than what would be expected. Our work serves as a scaffold to enable future researchers to more accurately address whether: 1) Collaboration with industry is still collaboration in the absence of an alternative or 2) if NLP inquiry has been captured by the motivations and research direction of private corporations.

Figures (6)

• Figure 1: Percentage of surveyed EMNLP 2022 paper citations with industry affiliation smoothed with a 1D Gaussian Filter ($\sigma = 2.5$). The "Expected" line is the percentage of industry affiliation across the entire ACL anthology over time. One could expect cited papers of a given year to have the same degree of industry affiliation. The citations are split into the following types: Datasets, Pre-Trained Models, Prior Bests, and Full Setting scores (See \ref{['sec:citation_def']} for definitions of citation types).
• Figure 2: Per paper average relative score increase distributions grouped by institution type for All-Time (top) and Unique (bottom) Setting SOTA papers. No non- profit papers surveyed claimed All-Time SOTA and the corresponding row is therefore excluded. Green triangles and orange vertical bars denote the mean and median respectively and the numerical value for the mean is labelled beneath it.
• Figure 3: Percentage of surveyed EMNLP 2022 paper citations with industry affiliation---labelled according to majority---smoothed with a 1D Gaussian Filter ($\sigma = 2.5$). The "Expected" line uses the original heuristic for quantifying industry as specified in \ref{['sec:affil_def']} since that is how abdalla-etal-2023-elephant reported.
• Figure 4: Per paper average relative score increase distributions grouped by institution type---labelled via majority heuristic---for All-Time (top) and Unique (bottom) Setting SOTA papers. Green triangles and orange vertical bars denote the mean and median respectively and the numerical value for the mean is labelled beneath it.
• Figure 5: Proportion of cited institute types split by EMNLP 2022 author affiliation. Citations are grouped by PTM (upper left), Datasets (upper right), Prior Bests (bottom left), and Full Settings (bottom right). $n$ is the total number of citations per bar. The sum of $n$ per subplot exceeds the 100 papers examined since each paper often cites more than one PTM or Dataset and reports more than one score for Prior Bests and Full Settings.