Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers

Abstract

Through an analysis of arXiv papers, we report several shifts in word usage that are likely driven by large language models (LLMs) but have not previously received sufficient attention, such as the increased frequency of "beyond" and "via" in titles and the decreased frequency of "the" and "of" in abstracts. Due to the similarities among different LLMs, experiments show that current classifiers struggle to accurately determine which specific model generated a given text in multi-class classification tasks. Meanwhile, variations across LLMs also result in evolving patterns of word usage in academic papers. By adopting a direct and highly interpretable linear approach and accounting for differences between models and prompts, we quantitatively assess these effects and show that real-world LLM usage is heterogeneous and dynamic.

Figures (18)

• Figure 1: Top-left & Middle-left: Word frequency comparison for generated titles or rewritten abstracts based on 2,000 real arXiv abstracts; error bars denote variance across models and prompts. Remaining panels: Temporal trends of word frequencies in real arXiv data, with the yellow dashed line fitted on data from 2015 to 2021 and extended to early 2026.
• Figure 2: Examples of word frequencies in paper titles.
• Figure 3: Frequency change ratios of the 20 most frequent words in 2,000 abstracts using multiple LLMs and the shorter prompt.
• Figure 4: Word frequencies in 2,000 abstracts or in the corresponding LLM-processed content. Error bars denote the standard deviation of word frequencies across outputs produced by different LLMs and/or prompts.
• Figure 5: The frequencies of some words that appeared at least 20 times in the 2,000 abstracts used for the simulation.