Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Why Does ChatGPT "Delve" So Much? Exploring the Sources of Lexical Overrepresentation in Large Language Models

Tom S. Juzek, Zina B. Ward

TL;DR

The paper addresses why certain high-frequency words appear disproportionately in scientific abstracts generated or influenced by large language models. It presents a formal, transferable method to identify focal words and applies it to PubMed abstracts to detect 21 words overrepresented in AI-written text. Through corpus analyses, model-comparison experiments, and an exploratory RLHF study, the authors find little evidence that model architecture, algorithms, or training data alone explain lexical overrepresentation, while RLHF effects are plausible but not conclusively demonstrated. The work highlights LLM-driven changes in scientific language, underscores transparency challenges in model development, and offers a replicable framework for probing linguistic biases across domains.

Abstract

Scientific English is currently undergoing rapid change, with words like "delve," "intricate," and "underscore" appearing far more frequently than just a few years ago. It is widely assumed that scientists' use of large language models (LLMs) is responsible for such trends. We develop a formal, transferable method to characterize these linguistic changes. Application of our method yields 21 focal words whose increased occurrence in scientific abstracts is likely the result of LLM usage. We then pose "the puzzle of lexical overrepresentation": WHY are such words overused by LLMs? We fail to find evidence that lexical overrepresentation is caused by model architecture, algorithm choices, or training data. To assess whether reinforcement learning from human feedback (RLHF) contributes to the overuse of focal words, we undertake comparative model testing and conduct an exploratory online study. While the model testing is consistent with RLHF playing a role, our experimental results suggest that participants may be reacting differently to "delve" than to other focal words. With LLMs quickly becoming a driver of global language change, investigating these potential sources of lexical overrepresentation is important. We note that while insights into the workings of LLMs are within reach, a lack of transparency surrounding model development remains an obstacle to such research.

Why Does ChatGPT "Delve" So Much? Exploring the Sources of Lexical Overrepresentation in Large Language Models

TL;DR

The paper addresses why certain high-frequency words appear disproportionately in scientific abstracts generated or influenced by large language models. It presents a formal, transferable method to identify focal words and applies it to PubMed abstracts to detect 21 words overrepresented in AI-written text. Through corpus analyses, model-comparison experiments, and an exploratory RLHF study, the authors find little evidence that model architecture, algorithms, or training data alone explain lexical overrepresentation, while RLHF effects are plausible but not conclusively demonstrated. The work highlights LLM-driven changes in scientific language, underscores transparency challenges in model development, and offers a replicable framework for probing linguistic biases across domains.

Abstract

Scientific English is currently undergoing rapid change, with words like "delve," "intricate," and "underscore" appearing far more frequently than just a few years ago. It is widely assumed that scientists' use of large language models (LLMs) is responsible for such trends. We develop a formal, transferable method to characterize these linguistic changes. Application of our method yields 21 focal words whose increased occurrence in scientific abstracts is likely the result of LLM usage. We then pose "the puzzle of lexical overrepresentation": WHY are such words overused by LLMs? We fail to find evidence that lexical overrepresentation is caused by model architecture, algorithm choices, or training data. To assess whether reinforcement learning from human feedback (RLHF) contributes to the overuse of focal words, we undertake comparative model testing and conduct an exploratory online study. While the model testing is consistent with RLHF playing a role, our experimental results suggest that participants may be reacting differently to "delve" than to other focal words. With LLMs quickly becoming a driver of global language change, investigating these potential sources of lexical overrepresentation is important. We note that while insights into the workings of LLMs are within reach, a lack of transparency surrounding model development remains an obstacle to such research.

Paper Structure

This paper contains 16 sections, 1 equation, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Corpus Analysis: Identification of Overrepresented Lexical Items
  3. The Puzzle of Lexical Overrepresentation
  4. Searching for Overrepresentation in Possible Training Data
  5. Model Choices: Architecture and Algorithms
  6. RLHF: An Experimental Approach
  7. Discussion and Concluding Remarks
  8. List Of Focal Words
  9. Analysis Of Further Corpora and GPT-4o
  10. Examples Of Critical Items
  11. A delve-initial item
  12. A non delve-initial item
  13. Per-word Entropy for various Llama Models
  14. The Rating Interface
  15. Ratings For The Distractor Items
  16. ...and 1 more sections

Figures (8)

  • Figure 1: We formalize a procedure for identifying words whose increasing prevalence is likely the result of LLM usage. Although our focus is Scientific English, the method can be applied across domains and languages.
  • Figure 2: Selected lexical entries: change over time.
  • Figure 3: Our method for the systematic identification of focal words.
  • Figure 4: Occurrences per million words in PubMed abstracts for our 21 focal words.
  • Figure 5: Experimental results: Preferences between focal-word and non-focal-word abstracts in delve-initial and other items.
  • ...and 3 more figures