Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Evaluation on Large Language Model Outputs: Discourse and Memorization

Adrian de Wynter, Xun Wang, Alex Sokolov, Qilong Gu, Si-Qing Chen

TL;DR

The paper empirically evaluates how memorization from training data affects outputs of nine large language models across five domains, using off-the-shelf plagiarism detectors and human annotation. It reveals that memorized content is common and variably correlates with text originality and discourse quality, with significant model-to-model differences. Explicit prompting to avoid memorized content can mitigate memorization, though effects on factuality and coherence are nuanced and domain-dependent. The work highlights important implications for evaluating LLMs, data deduplication, and safe deployment, raising questions about the balance between memorization, learning, and quality text generation.

Abstract

We present an empirical evaluation of various outputs generated by nine of the most widely-available large language models (LLMs). Our analysis is done with off-the-shelf, readily-available tools. We find a correlation between percentage of memorized text, percentage of unique text, and overall output quality, when measured with respect to output pathologies such as counterfactual and logically-flawed statements, and general failures like not staying on topic. Overall, 80.0% of the outputs evaluated contained memorized data, but outputs containing the most memorized content were also more likely to be considered of high quality. We discuss and evaluate mitigation strategies, showing that, in the models evaluated, the rate of memorized text being output is reduced. We conclude with a discussion on potential implications around what it means to learn, to memorize, and to evaluate quality text.

An Evaluation on Large Language Model Outputs: Discourse and Memorization

TL;DR

The paper empirically evaluates how memorization from training data affects outputs of nine large language models across five domains, using off-the-shelf plagiarism detectors and human annotation. It reveals that memorized content is common and variably correlates with text originality and discourse quality, with significant model-to-model differences. Explicit prompting to avoid memorized content can mitigate memorization, though effects on factuality and coherence are nuanced and domain-dependent. The work highlights important implications for evaluating LLMs, data deduplication, and safe deployment, raising questions about the balance between memorization, learning, and quality text generation.

Abstract

We present an empirical evaluation of various outputs generated by nine of the most widely-available large language models (LLMs). Our analysis is done with off-the-shelf, readily-available tools. We find a correlation between percentage of memorized text, percentage of unique text, and overall output quality, when measured with respect to output pathologies such as counterfactual and logically-flawed statements, and general failures like not staying on topic. Overall, 80.0% of the outputs evaluated contained memorized data, but outputs containing the most memorized content were also more likely to be considered of high quality. We discuss and evaluate mitigation strategies, showing that, in the models evaluated, the rate of memorized text being output is reduced. We conclude with a discussion on potential implications around what it means to learn, to memorize, and to evaluate quality text.
Paper Structure (42 sections, 3 figures, 9 tables)

This paper contains 42 sections, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Experiment Description
  3. Findings
  4. Related Work
  5. Methodology
  6. LLMs Evaluated
  7. Services Utilized
  8. Prompting and Querying Guidelines
  9. Text Analysis
  10. Domain Prompts
  11. Evaluation Criteria
  12. Main Results
  13. Variable Correlation
  14. Prompt Length and Memorization
  15. Mitigation Strategies for Memorization
  16. ...and 27 more sections

Figures (3)

  • Figure 1: Discourse quality as a function of the percentage of memorized text. The text output quality was higher when the model presented higher proportions of memorized content (e.g., GPT-3.51). This plot does not account for dialectical errors. The domains evaluated baseline, blog post, long text, scientific papers, and KR, are denoted by $O$, $B$, $L$, $S$, and $K$, respectively.
  • Figure 2: Unique (original) text percentage as a function of the percentage of memorized text. The uniqueness of the text returned dropped as a function of the amount of memorized content flagged by the services. This plot does not account for discourse quality or dialectical errors. The domains evaluated baseline, blog post, long text, scientific papers, and KR, are denoted by $O$, $B$, $L$, $S$, and $K$, respectively.
  • Figure 3: Memorization percentage as a function of the length of the prompt (in characters), per domain. Shorter prompts caused the models to output more memorized content. The general trend of the models to output memorized content with respect to the prompt length repeated itself over all the models evaluated, suggesting that this is a behavior intrinsic to LLMs.