Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Entropy in Large Language Models

Marco Scharringhausen

TL;DR

This study compares the source entropy of the output of large language models (LLM) to that of natural language (written or spoken) as represented by the Open American National Corpus (OANC).

Abstract

In this study, the output of large language models (LLM) is considered an information source generating an unlimited sequence of symbols drawn from a finite alphabet. Given the probabilistic nature of modern LLMs, we assume a probabilistic model for these LLMs, following a constant random distribution and the source itself thus being stationary. We compare this source entropy (per word) to that of natural language (written or spoken) as represented by the Open American National Corpus (OANC). Our results indicate that the word entropy of such LLMs is lower than the word entropy of natural speech both in written or spoken form. The long-term goal of such studies is to formalize the intuitions of information and uncertainty in large language training to assess the impact of training an LLM from LLM generated training data. This refers to texts from the world wide web in particular.

Entropy in Large Language Models

TL;DR

This study compares the source entropy of the output of large language models (LLM) to that of natural language (written or spoken) as represented by the Open American National Corpus (OANC).

Abstract

In this study, the output of large language models (LLM) is considered an information source generating an unlimited sequence of symbols drawn from a finite alphabet. Given the probabilistic nature of modern LLMs, we assume a probabilistic model for these LLMs, following a constant random distribution and the source itself thus being stationary. We compare this source entropy (per word) to that of natural language (written or spoken) as represented by the Open American National Corpus (OANC). Our results indicate that the word entropy of such LLMs is lower than the word entropy of natural speech both in written or spoken form. The long-term goal of such studies is to formalize the intuitions of information and uncertainty in large language training to assess the impact of training an LLM from LLM generated training data. This refers to texts from the world wide web in particular.
Paper Structure (8 sections, 3 equations, 2 figures, 3 tables)

This paper contains 8 sections, 3 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. LLMs in This Study
  4. Generation of LLM output
  5. Open American National Corpus
  6. Source entropy
  7. Results
  8. Conclusion

Figures (2)

  • Figure 1: Entropy rates for the OANC and the output of all LLMs (both Mistral and Blablador model family concatenated) for $T=0.3\dots1.5$ as well as for $T=0.3\dots1.0$. The difference between $T_{\rm{max}}=1.0$ and $T_{\rm{max}}=1.5$ is small (0.618 and 0.574, respectively), probably since the amount of LLM output with somewhat erroneous entropy (see \ref{['fig:fig2']}) is small. See table \ref{['tab:sourceentropy']} for numeric values of the entropy rates.
  • Figure 2: Entropy rates for the two model families Mistral and Blablador, distinguished by model temperature. The Blablador output for $T=1.5$ leads to very little output (140.000 words). This leads to an estimate of the entropy rate that is most likely wrong, see \ref{['sec:sourcentropy']} and the comment about sample sizes therein. The impact of model temperature on the entropy rate of the output is small. The values for the Mistral model family ($0.450\dots0.475$) are lower than for the Blablador family ($0.530\dots0.587$). It is not clear, however, wether this is a significant effect.