Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Physics in Next-token Prediction

Hongjun An, Yiliang Song, Xuelong Li

TL;DR

The law of information conservation within NTP is identified and the First Law of Information Capacity (IC-1) is proposed, demonstrating that the essence of intelligence emergence in auto-regressive models is fundamentally a process of information transfer.

Abstract

We discovered the underlying physics in Next-token Prediction (NTP). We identified the law of information conservation within NTP and proposed the First Law of Information Capacity (IC-1), demonstrating that the essence of intelligence emergence in auto-regressive models is fundamentally a process of information transfer. We also introduced Landauer's Principle into NTP, formulating the Second Law of Information Capacity (IC-2), which establishes the relationship between auto-regressive model training and energy consumption. Additionally, we presented several corollaries, which hold practical significance for production practices. Finally, we demonstrate the consistency between the Law of Information Capacity and the Scaling Law for Neural Language Models, the Knowledge Capacity Scaling Laws, and the Scaling Laws for Precision.

Physics in Next-token Prediction

TL;DR

The law of information conservation within NTP is identified and the First Law of Information Capacity (IC-1) is proposed, demonstrating that the essence of intelligence emergence in auto-regressive models is fundamentally a process of information transfer.

Abstract

We discovered the underlying physics in Next-token Prediction (NTP). We identified the law of information conservation within NTP and proposed the First Law of Information Capacity (IC-1), demonstrating that the essence of intelligence emergence in auto-regressive models is fundamentally a process of information transfer. We also introduced Landauer's Principle into NTP, formulating the Second Law of Information Capacity (IC-2), which establishes the relationship between auto-regressive model training and energy consumption. Additionally, we presented several corollaries, which hold practical significance for production practices. Finally, we demonstrate the consistency between the Law of Information Capacity and the Scaling Law for Neural Language Models, the Knowledge Capacity Scaling Laws, and the Scaling Laws for Precision.

Paper Structure

This paper contains 26 sections, 1 theorem, 20 equations, 2 figures.

Table of Contents

  1. Introduction
  2. The First Law of Information Capability: the Conservation of Information in NTP
  3. Preliminaries
  4. Baseline: Non-intelligent Transmission
  5. Intelligent Transmission based on Compression
  6. A Derivation of the First Law of Information Capability
  7. A Dynamic Perspective to the Process of Intelligence Emergence
  8. Initial State.
  9. Training State.
  10. Terminal State.
  11. The Second Law of Information Capacity: the Energy Relationship in NTP
  12. Corollaries to Guide Practice
  13. Estimating the Entropy of the Dataset
  14. Estimating the Quality of the Dataset
  15. Matching Suitable Model Size with Dataset Size
  16. ...and 11 more sections

Key Result

Corollary 4.1

The entropy of the dataset is approximately equal to the initial loss of the model training. Proof.

Figures (2)

  • Figure 1: We propose the First Law of Information Capacity (IC-1) and the Second Law of Information Capacity (IC-2), reveals the principle of information conservation and the energy relationship within NTP.
  • Figure 2: The plotted curve of $f(x)=x^{0.095}-x^{0.076}$. When $0 < x \ll 10^{15}$, $0 < f(x) \ll 10$.

Theorems & Definitions (1)

  • Corollary 4.1