Know Your Limits: Entropy Estimation Modeling for Compression and Generalization

TL;DR

This work investigates the fundamental role of informational entropy in language modeling and compression, introducing encoder-augmented causal decoders to enable efficient per token entropy estimation. It demonstrates that entropy estimation models train more efficiently than traditional causal models and can achieve superior scaling, leading to more accurate token level entropy estimates. The authors prove that training toward the dataset entropy yields ideal generalization and provide empirical evidence that entropy informed training improves generalization for causal models. The approach also includes practical techniques such as quantization aware training and second order entropy estimation to enhance robustness and tractability across large scale settings, with potential applicability to other modalities beyond language.

Abstract

Language prediction is constrained by informational entropy intrinsic to language, such that there exists a limit to how accurate any language model can become and equivalently a lower bound to language compression. The most efficient language compression algorithms today are causal (next token prediction) large language models, but the use of these models to form accurate estimates of language entropy is currently computationally infeasible. We introduce encoder-augmented causal decoder model architectures that exhibit superior training efficiency characteristics and achieve higher compression than causal transformers even when trained on modest hardware. We demonstrate how entropy estimates can be obtained on a per-token basis, and show that the generalization of models trained to approach the entropy of their training data necessarily exceeds the generalization of models trained to minimize loss beyond this value. We show empirically that causal models trained to approach but not exceed estimated per-token entropies exhibit greater generalization than models trained without taking entropy into account.

Figures (16)

• Figure 1: Transformer Autoencoders are poorly trainable with repeated but not unrolled embeddings. (a) Embedding unrolling method. (b) Experimental design and transformer autoencoder architecture. (c) Autoencoder training efficiencies on FineWeb-edu (number of layers, model dimension (width), and token context window $n_l=16, d_m=512, n_{ctx}=512$, respectively)
• Figure 2: Compressive autoencoder and causal language model training characteristics on FineWeb. All autoencoders are $d_m=512, n_l=16, n_{ctx}=512$ with a compressed embedding of size $d_e=128$, causal models are compute-matched to the autoencoders.
• Figure 3: Entropy Estimation model embedding introduction methods and training efficiency.
• Figure 4: Entropy estimation model compression scaling with tokens trained. Normalization performed according to Equation \ref{['eq32']}.
• Figure 5: Quantization-aware training via noise injection imparts quantization insensitivity to all layer activations and results in a minimal decrease in training efficiency.

Theorems & Definitions (2)

• Theorem 1
• proof