From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence

TL;DR

This work reframes information through a computationally bounded lens, defining epiplexity as the structural information extractable by a bounded observer and time-bounded entropy as remaining random content. By decomposing data into $S_T(X)$ and $H_T(X)$ and proposing practical estimation via prequential and requential coding, the authors explain how information can be created by computation, why data ordering and factorization matter, and how likelihood-based modeling can reveal learnable structure beyond the data-generating process. They demonstrate that epiplexity correlates with downstream generalization, especially OOD transfer, and provide theoretical results and data-driven procedures that support data selection and curriculum design for robust learning. The framework reconciles classical information theory with modern ML practice, offering a compute-aware foundation for data selection, synthetic data use, and emergent phenomena, while highlighting limitations and avenues for future theory and practice. Overall, epiplexity provides a principled, observer-aware metric to quantify and optimize the informative value of data under realistic compute budgets, with practical implications for pre-training, data curation, and transfer learning.

Abstract

Can we learn more from data than existed in the generating process itself? Can new and useful information be constructed from merely applying deterministic transformations to existing data? Can the learnable content in data be evaluated without considering a downstream task? On these questions, Shannon information and Kolmogorov complexity come up nearly empty-handed, in part because they assume observers with unlimited computational capacity and fail to target the useful information content. In this work, we identify and exemplify three seeming paradoxes in information theory: (1) information cannot be increased by deterministic transformations; (2) information is independent of the order of data; (3) likelihood modeling is merely distribution matching. To shed light on the tension between these results and modern practice, and to quantify the value of data, we introduce epiplexity, a formalization of information capturing what computationally bounded observers can learn from data. Epiplexity captures the structural content in data while excluding time-bounded entropy, the random unpredictable content exemplified by pseudorandom number generators and chaotic dynamical systems. With these concepts, we demonstrate how information can be created with computation, how it depends on the ordering of the data, and how likelihood modeling can produce more complex programs than present in the data generating process itself. We also present practical procedures to estimate epiplexity which we show capture differences across data sources, track with downstream performance, and highlight dataset interventions that improve out-of-distribution generalization. In contrast to principles of model selection, epiplexity provides a theoretical foundation for data selection, guiding how to select, generate, or transform data for learning systems.

Figures (20)

• Figure 1: Illustration of random vs structural information. (Left) Illustration of random vs structural information of different data for computationally-bounded observers, which we formalize with time-bounded entropy and epiplexity (\ref{['sec:epiplexity']}) and can be estimated from loss curves of neural networks trained on that data (\ref{['sec:measuring']}). (Top Right) Unlike other forms of information, time-bounded entropy and epiplexity can be increased through computational processes, such as simulating dynamical systems (cellular automation, Lorenz equations) and interventions like changing the data ordering, which can produce apparent randomness but also learnable, emergent structures like gliders and the Lorenz attractor invariant measure (\ref{['sec:paradox']}). (Bottom Right) Whereas time-bounded entropy captures the in-distribution randomness and unpredictability, epiplexity measures the amount of structural information the model extracts from the data to its weights, which can be useful for OOD tasks such as by reusing learned circuits shared between the in-distribution and OOD tasks.
• Figure 2: Estimate information in model
• Figure 3: Compute-optimal 2-part code
• Figure 4: Requential vs Prequential
• Figure 6: Information created with cellular automata. (Left) Example rollouts from random initial conditions of the class II rule 15, class III rule 30, and class IV rule 54. Time flows from up to down. (Right) Measuring epiplexity on data produced by these transformations, we see that rule 15 produces little information (low $\mathrm{H}_T$, low $\mathrm{S}_T)$, rule 30 produces lots of unpredictable random information (high $\mathrm{H}_T$, low $\mathrm{S}_T$), and rule 54 produces both random and structural information (medium $\mathrm{H}_T$, high $\mathrm{S}_T$). These observations are reflected in the training loss curve of LLMs, which saturates quickly for rule 15, makes no progress for rule 30, and makes continued progress with compute for rule 54.

Theorems & Definitions (33)

• Definition 1: Prefix Kolmogorov complexity Kolmogorov01011968chaitin1975theory
• Definition 2: Martin--Löf random sequence martin1966definition
• Definition 3: Non-uniform CSPRNG blum1984generategoldreich2001foundations1
• Definition 4: Non-uniform one-way function, OWF Yao1982Trapdoorgoldreich2001foundations1
• Definition 5: Naive Sophistication mota2013sophistication
• Definition 6: Two-part MDL rissanen2004minimumgrunwald2007minimum
• Definition 7: Time-bounded probabilistic model
• Definition 8: Epiplexity and Time-Bounded Entropy
• Theorem 9
• Theorem 10