Information-Theoretic Foundations for Machine Learning

TL;DR

The paper advances an information-theoretic, Bayesian framework for understanding learning by linking predictive performance to mutual information and rate-distortion bounds. It develops a general machinery to bound the irreducible and reducible components of estimation error across iid, sequential, meta-learning, and misspecified settings, with concrete results for linear, logistic, deep, and nonparametric models. By modeling data streams with latent parameters and using rate-distortion theory, it provides explicit bounds that scale with problem dimensions, data size, and model complexity, offering a principled way to anticipate sample efficiency and compute-coverage tradeoffs. The framework illuminates how information flows from data to predictions, informs bounds for minimax and meta-learning scenarios, and connects learning performance to optimal lossy compression, thereby guiding future theory and algorithm design in diverse ML contexts.

Abstract

The progress of machine learning over the past decade is undeniable. In retrospect, it is both remarkable and unsettling that this progress was achievable with little to no rigorous theory to guide experimentation. Despite this fact, practitioners have been able to guide their future experimentation via observations from previous large-scale empirical investigations. In this work, we propose a theoretical framework which attempts to provide rigor to existing practices in machine learning. To the theorist, we provide a framework which is mathematically rigorous and leaves open many interesting ideas for future exploration. To the practitioner, we provide a framework whose results are simple, and provide intuition to guide future investigations across a wide range of learning paradigms. Concretely, we provide a theoretical framework rooted in Bayesian statistics and Shannon's information theory which is general enough to unify the analysis of many phenomena in machine learning. Our framework characterizes the performance of an optimal Bayesian learner as it learns from a stream of experience. Unlike existing analyses that weaken with increasing data complexity, our theoretical tools provide accurate insights across diverse machine learning settings. Throughout this work, we derive theoretical results and demonstrate their generality by apply them to derive insights specific to settings. These settings range from learning from data which is independently and identically distributed under an unknown distribution, to data which is sequential, to data which exhibits hierarchical structure amenable to meta-learning, and finally to data which is not fully explainable under the learner's beliefs (misspecification). These results are particularly relevant as we strive to understand and overcome increasingly difficult machine learning challenges in this endlessly complex world.

Figures (10)

• Figure 1: The English alphabet consists of characters of varying frequency across common text corpuses. Notably, the vowels appear with greater frequency. A coding scheme ought to map these more frequently appearing characters to shorter strings.
• Figure 2: This venn diagram illustrates the relationships between the introduced information-theoretic quantities.
• Figure 3: We depict our linear regression data generating process above. It consists of an input vector $X$ of dimension $d$, an unknown weight vector $\theta$ of dimension $d$, and a final output $Y$ which is the sum of $\theta^\top X$ and independent Gaussian noise $W$.
• Figure 4: We depict our logistic regression data generating process above. It consists of an input vector $X$ of dimension $d$, an unknown weight vector $\theta$ of dimension $d$, and a final binary output $Y\in\{0,1\}$. $Y$ is sampled according to probabilities generated by the sigmoid function applied to $\theta^\top X$.
• Figure 5: We depict our deep neural network data generating process above. It consists of input dimension $d$, width $N$, depth $L$, with output dimension $1$, and ReLU activation units. We denote the weights at layer $\ell$ by $A^{(\ell)}$ and the output of layer $\ell$ by $U^{(\ell)}$. We assume that the final output $Y$ is the sum of the final network output $U^{(L)}$ and independent Gaussian noise $Z$.

Theorems & Definitions (152)

• Lemma 1
• proof
• Definition 2
• Theorem 3
• Lemma 4
• proof
• Lemma 5
• proof
• Lemma 6
• proof