Estimating Information Flow in Deep Neural Networks
Ziv Goldfeld, Ewout van den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, Yury Polyanskiy
TL;DR
This work challenges the traditional Information Bottleneck interpretation of deep learning by highlighting that mutual information I(X;T) is ill-posed for deterministic networks. It proposes a noisy DNN framework with Gaussian layerwise noise to render I(X;T) meaningful, and develops a rigorous, forward-pass sampling estimator based on differential entropy of Gaussian mixtures. The authors show that observed compression during training arises from progressive clustering of inputs from the same class in hidden representations, not from saturation alone, and that binning-based MI in deterministic networks mostly tracks this clustering. Across SZT models and a MNIST CNN, clustering emerges as the core geometric phenomenon guiding representation learning, while regularization that limits clustering can mitigate compression and influence generalization. The results advocate shifting focus from MI values to clustering dynamics as a more informative and actionable lens for understanding and regularizing deep representations.
Abstract
We study the flow of information and the evolution of internal representations during deep neural network (DNN) training, aiming to demystify the compression aspect of the information bottleneck theory. The theory suggests that DNN training comprises a rapid fitting phase followed by a slower compression phase, in which the mutual information $I(X;T)$ between the input $X$ and internal representations $T$ decreases. Several papers observe compression of estimated mutual information on different DNN models, but the true $I(X;T)$ over these networks is provably either constant (discrete $X$) or infinite (continuous $X$). This work explains the discrepancy between theory and experiments, and clarifies what was actually measured by these past works. To this end, we introduce an auxiliary (noisy) DNN framework for which $I(X;T)$ is a meaningful quantity that depends on the network's parameters. This noisy framework is shown to be a good proxy for the original (deterministic) DNN both in terms of performance and the learned representations. We then develop a rigorous estimator for $I(X;T)$ in noisy DNNs and observe compression in various models. By relating $I(X;T)$ in the noisy DNN to an information-theoretic communication problem, we show that compression is driven by the progressive clustering of hidden representations of inputs from the same class. Several methods to directly monitor clustering of hidden representations, both in noisy and deterministic DNNs, are used to show that meaningful clusters form in the $T$ space. Finally, we return to the estimator of $I(X;T)$ employed in past works, and demonstrate that while it fails to capture the true (vacuous) mutual information, it does serve as a measure for clustering. This clarifies the past observations of compression and isolates the geometric clustering of hidden representations as the true phenomenon of interest.