Understanding Encoder-Decoder Structures in Machine Learning Using Information Measures

TL;DR

This work develops an information-theoretic lens for understanding encoder-decoder structures in ML, centering on information sufficiency (IS) and mutual-information loss (MIL). It proves a functional characterization of IS-structured predictive models via $Y=f(W,\eta(X))$ and shows that mismatch to IS incurs a cross-entropy penalty quantified by $I(X;Y|U)$. An information-projection view links encoder expressiveness to KL divergence from the true model, with per-layer losses in deep nets and conditions for strong cross-entropy consistency. The study also connects IS with digital encoders, the information bottleneck, invariance, and robustness, and provides controlled empirical evidence using MLPs that IS priors improve learning efficiency, particularly in low-data regimes. Overall, the framework offers a principled way to design encoder-decoder architectures by aligning them with the predictive structure of the task and the information they preserve.

Abstract

We present new results to model and understand the role of encoder-decoder design in machine learning (ML) from an information-theoretic angle. We use two main information concepts, information sufficiency (IS) and mutual information loss (MIL), to represent predictive structures in machine learning. Our first main result provides a functional expression that characterizes the class of probabilistic models consistent with an IS encoder-decoder latent predictive structure. This result formally justifies the encoder-decoder forward stages many modern ML architectures adopt to learn latent (compressed) representations for classification. To illustrate IS as a realistic and relevant model assumption, we revisit some known ML concepts and present some interesting new examples: invariant, robust, sparse, and digital models. Furthermore, our IS characterization allows us to tackle the fundamental question of how much performance (predictive expressiveness) could be lost, using the cross entropy risk, when a given encoder-decoder architecture is adopted in a learning setting. Here, our second main result shows that a mutual information loss quantifies the lack of expressiveness attributed to the choice of a (biased) encoder-decoder ML design. Finally, we address the problem of universal cross-entropy learning with an encoder-decoder design where necessary and sufficiency conditions are established to meet this requirement. In all these results, Shannon's information measures offer new interpretations and explanations for representation learning.

Figures (4)

• Figure 1: The encoder-decoder structure of $\mu_{Y|X}(\cdot|\cdot)$ when $\mu_{X,Y} \in \mathcal{P}_{\eta}(\mathcal{X}\times \mathcal{Y})$ (see Def.\ref{['def_IS_redundant_models']}).
• Figure 2: Cross-entropy losses curves per-epoch for different MLP schemes, models and training data-lengths ($n$). The curves for $\text{MLP}32$, $\text{MLP}256$ and $\text{MLP}1024$ are presented in the first, second and third rows, respectively. The horizontal lines present the cross-entropy lower bounds of Theorem \ref{['th_main_IS_mismatch']} where $H(Y|X)=0.303532$, $H(Y|\tilde{X})=0.952762$ and $H(Y|\bar{X})=H(Y)=1.485475$. In the right caption, $\eta_5(\cdot)$ is a short-hand for $\eta_{1,2,3,4,5}(\cdot)$.
• Figure 3: Transform Sparse Neural Network (TS-NN).
• Figure 4: Visualization of i.i.d. realizations for the four model examples ($\mu_{X,Y}$) in Appendix \ref{['sec_model_examples']}. The color indicates the label identity of the sample ($y$). The red lines in (\ref{['sfig:singular-2d']}) and (\ref{['sfig:demo-2d']}) represent the boundaries of the 2D cells ($A_{\mathbf{i}}$).

Theorems & Definitions (15)

• proof
• proof
• proof
• proof
• proof
• proof
• proof
• proof
• proof
• proof