End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training

TL;DR

End-to-End (E2E) training generally outperforms layer-wise approaches, and this paper analyzes why via HSIC-based information-plane dynamics. It shows that E2E promotes layer-role differentiation, enabling distinct information-processing roles across layers and an information-bottleneck representation at the final layer, while middle-layer compression helps generalization. The study combines theoretical framing with experiments on LeNet5 and ResNet architectures, demonstrating that inter-layer cooperation is key to preserving task-relevant information and enabling efficient information propagation. The results suggest that information-bottleneck behavior in the last layer emerges from coordinated layer dynamics, offering insights for designing backpropagation-free training methods and future research directions such as Forward-Forward and HSIC-based regularization approaches.

Abstract

End-to-end (E2E) training, optimizing the entire model through error backpropagation, fundamentally supports the advancements of deep learning. Despite its high performance, E2E training faces the problems of memory consumption, parallel computing, and discrepancy with the functionalities of the actual brain. Various alternative methods have been proposed to overcome these difficulties; however, no one can yet match the performance of E2E training, thereby falling short in practicality. Furthermore, there is no deep understanding regarding differences in the trained model properties beyond the performance gap. In this paper, we reconsider why E2E training demonstrates a superior performance through a comparison with layer-wise training, a non-E2E method that locally sets errors. On the basis of the observation that E2E training has an advantage in propagating input information, we analyze the information plane dynamics of intermediate representations based on the Hilbert-Schmidt independence criterion (HSIC). The results of our normalized HSIC value analysis reveal the E2E training ability to exhibit different information dynamics across layers, in addition to efficient information propagation. Furthermore, we show that this layer-role differentiation leads to the final representation following the information bottleneck principle. It suggests the need to consider the cooperative interactions between layers, not just the final layer when analyzing the information bottleneck of deep learning.

Figures (19)

• Figure 1: Linear probing accuracies of ResNet50 trained on CIFAR10 dataset. Linear separability for test data and training data are presented. Left: E2E training with cross-entropy loss. Right: Layer-wise training with cross-entropy loss. The model was trained with an auxiliary linear classifier for each block.
• Figure 2: Comparison between E2E training and layer-wise training. The models have four layers, and each layer is denoted as $f_1, f_2, f_3, f_4$. Left: E2E training evaluates the loss only after the last representation ${\bm{z}}_4$. The gradient updates are back-propagated to the preceding layers. Middle: Layer-wise training calculates the loss for each layer's outputs and updates weights. Right: Layer-wise training with the auxiliary networks.
• Figure 3: HSIC plane dynamics of LeNet5 model. The color gradation shows the progress of training, i.e., the number of epochs. Inverted triangles with a blue-yellow-based colormap denote layer-wise training, whereas circles with a red-based colormap show E2E training.
• Figure 4: Toy data model. Left: Original data. Class $1$: $r=3.0$ and $\theta \sim \text{von-Mises}(0, 2.0)$. Class $-1$: $r=1.0$ and $\theta \sim \text{von-Mises}(\pi, 2.0)$. A small noise is added to the data. Right: Representation $Z_1$ when ${\bm{W}} = \bigl(\sqrt{1.9}00\sqrt{0.1} \bigr)$. The x-axis is set to the direction of auxiliary classifier ${\bm{v}}$.
• Figure 5: HSIC plane dynamics of the initial layer of Lenet5 models trained on CIFAR10 dataset. The size of auxiliary networks is different; the left shows the results for the linear head (same as figure \ref{['fig:hsic_lenet_cifar10']}), and the right shows the results for the two-layer MLP head.

Theorems & Definitions (15)

• Theorem 1: Informal version of Theorem \ref{['thm:sn_hsic_formal']}
• Definition 1: $\chi_2$-divergence
• Definition 2: chi-squared mutual information
• Theorem 2: gibbs2002choosing, Theorem 5
• Theorem 3: fukumizu2007kernel, Theorem 4
• Proposition 1
• proof
• Definition 3: soft nearest neighbor loss, original form frosst2019analyzing
• Definition 4: soft nearest neighbor loss
• Theorem 4: Formal version of Theorem \ref{['thm:sn_hsic']}.