Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Skip Connection: Pooling and Unpooling Design for Elimination Singularities

Chengkun Sun, Jinqian Pan, Zhuoli Jin, Russell Stevens Terry, Jiang Bian, Jie Xu

TL;DR

Pool Skip addresses elimination singularities that hamper training of deep CNNs by integrating a pooling-unpooling path with a small convolution and skip connections, underpinned by the Weight Inertia hypothesis and compensation theory. The approach stabilizes gradient flow and enables updates to previously inert weights, yielding performance gains across 2D classification, 2D segmentation, and 3D medical image segmentation tasks. Empirical results on CIFAR-10/100, Cityscapes, Pascal VOC, BTCV, and AMOS show consistent improvements across CNNs and, to a lesser extent, ViT variants, validating the practicality of pooling-based compensation in deep architectures. The work provides a theoretical framework and a lightweight architectural module that improves training robustness with potential broad applicability to other deep models.

Abstract

Training deep Convolutional Neural Networks (CNNs) presents unique challenges, including the pervasive issue of elimination singularities, consistent deactivation of nodes leading to degenerate manifolds within the loss landscape. These singularities impede efficient learning by disrupting feature propagation. To mitigate this, we introduce Pool Skip, an architectural enhancement that strategically combines a Max Pooling, a Max Unpooling, a 3 times 3 convolution, and a skip connection. This configuration helps stabilize the training process and maintain feature integrity across layers. We also propose the Weight Inertia hypothesis, which underpins the development of Pool Skip, providing theoretical insights into mitigating degradation caused by elimination singularities through dimensional and affine compensation. We evaluate our method on a variety of benchmarks, focusing on both 2D natural and 3D medical imaging applications, including tasks such as classification and segmentation. Our findings highlight Pool Skip's effectiveness in facilitating more robust CNN training and improving model performance.

Beyond Skip Connection: Pooling and Unpooling Design for Elimination Singularities

TL;DR

Pool Skip addresses elimination singularities that hamper training of deep CNNs by integrating a pooling-unpooling path with a small convolution and skip connections, underpinned by the Weight Inertia hypothesis and compensation theory. The approach stabilizes gradient flow and enables updates to previously inert weights, yielding performance gains across 2D classification, 2D segmentation, and 3D medical image segmentation tasks. Empirical results on CIFAR-10/100, Cityscapes, Pascal VOC, BTCV, and AMOS show consistent improvements across CNNs and, to a lesser extent, ViT variants, validating the practicality of pooling-based compensation in deep architectures. The work provides a theoretical framework and a lightweight architectural module that improves training robustness with potential broad applicability to other deep models.

Abstract

Training deep Convolutional Neural Networks (CNNs) presents unique challenges, including the pervasive issue of elimination singularities, consistent deactivation of nodes leading to degenerate manifolds within the loss landscape. These singularities impede efficient learning by disrupting feature propagation. To mitigate this, we introduce Pool Skip, an architectural enhancement that strategically combines a Max Pooling, a Max Unpooling, a 3 times 3 convolution, and a skip connection. This configuration helps stabilize the training process and maintain feature integrity across layers. We also propose the Weight Inertia hypothesis, which underpins the development of Pool Skip, providing theoretical insights into mitigating degradation caused by elimination singularities through dimensional and affine compensation. We evaluate our method on a variety of benchmarks, focusing on both 2D natural and 3D medical imaging applications, including tasks such as classification and segmentation. Our findings highlight Pool Skip's effectiveness in facilitating more robust CNN training and improving model performance.
Paper Structure (26 sections, 8 equations, 4 figures, 5 tables)

This paper contains 26 sections, 8 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Pooling Operations in CNNs
  4. Skip Connection and Batch Normalization
  5. Pool Skip Mechanism
  6. Weight Inertia Hypothesis
  7. Affine and Dimensional Compensation
  8. Experiments
  9. Image Classification
  10. Datasets
  11. Comparison Methods
  12. Experimental Results
  13. Natural Image Segmentation
  14. Datasets
  15. Comparison Methods
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Schematic representation of the computational process of Pool Skip.
  • Figure 2: An simple example of dimensional and affine and compensation.
  • Figure 3: The Top-1 error rates of deep ResNet on CIFAR10 and CIFAR100 datasets. The pool kernel size is 4 for CIFAR100 experiments and 2 for CIFAR10 experiments.
  • Figure 4: $\frac{l_2}{l_1}$ value quantitative comparison in ResNet350 and ResNet420 on CIFAR10 and CIFAR100 Datasets. The $\frac{l_2}{l_1}$ values were computed based on the output sequence of the network, with and without the incorporation of the Pool Skip. The plot highlights a moderate alleviation of the network degradation issue in shallow layers upon the integration of Pool Skip. Note: The horizontal axis represents the layers of the network along the output direction, from left to right. The "Pool_Skip_S4" means the size of Pool operation kernel is 4, "Pool_Skip_S4" does 2.