Auto-Compressing Networks

TL;DR

Auto-Compressing Networks (ACNs) replace short residuals with long skip connections from every layer to the output, enabling an architectural bias toward early-layer information and a phenomenon called auto-compression. Gradient analysis reveals a strong direct-gradient component that concentrates learning in early layers, producing layer-wise training dynamics and potential redundancy in deeper layers. Empirically, ACNs compress 30–80% of top layers while preserving accuracy, and they show improved robustness to noise, better generalization in low-data regimes, superior transfer, and reduced catastrophic forgetting across vision and language models, without specialized tuning. The approach is hardware-friendly and complements pruning and early-exit methods, offering a pathway to self-adapting depth and more efficient deep networks in real-world tasks.

Abstract

Deep neural networks with short residual connections have demonstrated remarkable success across domains, but increasing depth often introduces computational redundancy without corresponding improvements in representation quality. We introduce Auto-Compressing Networks (ACNs), an architectural variant where additive long feedforward connections from each layer to the output replace traditional short residual connections. By analyzing the distinct dynamics induced by this modification, we reveal a unique property we coin as auto-compression, the ability of a network to organically compress information during training with gradient descent, through architectural design alone. Through auto-compression, information is dynamically "pushed" into early layers during training, enhancing their representational quality and revealing potential redundancy in deeper ones. We theoretically show that this property emerges from layer-wise training patterns present in ACNs, where layers are dynamically utilized during training based on task requirements. We also find that ACNs exhibit enhanced noise robustness compared to residual networks, superior performance in low-data settings, improved transfer learning capabilities, and mitigate catastrophic forgetting suggesting that they learn representations that generalize better despite using fewer parameters. Our results demonstrate up to 18% reduction in catastrophic forgetting and 30-80% architectural compression while maintaining accuracy across vision transformers, MLP-mixers, and BERT architectures. These findings establish ACNs as a practical approach to developing efficient neural architectures that automatically adapt their computational footprint to task complexity, while learning robust representations suitable for noisy real-world tasks and continual learning scenarios.

Figures (7)

• Figure 1: (left) ACNs vs Residual Networks gradient flow across layers during training for MLP-Mixer architecture tolstikhin2021mlp on CIFAR-10 krizhevsky2009learning, showcasing implicit layer-wise training and information concentration on the bottom layers for ACNs. (right) ACNs vs Residual Networks incremental performance contribution across layers for ViT architecture dosovitskiy2021an on ImageNet-1K, revealing auto-compression by gradual layer-wise training in ACNs.
• Figure 2: Results for the CIFAR-10 task using the MLM-Mixer base architecture: (left) ACN variants achieve auto-compression needing only a few layers to achieve good performance. (right) The ratio of direct gradient DG to the total gradient FG is significantly higher in early layers for ACNs.
• Figure 3: Performance of intermediate layers of AC vs Residual Vision Transformers trained on Imagenet-1K.
• Figure 4: Performance of the intermediate layers as the number of classes (and examples) in the CIFAR-10 dataset increases from 2, to 5 to 10 classes: (a) Residual Mixer vs (b) AC-mixer (C denotes the number of classes in the subset).
• Figure 5: Downstream performance of AC-BERT vs residual BERT on three GLUE tasks.