Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical Residuals Exploit Brain-Inspired Compositionality

Francisco M. López, Jochen Triesch

TL;DR

The paper addresses the challenge of learning hierarchical, compositional representations in deep networks by introducing brain-inspired long-range residual connections. It presents Hierarchical Residual Networks (HiResNets), which couple blocks across levels via hierarchical projections formed by lightweight, compressive pathways. Empirical results on CIFAR-10, CIFAR-100, and TinyImageNet show HiResNets achieve higher accuracy and faster learning than ResNets and ResNeXt, with ablations highlighting the strongest benefit from output-side residuals. The work suggests that hierarchical skip connections are a parameter-efficient way to expand model capacity and motivates applying these ideas to deeper architectures and transformer-based models to better emulate hierarchical brain processing.

Abstract

We present Hierarchical Residual Networks (HiResNets), deep convolutional neural networks with long-range residual connections between layers at different hierarchical levels. HiResNets draw inspiration on the organization of the mammalian brain by replicating the direct connections from subcortical areas to the entire cortical hierarchy. We show that the inclusion of hierarchical residuals in several architectures, including ResNets, results in a boost in accuracy and faster learning. A detailed analysis of our models reveals that they perform hierarchical compositionality by learning feature maps relative to the compressed representations provided by the skip connections.

Hierarchical Residuals Exploit Brain-Inspired Compositionality

TL;DR

The paper addresses the challenge of learning hierarchical, compositional representations in deep networks by introducing brain-inspired long-range residual connections. It presents Hierarchical Residual Networks (HiResNets), which couple blocks across levels via hierarchical projections formed by lightweight, compressive pathways. Empirical results on CIFAR-10, CIFAR-100, and TinyImageNet show HiResNets achieve higher accuracy and faster learning than ResNets and ResNeXt, with ablations highlighting the strongest benefit from output-side residuals. The work suggests that hierarchical skip connections are a parameter-efficient way to expand model capacity and motivates applying these ideas to deeper architectures and transformer-based models to better emulate hierarchical brain processing.

Abstract

We present Hierarchical Residual Networks (HiResNets), deep convolutional neural networks with long-range residual connections between layers at different hierarchical levels. HiResNets draw inspiration on the organization of the mammalian brain by replicating the direct connections from subcortical areas to the entire cortical hierarchy. We show that the inclusion of hierarchical residuals in several architectures, including ResNets, results in a boost in accuracy and faster learning. A detailed analysis of our models reveals that they perform hierarchical compositionality by learning feature maps relative to the compressed representations provided by the skip connections.

Paper Structure

This paper contains 7 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Hierarchical Residual Networks
  3. Experiments
  4. HiResNets outperform baseline models
  5. Feature maps reveal hierarchical compositionality
  6. Output hierarchical residuals are most beneficial
  7. Conclusion

Figures (2)

  • Figure 1: HiResNet architecture based on the ResNet-18 with 3 residual blocks. The hierarchical residual projections consist of an average pooling to match the height and width, a $1\times 1$ convolution to match the number of channels, and batch normalization.
  • Figure 2: Accuracy comparisons on ResNets and ablated versions of HiResNets.