Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision Transformers with Hierarchical Attention

Yun Liu, Yu-Huan Wu, Guolei Sun, Le Zhang, Ajad Chhatkuli, Luc Van Gool

TL;DR

This paper proposes hierarchical MHSA (H-MHSA), a novel approach that computes sell-attention in a hierarchical fashion, and builds a family of hierarchical-attention-based transformer networks, namely HAT-Net, which provides a new perspective for vision transformers.

Abstract

This paper tackles the high computational/space complexity associated with Multi-Head Self-Attention (MHSA) in vanilla vision transformers. To this end, we propose Hierarchical MHSA (H-MHSA), a novel approach that computes self-attention in a hierarchical fashion. Specifically, we first divide the input image into patches as commonly done, and each patch is viewed as a token. Then, the proposed H-MHSA learns token relationships within local patches, serving as local relationship modeling. Then, the small patches are merged into larger ones, and H-MHSA models the global dependencies for the small number of the merged tokens. At last, the local and global attentive features are aggregated to obtain features with powerful representation capacity. Since we only calculate attention for a limited number of tokens at each step, the computational load is reduced dramatically. Hence, H-MHSA can efficiently model global relationships among tokens without sacrificing fine-grained information. With the H-MHSA module incorporated, we build a family of Hierarchical-Attention-based Transformer Networks, namely HAT-Net. To demonstrate the superiority of HAT-Net in scene understanding, we conduct extensive experiments on fundamental vision tasks, including image classification, semantic segmentation, object detection, and instance segmentation. Therefore, HAT-Net provides a new perspective for vision transformers. Code and pretrained models are available at https://github.com/yun-liu/HAT-Net.

Vision Transformers with Hierarchical Attention

TL;DR

This paper proposes hierarchical MHSA (H-MHSA), a novel approach that computes sell-attention in a hierarchical fashion, and builds a family of hierarchical-attention-based transformer networks, namely HAT-Net, which provides a new perspective for vision transformers.

Abstract

This paper tackles the high computational/space complexity associated with Multi-Head Self-Attention (MHSA) in vanilla vision transformers. To this end, we propose Hierarchical MHSA (H-MHSA), a novel approach that computes self-attention in a hierarchical fashion. Specifically, we first divide the input image into patches as commonly done, and each patch is viewed as a token. Then, the proposed H-MHSA learns token relationships within local patches, serving as local relationship modeling. Then, the small patches are merged into larger ones, and H-MHSA models the global dependencies for the small number of the merged tokens. At last, the local and global attentive features are aggregated to obtain features with powerful representation capacity. Since we only calculate attention for a limited number of tokens at each step, the computational load is reduced dramatically. Hence, H-MHSA can efficiently model global relationships among tokens without sacrificing fine-grained information. With the H-MHSA module incorporated, we build a family of Hierarchical-Attention-based Transformer Networks, namely HAT-Net. To demonstrate the superiority of HAT-Net in scene understanding, we conduct extensive experiments on fundamental vision tasks, including image classification, semantic segmentation, object detection, and instance segmentation. Therefore, HAT-Net provides a new perspective for vision transformers. Code and pretrained models are available at https://github.com/yun-liu/HAT-Net.

Paper Structure

This paper contains 12 sections, 15 equations, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Review of Vision Transformers
  5. Hierarchical Multi-Head Self-Attention
  6. Network Architecture
  7. Experiments
  8. Image Classification
  9. Semantic Segmentation
  10. Object Detection and Instance Segmentation
  11. Ablation Studies
  12. Conclusion

Figures (1)

  • Figure 1: Illustration of the proposed HAT-Net. GAP: global average pooling; FC: fully-connected layer. $\times L_i$ means that the transformer block is repeated for $L_i$ times. $H$ and $W$ denote the height and width of the input image, respectively.