Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Memory Efficient Transformer Adapter for Dense Predictions

Dong Zhang, Rui Yan, Pingcheng Dong, Kwang-Ting Cheng

TL;DR

This work tackles memory-access bottlenecks in Vision Transformer adapters used for dense predictions. It introduces META, a memory-efficient transformer adapter, built around the MEA block that shares layer normalization between self-attention and FFN, employs cross-shaped self-attention to reduce reshaping, adds a lightweight Conv branch for local bias, and uses a cascaded injector/extractor framework to diversify head features. Empirically, META achieves new state-of-the-art accuracy-efficiency trade-offs on object detection, instance segmentation, and semantic segmentation across MS-COCO and ADE20K, while lowering memory consumption and maintaining or increasing predictive quality. The authors also provide theoretical analysis suggesting improved generalization and adaptability due to increased information entropy from the fused ViT and convolutional representations, supporting META's practical impact for fast, scalable dense prediction with pre-trained ViT backbones.

Abstract

While current Vision Transformer (ViT) adapter methods have shown promising accuracy, their inference speed is implicitly hindered by inefficient memory access operations, e.g., standard normalization and frequent reshaping. In this work, we propose META, a simple and fast ViT adapter that can improve the model's memory efficiency and decrease memory time consumption by reducing the inefficient memory access operations. Our method features a memory-efficient adapter block that enables the common sharing of layer normalization between the self-attention and feed-forward network layers, thereby reducing the model's reliance on normalization operations. Within the proposed block, the cross-shaped self-attention is employed to reduce the model's frequent reshaping operations. Moreover, we augment the adapter block with a lightweight convolutional branch that can enhance local inductive biases, particularly beneficial for the dense prediction tasks, e.g., object detection, instance segmentation, and semantic segmentation. The adapter block is finally formulated in a cascaded manner to compute diverse head features, thereby enriching the variety of feature representations. Empirically, extensive evaluations on multiple representative datasets validate that META substantially enhances the predicted quality, while achieving a new state-of-the-art accuracy-efficiency trade-off. Theoretically, we demonstrate that META exhibits superior generalization capability and stronger adaptability.

Memory Efficient Transformer Adapter for Dense Predictions

TL;DR

This work tackles memory-access bottlenecks in Vision Transformer adapters used for dense predictions. It introduces META, a memory-efficient transformer adapter, built around the MEA block that shares layer normalization between self-attention and FFN, employs cross-shaped self-attention to reduce reshaping, adds a lightweight Conv branch for local bias, and uses a cascaded injector/extractor framework to diversify head features. Empirically, META achieves new state-of-the-art accuracy-efficiency trade-offs on object detection, instance segmentation, and semantic segmentation across MS-COCO and ADE20K, while lowering memory consumption and maintaining or increasing predictive quality. The authors also provide theoretical analysis suggesting improved generalization and adaptability due to increased information entropy from the fused ViT and convolutional representations, supporting META's practical impact for fast, scalable dense prediction with pre-trained ViT backbones.

Abstract

While current Vision Transformer (ViT) adapter methods have shown promising accuracy, their inference speed is implicitly hindered by inefficient memory access operations, e.g., standard normalization and frequent reshaping. In this work, we propose META, a simple and fast ViT adapter that can improve the model's memory efficiency and decrease memory time consumption by reducing the inefficient memory access operations. Our method features a memory-efficient adapter block that enables the common sharing of layer normalization between the self-attention and feed-forward network layers, thereby reducing the model's reliance on normalization operations. Within the proposed block, the cross-shaped self-attention is employed to reduce the model's frequent reshaping operations. Moreover, we augment the adapter block with a lightweight convolutional branch that can enhance local inductive biases, particularly beneficial for the dense prediction tasks, e.g., object detection, instance segmentation, and semantic segmentation. The adapter block is finally formulated in a cascaded manner to compute diverse head features, thereby enriching the variety of feature representations. Empirically, extensive evaluations on multiple representative datasets validate that META substantially enhances the predicted quality, while achieving a new state-of-the-art accuracy-efficiency trade-off. Theoretically, we demonstrate that META exhibits superior generalization capability and stronger adaptability.

Paper Structure

This paper contains 21 sections, 1 theorem, 8 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Memory Efficient Transformer Adapter (META)
  4. Overall Architecture and Configurations
  5. MEA Block
  6. Cascaded MEA Injector
  7. Cascaded MEA Extractor
  8. Experiments
  9. Datasets and Evaluation Metrics
  10. Experiments on Object Detection (ODet) and Instance Segmentation (ISeg)
  11. Experiments on Semantic Segmentation (SSeg)
  12. Ablation Analysis
  13. Conclusion
  14. Theoretical Analysis of META
  15. Introduction of the Experimental Datasets
  16. ...and 6 more sections

Key Result

Lemma 6.1

In any case of mutual information, the MEA block will gain larger information entropy after fusing $\textbf{X}_{vit}$ and $\textbf{X}_{con}$.

Figures (4)

  • Figure 1: Qualitative performance comparisons of different models with respect to training parameters, application gaps, memory access costs, and inference time costs.
  • Figure 2: Overall architecture of META. Our primary contribution is the proposal of a MEA block in (a), which serves as the fundamental component for the injector in (b) and extractor in (c).
  • Figure 3: Class activation map comparisons of instance segmentation before and after adding the Conv branch. The sample images are from the training set of MS-COCO caesar2018coco.
  • Figure 4: Qualitative results. The sample images are from the val set of MS-COCO caesar2018coco for instance segmentation, and are from the val set of ADE20K zhou2017scene for semantic segmentation. "w/o Conv" denotes that the Conv branch is not used in the experiments. "20K" and "MM" refers to the backbone network being pre-trained on ImageNet-22k steiner2021train and Multi-Modal zhu2022uni, respectively.

Theorems & Definitions (2)

  • Lemma 6.1
  • proof