Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HIRI-ViT: Scaling Vision Transformer with High Resolution Inputs

Ting Yao, Yehao Li, Yingwei Pan, Tao Mei

TL;DR

HIRI-ViT introduces a principled five-stage CNN+ViT backbone designed for high-resolution inputs by decomposing CNN operations into two parallel branches that process high- and low-resolution features. The architecture features a High Resolution Stem and High Resolution Blocks, along with Inverted Residual Downsampling, BN/LN normalization choices, and a bi-directional EMA distillation strategy to stabilize training. Empirical results across ImageNet-1K, COCO, and ADE20K demonstrate state-of-the-art accuracy within practical computational budgets, notably achieving 84.3% Top-1 on ImageNet-1K at 448×448 with around 5 GFLOPs for the S variant and 85.7% for larger configurations at similar resolutions. The design generalizes to other backbones, underscoring the versatility of the two-branch, high-resolution scaling approach for CNN+ViT hybrids.

Abstract

The hybrid deep models of Vision Transformer (ViT) and Convolution Neural Network (CNN) have emerged as a powerful class of backbones for vision tasks. Scaling up the input resolution of such hybrid backbones naturally strengthes model capacity, but inevitably suffers from heavy computational cost that scales quadratically. Instead, we present a new hybrid backbone with HIgh-Resolution Inputs (namely HIRI-ViT), that upgrades prevalent four-stage ViT to five-stage ViT tailored for high-resolution inputs. HIRI-ViT is built upon the seminal idea of decomposing the typical CNN operations into two parallel CNN branches in a cost-efficient manner. One high-resolution branch directly takes primary high-resolution features as inputs, but uses less convolution operations. The other low-resolution branch first performs down-sampling and then utilizes more convolution operations over such low-resolution features. Experiments on both recognition task (ImageNet-1K dataset) and dense prediction tasks (COCO and ADE20K datasets) demonstrate the superiority of HIRI-ViT. More remarkably, under comparable computational cost ($\sim$5.0 GFLOPs), HIRI-ViT achieves to-date the best published Top-1 accuracy of 84.3% on ImageNet with 448$\times$448 inputs, which absolutely improves 83.4% of iFormer-S by 0.9% with 224$\times$224 inputs.

HIRI-ViT: Scaling Vision Transformer with High Resolution Inputs

TL;DR

HIRI-ViT introduces a principled five-stage CNN+ViT backbone designed for high-resolution inputs by decomposing CNN operations into two parallel branches that process high- and low-resolution features. The architecture features a High Resolution Stem and High Resolution Blocks, along with Inverted Residual Downsampling, BN/LN normalization choices, and a bi-directional EMA distillation strategy to stabilize training. Empirical results across ImageNet-1K, COCO, and ADE20K demonstrate state-of-the-art accuracy within practical computational budgets, notably achieving 84.3% Top-1 on ImageNet-1K at 448×448 with around 5 GFLOPs for the S variant and 85.7% for larger configurations at similar resolutions. The design generalizes to other backbones, underscoring the versatility of the two-branch, high-resolution scaling approach for CNN+ViT hybrids.

Abstract

The hybrid deep models of Vision Transformer (ViT) and Convolution Neural Network (CNN) have emerged as a powerful class of backbones for vision tasks. Scaling up the input resolution of such hybrid backbones naturally strengthes model capacity, but inevitably suffers from heavy computational cost that scales quadratically. Instead, we present a new hybrid backbone with HIgh-Resolution Inputs (namely HIRI-ViT), that upgrades prevalent four-stage ViT to five-stage ViT tailored for high-resolution inputs. HIRI-ViT is built upon the seminal idea of decomposing the typical CNN operations into two parallel CNN branches in a cost-efficient manner. One high-resolution branch directly takes primary high-resolution features as inputs, but uses less convolution operations. The other low-resolution branch first performs down-sampling and then utilizes more convolution operations over such low-resolution features. Experiments on both recognition task (ImageNet-1K dataset) and dense prediction tasks (COCO and ADE20K datasets) demonstrate the superiority of HIRI-ViT. More remarkably, under comparable computational cost (5.0 GFLOPs), HIRI-ViT achieves to-date the best published Top-1 accuracy of 84.3% on ImageNet with 448448 inputs, which absolutely improves 83.4% of iFormer-S by 0.9% with 224224 inputs.
Paper Structure (22 sections, 5 equations, 6 figures, 9 tables)

This paper contains 22 sections, 5 equations, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Convolutional Neural Network
  4. Vision Transformer
  5. High-resolution Representation Learning
  6. Preliminaries
  7. HIRI-ViT
  8. High Resolution Stem
  9. High Resolution Block
  10. Inverted Residual Downsampling
  11. Normalization of Block
  12. EMA Distillation
  13. Architecture Details
  14. Experiments
  15. Image Classification on ImageNet-1K
  16. ...and 7 more sections

Figures (6)

  • Figure 1: Performance and computational cost (i.e., GFLOPs and Inference time) of Swin Transformer and our HIRI-ViT with different input resolutions (224$\times$224, 384$\times$384, and 448$\times$448) on ImageNet-1K dataset.
  • Figure 2: Comparison between (a) the typical multi-stage Vision Transformer (M-ViT) with regular resolution inputs and (b) our HIRI-ViT with high resolution inputs. HR-stem: High Resolution Stem. IRDS: Inverted Residual Downsampling. HR Block: High Resolution Block. CFFN: Convolutional Feed-Forward Network.
  • Figure 3: An illustration of (a) ViT-stem and (b) Conv-stem in typical Vision Transformer, and (c) our HR-stem in HIRI-ViT.
  • Figure 4: An illustration of HR block in HIRI-ViT.
  • Figure 5: An illustration of (a) downsampling (DS) in typical Vision Transformer and two variants of our Inverted Residual Downsampling (i.e., (b) IRDS-a and (c) IRDS-b) in HIRI-ViT.
  • ...and 1 more figures