Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nested-TNT: Hierarchical Vision Transformers with Multi-Scale Feature Processing

Yuang Liu, Zhiheng Qiu, Xiaokai Qin

TL;DR

This paper tackles the coarse-grained feature extraction of Vision Transformers by introducing Nested-TNT, which fuses TNT's fine-grained visual words with Nested ViT's cross-layer attention. The architecture stacks inner word-level transformers inside visual sentences and an outer sentence-level transformer, connected via nested multi-head attention to enable information sharing across adjacent layers. Empirical results on CIFAR100, CIFAR10, and Flowers102 show that Nested-TNT achieves higher top-1 accuracy than ViT and TNT, at the cost of increased parameters and slower throughput. The approach demonstrates that jointly capturing detailed local patterns and global context yields strong improvements in image classification, while pointing to future work on speed, scaling, and broader vision tasks.

Abstract

Transformer has been applied in the field of computer vision due to its excellent performance in natural language processing, surpassing traditional convolutional neural networks and achieving new state-of-the-art. ViT divides an image into several local patches, known as "visual sentences". However, the information contained in the image is vast and complex, and focusing only on the features at the "visual sentence" level is not enough. The features between local patches should also be taken into consideration. In order to achieve further improvement, the TNT model is proposed, whose algorithm further divides the image into smaller patches, namely "visual words," achieving more accurate results. The core of Transformer is the Multi-Head Attention mechanism, and traditional attention mechanisms ignore interactions across different attention heads. In order to reduce redundancy and improve utilization, we introduce the nested algorithm and apply the Nested-TNT to image classification tasks. The experiment confirms that the proposed model has achieved better classification performance over ViT and TNT, exceeding 2.25%, 1.1% on dataset CIFAR10 and 2.78%, 0.25% on dataset FLOWERS102 respectively.

Nested-TNT: Hierarchical Vision Transformers with Multi-Scale Feature Processing

TL;DR

This paper tackles the coarse-grained feature extraction of Vision Transformers by introducing Nested-TNT, which fuses TNT's fine-grained visual words with Nested ViT's cross-layer attention. The architecture stacks inner word-level transformers inside visual sentences and an outer sentence-level transformer, connected via nested multi-head attention to enable information sharing across adjacent layers. Empirical results on CIFAR100, CIFAR10, and Flowers102 show that Nested-TNT achieves higher top-1 accuracy than ViT and TNT, at the cost of increased parameters and slower throughput. The approach demonstrates that jointly capturing detailed local patterns and global context yields strong improvements in image classification, while pointing to future work on speed, scaling, and broader vision tasks.

Abstract

Transformer has been applied in the field of computer vision due to its excellent performance in natural language processing, surpassing traditional convolutional neural networks and achieving new state-of-the-art. ViT divides an image into several local patches, known as "visual sentences". However, the information contained in the image is vast and complex, and focusing only on the features at the "visual sentence" level is not enough. The features between local patches should also be taken into consideration. In order to achieve further improvement, the TNT model is proposed, whose algorithm further divides the image into smaller patches, namely "visual words," achieving more accurate results. The core of Transformer is the Multi-Head Attention mechanism, and traditional attention mechanisms ignore interactions across different attention heads. In order to reduce redundancy and improve utilization, we introduce the nested algorithm and apply the Nested-TNT to image classification tasks. The experiment confirms that the proposed model has achieved better classification performance over ViT and TNT, exceeding 2.25%, 1.1% on dataset CIFAR10 and 2.78%, 0.25% on dataset FLOWERS102 respectively.
Paper Structure (10 sections, 8 equations, 1 figure, 5 tables)

This paper contains 10 sections, 8 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Experiments
  5. Datasets
  6. Experimental Settings
  7. Results
  8. Conclusion
  9. Limitation and Future Work
  10. About Authors

Figures (1)

  • Figure 1: Nested Multi-head Attention TNT Structure