Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LeMeViT: Efficient Vision Transformer with Learnable Meta Tokens for Remote Sensing Image Interpretation

Wentao Jiang, Jing Zhang, Di Wang, Qiming Zhang, Zengmao Wang, Bo Du

TL;DR

LeMeViT addresses the heavy computation of self-attention in Vision Transformers on remote sensing data by introducing learnable meta tokens and a Dual Cross-Attention (DCA) mechanism that exchanges information between dense image tokens and a small set of meta tokens. This design replaces the quadratic $O(N^2)$ self-attention with a linear-time cross-attention regime, enabling a hierarchical four-stage architecture with variants Tiny, Small, and Base, and achieving about a $1.7\times$ speedup with fewer parameters while maintaining competitive accuracy. The core contributions include the learnable meta-tokens concept, the DCA block, and a hardware-friendly arrangement that uses standard operators, leading to strong performance gains in ImageNet classification and remote-sensing tasks such as object detection, semantic segmentation, and change detection. The approach demonstrates robust transferability to RS benchmarks and downstream tasks, providing a practical balance between efficiency and accuracy for large-scale RS image interpretation.

Abstract

Due to spatial redundancy in remote sensing images, sparse tokens containing rich information are usually involved in self-attention (SA) to reduce the overall token numbers within the calculation, avoiding the high computational cost issue in Vision Transformers. However, such methods usually obtain sparse tokens by hand-crafted or parallel-unfriendly designs, posing a challenge to reach a better balance between efficiency and performance. Different from them, this paper proposes to use learnable meta tokens to formulate sparse tokens, which effectively learn key information meanwhile improving the inference speed. Technically, the meta tokens are first initialized from image tokens via cross-attention. Then, we propose Dual Cross-Attention (DCA) to promote information exchange between image tokens and meta tokens, where they serve as query and key (value) tokens alternatively in a dual-branch structure, significantly reducing the computational complexity compared to self-attention. By employing DCA in the early stages with dense visual tokens, we obtain the hierarchical architecture LeMeViT with various sizes. Experimental results in classification and dense prediction tasks show that LeMeViT has a significant $1.7 \times$ speedup, fewer parameters, and competitive performance compared to the baseline models, and achieves a better trade-off between efficiency and performance.

LeMeViT: Efficient Vision Transformer with Learnable Meta Tokens for Remote Sensing Image Interpretation

TL;DR

LeMeViT addresses the heavy computation of self-attention in Vision Transformers on remote sensing data by introducing learnable meta tokens and a Dual Cross-Attention (DCA) mechanism that exchanges information between dense image tokens and a small set of meta tokens. This design replaces the quadratic self-attention with a linear-time cross-attention regime, enabling a hierarchical four-stage architecture with variants Tiny, Small, and Base, and achieving about a speedup with fewer parameters while maintaining competitive accuracy. The core contributions include the learnable meta-tokens concept, the DCA block, and a hardware-friendly arrangement that uses standard operators, leading to strong performance gains in ImageNet classification and remote-sensing tasks such as object detection, semantic segmentation, and change detection. The approach demonstrates robust transferability to RS benchmarks and downstream tasks, providing a practical balance between efficiency and accuracy for large-scale RS image interpretation.

Abstract

Due to spatial redundancy in remote sensing images, sparse tokens containing rich information are usually involved in self-attention (SA) to reduce the overall token numbers within the calculation, avoiding the high computational cost issue in Vision Transformers. However, such methods usually obtain sparse tokens by hand-crafted or parallel-unfriendly designs, posing a challenge to reach a better balance between efficiency and performance. Different from them, this paper proposes to use learnable meta tokens to formulate sparse tokens, which effectively learn key information meanwhile improving the inference speed. Technically, the meta tokens are first initialized from image tokens via cross-attention. Then, we propose Dual Cross-Attention (DCA) to promote information exchange between image tokens and meta tokens, where they serve as query and key (value) tokens alternatively in a dual-branch structure, significantly reducing the computational complexity compared to self-attention. By employing DCA in the early stages with dense visual tokens, we obtain the hierarchical architecture LeMeViT with various sizes. Experimental results in classification and dense prediction tasks show that LeMeViT has a significant speedup, fewer parameters, and competitive performance compared to the baseline models, and achieves a better trade-off between efficiency and performance.
Paper Structure (17 sections, 4 equations, 5 figures, 6 tables)

This paper contains 17 sections, 4 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vanilla Vision Transformer
  4. Efficient Vision Transformer
  5. ViT for Remote Sensing
  6. Method
  7. Overview and Preliminaries
  8. Key Components in LeMeViT
  9. Details of Architecture Design
  10. Computational Complexity Analysis
  11. Experiments
  12. Image Classification on ImageNet-1K
  13. Remote Sensing Scene Recognition
  14. Remote Sensing Downstream Tasks
  15. Ablation Study
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Due to the high correlation between neighboring pixels and image patches, as well as the repetitive nature of textures and patterns in remote sensing images, there is a significant amount of spatial redundancy. This results in redundant computation in self-attention mechanism.
  • Figure 2: Existing methods commonly use downsampling or clustering to reduce the number of image tokens within the current block, which relies on strong priors or is parallel-computation unfriendly. Our method learns meta tokens to sparsely represent dense image tokens. Meta tokens exchange information with image tokens via the computationally efficient Dual Cross-Attention Block in an end-to-end way, promoting information flow stage-by-stage.
  • Figure 3: The Overall Architecture of LeMeViT. LeMeViT consists of three different attention blocks, arranged from left to right as Cross Attention Block, Dual Cross Attention Block, and Standard Attention Block. Specific details of attention computation method are provided.
  • Figure 4: Visualization of comparison between different models. The size of scatter represents the parameter count of the model.
  • Figure 5: Visualization of the attention maps between three meta tokens in the last layer and image tokens. (a) illustrates the attention maps on natural images, while (b) illustrates attention maps on remote sensing images.