Unified Local and Global Attention Interaction Modeling for Vision Transformers

TL;DR

This work targets the limited ability of vision transformers to exchange local and global information before self-attention, especially for visually similar but semantically distinct objects. It introduces two pre-attention modules—Aggressive Convolutional Pooling (ACP) for multi-scale local/global feature mixing and Conceptual Attention Transformation (CAT) for integrating global semantic concepts—forming the Enhanced Interaction Vision Transformer (EI-ViT). Through extensive experiments across ViT, Swin, and DAT++ backbones on diverse datasets (including a new cancer-tumor COD dataset), EI modules yield consistent improvements in detection metrics, particularly at higher IoU thresholds, while maintaining competitive parameter budgets. Qualitative analyses (PCA, CKA, attention maps) show richer feature diversity and more discriminative attention before self-attention, supporting the empirical gains. The approach demonstrates strong potential for medical and concealed-object detection, with notable cross-dataset generalization and a publicly released dataset and codebase. All mathematical notions are expressed with clear notation to facilitate rigorous interpretation and reproducibility.

Abstract

We present a novel method that extends the self-attention mechanism of a vision transformer (ViT) for more accurate object detection across diverse datasets. ViTs show strong capability for image understanding tasks such as object detection, segmentation, and classification. This is due in part to their ability to leverage global information from interactions among visual tokens. However, the self-attention mechanism in ViTs are limited because they do not allow visual tokens to exchange local or global information with neighboring features before computing global attention. This is problematic because tokens are treated in isolation when attending (matching) to other tokens, and valuable spatial relationships are overlooked. This isolation is further compounded by dot-product similarity operations that make tokens from different semantic classes appear visually similar. To address these limitations, we introduce two modifications to the traditional self-attention framework; a novel aggressive convolution pooling strategy for local feature mixing, and a new conceptual attention transformation to facilitate interaction and feature exchange between semantic concepts. Experimental results demonstrate that local and global information exchange among visual features before self-attention significantly improves performance on challenging object detection tasks and generalizes across multiple benchmark datasets and challenging medical datasets. We publish source code and a novel dataset of cancerous tumors (chimeric cell clusters).

Figures (14)

• Figure 1: Overview of the Enhanced Interaction Vision Transformer: (a) The input image is tokenized using patch embedding to initialize the network. (b) The Aggressive Convolutional Pooling module (ACP) (Sec. \ref{['subsec:aggressive-attention-pooling']}) iteratively enriches features by expanding the receptive field through convolution and pooling operations. The resulting features are normalized via layer normalization and passed to the Conceptual Attention Transformer (CAT) (Sec. \ref{['subsec:conceptual-attention-transformation']}), which refines their representation through global interactions with high-level concept classes. (c) The refined features are processed through a multi-head self-attention mechanism, followed by (d) MLP feedforward layers, to generate the final output features of the enhanced interaction transformer block.
• Figure 2: Conceptual Attention Transformation (CAT) Module: (a) CAT operates on input feature maps $f \in \mathbb{R}^{C \times H \times W}$. (b) Positional mixing is first applied to integrate positional attributes into the input features. (c) These features are then used to compute global concept tokens. The computed global concepts are subsequently used to enhance the input feature (d), producing enhanced feature maps with global interaction information of the same shape as the input $f$.
• Figure 3: Global Concept Tokens: Input feature maps are processed by a linear layer followed by a Softmax function to generate conceptual attention maps. Each feature is a treated as a semantic vector projected onto a hyperplane.
• Figure 4: Examples of each dataset are as follows: (a) CCellBio Dataset: This dataset contains $3,643$ test images and $26,991$ training images, all showing cancer tumors in stained tissue scans. (b) COD10K-V2: This dataset contains $4,000$ test images and $6,000$ training images and focuses on concealed objects in both natural and artificial environments, with an example image displaying a camouflaged crab in the top-left quarter. (c) Brain Tumor Dataset: This medical dataset includes T1-weighted contrast-enhanced images, featuring three types of brain tumors: meningioma, glioma, and pituitary tumor with $613$ test images and $2,451$ training images. (d) NIH Chest XRay Dataset containing XRay Chest imaging with $1,000$ bounding box annotations with $176$ test images and $704$ training images. (e) RSNA Pneumonia Dataset to detect lung opacity with $1,911$ testing, $7,644$.
• Figure 5: Number of parameters for baseline and enhanced interaction models for 300x300 image input (left) and 512x512 image input (right).