Visual Attention Methods in Deep Learning: An In-Depth Survey

TL;DR

This survey comprehensively catalogs and analyzes a wide spectrum of visual attention techniques in deep learning beyond Transformers, organizing them into soft/deterministic, hard/stochastic, and category-based families. It details core building blocks, formulations, and typical vision applications, while discussing strengths, limitations, and practical challenges such as computational cost and data requirements. The work covers more than 70 attention methods, including channel/spatial/self-attention, multi-modal fusion, Bayesian and Gaussian stochastic approaches, and auto-learning and clustering-based mechanisms. By proposing a hierarchical framework and highlighting open questions, it provides practical guidance for designing efficient, effective attention-enabled vision models and identifies key gaps for future research.

Abstract

Inspired by the human cognitive system, attention is a mechanism that imitates the human cognitive awareness about specific information, amplifying critical details to focus more on the essential aspects of data. Deep learning has employed attention to boost performance for many applications. Interestingly, the same attention design can suit processing different data modalities and can easily be incorporated into large networks. Furthermore, multiple complementary attention mechanisms can be incorporated into one network. Hence, attention techniques have become extremely attractive. However, the literature lacks a comprehensive survey on attention techniques to guide researchers in employing attention in their deep models. Note that, besides being demanding in terms of training data and computational resources, transformers only cover a single category in self-attention out of the many categories available. We fill this gap and provide an in-depth survey of 50 attention techniques, categorizing them by their most prominent features. We initiate our discussion by introducing the fundamental concepts behind the success of the attention mechanism. Next, we furnish some essentials such as the strengths and limitations of each attention category, describe their fundamental building blocks, basic formulations with primary usage, and applications specifically for computer vision. We also discuss the challenges and general open questions related to attention mechanisms. Finally, we recommend possible future research directions for deep attention. All the information about visual attention methods in deep learning is provided at \href{https://github.com/saeed-anwar/VisualAttention}{https://github.com/saeed-anwar/VisualAttention}

Figures (10)

• Figure 1: Visual charts show the increase in the number of attention-related papers in the top conferences and journals.
• Figure 2: A taxonomy of attention types. The attentions are categorized based on the methodology adopted to perform attention. Some attention techniques can be accommodated in multiple categories; in this case, the attention is grouped based on the most dominant characteristic and primary application.
• Figure 3: Core structures of the channel-based attention methods. Different methods to generate the attention scores, including squeeze and excitation se, splitting and squeezing zhang2020resnest, calculating the second order fu2019dual or efficient squeezing and excitation wang2020eca. Images are taken from the original papers and are best viewed in color.
• Figure 4: The structures of the spatial-based attention methods, including RANet shen2020ranet, and Co-excite NEURIPS2019_92af93f7. These methods focus on attending to the most important parts of the spatial map. The images are taken from woo2018cbamshen2020ranetNEURIPS2019_92af93f7.
• Figure 5: The architectures of self-attention methods: Transformers m_transformers, Axial attention axial_attention, X-Linear pan2020x, Slot slot and RFA peng2021random (pictures taken from the corresponding articles). These methods are self-attention, which generates the scores by measuring the similarity between two maps of the same input. However, there is a difference in the way of processing.