Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insights

TL;DR

Vision Transformers achieve strong performance but incur quadratic $O(N^2)$ self‑attention costs, motivating a comprehensive survey of efficiency‑oriented designs. The authors introduce a unified taxonomy that groups approaches into Self‑Attention Complexity Reduction, Hierarchical Transformers, Channel & Spatial Transformers, Rethinking Tokenization, and Other, and review representative architectures such as Swin, XCiT, CrossViT, MISSFormer, DaViT, DynamicViT, and BiFormer. They analyze ViT blocks across parameters, FLOPs, MACs, and latency, and provide insights into tradeoffs, timelines, and open challenges, complemented by an open‑source GitHub resource for implementations. The work emphasizes real‑world impact in medical imaging, remote sensing, and real‑time mobile vision, and highlights how local/global attention, tokenization strategies, and CNN–Transformer hybrids can deliver substantial efficiency gains without sacrificing accuracy. Overall, the survey serves as a practical roadmap for designing scalable, efficient ViTs and identifies key avenues for future research in computation, data efficiency, multi‑modal learning, and explainability.

Abstract

Intrigued by the inherent ability of the human visual system to identify salient regions in complex scenes, attention mechanisms have been seamlessly integrated into various Computer Vision (CV) tasks. Building upon this paradigm, Vision Transformer (ViT) networks exploit attention mechanisms for improved efficiency. This review navigates the landscape of redesigned attention mechanisms within ViTs, aiming to enhance their performance. This paper provides a comprehensive exploration of techniques and insights for designing attention mechanisms, systematically reviewing recent literature in the field of CV. This survey begins with an introduction to the theoretical foundations and fundamental concepts underlying attention mechanisms. We then present a systematic taxonomy of various attention mechanisms within ViTs, employing redesigned approaches. A multi-perspective categorization is proposed based on their application, objectives, and the type of attention applied. The analysis includes an exploration of the novelty, strengths, weaknesses, and an in-depth evaluation of the different proposed strategies. This culminates in the development of taxonomies that highlight key properties and contributions. Finally, we gather the reviewed studies along with their available open-source implementations at our \href{https://github.com/mindflow-institue/Awesome-Attention-Mechanism-in-Medical-Imaging}{GitHub}\footnote{\url{https://github.com/xmindflow/Awesome-Attention-Mechanism-in-Medical-Imaging}}. We aim to regularly update it with the most recent relevant papers.

Figures (13)

• Figure 1: (a) The task model and (b) The generalized attention module. From AttentionSUrvey.
• Figure 2: The Vision Transformer architecture from dosovitskiy2020vit is located on the center. Scaled dot-product attention and multi-head attention are on the top.
• Figure 3: The suggested taxonomy for attention mechanisms used within ViTs consists of four distinct groups: I) Computation Reduction, II) Hierarchical, III) Channel & Spatial, IV) Other. To maintain conciseness, we assign ascending prefix numbers to each category in the paper's name and cite each study accordingly as follows: 1. shen2021efficient, 2. ali2021xcit, 3. wu2022p2t, 4. wang2022kvt, 5. dong2022cswin, 6. chen2021crossvit, 7. tang2022quadtree, 8. huang2022missformer, 9. hassani2023neighborhood, 10. hassani2023dilated, 11. you2023castlingvit, 12. li2023rethinking, 13. jiao2023dilateformer, 14. shaker2023swiftformer, 15. liu2023efficientvit, 16. han2023flatten, 17. jiang2021all, 18. fang2022msg, 19. rao2021dynamicvit, 20. guo2022cmt, 21. zhang2021token, 22. xu2022evo, 23. tu2022maxvit, 24. zhang2023vsa, 25. pan2023fast, 26. rao2022hornet, 27. zhou2023token, 28. zhu2023biformer, 29. li2023bvit 30. wang2022pvt, 31. liu2021swin, 32. heo2021rethinking, 33. chen2021regionvit, 34. zhou2023nnformer, 35. yu2022metaformer, 36. liu2021swinv2, 37. hatamizadeh2023global, 38. pan2023slide, 39. vasu2023fastvit, 40. hatamizadeh2023fastervit, 41. huang2022channelized, 42. ding2022davit, 43. ma2022knowing, 44. shaker2022unetr++, 45. valanarasu2021medical, 46. zhou2021deepvit, 47. wu2021cvt, 48. graham2021levit, 49. xia2022vision 50. yang2022focal, 51. feng2022evit, 52. zhou2022spikformer, 53. Guo2023, 54. fan2023lightweight, 55. azad2023selfattention.
• Figure 4: Standard dot-product attention on the left and efficient attention on the right. From shen2021efficient.
• Figure 5: Regular self attention (top right) and cross-covariance attention (bottom right). From ali2021xcit.