Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cost-Effective Attention Mechanisms for Low Resource Settings: Necessity & Sufficiency of Linear Transformations

Peyman Hosseini, Mehran Hosseini, Ignacio Castro, Matthew Purver

TL;DR

This paper tackles the high memory and compute demands of scaled dot-product attention (SDPA) in low-resource environments. It introduces three SDPA variants—Optimized Attention, Efficient Attention, and Super Attention—that modify the arrangement of linear transformations (including absorbing projections into other matrices and adding a left-mixing alignment kernel) to cut parameters by 25-50% and reduce FLOPs, while preserving or improving model quality. Across NLP benchmarks (sentiment, MT, generative LM, NLI) and Vision Transformers, Optimized and Efficient Attentions match standard SDPA with far fewer parameters and faster inference, while Super Attention frequently outperforms the baseline by up to about 10% in accuracy and is also more edge-friendly. The findings suggest these attention variants enable more capable transformer models to run on devices with limited computation and memory, broadening practical deployment in real-world, resource-constrained settings.

Abstract

From natural language processing to vision, Scaled Dot Product Attention (SDPA) is the backbone of most modern deep learning applications. Unfortunately, its memory and computational requirements can be prohibitive in low-resource settings. In this paper, we improve its efficiency without sacrificing its versatility. We propose three attention variants where we remove consecutive linear transformations or add a novel one, and evaluate them on a range of standard NLP and vision tasks. Our proposed models are substantially lighter than standard SDPA (and have 25-50% fewer parameters). We show that the performance cost of these changes is negligible relative to size reduction and that in one case (Super Attention) we succeed in outperforming SDPA by up to 10% while improving its speed and reducing its parameters by 25%.

Cost-Effective Attention Mechanisms for Low Resource Settings: Necessity & Sufficiency of Linear Transformations

TL;DR

This paper tackles the high memory and compute demands of scaled dot-product attention (SDPA) in low-resource environments. It introduces three SDPA variants—Optimized Attention, Efficient Attention, and Super Attention—that modify the arrangement of linear transformations (including absorbing projections into other matrices and adding a left-mixing alignment kernel) to cut parameters by 25-50% and reduce FLOPs, while preserving or improving model quality. Across NLP benchmarks (sentiment, MT, generative LM, NLI) and Vision Transformers, Optimized and Efficient Attentions match standard SDPA with far fewer parameters and faster inference, while Super Attention frequently outperforms the baseline by up to about 10% in accuracy and is also more edge-friendly. The findings suggest these attention variants enable more capable transformer models to run on devices with limited computation and memory, broadening practical deployment in real-world, resource-constrained settings.

Abstract

From natural language processing to vision, Scaled Dot Product Attention (SDPA) is the backbone of most modern deep learning applications. Unfortunately, its memory and computational requirements can be prohibitive in low-resource settings. In this paper, we improve its efficiency without sacrificing its versatility. We propose three attention variants where we remove consecutive linear transformations or add a novel one, and evaluate them on a range of standard NLP and vision tasks. Our proposed models are substantially lighter than standard SDPA (and have 25-50% fewer parameters). We show that the performance cost of these changes is negligible relative to size reduction and that in one case (Super Attention) we succeed in outperforming SDPA by up to 10% while improving its speed and reducing its parameters by 25%.
Paper Structure (38 sections, 10 equations, 8 figures, 11 tables)

This paper contains 38 sections, 10 equations, 8 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Revising the Attention Mechanism
  4. Optimized Attention: Absorbing $W^V_i$'s into $W^0$
  5. Efficient Attention: Absorbing $W^K$ into $W^Q$
  6. Super Attention: Introducing $W^A$
  7. Evaluation
  8. Evaluation Methodology.
  9. Experimental Setup.
  10. NLP Benchmarks
  11. Sentiment Classification.
  12. Machine Translation (MT)
  13. Generative LM and NLI.
  14. NLP Results Analysis.
  15. Vision Transformers
  16. ...and 23 more sections

Figures (8)

  • Figure 1: Standard multi-head scaled dot product attention (\ref{['subfig: Standard Attention']}) alongside the proposed variations: Optimized Attention (\ref{['subfig: Optimised\n Attention']}), Efficient Attention (\ref{['subfig: Efficient\n Attention']}), and Super Attention (\ref{['subfig: Super\n Attention']}). The "Linear" block denotes a linear transformation right while "Linear*" denotes a linear transformation from left.
  • Figure 2: 3D plots visualizing the number of FLOPs for a forward + backward pass given different sequence lengths and projection dimensions in single-head setting for Efficient and Standard attention. Efficient Att. needs substantially fewer FLOPs for completing a forward + backward pass. \ref{['fig: Flops All Archs Full']} compares all architectures.
  • Figure 3: Summary of relative inference latency of models using different attention variants relative to standard attention on different datasets on Edge Device (Apple Laptop M2). Efficient Att. is the fastest (Optimized and Super Att. are also faster than standard attention). More details and numerical results are available in \ref{['tbl: inference time']}.
  • Figure 4: Performance of different architectures on the Amazon Reviews as the size of models grows from 5 Million parameters to 25 Million parameters. In terms of test accuracy and loss, Super Attention shows increasingly better performance compared to all other architectures which are performing on par with each other. In terms of inference speed, all variants (especially Efficient) perform better than the Standard attention.
  • Figure 5: Number of Flops required to complete a single forward plus backward pass for each attention mechanism. While the complexity and therefore, the number of FLOPs increases linearly as the projection dimension increases for all attention mechanisms, the slope of the increase varies significantly as depicted in this plot. Efficient Attention and Super Attention (Optimized Attention is not shown as it is exactly similar to Super Attention) require significantly fewer FLOPs as the projection dimension increases compared to standard attention. Here sequence length is set to 64 ($\ell=64$). Trying different values for $\ell$ changes the scale of the $y$-axis but the chart looks the same.
  • ...and 3 more figures

Theorems & Definitions (10)

  • Definition 2.1: Standard Attention
  • Definition 3.1: Optimized Attention
  • Remark 3.2
  • proof
  • Definition 3.3: Efficient Attention
  • Remark 3.4
  • proof
  • Definition 3.5: Super Attention
  • Remark 3.6
  • proof