Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HLA: Hadamard Linear Attention

Hanno Ackermann, Hong Cai, Mohsen Ghafoorian, Amirhossein Habibian

TL;DR

The paper introduces Hadamard Linear Attention (HLA) to address the quadratic bottleneck of standard softmax attention by applying a nonlinear Hadamard product after computing pairwise interactions, achieving a linear-time scheme with $O(N)$ complexity. By using multiple kernel factors, HLA raises the degree of the resulting rational attention function, increasing expressivity while maintaining efficiency through a tensor-based formulation that avoids expensive reshaping. The authors validate HLA on a large video diffusion model, showing competitive quality with substantially reduced compute (roughly a fraction of the FLOPs compared to quadratic attention). They also provide analysis on sequential updates, causal variants, and value modulation, illustrating that higher-factor HLA can scale model capacity without sacrificing speed, making it suitable for long-sequence tasks.

Abstract

The attention mechanism is an important reason for the success of transformers. It relies on computing pairwise relations between tokens. To reduce the high computational cost of standard quadratic attention, linear attention has been proposed as an efficient approximation. It employs kernel functions that are applied independently to the inputs before the pairwise similarities are calculated. That allows for an efficient computational procedure which, however, amounts to a low-degree rational function approximating softmax. We propose Hadamard Linear Attention (HLA). Unlike previous works on linear attention, the nonlinearity in HLA is not applied separately to queries and keys, but, analogously to standard softmax attention, after the pairwise similarities have been computed. It will be shown that the proposed nonlinearity amounts to a higher-degree rational function to approximate softmax. An efficient computational scheme for the proposed method is derived that is similar to that of standard linear attention. In contrast to other approaches, no time-consuming tensor reshaping is necessary to apply the proposed algorithm. The effectiveness of the approach is demonstrated by applying it to a large diffusion transformer model for video generation, an application that involves very large amounts of tokens.

HLA: Hadamard Linear Attention

TL;DR

The paper introduces Hadamard Linear Attention (HLA) to address the quadratic bottleneck of standard softmax attention by applying a nonlinear Hadamard product after computing pairwise interactions, achieving a linear-time scheme with complexity. By using multiple kernel factors, HLA raises the degree of the resulting rational attention function, increasing expressivity while maintaining efficiency through a tensor-based formulation that avoids expensive reshaping. The authors validate HLA on a large video diffusion model, showing competitive quality with substantially reduced compute (roughly a fraction of the FLOPs compared to quadratic attention). They also provide analysis on sequential updates, causal variants, and value modulation, illustrating that higher-factor HLA can scale model capacity without sacrificing speed, making it suitable for long-sequence tasks.

Abstract

The attention mechanism is an important reason for the success of transformers. It relies on computing pairwise relations between tokens. To reduce the high computational cost of standard quadratic attention, linear attention has been proposed as an efficient approximation. It employs kernel functions that are applied independently to the inputs before the pairwise similarities are calculated. That allows for an efficient computational procedure which, however, amounts to a low-degree rational function approximating softmax. We propose Hadamard Linear Attention (HLA). Unlike previous works on linear attention, the nonlinearity in HLA is not applied separately to queries and keys, but, analogously to standard softmax attention, after the pairwise similarities have been computed. It will be shown that the proposed nonlinearity amounts to a higher-degree rational function to approximate softmax. An efficient computational scheme for the proposed method is derived that is similar to that of standard linear attention. In contrast to other approaches, no time-consuming tensor reshaping is necessary to apply the proposed algorithm. The effectiveness of the approach is demonstrated by applying it to a large diffusion transformer model for video generation, an application that involves very large amounts of tokens.
Paper Structure (31 sections, 3 theorems, 18 equations, 12 figures, 5 tables, 1 algorithm)

This paper contains 31 sections, 3 theorems, 18 equations, 12 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Attention
  4. Softmax Attention
  5. Linear Attention
  6. Hadamard Linear Attention
  7. Definition
  8. Efficient Attention
  9. Polynomial Degree and Complexity
  10. Causal Attention, Decay Factors and Sequential Updates
  11. Sequential Updates
  12. Value Modulation
  13. Experiments
  14. Implementation Details
  15. Data
  16. ...and 16 more sections

Key Result

Lemma 4.1

Given $F=2$ factors for the Hadamard product in equation eq:hla, we may express the product involving the $d$-dimensional vectors $q$, $r^1$ and $r^2$ as where the sum is over all the $\hbox{len}(q)^2$ elements. For $F>2$, the relation involving the $d$-dimensional vectors $q$ and $r^f$ generalizes to where the symbol $\otimes$ denotes the outer tensorial product and the summation is over all th

Figures (12)

  • Figure 1: Results generated using the proposed video diffusion transformer (HLA-3F-R1-10, see sec. \ref{['sec:implementation.details']} for more details) using Hadamard Linear Attention.
  • Figure 2: Examples of generated videos by HLA-3F-R1-10 (upper videos) and HLA-3F-R1-21 (lower videos). It can be seen that the methods using the proposed attention mechanism indeed generate high-quality videos. Please see the supplementary material for more examples.
  • Figure 3: Definition of networks $\phi_{q,k_{1,2,3},v}$
  • Figure 4: VBench scores by categories. Dark blue line corresponds to HLA-3F-R1-10, red to HLA-3F-R1-21 and light blue to HLA-3F-R2-15.
  • Figure 5: First image of a video generated by HLA-3F-R1-21 in 50 steps.
  • ...and 7 more figures

Theorems & Definitions (5)

  • Lemma 4.1
  • proof
  • Theorem 4.2
  • proof
  • Lemma 4.3