Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain

Hyowon Wi, Jeongwhan Choi, Noseong Park

TL;DR

This work reframes self-attention in Transformers as a directed-graph filter and introduces Attentive Graph Filter (AGF), which learns a sophisticated graph filter directly in the singular value domain with token-specific spectra. By employing Jacobi polynomial bases and an orthogonality-regularized SVD-like factorization, AGF can exploit both low- and high-frequency information while maintaining a linear-time complexity $O(nd^2)$ and modest memory. The approach yields strong empirical gains on time-series benchmarks (UEA) and long-sequence tasks (Long Range Arena), and it also helps mitigate over-smoothing in deep Transformer models. Overall, AGF offers a principled, spectral-domain enhancement to linear Transformers with practical benefits for long-context tasks and scalable deployment.

Abstract

Transformers have demonstrated remarkable performance across diverse domains. The key component of Transformers is self-attention, which learns the relationship between any two tokens in the input sequence. Recent studies have revealed that the self-attention can be understood as a normalized adjacency matrix of a graph. Notably, from the perspective of graph signal processing (GSP), the self-attention can be equivalently defined as a simple graph filter, applying GSP using the value vector as the signal. However, the self-attention is a graph filter defined with only the first order of the polynomial matrix, and acts as a low-pass filter preventing the effective leverage of various frequency information. Consequently, existing self-attention mechanisms are designed in a rather simplified manner. Therefore, we propose a novel method, called \underline{\textbf{A}}ttentive \underline{\textbf{G}}raph \underline{\textbf{F}}ilter (AGF), interpreting the self-attention as learning the graph filter in the singular value domain from the perspective of graph signal processing for directed graphs with the linear complexity w.r.t. the input length $n$, i.e., $\mathcal{O}(nd^2)$. In our experiments, we demonstrate that AGF achieves state-of-the-art performance on various tasks, including Long Range Arena benchmark and time series classification.

Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain

TL;DR

This work reframes self-attention in Transformers as a directed-graph filter and introduces Attentive Graph Filter (AGF), which learns a sophisticated graph filter directly in the singular value domain with token-specific spectra. By employing Jacobi polynomial bases and an orthogonality-regularized SVD-like factorization, AGF can exploit both low- and high-frequency information while maintaining a linear-time complexity and modest memory. The approach yields strong empirical gains on time-series benchmarks (UEA) and long-sequence tasks (Long Range Arena), and it also helps mitigate over-smoothing in deep Transformer models. Overall, AGF offers a principled, spectral-domain enhancement to linear Transformers with practical benefits for long-context tasks and scalable deployment.

Abstract

Transformers have demonstrated remarkable performance across diverse domains. The key component of Transformers is self-attention, which learns the relationship between any two tokens in the input sequence. Recent studies have revealed that the self-attention can be understood as a normalized adjacency matrix of a graph. Notably, from the perspective of graph signal processing (GSP), the self-attention can be equivalently defined as a simple graph filter, applying GSP using the value vector as the signal. However, the self-attention is a graph filter defined with only the first order of the polynomial matrix, and acts as a low-pass filter preventing the effective leverage of various frequency information. Consequently, existing self-attention mechanisms are designed in a rather simplified manner. Therefore, we propose a novel method, called \underline{\textbf{A}}ttentive \underline{\textbf{G}}raph \underline{\textbf{F}}ilter (AGF), interpreting the self-attention as learning the graph filter in the singular value domain from the perspective of graph signal processing for directed graphs with the linear complexity w.r.t. the input length , i.e., . In our experiments, we demonstrate that AGF achieves state-of-the-art performance on various tasks, including Long Range Arena benchmark and time series classification.
Paper Structure (58 sections, 2 theorems, 20 equations, 3 figures, 21 tables)

This paper contains 58 sections, 2 theorems, 20 equations, 3 figures, 21 tables.

Table of Contents

  1. Introduction
  2. Linear Transformers approximate the self-attention.
  3. The self-attention is a low-pass filter (see Thm. \ref{['thm:sa_is_lpf']}).
  4. Our proposed linear Transformer learns an advanced self-attention (see Thm. \ref{['thm:coeff']}).
  5. Background
  6. Self-attention in Transformer
  7. Linear Transformer
  8. Graph Convolutional Filter
  9. Proposed Method
  10. Self-attention as a graph filter
  11. Polynomial graph filter
  12. Attentive Graph Filter
  13. Objective Function
  14. Time and Space Complexities of AGF
  15. Properties of AGF
  16. ...and 43 more sections

Key Result

Theorem 1

Let $\mathbf{M}=\textrm{softmax}(\mathbf{Z})$ for any matrix $\mathbf{Z}\in\mathbb{R}^{n\times n}$. Then $\mathbf{M}$ inherently acts as a low pass filter. For all $\mathbf{x}\in \mathbb{R}^N$, in other words, $\text{lim}_{t\rightarrow \infty}\Vert \text{HFC}[\mathbf{M}^t(\mathbf{x})]\Vert_2/\Vert \

Figures (3)

  • Figure 1: Illustration of the vanilla self-attention in Transformers and graph signal processing: (a) shows the softmax operation after the dot-product of query and key vectors, followed by the multiplication with the value vector; (b) and (c) show graph signal processing in undirected and directed graphs, respectively. For undirected graphs, the signal is filtered in the eigenvalue domain, while for directed graphs, the signal is filtered in the singular value domain.
  • Figure 2: The proposed AGF performs the directed GSP in the singular value domain by learning $U(\mathbf{X})$, $\Sigma(\mathbf{\mathbf{X}})$, and $V(\mathbf{X})$ (cf. Eqs. \ref{['eq:singular']} to \ref{['eq:singular2']}). The $n$ different sets of singular values in $\Sigma(\mathbf{X})$ are used for token-specific processing. In other words, $n$ different graph filters are used for $n$ different tokens in order to increase the representation learning capability of AGF.
  • Figure 3: Filter response and cosine similarity on ImageNet-1k for DeiT-small and DeiT-small + AGF

Theorems & Definitions (4)

  • Theorem 1: Self-attention is a low-pass filter
  • Theorem 2: Adapted from chien2021GPRGNN
  • proof
  • proof