Graph Laplacian Wavelet Transformer via Learnable Spectral Decomposition

TL;DR

The Graph Wavelet Transformer (GWT) addresses the $O(N^2)$ time/memory cost of dot-product self-attention by substituting a bank of learnable multi-scale graph-wavelet filters defined over the graph Laplacian $L$ derived from linguistic parses. By applying spectral filters $g_k(\lambda)$ in the graph-frequency domain, then mixing via per-filter coefficients, GWT achieves efficient, interpretable multi-resolution modeling of graph-structured sequences, with practical reductions in memory and latency. On WMT14 En–De, GWT attains $BLEU=28.1\pm0.1$ compared to $27.3\pm0.2$ for the baseline, while reducing parameters and improving throughput; ablations confirm the importance of multiple scales. The approach offers a principled, scalable alternative to dense attention for structured data, with potential for dynamic scale adaptation and broader graph-based applications.

Abstract

Existing sequence to sequence models for structured language tasks rely heavily on the dot product self attention mechanism, which incurs quadratic complexity in both computation and memory for input length N. We introduce the Graph Wavelet Transformer (GWT), a novel architecture that replaces this bottleneck with a learnable, multi scale wavelet transform defined over an explicit graph Laplacian derived from syntactic or semantic parses. Our analysis shows that multi scale spectral decomposition offers an interpretable, efficient, and expressive alternative to quadratic self attention for graph structured sequence modeling.

Figures (1)

• Figure 1: Graph Wavelet Transformer Architecture. Starting from input embeddings $X \in \mathbb{R}^{N\times d}$ (top left), we first build a directed graph $\mathcal{G}=(V,E)$ from a syntactic or semantic parse. From $\mathcal{G}$ we compute the normalized graph Laplacian $L = I - D^{-1/2} A\,D^{-1/2}$, whose eigendecomposition $L = U\,\Lambda\,U^\top$, $\Lambda = \mathrm{diag}(\lambda_1,\dots,\lambda_N)$ yields the graph Fourier basis $U$. In the multi-scale wavelet block (shaded), $K$ learnable bandpass filters $g_k(\lambda)$ are applied in the spectral domain, producing $\hat{X}^{(k)} = U\,g_k(\Lambda)\,U^\top\,X$ for $k=1,\dots,K$. These $K$ subband signals are then linearly combined via scale-specific mixing weights $\alpha^{(k)}\in\mathbb{R}^d$: $Y = \sum_{k=1}^K \hat{X}^{(k)}\,\mathrm{diag}(\alpha^{(k)})$. Finally, a position-wise feed-forward network (MLP + residual) produces the output representation $Y_{\mathrm{out}}$. This spectral decomposition and mixing pipeline replaces the usual $O(N^2)$ self-attention mechanism with a compact, multi-scale alternative that captures both local (high-frequency) and global (low-frequency) graph structure.