A Reproduction Study: The Kernel PCA Interpretation of Self-Attention Fails Under Scrutiny

TL;DR

This reproduction study critically evaluates the kernel PCA interpretation of self-attention, testing whether attention value vectors align with KPCA eigenvectors, whether reconstruction-based projection losses imply convergence, and whether eigenvalue statistics justify the KPCA claim. Across 10 transformer architectures, the authors find negligible alignment between $V$ and the KPCA counterparts, demonstrate that the reduction in $J_{\text{proj}}$ arises from scale effects rather than true convergence, and reveal substantial inconsistencies in the claimed eigenvalue behavior of $\tilde{K}_{\varphi}$. The results undermine the claimed KPCA equivalence for self-attention and highlight implementation and visualization choices that can mislead interpretations. The work emphasizes the need for robust, reproducible analyses and offers anonymized code to support transparent peer review, suggesting the KPCA view may be more limited in scope than originally proposed.

Abstract

In this reproduction study, we revisit recent claims that self-attention implements kernel principal component analysis (KPCA) (Teo et al., 2024), positing that (i) value vectors $V$ capture the eigenvectors of the Gram matrix of the keys, and (ii) that self-attention projects queries onto the principal component axes of the key matrix $K$ in a feature space. Our analysis reveals three critical inconsistencies: (1) No alignment exists between learned self-attention value vectors and what is proposed in the KPCA perspective, with average similarity metrics (optimal cosine similarity $\leq 0.32$, linear CKA (Centered Kernel Alignment) $\leq 0.11$, kernel CKA $\leq 0.32$) indicating negligible correspondence; (2) Reported decreases in reconstruction loss $J_\text{proj}$, arguably justifying the claim that the self-attention minimizes the projection error of KPCA, are misinterpreted, as the quantities involved differ by orders of magnitude ($\sim\!10^3$); (3) Gram matrix eigenvalue statistics, introduced to justify that $V$ captures the eigenvector of the gram matrix, are irreproducible without undocumented implementation-specific adjustments. Across 10 transformer architectures, we conclude that the KPCA interpretation of self-attention lacks empirical support.

Figures (5)

• Figure 1: Reconstruction loss ($J_{\text{proj}}$) over training epochs for ViT-Tiny and DeiT-Tiny models, along with the values of the individual squared norms, shown with markers. Circle markers indicate average of squared output norms ($\|\mathbf{h}_i\|^2$) and triangle markers (extremely low values around $10^{-3}$) show the average of squared feature map norms ($\|\varphi(\mathbf{q}_i)\|^2$).
• Figure 2: Comparison of squared norms across transformer layers. The plots show medians (solid lines) and 95% percentiles (shaded regions) of $\left\| \varphi(q_i) \right\|^2$ (blue) and $\left\| h_i \right\|^2$ (red) for 9 pre-trained transformer models for an input image. Values are displayed in log-scale due to the small magnitude of $\left\| \varphi(q_i) \right\|^2$. Log scaling highlights vanishing $\|\varphi(q_i)\|^2$ magnitudes. Notice the (1) high variance in $\varphi(q_i)$ projections vs. stable attention outputs, (2) no layer-wise convergence despite architectural scaling (DeiT/ViT, Tiny$\to$Base)
• Figure 3: (ViT Tiny) Top row: Mean and standard deviation of the absolute differences of entries in the $\gamma$ vector from true eigenvectors of $\tilde{K}_{\varphi}$ (matrix $A$). Bottom row: Corresponding results for random-direction eigenvectors ($A_{\text{random}}$) with matched row norms. Left panels initially suggest both satisfy $\frac{\tilde{K}_{\varphi} \hat{a}_d}{N \hat{a}_d} = \gamma = [\gamma_1, \ldots, \gamma_N]$ with $\gamma_1 = \cdots = \gamma_N = \text{const.}$; however, absolute differences reveal orders-of-magnitude deviation ($10^{-7}$ vs. $10^{-11}$). Right panels (relative error to $\max(|\gamma_i|, |\gamma_j|)$) demonstrate the condition violation more explicitly through significantly higher relative errors for $A_{\text{random}}$, showing small $\tilde{K}_{\varphi}$ eigenvalues permit visual resemblance despite failing the eigenvector criterion.
• Figure 4: Relative absolute reconstruction train/test errors with respect to $\|\varphi(\mathbf{q}_i)\|^2$ and $\|h_i\|^2$ for ViT-Tiny. Errors with respect to $\|\varphi(\mathbf{q}_i)\|^2$ are in scale $10^{-6}$. For clarity, the lower panel excludes the first 10 epochs to mitigate outlier effects and enhance trend visibility.
• Figure 5: Distribution of $\left| \varphi(q_i) \right|^2$ (blue) and $\left| h_i \right|^2$ (red) across layers of nine pre-trained encoder-only language models (log scale) (ordered by the parameter count). Each plot shows the median (solid line) and 95th percentile (shaded region) of the squared norm values across tokens. Despite differences in architecture and scale, all models exhibit a similar pattern: large variability in $\left| \varphi(q_i) \right|^2$ compared to the more stable $\left| h_i \right|^2$, and no consistent convergence behavior across layers. This mirrors observations made in vision transformers.