Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Manifold Approximation leads to Robust Kernel Alignment

Mohammad Tariqul Islam, Du Liu, Deblina Sarkar

TL;DR

This work addresses the fragility of Centered Kernel Alignment (CKA) to data scale and manifold structure by introducing Manifold-approximated Kernel Alignment (MKA), a manifold-aware, non-Mercer kernel method. MKA uses a UMAP-inspired KNN graph to define a sparse, inductive kernel $K_U$ and computes alignment with $L_U$ via a row-centered HSIC-based ratio, yielding a symmetric, robust similarity measure even with non-symmetric kernels. The paper provides a theoretical framework, a fast computation route, and extensive empirical validation across synthetic shapes, rings/clusters, ReSi benchmarks, neural networks, and different domains (vision, NLP, graphs), showing MKA often outperforms or matches alternatives with far less sensitivity to hyperparameters. The findings suggest manifold-aware alignment provides a more stable foundation for comparing representations and can benefit diverse areas such as representation learning, neuroscience, and graph-based learning. The authors also release code for MKA and discuss avenues for future work, including exploring other kernels and debiasing strategies.

Abstract

Centered kernel alignment (CKA) is a popular metric for comparing representations, determining equivalence of networks, and neuroscience research. However, CKA does not account for the underlying manifold and relies on numerous heuristics that cause it to behave differently at different scales of data. In this work, we propose Manifold approximated Kernel Alignment (MKA), which incorporates manifold geometry into the alignment task. We derive a theoretical framework for MKA. We perform empirical evaluations on synthetic datasets and real-world examples to characterize and compare MKA to its contemporaries. Our findings suggest that manifold-aware kernel alignment provides a more robust foundation for measuring representations, with potential applications in representation learning.

Manifold Approximation leads to Robust Kernel Alignment

TL;DR

This work addresses the fragility of Centered Kernel Alignment (CKA) to data scale and manifold structure by introducing Manifold-approximated Kernel Alignment (MKA), a manifold-aware, non-Mercer kernel method. MKA uses a UMAP-inspired KNN graph to define a sparse, inductive kernel and computes alignment with via a row-centered HSIC-based ratio, yielding a symmetric, robust similarity measure even with non-symmetric kernels. The paper provides a theoretical framework, a fast computation route, and extensive empirical validation across synthetic shapes, rings/clusters, ReSi benchmarks, neural networks, and different domains (vision, NLP, graphs), showing MKA often outperforms or matches alternatives with far less sensitivity to hyperparameters. The findings suggest manifold-aware alignment provides a more stable foundation for comparing representations and can benefit diverse areas such as representation learning, neuroscience, and graph-based learning. The authors also release code for MKA and discuss avenues for future work, including exploring other kernels and debiasing strategies.

Abstract

Centered kernel alignment (CKA) is a popular metric for comparing representations, determining equivalence of networks, and neuroscience research. However, CKA does not account for the underlying manifold and relies on numerous heuristics that cause it to behave differently at different scales of data. In this work, we propose Manifold approximated Kernel Alignment (MKA), which incorporates manifold geometry into the alignment task. We derive a theoretical framework for MKA. We perform empirical evaluations on synthetic datasets and real-world examples to characterize and compare MKA to its contemporaries. Our findings suggest that manifold-aware kernel alignment provides a more robust foundation for measuring representations, with potential applications in representation learning.
Paper Structure (63 sections, 4 theorems, 14 equations, 10 figures, 17 tables)

This paper contains 63 sections, 4 theorems, 14 equations, 10 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Centered Kernel Alignment (CKA)
  4. Manifold-approximated Kernel Alignment (MKA)
  5. Experiments
  6. Equivalence of Shapes
  7. Ranking Structures
  8. Characterizing The Algorithms
  9. Evaluation using Representation Similarity (ReSi) Benchmark
  10. Neural Network Representations
  11. Computational complexity
  12. Discussion and Conclusions
  13. Proofs
  14. Clusters Data
  15. Details of Swiss-roll and S-Curve
  16. ...and 48 more sections

Key Result

Theorem 3.1

$\operatorname{CKA}(K_{\operatorname{RBF}},L) = \operatorname{CKA}(K_{\operatorname{LIN}},L)+O(1/\sigma^2)$ as $\sigma\to\infty$. Here, $K_{\operatorname{RBF}}$ is the RBF kernel matrix with bandwidth $\sigma$, $K_{\operatorname{LIN}}$ is the linear kernel matrix, and $L$ is any positive definite sy

Figures (10)

  • Figure 1: Equivalence of two different shapes with 1-D manifolds. (a) Swiss-roll. (b) S-curve by varying parameter $r$. (c) Alignment for the methods as S-curve parameter, $r$, varies. (d) Alignment for different methods as the number of nearest neighbors, $k$, varies. Note that $\operatorname{CKA}$, RTD, and SVCCA do not have any notion of nearest neighbors; thus, we have plotted these values at the end of the x-axis.
  • Figure 2: Alignment for the "rings" data. (a) Point clouds used in the clusters experiment. (b) Alignment using various methods, along with Kendall's rank correlation ($\tau$, higher is better). (c-e) Alignment by varying nearest neighbors, $k$, in (c) IMD, (d) kCKA, and (e) MKA. MKA shows the most robustness to the parameter $k$.
  • Figure 3: Characterizing MKA using synthetic datasets and comparison to other methods. (a) Top: A Gaussian spot; colors identify the position of the points on the x-axis. Middle: Perturbed Gaussian spot. We added noise to the points of the top figure so that the colors slightly overlap. Bottom: A Gaussian spot with no correspondence to the spot on the top. (b-e) Alignment between a Gaussian spot and when it is perturbed when (b) number of samples, $N$ ($d=1000$), and (c) number of dimensions, $d$ ($N=5000$), varies for various methods, and their performance as number of nearest neighbor, $k$, varies for (d) $d=2$ and (e) $d=100$ ($N=5000$). (f-i) Alignment under lost correspondence when (b) number of samples, $N$ ($d=1000$), and (c) number of dimensions, $d$ ($N=5000$), varies for various methods, and their performance as number of nearest neighbor, $k$, varies for (d) $d=2$ and (e) $d=100$ ($N=5000$). (j) Two uniform spots are located nearby (top) and translated far away (bottom). (k-n) Alignment under translation when (k) number of samples, $N$ ($d=1000$), (l) number of dimensions, $d$ ($N=5000$), (m) translation distance, $t$, and (n) number of nearest neighbors, $k$, varies. Error bars are drawn up to one standard deviation (5 trials for each experiment).
  • Figure 4: Aggregated ranks of alignment measures using the ReSi benchmark across different models and tests, separated by domains: (a) vision, (b) natural language processing, and (c) graph. Boxplots indicate quartiles of rank distributions; the whiskers extend up to 1.5 times the interquartile range. The black dots indicate the mean rank.
  • Figure 6: Alignment for the "clusters" data. (a) Point clouds used in the clusters experiment. (b) Alignment using various methods, along with Kendall's rank correlation (higher is better). (c-e) Alignment by varying nearest neighbors, $k$, in (c) IMD, (d) kCKA, and (e) MKA. MKA shows the most robustness to parameters.
  • ...and 5 more figures

Theorems & Definitions (6)

  • Theorem 3.1: alvarez2022gaussian
  • Theorem 4.1
  • Corollary 4.2
  • proof : (Proof of Theorem \ref{['thm:mka_simple']})
  • proof : (Proof of Corollary \ref{['thm:mka_range']})
  • Corollary E.1: Linear vs. Non-linear kCKA