Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Invariant Kernels: Rank Stabilization and Generalization Across Dimensions

Mateo Díaz, Dmitriy Drusvyatskiy, Jack Kendrick, Rekha R. Thomas

TL;DR

This paper investigates how symmetry in data—captured by group invariances—drives the rank and generalization properties of fixed-degree polynomial kernels. By linking kernel rank to the dimension of invariant polynomial spaces ${\mathbb{R}}[V]_m^G$, it shows that in several natural settings (permutations, set-permutations, graphs, and point clouds) the invariant dimension, and hence the kernel rank, can be independent of the ambient data dimension, enabling dimension-free learning and efficient computation. The authors develop free invariant bases (elementary symmetric polynomials and polarized variants) and show how to express invariant kernels as dimension-stable bilinear forms using these bases, together with a Monte Carlo approach to estimate invariant-dimensions and a cross-dimension (minimax) generalization framework. They connect rank stabilization to representation stability, providing a principled path to learning across varying dimensions with finite-parameter representations, and validate the theory through numerical experiments on set-classification, cross-dimension regression, and invariant-dimension estimation. The results offer a rigorous, scalable approach for leveraging invariances in kernel methods, with practical implications for multi-dimensional learning across domains such as graphs, point clouds, and unordered sets.

Abstract

Symmetry arises often when learning from high dimensional data. For example, data sets consisting of point clouds, graphs, and unordered sets appear routinely in contemporary applications, and exhibit rich underlying symmetries. Understanding the benefits of symmetry on the statistical and numerical efficiency of learning algorithms is an active area of research. In this work, we show that symmetry has a pronounced impact on the rank of kernel matrices. Specifically, we compute the rank of a polynomial kernel of fixed degree that is invariant under various groups acting independently on its two arguments. In concrete circumstances, including the three aforementioned examples, symmetry dramatically decreases the rank making it independent of the data dimension. In such settings, we show that a simple regression procedure is minimax optimal for estimating an invariant polynomial from finitely many samples drawn across different dimensions. We complete the paper with numerical experiments that illustrate our findings.

Invariant Kernels: Rank Stabilization and Generalization Across Dimensions

TL;DR

This paper investigates how symmetry in data—captured by group invariances—drives the rank and generalization properties of fixed-degree polynomial kernels. By linking kernel rank to the dimension of invariant polynomial spaces , it shows that in several natural settings (permutations, set-permutations, graphs, and point clouds) the invariant dimension, and hence the kernel rank, can be independent of the ambient data dimension, enabling dimension-free learning and efficient computation. The authors develop free invariant bases (elementary symmetric polynomials and polarized variants) and show how to express invariant kernels as dimension-stable bilinear forms using these bases, together with a Monte Carlo approach to estimate invariant-dimensions and a cross-dimension (minimax) generalization framework. They connect rank stabilization to representation stability, providing a principled path to learning across varying dimensions with finite-parameter representations, and validate the theory through numerical experiments on set-classification, cross-dimension regression, and invariant-dimension estimation. The results offer a rigorous, scalable approach for leveraging invariances in kernel methods, with practical implications for multi-dimensional learning across domains such as graphs, point clouds, and unordered sets.

Abstract

Symmetry arises often when learning from high dimensional data. For example, data sets consisting of point clouds, graphs, and unordered sets appear routinely in contemporary applications, and exhibit rich underlying symmetries. Understanding the benefits of symmetry on the statistical and numerical efficiency of learning algorithms is an active area of research. In this work, we show that symmetry has a pronounced impact on the rank of kernel matrices. Specifically, we compute the rank of a polynomial kernel of fixed degree that is invariant under various groups acting independently on its two arguments. In concrete circumstances, including the three aforementioned examples, symmetry dramatically decreases the rank making it independent of the data dimension. In such settings, we show that a simple regression procedure is minimax optimal for estimating an invariant polynomial from finitely many samples drawn across different dimensions. We complete the paper with numerical experiments that illustrate our findings.

Paper Structure

This paper contains 42 sections, 23 theorems, 124 equations, 3 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Summary of results
  3. Permutation Invariant Functions
  4. Set-permutation Invariant Functions.
  5. Functions on Graphs.
  6. Functions on Point Clouds
  7. A Monte Carlo rank estimator
  8. Connection to representation stability
  9. Computing invariant bases and kernels
  10. Generalization of invariant polynomial regression across dimensions
  11. Related literature
  12. Representation stability and generalization across dimensions.
  13. Polynomial kernels and their rank
  14. Group invariance and symmetrization.
  15. Invariant polynomials and kernel rank
  16. ...and 27 more sections

Key Result

Lemma 2.2

The kernel $K^*$ is $G$-invariant. Moreover, if $K$ is PSD then so is $K^*$.

Figures (3)

  • Figure 1: Comparison of performance between $(i)$ the set-permutation invariant kernel \ref{['eq:set-kernel']} and $(ii)$ the non-invariant Taylor features kernel for classifying points drawn from a mixture of Gaussians (Illustration \ref{['ex: invariant tf']}). We present two experimental scenarios: Figure \ref{['fig:classification-var-degree']} (left) shows results with fixed dimension $k=2$ across varying degrees $m \in \{1, \dots, 5\}$, while Figure \ref{['fig:classification-var-k']} (right) shows results with fixed degree $m=2$ across varying dimensions $k \in \{1, \dots, m\}$. In each setting, the invariant kernel outperforms the non-invariant kernel.
  • Figure 2: Mean Square Error (left) and Mean Percentage Error (right) versus test dimension using the estimator defined \ref{['eq:informal-estimator']}. For each degree, data was collected from ten randomly generated symmetric polynomials. Each estimator was trained on 10,000 noisy data points in dimension $d=100$ and tested on 200 noisy data points in dimensions $d=100, 200, \ldots, 1000.$ The highlighted regions indicate a 95% confidence interval for the values.
  • Figure 3: Monte Carlo estimates of $\dim({\mathbb{R}}[V]_4^G)$ for orthogonal transformations, coordinate permutations and graph permutations. The estimates are calculated using the empirical average approximation to the formula in Theorem \ref{['thm:calculate_dim_intro']} with an increasing number of samples. In each case, we see that the estimate stabilises around the correct dimension relatively quickly. The highlighted regions are the 95% confidence intervals for the estimate.

Theorems & Definitions (46)

  • Lemma 2.2
  • Definition 2.3: Polynomial kernel
  • Lemma 2.5: Bilinear form
  • Lemma 2.6: Exact rank formula
  • Theorem 2.7: Rank and polynomial invariants
  • Lemma 2.8
  • proof
  • Definition 4.1: Character
  • Lemma 4.2: Invariants and characters
  • Corollary 4.3
  • ...and 36 more