Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning in RKHM: a $C^*$-Algebraic Twist for Kernel Machines

Yuka Hashimoto, Masahiro Ikeda, Hachem Kadri

TL;DR

This work extends kernel-based learning to Reproducing Kernel Hilbert C*-Modules (RKHM) by leveraging $C^*$-algebra-valued kernels, enabling larger and more expressive representation spaces than traditional RKHS or vvRKHS.It defines $C^*$-algebra-valued positive definite kernels (linear, polynomial, Gaussian), derives an $ ext{A}$-valued generalization bound via a Rademacher complexity framework, and analyzes computational strategies including FFT-based and conjugate-gradient approaches.The framework recovers and generalizes convolutional neural network (CNN) concepts and the convolutional kernel as special cases, while allowing richer interactions among Fourier components, and demonstrates superior performance on synthetic data and competitive MNIST results when compared to standard CNNs.By unifying kernel methods with $C^*$-algebras, the paper provides a principled pathway to analyze and learn from high-dimensional structured data (notably images), with potential for further efficiency via random feature approximations.

Abstract

Supervised learning in reproducing kernel Hilbert space (RKHS) and vector-valued RKHS (vvRKHS) has been investigated for more than 30 years. In this paper, we provide a new twist to this rich literature by generalizing supervised learning in RKHS and vvRKHS to reproducing kernel Hilbert $C^*$-module (RKHM), and show how to construct effective positive-definite kernels by considering the perspective of $C^*$-algebra. Unlike the cases of RKHS and vvRKHS, we can use $C^*$-algebras to enlarge representation spaces. This enables us to construct RKHMs whose representation power goes beyond RKHSs, vvRKHSs, and existing methods such as convolutional neural networks. Our framework is suitable, for example, for effectively analyzing image data by allowing the interaction of Fourier components.

Learning in RKHM: a $C^*$-Algebraic Twist for Kernel Machines

TL;DR

This work extends kernel-based learning to Reproducing Kernel Hilbert C*-Modules (RKHM) by leveraging $C^*$-algebra-valued kernels, enabling larger and more expressive representation spaces than traditional RKHS or vvRKHS.It defines $C^*$-algebra-valued positive definite kernels (linear, polynomial, Gaussian), derives an $ ext{A}$-valued generalization bound via a Rademacher complexity framework, and analyzes computational strategies including FFT-based and conjugate-gradient approaches.The framework recovers and generalizes convolutional neural network (CNN) concepts and the convolutional kernel as special cases, while allowing richer interactions among Fourier components, and demonstrates superior performance on synthetic data and competitive MNIST results when compared to standard CNNs.By unifying kernel methods with $C^*$-algebras, the paper provides a principled pathway to analyze and learn from high-dimensional structured data (notably images), with potential for further efficiency via random feature approximations.

Abstract

Supervised learning in reproducing kernel Hilbert space (RKHS) and vector-valued RKHS (vvRKHS) has been investigated for more than 30 years. In this paper, we provide a new twist to this rich literature by generalizing supervised learning in RKHS and vvRKHS to reproducing kernel Hilbert -module (RKHM), and show how to construct effective positive-definite kernels by considering the perspective of -algebra. Unlike the cases of RKHS and vvRKHS, we can use -algebras to enlarge representation spaces. This enables us to construct RKHMs whose representation power goes beyond RKHSs, vvRKHSs, and existing methods such as convolutional neural networks. Our framework is suitable, for example, for effectively analyzing image data by allowing the interaction of Fourier components.
Paper Structure (19 sections, 21 theorems, 48 equations, 6 figures, 2 tables)

This paper contains 19 sections, 21 theorems, 48 equations, 6 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. PRELIMINARIES
  3. $C^*$-Algebra and Hilbert $C^*$-Module
  4. Reproducing Kernel Hilbert $C^*$-Module
  5. $C^*$-ALGEBRA-VALUED POSITIVE DEFINITE KERNELS
  6. Comparison with vvRKHS
  7. SUPERVISED LEARNING IN RKHM
  8. Problem Setting
  9. Generalization Bound
  10. Computational Complexity
  11. CONNECTION WITH EXISTING METHODS
  12. Connection with Convolutional Neural Network
  13. Connection with Convolutional Kernel
  14. NUMERICAL RESULTS
  15. Experiments with Synthetic Data
  16. ...and 4 more sections

Key Result

Lemma 2.2

Any circulant matrix $x\in Circ(p)$ has an eigenvalue decomposition $x=F\Lambda_x F^*$, where

Figures (6)

  • Figure 1: Representing samples in RKHM
  • Figure 2: Product in $\mathcal{A}_1$ and $\mathcal{A}_2$ in Example \ref{['ex:FC']}
  • Figure 3: Comparison of RKHM with vvRKHS
  • Figure 4: Mean test error versus hyperparameters (Mean value $\pm$ standard deviation of $5$ runs).
  • Figure 5: Comparison between RKHM and CNN
  • ...and 1 more figures

Theorems & Definitions (40)

  • Definition 2.1: Group $C^*$-algebra on a finite cyclic group
  • Lemma 2.2
  • Lemma 2.3
  • Definition 2.4: Positive
  • Definition 2.5: Minimum
  • Definition 2.6: Hilbert $C^*$-module
  • Definition 2.7: $\mathcal{A}$-valued positive definite kernel
  • Proposition 2.8: Representer theorem
  • Example 3.1
  • Definition 3.2
  • ...and 30 more