Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Noncommutative $C^*$-algebra Net: Learning Neural Networks with Powerful Product Structure in $C^*$-algebra

Ryuichiro Hataya, Yuka Hashimoto

TL;DR

The paper studies neural networks with parameters drawn from noncommutative $C^*$-algebras to induce interactions among sub-models and enable group-equivariant behavior. It introduces noncommutative $C^*$-algebra nets and provides concrete instantiations over matrix algebras and group $C^*$-algebras, plus a universality result for function approximation with interactions. Empirical results in image classification and neural implicit representations show that noncommutative nets gain performance through sub-model interactions and can realize permutation-equivariant structures without bespoke architectures. The work offers a unified algebraic perspective on learning with interactions and symmetries, with potential for functional-data applications, while acknowledging significant computational costs and outlining directions for scalable and infinite-dimensional extensions.

Abstract

We propose a new generalization of neural network parameter spaces with noncommutative $C^*$-algebra, which possesses a rich noncommutative structure of products. We show that this noncommutative structure induces powerful effects in learning neural networks. Our framework has a wide range of applications, such as learning multiple related neural networks simultaneously with interactions and learning equivariant features with respect to group actions. Numerical experiments illustrate the validity of our framework and its potential power.

Noncommutative $C^*$-algebra Net: Learning Neural Networks with Powerful Product Structure in $C^*$-algebra

TL;DR

The paper studies neural networks with parameters drawn from noncommutative -algebras to induce interactions among sub-models and enable group-equivariant behavior. It introduces noncommutative -algebra nets and provides concrete instantiations over matrix algebras and group -algebras, plus a universality result for function approximation with interactions. Empirical results in image classification and neural implicit representations show that noncommutative nets gain performance through sub-model interactions and can realize permutation-equivariant structures without bespoke architectures. The work offers a unified algebraic perspective on learning with interactions and symmetries, with potential for functional-data applications, while acknowledging significant computational costs and outlining directions for scalable and infinite-dimensional extensions.

Abstract

We propose a new generalization of neural network parameter spaces with noncommutative -algebra, which possesses a rich noncommutative structure of products. We show that this noncommutative structure induces powerful effects in learning neural networks. Our framework has a wide range of applications, such as learning multiple related neural networks simultaneously with interactions and learning equivariant features with respect to group actions. Numerical experiments illustrate the validity of our framework and its potential power.
Paper Structure (22 sections, 2 theorems, 10 equations, 9 figures, 7 tables)

This paper contains 22 sections, 2 theorems, 10 equations, 9 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. $C^*$-algebra
  4. $C^*$-algebra net
  5. The case of diagonal matrices
  6. Noncommutative $C^*$-algebra Net
  7. Examples of $C^*$-algebras for neural networks
  8. Nondiagonal matrices
  9. Block diagonal matrices
  10. Circulant matrices
  11. Group $C^*$-algebra on a symmetric group
  12. Bounded linear operators on a Hilbert space
  13. Approximation of functions with interactions by $C^*$-algebra net
  14. Experiments
  15. $C^*$-algebra nets over matrices
  16. ...and 7 more sections

Key Result

Lemma 1

Any circulant matrix $a$ is decomposed as $a=F\Lambda_a F^*$, where

Figures (9)

  • Figure 1: Difference between commutative and noncommutative $C^*$-algebra nets from the perspective of interactions among sub-models.
  • Figure 2: Average PSNR of implicit representations of the image below (top) and reconstructions of the ground truth image at every 100 iterations (bottom). The noncommutative $C^*$-algebra net learns the geometry and colors of the image faster than its commutative counterpart.
  • Figure 3: Ground truth images and their implicit representations of commutative and noncommutative $C^*$-algebra nets after 500 iterations of training. The noncommutative $C^*$-algebra net reproduces colors more faithfully.
  • Figure 4: Synthesized views of 3D implicit representations of commutative and noncommutative $C^*$-algebra nets after 5000 iterations of training. The noncommutative $C^*$-algebra net can produce finer details. Note that the commutative $C^*$-algebra net could not synthesize the chair on the left.
  • Figure 5: Average test accuracy curves of invariant DeepSet, equivariant DeepSet, and a group $C^*$-algebra net trained on 10k data of the sum-of-digits task. The group $C^*$-algebra net can learn more efficiently and effectively.
  • ...and 4 more figures

Theorems & Definitions (11)

  • Definition 1: $C^*$-algebra
  • Example 1: Commutative $C^*$-algebra
  • Example 2: Noncommutative $C^*$-algebra
  • Example 3: Group $C^*$-algebra
  • Example 4: Clifford algebra
  • Lemma 1
  • Remark 1
  • Remark 2
  • Remark 3
  • Proposition 1
  • ...and 1 more