Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Monomial Matrix Group Equivariant Neural Functional Networks

Viet-Hoang Tran, Thieu N. Vo, Tho H. Tran, An T. Nguyen, Tan M. Nguyen

TL;DR

This work extends neural functional networks by incorporating weight-space symmetries beyond permutations, using monomial matrices to capture both neuron-permutation plus scaling/sign-flipping actions. It introduces Monomial-NFNs, comprising $G$-equivariant linear layers and $G$-invariant pooling modules, with a linear-parameter design that scales gracefully with network size. Theoretical results identify maximal symmetry subgroups preserved by common activations and demonstrate that many weight-space symmetries are encompassed by monomial groups for both FCNNs and CNNs. Empirically, Monomial-NFNs achieve competitive generalization, robust weight-space editing, and INR classification performance while reducing parameter counts relative to permutation-only NFN baselines. The framework opens avenues for more expressive and efficient NFNs, though it acknowledges potential expressivity limits and maximality questions that warrant future study.

Abstract

Neural functional networks (NFNs) have recently gained significant attention due to their diverse applications, ranging from predicting network generalization and network editing to classifying implicit neural representation. Previous NFN designs often depend on permutation symmetries in neural networks' weights, which traditionally arise from the unordered arrangement of neurons in hidden layers. However, these designs do not take into account the weight scaling symmetries of $\ReLU$ networks, and the weight sign flipping symmetries of $\sin$ or $\Tanh$ networks. In this paper, we extend the study of the group action on the network weights from the group of permutation matrices to the group of monomial matrices by incorporating scaling/sign-flipping symmetries. Particularly, we encode these scaling/sign-flipping symmetries by designing our corresponding equivariant and invariant layers. We name our new family of NFNs the Monomial Matrix Group Equivariant Neural Functional Networks (Monomial-NFN). Because of the expansion of the symmetries, Monomial-NFN has much fewer independent trainable parameters compared to the baseline NFNs in the literature, thus enhancing the model's efficiency. Moreover, for fully connected and convolutional neural networks, we theoretically prove that all groups that leave these networks invariant while acting on their weight spaces are some subgroups of the monomial matrix group. We provide empirical evidence to demonstrate the advantages of our model over existing baselines, achieving competitive performance and efficiency.

Monomial Matrix Group Equivariant Neural Functional Networks

TL;DR

This work extends neural functional networks by incorporating weight-space symmetries beyond permutations, using monomial matrices to capture both neuron-permutation plus scaling/sign-flipping actions. It introduces Monomial-NFNs, comprising -equivariant linear layers and -invariant pooling modules, with a linear-parameter design that scales gracefully with network size. Theoretical results identify maximal symmetry subgroups preserved by common activations and demonstrate that many weight-space symmetries are encompassed by monomial groups for both FCNNs and CNNs. Empirically, Monomial-NFNs achieve competitive generalization, robust weight-space editing, and INR classification performance while reducing parameter counts relative to permutation-only NFN baselines. The framework opens avenues for more expressive and efficient NFNs, though it acknowledges potential expressivity limits and maximality questions that warrant future study.

Abstract

Neural functional networks (NFNs) have recently gained significant attention due to their diverse applications, ranging from predicting network generalization and network editing to classifying implicit neural representation. Previous NFN designs often depend on permutation symmetries in neural networks' weights, which traditionally arise from the unordered arrangement of neurons in hidden layers. However, these designs do not take into account the weight scaling symmetries of networks, and the weight sign flipping symmetries of or networks. In this paper, we extend the study of the group action on the network weights from the group of permutation matrices to the group of monomial matrices by incorporating scaling/sign-flipping symmetries. Particularly, we encode these scaling/sign-flipping symmetries by designing our corresponding equivariant and invariant layers. We name our new family of NFNs the Monomial Matrix Group Equivariant Neural Functional Networks (Monomial-NFN). Because of the expansion of the symmetries, Monomial-NFN has much fewer independent trainable parameters compared to the baseline NFNs in the literature, thus enhancing the model's efficiency. Moreover, for fully connected and convolutional neural networks, we theoretically prove that all groups that leave these networks invariant while acting on their weight spaces are some subgroups of the monomial matrix group. We provide empirical evidence to demonstrate the advantages of our model over existing baselines, achieving competitive performance and efficiency.
Paper Structure (51 sections, 5 theorems, 72 equations, 2 figures, 13 tables)

This paper contains 51 sections, 5 theorems, 72 equations, 2 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Monomial Matrices Perserved by a Nonlinear Activation
  4. Monomial Matrices and Monomial Matrix Group Actions
  5. Monomial Matrices Preserved by a Nonlinear Activation
  6. Weight Spaces and Monomial Matrix Group Actions on Weight Spaces
  7. Weight Spaces of FCNNs and CNNs
  8. Monomial Matrix Group Action on Weight Spaces
  9. Monomial Matrix Group Equivariant and Invariant NFNs
  10. Equivariant Layers
  11. Invariant Layers
  12. Monomial Matrix Group Equivariant Neural Functionals (Monomial-NFNs)
  13. Experimental Results
  14. Predicting CNN Generalization from Weights
  15. Classifying implicit neural representations of images
  16. ...and 36 more sections

Key Result

Proposition 3.2

Let $\mathbf{x} \in \mathbb{R}^n$ and $A = (A_{ij}) \in \mathbb{R}^{n \times m}$. Then for $D = \operatorname{diag}(d_1, \ldots, d_n) \in \Delta_n$, $\overline{D} = \operatorname{diag}(\overline{d}_1, \ldots, \overline{d}_m) \in \Delta_m$, $P_\pi \in \mathcal{P}_n$, and $P_\sigma \in \mathcal{P}_m$,

Figures (2)

  • Figure 1: CNN prediction on $\operatorname{ReLU}$ subset of Small CNN Zoo with different ranges of augmentations. Here the x-axis is the augment upper scale, presented in log scale. The metric used is Kendall's $\tau$.
  • Figure 2: Random qualitative samples of INR editing behavior on the Dilate (MNIST) and Contrast (CIFAR-10) editing tasks.

Theorems & Definitions (16)

  • Definition 3.1: See rotman2012introduction
  • Proposition 3.2
  • Definition 3.3
  • Proposition 3.4
  • Remark 3.5
  • Remark 4.1
  • Definition 4.2: Group action on weight spaces
  • Remark 4.3
  • Proposition 4.4: $G$-equivariance of neural functionals
  • Remark 4.5: Maximality of $G$
  • ...and 6 more