A Multiplicative Neural Network Architecture: Locality and Regularity of Appriximation

TL;DR

The paper investigates how neural network architecture influences locality and regularity in function approximation. It introduces a multiplicative neural network framework whose units yield localized support, and proves a universal approximation theorem for $H_p^\gamma(\mathbb{R}^m)$, linking density to Sobolev-Zygmund regularity via $\mathcal{C}^{\gamma - m/p}$. Through numerical experiments on two geometrically distinct targets, the authors show that multiplicative units localize residual errors near irregular regions and achieve more stable convergence of regularity-sensitive metrics, as captured by Zygmund seminorms. This work establishes a concrete connection between architectural design and analytical properties of the approximated functions, highlighting potential gains in localization and regularity control for neural approximations.

Abstract

We introduce a multiplicative neural network architecture in which multiplicative interactions constitute the fundamental representation, rather than appearing as auxiliary components within an additive model. We establish a universal approximation theorem for this architecture and analyze its approximation properties in terms of locality and regularity in Bessel potential spaces. To complement the theoretical results, we conduct numerical experiments on representative targets exhibiting sharp transition layers or pointwise loss of higher-order regularity. The experiments focus on the spatial structure of approximation errors and on regularity-sensitive quantities, in particular the convergence of Zygmund-type seminorms. The results show that the proposed multiplicative architecture yields residual error structures that are more tightly aligned with regions of reduced regularity and exhibits more stable convergence in regularity-sensitive metrics. These results demonstrate that adopting a multiplicative representation format has concrete implications for the localization and regularity behavior of neural network approximations, providing a direct connection between architectural design and analytical properties of the approximating functions.

Figures (4)

• Figure 1: Figure 1: Absolute error for the mollified circle under the $L^2$ loss. Top: MLP; bottom: MMLP; left: tanh; right: gaussian. The MMLP produces a more structured and geometrically aligned error pattern, whereas the MLP exhibits more spatially diffuse residuals away from the transition region.
• Figure 2: Figure 2: Absolute error for the cone target under the $L^2$ loss. Top: MLP; bottom: MMLP; left: tanh; right: gaussian. The MMLP produces residual errors that are more tightly localized and geometrically organized around the origin, whereas the MLP exhibits more spatially dispersed error patterns.
• Figure 3: Convergence of regularity-sensitive error measures for the cone target under the $L_2$ loss. Top: $H^2_2$-type error; bottom: Zygmund seminorm $\mathcal{C}^{0.8}$ error. The MMLP exhibits a more sustained decay of the Zygmund seminorm, whereas the MLP shows earlier saturation.
• Figure 4: Convergence of regularity-sensitive error measures for the cone target under the $H^2_2$ loss. Top: $H^2_2$-type error; bottom: Zygmund seminorm $\mathcal{C}^{0.8}$ error. The MMLP exhibits a more sustained decay of the Zygmund seminorm, whereas the MLP shows earlier saturation.

Theorems & Definitions (20)

• Theorem 1.1: Universal Approximation -- Cybenko
• Definition 1.2: $L_p$ spaces
• Definition 1.3: Bessel Potential Space
• Remark 1.4: Relation to classical Sobolev spaces
• Definition 1.5: Zygmund Space
• Remark 1.6
• Lemma 1.7
• Theorem 1.8
• Remark 1.9: Invertibility of weight matrices
• Remark 1.10: Meaning and role of the parameters in Theorem \ref{['main theorem']}