Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unveiling the Potential of Superexpressive Networks in Implicit Neural Representations

Uvini Balasuriya Mudiyanselage, Woojin Cho, Minju Jo, Noseong Park, Kookjin Lee

TL;DR

This paper investigates NestNets, a superexpressive INR architecture that augments MLPs with an additional height dimension $s$ and trainable subnetworks as activation, aiming to boost expressive power in implicit representations. It combines Fourier feature mappings for coordinate encoding with nested activations and evaluates on signal representation, computer vision INR tasks, and physics-informed learning, comparing to baselines such as WIRE, SIREN, MFN, and FFN. Across tasks, NestNets deliver higher PSNR/SSIM/IOU and markedly lower PDE residuals; on a 1D convection PDE with $\beta=10$, the relative error drops to $0.0342$ versus $0.3719$–$0.5918$ for baselines. The work highlights learned, nontrivial activation shapes arising from the nested architecture, suggesting broader applicability to INR-based rendering and scientific ML.

Abstract

In this study, we examine the potential of one of the ``superexpressive'' networks in the context of learning neural functions for representing complex signals and performing machine learning downstream tasks. Our focus is on evaluating their performance on computer vision and scientific machine learning tasks including signal representation/inverse problems and solutions of partial differential equations. Through an empirical investigation in various benchmark tasks, we demonstrate that superexpressive networks, as proposed by [Zhang et al. NeurIPS, 2022], which employ a specialized network structure characterized by having an additional dimension, namely width, depth, and ``height'', can surpass recent implicit neural representations that use highly-specialized nonlinear activation functions.

Unveiling the Potential of Superexpressive Networks in Implicit Neural Representations

TL;DR

This paper investigates NestNets, a superexpressive INR architecture that augments MLPs with an additional height dimension and trainable subnetworks as activation, aiming to boost expressive power in implicit representations. It combines Fourier feature mappings for coordinate encoding with nested activations and evaluates on signal representation, computer vision INR tasks, and physics-informed learning, comparing to baselines such as WIRE, SIREN, MFN, and FFN. Across tasks, NestNets deliver higher PSNR/SSIM/IOU and markedly lower PDE residuals; on a 1D convection PDE with , the relative error drops to versus – for baselines. The work highlights learned, nontrivial activation shapes arising from the nested architecture, suggesting broader applicability to INR-based rendering and scientific ML.

Abstract

In this study, we examine the potential of one of the ``superexpressive'' networks in the context of learning neural functions for representing complex signals and performing machine learning downstream tasks. Our focus is on evaluating their performance on computer vision and scientific machine learning tasks including signal representation/inverse problems and solutions of partial differential equations. Through an empirical investigation in various benchmark tasks, we demonstrate that superexpressive networks, as proposed by [Zhang et al. NeurIPS, 2022], which employ a specialized network structure characterized by having an additional dimension, namely width, depth, and ``height'', can surpass recent implicit neural representations that use highly-specialized nonlinear activation functions.

Paper Structure

This paper contains 30 sections, 1 equation, 17 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Nested Networks
  4. Fourier Feature Mapping
  5. Training objectives
  6. Experiments
  7. NestNet specification.
  8. Baselines.
  9. Training.
  10. Computer Vision Downstream Tasks
  11. Physics-informed Neural Networks
  12. Conclusion
  13. Acknowledgment
  14. Detailed description on experimental setup and additional results
  15. Signal Representation
  16. ...and 15 more sections

Figures (17)

  • Figure 1: A NestNet of height 2. (a) The input and output of the network are coordinates $\pmb{x}=(x,y)$ and the signal at that coordinate $\pmb{u}(\pmb{x})$. Two subnetworks $\varrho_1$ and $\varrho_2$ (which are regular MLPs) serve as nonlinear activations. (b) A high-level illustration of a NestNet with width = depth = 3 and height = 2. Each blue node represents a regular MLP, used as a learnable activation function applied element-wise to pre-activations in the main network.
  • Figure 2: [Computer Vision tasks] (From top to bottom rows) image representation, occupancy volume presentation, single-image super resolution, multi-image super resolution, image-denoising, and CT reconstruction. NestNets produce consistently better results qualitatively as well as quantitatively (PSNR, SSIM, and IOU reported in Appendix \ref{['app:results']}).
  • Figure 3: [PINNs] The solution snapshots of the 1D convection equation ($\beta=10$). The relative errors for (NestNet, MLP, SIREN, FFN, WIRE) are (0.0342, 0.3719, 0.4031, 0.5918, 0.5525).
  • Figure 4: [Image representation] The numbers above the figures report PSNR and SSIM (in the parenthesis).
  • Figure 5: [Signal representation] Image representation accuracy is measured over training epochs in terms of PSNR (left, \ref{['fig:training_loss_psnr']}) and occupancy volume representation accuracy is measured over training epochs in terms of IoU (right, \ref{['fig:training_loss_iou']}).
  • ...and 12 more figures