A Sampling Theory Perspective on Activations for Implicit Neural Representations

TL;DR

This work offers a unified theory for activation design in implicit neural representations by casting activations as generator functions within a sampling-theory framework. It proves that $\mathrm{sinc}$ activations are optimal for $L^2$ signal reconstruction under Riesz-basis and partition-of-unity conditions, and extends the results from shallow to deep networks, including sinc-based positional embeddings. The authors demonstrate practical validity across image reconstruction, neural radiance fields, and dynamical-system modeling, including robust latent-variable dynamics discovery and improved SINDy-based equation discovery. The approach links signal processing and dynamical-systems modeling, providing principled guidance for high-frequency content modeling and offering a pathway to more reliable data-driven dynamics from partial or noisy observations.

Abstract

Implicit Neural Representations (INRs) have gained popularity for encoding signals as compact, differentiable entities. While commonly using techniques like Fourier positional encodings or non-traditional activation functions (e.g., Gaussian, sinusoid, or wavelets) to capture high-frequency content, their properties lack exploration within a unified theoretical framework. Addressing this gap, we conduct a comprehensive analysis of these activations from a sampling theory perspective. Our investigation reveals that sinc activations, previously unused in conjunction with INRs, are theoretically optimal for signal encoding. Additionally, we establish a connection between dynamical systems and INRs, leveraging sampling theory to bridge these two paradigms.

Figures (7)

• Figure 1: Comparison of Image reconstruction across different INRs over DIVK dataset. We run a grid search to find the optimal parameters for each INR. Note that a single optimal parameter setting is used for each activation, across all the images in the dataset.
• Figure 2: Discovering the dynamics from partial observations. We use the Vanderpol system for this illustration. Top row: the original attractor and the diffeomorphism obtained by the SVD decomposition of the Hankel matrix (see Sec. \ref{['sec:dynamics']}) without noise. Third row: The same procedure is used to obtain the reconstructions with noisy, random, and sparse samples. Second row: First, a $\mathrm{sinc}$-INR is used to obtain a continuous reconstruction of the signal from discrete samples, which is then used as a surrogate signal to resample measurements. Afterwards, the diffeomorphisms are obtained using those measurements. As shown, $\mathrm{sinc}$-INRs are able to recover the dynamics more robustly with noisy, sparse, and random samples.
• Figure 3: Quantitative comparison on discovering the dynamics of latent variables using INRs vs classical methods.
• Figure 4: We use $\mathrm{sinc}$-INRs to improve the results of the SINDy algorithm. The top block and the bottom block demonstrate experiments on the Lorenz system and the Rossler system, respectively. In each block, the top row and the bottom row represent the results of the baseline SINDy algorithm and the improved version (using coordinate networks). As evident, coordinate networks can be used to obtain significantly robust results.
• Figure 5: Robust recovery of dynamical systems from partial observations (Lorenz system). Top row: coordinate network. Bottom row: classical method.

Theorems & Definitions (23)

• Definition 3.1
• Definition 3.2
• Proposition 3.3
• Theorem 3.4
• Definition 3.5
• Example 3.6
• Theorem 3.7
• Lemma 3.8
• Proposition 3.9
• Theorem 3.10