HOSC: A Periodic Activation with Saturation Control for High-Fidelity Implicit Neural Representations

TL;DR

Implicit neural representations suffer from spectral bias and gradient instability, motivating a need for explicit gradient control. We propose HOSC, a periodic activation $\text{HOSC}_\beta(x)=\tanh(\beta \sin(\omega_0 x))$ that yields a tight activation Lipschitz bound $L_{\text{act}}=\beta\omega_0$, decoupling spectral support from gradient scale. We provide analytic gradient bounds, NTK insights, and comprehensive experiments across images, audio, video, NeRFs, and SDFs, demonstrating robust improvements—especially in high-frequency or high-gradient regimes—and practical hyperparameter guidance. The activation is a simple drop-in replacement for SIREN that is compatible with existing encodings and architectures, enabling multi-modal INR performance with controlled stability and locality. The results suggest that explicit gradient control via activation design is a valuable, orthogonal complement to frequency-based INR strategies and encodings, with potential for future extensions like learnable carriers or spatially adaptive gating.

Abstract

Periodic activations such as sine preserve high-frequency information in implicit neural representations (INRs) through their oscillatory structure, but often suffer from gradient instability and limited control over multi-scale behavior. We introduce the Hyperbolic Oscillator with Saturation Control (HOSC) activation, $\text{HOSC}(x) = \tanh\bigl(β\sin(ω_0 x)\bigr)$, which exposes an explicit parameter $β$ that controls the Lipschitz bound of the activation by $βω_0$. This provides a direct mechanism to tune gradient magnitudes while retaining a periodic carrier. We provide a mathematical analysis and conduct a comprehensive empirical study across images, audio, video, NeRFs, and SDFs using standardized training protocols. Comparative analysis against SIREN, FINER, and related methods shows where HOSC provides substantial benefits and where it achieves competitive parity. Results establish HOSC as a practical periodic activation for INR applications, with domain-specific guidance on hyperparameter selection. For code visit the project page https://hosc-nn.github.io/ .

Figures (12)

• Figure 1: 2D image fitting with an identical coordinate MLP and fixed frequency $\omega_0 = 30$, varying the activation. From left to right: ground truth, SIREN $\sin(\omega_0 x)$, scaled sine $\beta\sin(\omega_0 x)$, and HOSC $\tanh(\beta\sin(\omega_0 z))$. SIREN and low-$\beta$ sine underfit high-frequency detail, large-$\beta$ sine produces unstable artifacts, while HOSC recovers sharp structure by combining high-frequency support with saturated, Lipschitz-controlled gradients.
• Figure 2: HOSC gradient control. Wrapping the sine in a hyperbolic tangent results in a periodic activation function that achieves saturation control through $\beta$. Derivative magnitude with Lipschitz bounds $L=\beta\omega_0$ (dashed). Increasing $\beta$ amplifies and localizes gradients at zero-crossings while maintaining bounded outputs.
• Figure 3: Empirical NTK behaviour of HOSC. Top: Correlation-normalized kernels for SIREN and HOSC with $\beta\in\{0.5,2,10\}$, averaged over multiple seeds. Larger $\beta$ sharpens the diagonal and attenuates off–diagonal structure. Bottom: Mean diagonal values and diagonal-dominance ratio (diag / off-diagonal std) as a function of $\beta$. Both increase steadily, indicating that $\beta$ acts as a controllable kernel-scale and interaction-strength knob.
• Figure 4: Effect of $\beta$ for HOSC and $\beta$-sin. Single DIV2K image, $\omega_0=30$. Top row: GT and HOSC for $\beta\in\{0.1,1,7,14\}$. Bottom row: GT and $\beta$-sin for the same $\beta$. All runs use the same INR architecture and training setup.
• Figure 5: Effect of $\omega_0$ for HOSC and SIREN. Single DIV2K image. Top row: HOSC with fixed $\beta=14$ and $\omega_0\in\{1,15,30,50\}$. Bottom row: SIREN with $\beta=1$ and the same $\omega_0$ values, using the same INR architecture and training setup.