SL$^{2}$A-INR: Single-Layer Learnable Activation for Implicit Neural Representation

TL;DR

SL$^{2}$A-INR tackles the spectral-bias challenge in implicit neural representations by introducing a Learnable Activation (LA) Block based on high-degree Chebyshev polynomials and coupling it with a lightweight, low-rank Fusion Block. The LA Block adaptively learns activation polynomials to capture high-frequency details, while the Fusion Block modulates its layers with LA information and preserves efficiency through low-rank linear layers and skip connections. Across 2D image fitting, 3D occupancy reconstruction, and Neural Radiance Fields, SL$^{2}$A delivers superior accuracy, robustness, and convergence, establishing new benchmarks and demonstrating strong generalization, including single-image super-resolution. The work provides detailed ablations, NTK-informed analysis, and practical guidelines for hyperparameters, highlighting the effectiveness of spectral-bias tuning via learnable activations for diverse INR tasks.

Abstract

Implicit Neural Representation (INR), leveraging a neural network to transform coordinate input into corresponding attributes, has recently driven significant advances in several vision-related domains. However, the performance of INR is heavily influenced by the choice of the nonlinear activation function used in its multilayer perceptron (MLP) architecture. To date, multiple nonlinearities have been investigated, but current INRs still face limitations in capturing high-frequency components and diverse signal types. We show that these challenges can be alleviated by introducing a novel approach in INR architecture. Specifically, we propose SL$^{2}$A-INR, a hybrid network that combines a single-layer learnable activation function with an MLP that uses traditional ReLU activations. Our method performs superior across diverse tasks, including image representation, 3D shape reconstruction, and novel view synthesis. Through comprehensive experiments, SL$^{2}$A-INR sets new benchmarks in accuracy, quality, and robustness for INR. Our Code is publicly available on~\href{https://github.com/Iceage7/SL2A-INR}{\textcolor{magenta}{GitHub}}.

Figures (11)

• Figure 1: An implicit neural representation method that enables flexible tuning of spectral bias through SL$^2$A-INR. The top diagram illustrates the network architecture, which begins with a Learnable Activation Block ($\Psi$) parameterized by Chebyshev polynomials of degree $K$, followed by a feature fusion block. This fusion block is implemented as a vanilla feed-forward network, where the input to each layer is modulated by the output of the first block ($\Psi$). The bottom panel provides a qualitative comparison of reconstruction quality. As the polynomial degree $K$ increases from 4 to 64, the results show a significant enhancement in detail and expressive power, highlighting the improved capacity for finer representations with larger $K$ values. Image adapted from chen2022cross.
• Figure 2: Comparison of convergence rate and frequency approximation error of different methods in 1d function fitting.
• Figure 3: Comparison of image representation of SL$^{2}$A with other methods.
• Figure 4: Results for occupancy (dragon shape) with different methods. The numbers in parentheses represent the IoU metric.
• Figure 5: Heatmap of mean PSNR across different batch sizes and learning rates (LR) for various methods on 5 random images from the DIV2K timofte2017ntire dataset. Our method, SL$^2$A, demonstrates greater robustness to variations in the learning rate and batch size compared to most other methods.