Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HOIN: High-Order Implicit Neural Representations

Yang Chen, Ruituo Wu, Yipeng Liu, Ce Zhu

TL;DR

Spectral bias limits implicit neural representations (INRs) from accurately recovering high-frequency content in inverse problems. HOIN introduces a coding layer and High-Order Interaction Block to enlarge the INR function space and promote high-frequency learning, supported by Neural Tangent Kernel ($\mathcal{K}_{\mathrm{NTK}}$) analysis showing a diagonal-dominant structure. The paper provides theoretical results on expression ability, high-order derivatives, and the NTK perspective, proving HO blocks expand the leading functional space and improve high-frequency learning; it also demonstrates 1–3 dB improvements across tasks such as image denoising, super-resolution, CT reconstruction, and inpainting, while maintaining training efficiency. Overall, HOIN offers a universal INR framework that mitigates spectral bias and accelerates inverse problem solving across diverse domains.

Abstract

Implicit neural representations (INR) suffer from worsening spectral bias, which results in overly smooth solutions to the inverse problem. To deal with this problem, we propose a universal framework for processing inverse problems called \textbf{High-Order Implicit Neural Representations (HOIN)}. By refining the traditional cascade structure to foster high-order interactions among features, HOIN enhances the model's expressive power and mitigates spectral bias through its neural tangent kernel's (NTK) strong diagonal properties, accelerating and optimizing inverse problem resolution. By analyzing the model's expression space, high-order derivatives, and the NTK matrix, we theoretically validate the feasibility of HOIN. HOIN realizes 1 to 3 dB improvements in most inverse problems, establishing a new state-of-the-art recovery quality and training efficiency, thus providing a new general paradigm for INR and paving the way for it to solve the inverse problem.

HOIN: High-Order Implicit Neural Representations

TL;DR

Spectral bias limits implicit neural representations (INRs) from accurately recovering high-frequency content in inverse problems. HOIN introduces a coding layer and High-Order Interaction Block to enlarge the INR function space and promote high-frequency learning, supported by Neural Tangent Kernel () analysis showing a diagonal-dominant structure. The paper provides theoretical results on expression ability, high-order derivatives, and the NTK perspective, proving HO blocks expand the leading functional space and improve high-frequency learning; it also demonstrates 1–3 dB improvements across tasks such as image denoising, super-resolution, CT reconstruction, and inpainting, while maintaining training efficiency. Overall, HOIN offers a universal INR framework that mitigates spectral bias and accelerates inverse problem solving across diverse domains.

Abstract

Implicit neural representations (INR) suffer from worsening spectral bias, which results in overly smooth solutions to the inverse problem. To deal with this problem, we propose a universal framework for processing inverse problems called \textbf{High-Order Implicit Neural Representations (HOIN)}. By refining the traditional cascade structure to foster high-order interactions among features, HOIN enhances the model's expressive power and mitigates spectral bias through its neural tangent kernel's (NTK) strong diagonal properties, accelerating and optimizing inverse problem resolution. By analyzing the model's expression space, high-order derivatives, and the NTK matrix, we theoretically validate the feasibility of HOIN. HOIN realizes 1 to 3 dB improvements in most inverse problems, establishing a new state-of-the-art recovery quality and training efficiency, thus providing a new general paradigm for INR and paving the way for it to solve the inverse problem.
Paper Structure (42 sections, 5 theorems, 16 equations, 15 figures, 7 tables)

This paper contains 42 sections, 5 theorems, 16 equations, 15 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Inverse Problem
  4. Implicit Neural Representation Details
  5. Motivation
  6. HOIN: High-Order Implicit Neural Representations
  7. Overview
  8. Encoding Layer
  9. High-Order Interaction Block
  10. Rethinking Plain Block and Residual Block
  11. HO Block
  12. Theoretical Analysis of HOIN
  13. Expression Ability Exploration
  14. Derivative Analysis
  15. Neural Tangent Kernel Perspective
  16. ...and 27 more sections

Key Result

theorem 1

For an activation function with leading degree $r \geq 1$ and network architecture $\mathcal{D} =\left\{ D_1,...,D_l \right\}$, the leading functional variety of Plain Block, $\mathcal{V} _{\mathcal{D} ,r}^{P}$, HO Block, $\mathcal{V} _{\mathcal{D} ,r}^{HO}$, and Residual Block, $\mathcal{V} _{\math Proof. See Supplementary

Figures (15)

  • Figure 1: Overview of HOIN. We select the corresponding encoding layer based on the type of inverse problem, mapping the coordinate input $\mathbf{x}$ into a higher dimensional space $\gamma(\mathbf{x})$. Then, the low-frequency and high-frequency information in the signal is captured through a High-Order Block structure. During training, we find the peak performance point, stop the fitting process there, and find the solution $F_{\mathbf{\theta }}(\mathbf{x})$. $\odot$ denotes Hadamard product, $\boxplus$ is addition, $\varphi$ is the nonlinear activation function.
  • Figure 2: (a) Comparison of learning speeds at different frequencies. The target image is transformed into 10 frequency bands through the Fourier transform (x-axis, 0 represents the lowest frequency band), and we compare the learned components with the proper amplitude. On the color chart scale, 1 represents a perfect approximation. HO block can effectively alleviate spectral bias. Hash encoding does not exhibit spectral bias. (b) PSNR learning curves for different blocks. HO block maintains the highest PSNR.
  • Figure 3: (a) Visualization of NTK and corresponding eigenvalues in different models. (b) Draw the corresponding feature values. Because the maximum eigenvalue is much larger than the minimum eigenvalue, all eigenvalues are processed by logarithmic functions for visualization. HO blocks significantly enhance the eigenvalues on the diagonal of the NTK matrix, thus enhancing the ability of the INR to capture high-frequency information. Plain, residual, and high-order blocks are abbreviated as P, R, and HO.
  • Figure 4: Visualization of Image Representation. Here, we demonstrate the representation errors of different models. The brighter areas indicate higher representation errors. HO-FFN accurately reconstructs all the detailed information of the image.
  • Figure 5: Visualization of Computed Tomography Reconstruction. Here, we demonstrate various methods for CT-based reconstruction of $256 \times 256$ images at 100 angles. HO-FFN maintains the best reconstruction results.
  • ...and 10 more figures

Theorems & Definitions (6)

  • theorem 1
  • theorem 2
  • definition 1
  • proposition 1
  • theorem 3
  • theorem 4