Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Encoding Semantic Priors into the Weights of Implicit Neural Representation

Zhicheng Cai, Qiu Shen

TL;DR

The paper tackles the lack of semantic information in implicit neural representations by introducing SPW, a reparameterization that encodes semantic priors into INR weights through a fixed Semantic Neural Network (SNN) and trainable Weight Generation Networks (WGN). The semantic vector derived from multi-stage SNN features is fed to WGNs to produce per-layer INR weights, enabling semantic-aware mappings without increasing inference cost. Across image fitting, CT and MRI reconstruction, and NeRF-based view synthesis, SPW consistently improves performance and reduces weight redundancy, while ablations show the importance of high-level semantic features and a 3-layer WGN configuration. This approach broadens INR applicability and enhances representational capacity for vision-related signal representations with practical efficiency gains.

Abstract

Implicit neural representation (INR) has recently emerged as a promising paradigm for signal representations, which takes coordinates as inputs and generates corresponding signal values. Since these coordinates contain no semantic features, INR fails to take any semantic information into consideration. However, semantic information has been proven critical in many vision tasks, especially for visual signal representation. This paper proposes a reparameterization method termed as SPW, which encodes the semantic priors to the weights of INR, thus making INR contain semantic information implicitly and enhancing its representational capacity. Specifically, SPW uses the Semantic Neural Network (SNN) to extract both low- and high-level semantic information of the target visual signal and generates the semantic vector, which is input into the Weight Generation Network (WGN) to generate the weights of INR model. Finally, INR uses the generated weights with semantic priors to map the coordinates to the signal values. After training, we only retain the generated weights while abandoning both SNN and WGN, thus SPW introduces no extra costs in inference. Experimental results show that SPW can improve the performance of various INR models significantly on various tasks, including image fitting, CT reconstruction, MRI reconstruction, and novel view synthesis. Further experiments illustrate that model with SPW has lower weight redundancy and learns more novel representations, validating the effectiveness of SPW.

Encoding Semantic Priors into the Weights of Implicit Neural Representation

TL;DR

The paper tackles the lack of semantic information in implicit neural representations by introducing SPW, a reparameterization that encodes semantic priors into INR weights through a fixed Semantic Neural Network (SNN) and trainable Weight Generation Networks (WGN). The semantic vector derived from multi-stage SNN features is fed to WGNs to produce per-layer INR weights, enabling semantic-aware mappings without increasing inference cost. Across image fitting, CT and MRI reconstruction, and NeRF-based view synthesis, SPW consistently improves performance and reduces weight redundancy, while ablations show the importance of high-level semantic features and a 3-layer WGN configuration. This approach broadens INR applicability and enhances representational capacity for vision-related signal representations with practical efficiency gains.

Abstract

Implicit neural representation (INR) has recently emerged as a promising paradigm for signal representations, which takes coordinates as inputs and generates corresponding signal values. Since these coordinates contain no semantic features, INR fails to take any semantic information into consideration. However, semantic information has been proven critical in many vision tasks, especially for visual signal representation. This paper proposes a reparameterization method termed as SPW, which encodes the semantic priors to the weights of INR, thus making INR contain semantic information implicitly and enhancing its representational capacity. Specifically, SPW uses the Semantic Neural Network (SNN) to extract both low- and high-level semantic information of the target visual signal and generates the semantic vector, which is input into the Weight Generation Network (WGN) to generate the weights of INR model. Finally, INR uses the generated weights with semantic priors to map the coordinates to the signal values. After training, we only retain the generated weights while abandoning both SNN and WGN, thus SPW introduces no extra costs in inference. Experimental results show that SPW can improve the performance of various INR models significantly on various tasks, including image fitting, CT reconstruction, MRI reconstruction, and novel view synthesis. Further experiments illustrate that model with SPW has lower weight redundancy and learns more novel representations, validating the effectiveness of SPW.
Paper Structure (14 sections, 1 equation, 5 figures, 3 tables)

This paper contains 14 sections, 1 equation, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Proposed Method
  4. Generate the Semantic Vector
  5. Generate the Weights with Semantic Priors
  6. Training and Practical Application
  7. Experiments
  8. Image Fitting
  9. CT Reconstruction.
  10. MRI Reconstruction.
  11. Novel View Synthesis.
  12. Visualization
  13. Ablation Study
  14. Conclusion

Figures (5)

  • Figure 1: Pipeline of SPW. The structure of each component is also exhibited.
  • Figure 2: Rate distortion plots of various INR models w./w.o. SPW under different bpps trained on the Kodak dataset.
  • Figure 3: The similarity matrices of different layers of SIREN and SPW-SIREN trained on Kodak. A point with a darker color represents a larger value of KL divergence, hence a lower similarity. SPW SIREN has lower similarity compared to SIREN, indicating lower weight redundancy.
  • Figure 4: The weight distribution of SIREN, SPW SIREN, PE-MLP and SPW PE-MLP trained on Kodak dataset. The INR models with SPW have larger weight entropy compared to their original counterparts.
  • Figure 5: Activated feature maps output by the first layer of SIREN and SPW SIREN. SIREN learns similar redundant representations as marked by the boxes. While SPW SIREN can learn more distinctive representations.