Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpikingNeRF: Making Bio-inspired Neural Networks See through the Real World

Xingting Yao, Qinghao Hu, Fei Zhou, Tielong Liu, Zitao Mo, Zeyu Zhu, Zhengyang Zhuge, Jian Cheng

TL;DR

This paper proposes SpikingNeRF, which aligns the temporal dimension of spiking neural networks (SNNs) with the radiance rays, to seamlessly accommodate SNNs to the reconstruction of neural radiance fields (NeRF), and develops the temporal padding strategy to tackle the masked samples to maintain regular temporal length.

Abstract

In this paper, we propose SpikingNeRF, which aligns the temporal dimension of spiking neural networks (SNNs) with the radiance rays, to seamlessly accommodate SNNs to the reconstruction of neural radiance fields (NeRF). Thus, the computation turns into a spike-based, multiplication-free manner, reducing energy consumption and making high-quality 3D rendering, for the first time, accessible to neuromorphic hardware. In SpikingNeRF, each sampled point on the ray is matched to a particular time step and represented in a hybrid manner where the voxel grids are maintained as well. Based on the voxel grids, sampled points are determined whether to be masked out for faster training and inference. However, this masking operation also incurs irregular temporal length, making it intractable for hardware processors, e.g., GPUs, to conduct parallel training. To address this problem, we develop the temporal padding strategy to tackle the masked samples to maintain regular temporal length, i.e., regular tensors, and further propose the temporal condensing strategy to form a denser data structure for hardware-friendly computation. Experiments on various datasets demonstrate that our method can reduce energy consumption by an average of 70.79\% and obtain comparable synthesis quality with the ANN baseline. Verification on the neuromorphic hardware accelerator also shows that SpikingNeRF can further benefit from neuromorphic computing over the ANN baselines on energy efficiency. Codes and the appendix are in \url{https://github.com/Ikarosy/SpikingNeRF-of-CASIA}.

SpikingNeRF: Making Bio-inspired Neural Networks See through the Real World

TL;DR

This paper proposes SpikingNeRF, which aligns the temporal dimension of spiking neural networks (SNNs) with the radiance rays, to seamlessly accommodate SNNs to the reconstruction of neural radiance fields (NeRF), and develops the temporal padding strategy to tackle the masked samples to maintain regular temporal length.

Abstract

In this paper, we propose SpikingNeRF, which aligns the temporal dimension of spiking neural networks (SNNs) with the radiance rays, to seamlessly accommodate SNNs to the reconstruction of neural radiance fields (NeRF). Thus, the computation turns into a spike-based, multiplication-free manner, reducing energy consumption and making high-quality 3D rendering, for the first time, accessible to neuromorphic hardware. In SpikingNeRF, each sampled point on the ray is matched to a particular time step and represented in a hybrid manner where the voxel grids are maintained as well. Based on the voxel grids, sampled points are determined whether to be masked out for faster training and inference. However, this masking operation also incurs irregular temporal length, making it intractable for hardware processors, e.g., GPUs, to conduct parallel training. To address this problem, we develop the temporal padding strategy to tackle the masked samples to maintain regular temporal length, i.e., regular tensors, and further propose the temporal condensing strategy to form a denser data structure for hardware-friendly computation. Experiments on various datasets demonstrate that our method can reduce energy consumption by an average of 70.79\% and obtain comparable synthesis quality with the ANN baseline. Verification on the neuromorphic hardware accelerator also shows that SpikingNeRF can further benefit from neuromorphic computing over the ANN baselines on energy efficiency. Codes and the appendix are in \url{https://github.com/Ikarosy/SpikingNeRF-of-CASIA}.
Paper Structure (11 sections, 8 equations, 4 figures, 6 tables, 1 algorithm)

This paper contains 11 sections, 8 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Methodology
  4. Data encoding
  5. Time-ray alignment (TRA)
  6. Temporal condensing-and-padding (TCP)
  7. Overall algorithm
  8. Experiments
  9. Experimental settings
  10. Comparisons and ablations
  11. Conclusion

Figures (4)

  • Figure 1: Comparisons of our SpikingNeRF with other NeRF-based works in synthesis quality and model rendering energy. Different colors represent different works, and our SpikingNeRF with two different frameworks are denoted in red and violet, respectively. A detailed notation explanation is specified in the Experiments section. Different testing datasets are denoted by different shapes.
  • Figure 2: Conventional data encoding schemes. For direct-encoding, only the operation #1 is necessary that it duplicates the input data $T$ times to fit the length of the temporal dimension. For Poisson-encoding, both operation #1 and #2 are utilized to generate the input spike train. The “Mean” or “Voting” operation is able to decode the SNN output.
  • Figure 3: Overview of the proposed SpikingNeRF. (a) The rendering process of SpikingNeRF. The whole 3D volumetric parameters are stored in the voxel grids. The irrelevant or unimportant samples are masked before the sMLP querying. The expected scenes are rendered with the volumetric information yielded by the sMLP. (b) Alignment between the temporal dimension and the ray. The sMLP queries each sampled point step-by-step to yield the volumetric information. (c) Proposed temporal padding (left) and temporal condensing-and-padding (right) methods. For simplification, the channel length of the volumetric parameters is set to 1.
  • Figure :