Spike-NeRF: Neural Radiance Field Based On Spike Camera

TL;DR

Spike-NeRF is proposed, the first Neural Radiance Field derived from spike data, to achieve 3D reconstruction and novel viewpoint synthesis for high-speed scenes and demonstrates that Spike-NeRF produces more visually appealing results than the existing methods and the baseline the authors proposed in high-speed scenes.

Abstract

As a neuromorphic sensor with high temporal resolution, spike cameras offer notable advantages over traditional cameras in high-speed vision applications such as high-speed optical estimation, depth estimation, and object tracking. Inspired by the success of the spike camera, we proposed Spike-NeRF, the first Neural Radiance Field derived from spike data, to achieve 3D reconstruction and novel viewpoint synthesis of high-speed scenes. Instead of the multi-view images at the same time of NeRF, the inputs of Spike-NeRF are continuous spike streams captured by a moving spike camera in a very short time. To reconstruct a correct and stable 3D scene from high-frequency but unstable spike data, we devised spike masks along with a distinctive loss function. We evaluate our method qualitatively and numerically on several challenging synthetic scenes generated by blender with the spike camera simulator. Our results demonstrate that Spike-NeRF produces more visually appealing results than the existing methods and the baseline we proposed in high-speed scenes. Our code and data will be released soon.

Figures (7)

• Figure 1: Existing works on NeRF (orange background) are reconstructed from image sequences generated by traditional cameras which record the luminance intensity during the exposure time at a fixed frame rate producing strong blur in high-speed scenes. Our approach (blue background) produces significantly better and sharper results by using dense spike streams instead of image sequences.
• Figure 2: Overview of our Spike-NeRF. Same as NeRF, we use color(C) and density($\sigma$) generated by MLPs as the input of volume renderer (equation (\ref{['nerf:qiuhe']})) and spiking volume renderer (equation (\ref{['spikenerf:qiuhe']})). We proposed reconstruct loss between the volume renderer result and masked images which are reconstructed from g.t. spike streams. Spike loss between the spike rendering result generated by our spiking volume renderer and g.t. spike streams is computed too.
• Figure 3: Comparisons on novel view synthesis. We compare our results with three baselines:NeRF, BAD-NeRF, and NeRF+Spk2ImgNet.More details are shown in the green box. NeRF and BAD-NeRF's results have significant blur while NeRF+Spk2ImgNet's results show artifacts. Our results are sharp. The supplement shows more details.
• Figure 4: Effective areas (white when r+g+b$>$0 and black when r+g+b=0) for different solutions. Compared with GT, there are obvious error areas when not using masks, and cavities when using single-frame masks. Our solution solves the above problems.
• Figure 5: Comparison between spike and image. Compared with images, spikes lack texture details and have a lot of noise because a single frame of spike has less information than an image.