Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NeRFCom: Feature Transform Coding Meets Neural Radiance Field for Free-View 3D Scene Semantic Transmission

Weijie Yue, Zhongwei Si, Bolin Wu, Sixian Wang, Xiaoqi Qin, Kai Niu, Jincheng Dai, Ping Zhang

TL;DR

NeRFCom tackles the challenge of transmitting high-dimensional NeRF features for free-view 3D scene synthesis over imperfect channels. It integrates a neural nonlinear transform, a learned entropy model, and a variable-rate joint source-channel coding (JSCC) module, all optimized end-to-end under a rate-distortion objective to allocate bandwidth where it most improves 3D fidelity. The approach yields graceful degradation under noise, efficient bandwidth use by concentrating resources on scene-relevant regions, and superior rate-distortion performance compared with baselines that rely on separate source and channel coding. This enables robust, bandwidth-efficient free-view synthesis suitable for VR/AR streaming, with practical implications for bandwidth-constrained 3D content transmission.

Abstract

We introduce NeRFCom, a novel communication system designed for end-to-end 3D scene transmission. Compared to traditional systems relying on handcrafted NeRF semantic feature decomposition for compression and well-adaptive channel coding for transmission error correction, our NeRFCom employs a nonlinear transform and learned probabilistic models, enabling flexible variable-rate joint source-channel coding and efficient bandwidth allocation aligned with the NeRF semantic feature's different contribution to the 3D scene synthesis fidelity. Experimental results demonstrate that NeRFCom achieves free-view 3D scene efficient transmission while maintaining robustness under adverse channel conditions.

NeRFCom: Feature Transform Coding Meets Neural Radiance Field for Free-View 3D Scene Semantic Transmission

TL;DR

NeRFCom tackles the challenge of transmitting high-dimensional NeRF features for free-view 3D scene synthesis over imperfect channels. It integrates a neural nonlinear transform, a learned entropy model, and a variable-rate joint source-channel coding (JSCC) module, all optimized end-to-end under a rate-distortion objective to allocate bandwidth where it most improves 3D fidelity. The approach yields graceful degradation under noise, efficient bandwidth use by concentrating resources on scene-relevant regions, and superior rate-distortion performance compared with baselines that rely on separate source and channel coding. This enables robust, bandwidth-efficient free-view synthesis suitable for VR/AR streaming, with practical implications for bandwidth-constrained 3D content transmission.

Abstract

We introduce NeRFCom, a novel communication system designed for end-to-end 3D scene transmission. Compared to traditional systems relying on handcrafted NeRF semantic feature decomposition for compression and well-adaptive channel coding for transmission error correction, our NeRFCom employs a nonlinear transform and learned probabilistic models, enabling flexible variable-rate joint source-channel coding and efficient bandwidth allocation aligned with the NeRF semantic feature's different contribution to the 3D scene synthesis fidelity. Experimental results demonstrate that NeRFCom achieves free-view 3D scene efficient transmission while maintaining robustness under adverse channel conditions.

Paper Structure

This paper contains 17 sections, 6 equations, 6 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. System Model and Methods
  3. System Model
  4. Variable-Rate Feature Coded Transmission
  5. Optimization Goal and Strategy
  6. Feature Transmission Loss
  7. End-to-End Reconstruction Loss
  8. Experimental Results
  9. Experimental Setup
  10. Training Details
  11. Comparison Schemes
  12. Metrics
  13. Experimental Results
  14. Free-View 3D Scene Transmission Results
  15. Variable-Rate Feature Coding Results
  16. ...and 2 more sections

Figures (6)

  • Figure 1: The framework of our end-to-end free-view 3D scene transmission system.
  • Figure 2: Free-view 3D scene transmission results. The first image depicts the original 3D scene (a microphone), and the subsequent images display the rendering results from various viewpoints using the transmitted features.
  • Figure 3: Bandwidth allocation for different view images. Less bandwidth is allocated to blank areas in the viewpoint images, while more bandwidth is allocated to scene-relevant areas. The allocation is correlated with the spatial distribution of NeRF features.
  • Figure 4: PSNR and SSIM performance versus the average channel bandwidth ratio (CBR) at channel $\text{SNR}=10 \text{ dB}$.
  • Figure 5: Examples of visual comparisons, with each result zoomed in on three specific details. The corresponding metric values and relative percentages are displayed below each result.
  • ...and 1 more figures