Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Dynamic-NeRF Based Volumetric Video Coding with Rate Distortion Optimization

Zhiyu Zhang, Guo Lu, Huanxiong Liang, Anni Tang, Qiang Hu, Li Song

TL;DR

This work decomposes the NeRF representation into the coefficient fields and the basis fields, incrementally updating the basis fields in the temporal domain to achieve dynamic modeling and performs end-to-end joint optimization on the modeling and compression process to further improve the compression efficiency.

Abstract

Volumetric videos, benefiting from immersive 3D realism and interactivity, hold vast potential for various applications, while the tremendous data volume poses significant challenges for compression. Recently, NeRF has demonstrated remarkable potential in volumetric video compression thanks to its simple representation and powerful 3D modeling capabilities, where a notable work is ReRF. However, ReRF separates the modeling from compression process, resulting in suboptimal compression efficiency. In contrast, in this paper, we propose a volumetric video compression method based on dynamic NeRF in a more compact manner. Specifically, we decompose the NeRF representation into the coefficient fields and the basis fields, incrementally updating the basis fields in the temporal domain to achieve dynamic modeling. Additionally, we perform end-to-end joint optimization on the modeling and compression process to further improve the compression efficiency. Extensive experiments demonstrate that our method achieves higher compression efficiency compared to ReRF on various datasets.

Efficient Dynamic-NeRF Based Volumetric Video Coding with Rate Distortion Optimization

TL;DR

This work decomposes the NeRF representation into the coefficient fields and the basis fields, incrementally updating the basis fields in the temporal domain to achieve dynamic modeling and performs end-to-end joint optimization on the modeling and compression process to further improve the compression efficiency.

Abstract

Volumetric videos, benefiting from immersive 3D realism and interactivity, hold vast potential for various applications, while the tremendous data volume poses significant challenges for compression. Recently, NeRF has demonstrated remarkable potential in volumetric video compression thanks to its simple representation and powerful 3D modeling capabilities, where a notable work is ReRF. However, ReRF separates the modeling from compression process, resulting in suboptimal compression efficiency. In contrast, in this paper, we propose a volumetric video compression method based on dynamic NeRF in a more compact manner. Specifically, we decompose the NeRF representation into the coefficient fields and the basis fields, incrementally updating the basis fields in the temporal domain to achieve dynamic modeling. Additionally, we perform end-to-end joint optimization on the modeling and compression process to further improve the compression efficiency. Extensive experiments demonstrate that our method achieves higher compression efficiency compared to ReRF on various datasets.
Paper Structure (10 sections, 9 equations, 4 figures, 2 tables)

This paper contains 10 sections, 9 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Preliminaries
  4. Compact Dynamic NeRF Representation
  5. End-to-end Optimization
  6. Experiments
  7. Implementation Details
  8. Experimental Results
  9. Ablation Studies
  10. Conclusion

Figures (4)

  • Figure 1: The rendering results of our method in comparison with ReRF.
  • Figure 2: Training pipeline for the proposed method. (a) Initially, we load the basis field at the previous time step from the decoded frame buffer. During training, we only update the coefficient field and the residual field. (b) During end-to-end optimization, we estimate the rate of the coefficient field and the residual field as loss, using simulated quantization during the forward pass.
  • Figure 3: The performance comparison of ReRF and our method in both ReRF and Dna-rendering dataset. BD-rate: Dna-rendering train:-66.54%, Dna-rendering test:-28.71%, ReRF train:-45.50%, ReRF test:-33.24%.
  • Figure 4: Comparing the multi-view rendering results of our approach and ReRF on the DNA-rendering sequence, under similar bitrate.