Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Dynamic Novel View Synthesis Technologies for Cinematography

Adrian Azzarelli, Nantheera Anantrasirichai, David R Bull

TL;DR

The paper investigates dynamic novel view synthesis for cinematography by contrasting NeRF and Gaussian Splatting (GS) representations, and by evaluating dynamic 4D approaches that separate canonical fields from motion via deformation fields or low-rank planes. It presents a spectrum of dynamic representations, from continuous deformation to key-frame interpolation and 4D primitives, and discusses data acquisition strategies, including depth initialization and COLMAP-based geometry. A production-style exhibition demonstrates three NVS configurations on a short scene, highlighting how dynamic NVS enables smooth camera trajectories, lighting variation, and non-rigid motion reconstruction, while also revealing challenges like temporal jitter and pose misalignment. The work underscores dynamic NVS as a practical tool for cinematography, capable of delivering high-quality background and motion effects with modest image counts, and points to directions such as improved calibration, dynamic masking, and depth-informed initialization to further close the gap with real-world shooting constraints.

Abstract

Novel view synthesis (NVS) has shown significant promise for applications in cinematographic production, particularly through the exploitation of Neural Radiance Fields (NeRF) and Gaussian Splatting (GS). These methods model real 3D scenes, enabling the creation of new shots that are challenging to capture in the real world due to set topology or expensive equipment requirement. This innovation also offers cinematographic advantages such as smooth camera movements, virtual re-shoots, slow-motion effects, etc. This paper explores dynamic NVS with the aim of facilitating the model selection process. We showcase its potential through a short montage filmed using various NVS models.

Exploring Dynamic Novel View Synthesis Technologies for Cinematography

TL;DR

The paper investigates dynamic novel view synthesis for cinematography by contrasting NeRF and Gaussian Splatting (GS) representations, and by evaluating dynamic 4D approaches that separate canonical fields from motion via deformation fields or low-rank planes. It presents a spectrum of dynamic representations, from continuous deformation to key-frame interpolation and 4D primitives, and discusses data acquisition strategies, including depth initialization and COLMAP-based geometry. A production-style exhibition demonstrates three NVS configurations on a short scene, highlighting how dynamic NVS enables smooth camera trajectories, lighting variation, and non-rigid motion reconstruction, while also revealing challenges like temporal jitter and pose misalignment. The work underscores dynamic NVS as a practical tool for cinematography, capable of delivering high-quality background and motion effects with modest image counts, and points to directions such as improved calibration, dynamic masking, and depth-informed initialization to further close the gap with real-world shooting constraints.

Abstract

Novel view synthesis (NVS) has shown significant promise for applications in cinematographic production, particularly through the exploitation of Neural Radiance Fields (NeRF) and Gaussian Splatting (GS). These methods model real 3D scenes, enabling the creation of new shots that are challenging to capture in the real world due to set topology or expensive equipment requirement. This innovation also offers cinematographic advantages such as smooth camera movements, virtual re-shoots, slow-motion effects, etc. This paper explores dynamic NVS with the aim of facilitating the model selection process. We showcase its potential through a short montage filmed using various NVS models.

Paper Structure

This paper contains 16 sections, 7 equations, 7 figures.

Table of Contents

  1. Introduction
  2. Technical Background
  3. Neural Radiance Fields
  4. Gaussian Splatting
  5. NeRF vs. GS
  6. Dynamic NVS for Cinematography
  7. Dynamic Representations
  8. Dynamic Scenes vs. Articulated 3D Objects
  9. Dynamic Scenes
  10. Articulated Humans/Objects
  11. Data Acquisition
  12. Exhibition: An emotional sip of tea
  13. Part I
  14. Part II
  15. Part III
  16. ...and 1 more sections

Figures (7)

  • Figure 1: Fly-through shot on a toy-scene
  • Figure 2: An illustration of the classical NVS pipeline. The learnable parts of the pipeline are highlighted in red
  • Figure 3: An illustration of the original NeRF and GS representations
  • Figure 4: Illustrations of the temporal deformation, hex-plane decomposition and key-frame interpolation approaches for dynamic neural 4D representation
  • Figure 5: Production plan: Our plan uses a storyboard for the actor and post-production staff to follow. We attached the scene constraints and visual plans for the camera man to collect each dataset
  • ...and 2 more figures