Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ev4DGS: Novel-view Rendering of Non-Rigid Objects from Monocular Event Streams

Takuya Nakabayashi, Navami Kairanda, Hideo Saito, Vladislav Golyanik

TL;DR

This work introduces Ev4DGS, the first method for novel-view rendering of non-rigidly deforming objects from monocular event streams. It combines a time-dependent coarse deformation model with a deformable 4D Gaussian Splatting representation, trained end-to-end with event-based supervision and masks derived directly from events, eliminating the need for RGB inputs. The approach demonstrates state-of-the-art performance on synthetic and real datasets, validated through ablations that highlight the importance of silhouette supervision and staged optimization. Ev4DGS represents a significant step toward practical non-rigid 3D reconstruction from lightweight, low-power event sensors, enabling applications in AR/VR and robotics while reducing reliance on RGB cameras.

Abstract

Event cameras offer various advantages for novel view rendering compared to synchronously operating RGB cameras, and efficient event-based techniques supporting rigid scenes have been recently demonstrated in the literature. In the case of non-rigid objects, however, existing approaches additionally require sparse RGB inputs, which can be a substantial practical limitation; it remains unknown if similar models could be learned from event streams only. This paper sheds light on this challenging open question and introduces Ev4DGS, i.e., the first approach for novel view rendering of non-rigidly deforming objects in the explicit observation space (i.e., as RGB or greyscale images) from monocular event streams. Our method regresses a deformable 3D Gaussian Splatting representation through 1) a loss relating the outputs of the estimated model with the 2D event observation space, and 2) a coarse 3D deformation model trained from binary masks generated from events. We perform experimental comparisons on existing synthetic and newly recorded real datasets with non-rigid objects. The results demonstrate the validity of Ev4DGS and its superior performance compared to multiple naive baselines that can be applied in our setting. We will release our models and the datasets used in the evaluation for research purposes; see the project webpage: https://4dqv.mpi-inf.mpg.de/Ev4DGS/.

Ev4DGS: Novel-view Rendering of Non-Rigid Objects from Monocular Event Streams

TL;DR

This work introduces Ev4DGS, the first method for novel-view rendering of non-rigidly deforming objects from monocular event streams. It combines a time-dependent coarse deformation model with a deformable 4D Gaussian Splatting representation, trained end-to-end with event-based supervision and masks derived directly from events, eliminating the need for RGB inputs. The approach demonstrates state-of-the-art performance on synthetic and real datasets, validated through ablations that highlight the importance of silhouette supervision and staged optimization. Ev4DGS represents a significant step toward practical non-rigid 3D reconstruction from lightweight, low-power event sensors, enabling applications in AR/VR and robotics while reducing reliance on RGB cameras.

Abstract

Event cameras offer various advantages for novel view rendering compared to synchronously operating RGB cameras, and efficient event-based techniques supporting rigid scenes have been recently demonstrated in the literature. In the case of non-rigid objects, however, existing approaches additionally require sparse RGB inputs, which can be a substantial practical limitation; it remains unknown if similar models could be learned from event streams only. This paper sheds light on this challenging open question and introduces Ev4DGS, i.e., the first approach for novel view rendering of non-rigidly deforming objects in the explicit observation space (i.e., as RGB or greyscale images) from monocular event streams. Our method regresses a deformable 3D Gaussian Splatting representation through 1) a loss relating the outputs of the estimated model with the 2D event observation space, and 2) a coarse 3D deformation model trained from binary masks generated from events. We perform experimental comparisons on existing synthetic and newly recorded real datasets with non-rigid objects. The results demonstrate the validity of Ev4DGS and its superior performance compared to multiple naive baselines that can be applied in our setting. We will release our models and the datasets used in the evaluation for research purposes; see the project webpage: https://4dqv.mpi-inf.mpg.de/Ev4DGS/.

Paper Structure

This paper contains 20 sections, 10 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Monocular RGB-based Dynamic Novel-view Synthesis
  4. Event-based Novel-view Synthesis
  5. Rigid Scenes.
  6. Non-rigid Scenes.
  7. Preliminaries
  8. Method
  9. Coarse Stage: Deformation Model
  10. Fine Stage: Dynamic Novel View Synthesis
  11. Experiments
  12. Datasets
  13. Compared Methods
  14. Results and Comparisons
  15. Ablations
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Overview of the proposed Ev4DGS method. We divide the event-based reconstruction of non-rigid scenes into two stages. In the first stage, we train the coarse deformation model, which represents a non-rigid object shape as a set of points and enables the representation of large 3D deformations in a scene. In the second stage, we obtain the 3DGS representation from an event stream. Best viewed with zoom.
  • Figure 2: Qualitative results on the synthetic dataset (novel views). Our Ev4DGS outputs high-quality novel-view rendering as expected in this challenging setting. Competing methods miss object parts and details in the object interiors.
  • Figure 3: Qualitative results on our real dataset. Our Ev4DGS synthesises the spatially (shape, texture) and temporally coherent novel views that match the GT test views, while the competing methods fail.
  • Figure 4: Hyperparameter selection and the average performance of the proposed method on three synthetic sequences.
  • Figure I: Real data recording setup.