Event-boosted Deformable 3D Gaussians for Dynamic Scene Reconstruction

TL;DR

This paper tackles the challenge of reconstructing dynamic scenes from data with both high temporal resolution and motion blur by combining event cameras with deformable 3D-Gaussian Splatting (3D-GS). It introduces GS-threshold Joint Modeling (GTJM) to jointly estimate event thresholds and geometry using RGB cues and pseudo-intermediate frames, and Dynamic-Static Decomposition (DSD) to isolate dynamic regions for targeted deformation, plus a one-minute 3D-to-2D decomposition. The method is evaluated on a new event-inclusive 4D benchmark with synthetic and real scenes, achieving state-of-the-art PSNR/SSIM/LPIPS and faster rendering compared with RGB-only and prior event-based baselines (e.g., gains of several dB in PSNR). These contributions enable more faithful and efficient dynamic scene capture for AR/VR, robotics, and animation.

Abstract

Deformable 3D Gaussian Splatting (3D-GS) is limited by missing intermediate motion information due to the low temporal resolution of RGB cameras. To address this, we introduce the first approach combining event cameras, which capture high-temporal-resolution, continuous motion data, with deformable 3D-GS for dynamic scene reconstruction. We observe that threshold modeling for events plays a crucial role in achieving high-quality reconstruction. Therefore, we propose a GS-Threshold Joint Modeling strategy, creating a mutually reinforcing process that greatly improves both 3D reconstruction and threshold modeling. Moreover, we introduce a Dynamic-Static Decomposition strategy that first identifies dynamic areas by exploiting the inability of static Gaussians to represent motions, then applies a buffer-based soft decomposition to separate dynamic and static areas. This strategy accelerates rendering by avoiding unnecessary deformation in static areas, and focuses on dynamic areas to enhance fidelity. Additionally, we contribute the first event-inclusive 4D benchmark with synthetic and real-world dynamic scenes, on which our method achieves state-of-the-art performance.

Figures (11)

• Figure 1: Overview of GS-threshold joint modeling strategy. $\mathcal{L}_{rgb}$ optimizes 3D-GS, $\mathcal{L}_{thres}$ optimizes the threshold, and $\mathcal{L}_{event}$ jointly optimizes both 3D-GS and threshold.
• Figure 2: (a) The effect of different ranges of threshold variation for 3D reconstruction. (b) The effect of different number of RGB frames for threshold estimation.
• Figure 3: The effect of dynamic-static decomposition strategy, which improves the rendering quality of dynamic regions.
• Figure 4: Overview of dynamic-static decomposition strategy. First, we decompose dynamic and static regions in 2D images based on the inherent inability of static Gaussians to represent motions. Next, we establish a correspondence to extend 2D decomposition to 3D Gaussians. Finally, the decomposed dynamic and static Gaussians are jointly rendered to reconstruct the complete dynamic scene.
• Figure 5: Real-world data acquisition setup (left) and our hybrid camera system (right).