Leveraging 2D Priors and SDF Guidance for Dynamic Urban Scene Rendering

TL;DR

This work tackles dynamic urban scene rendering without heavy 3D motion annotations or LiDAR, by marrying Signed Distance Functions with 3D Gaussian Splatting (UGSDF) and leveraging 2D priors such as depth networks and point trackers. The method builds a canonical object representation, learns a motion model from basis trajectories, and employs a deformation network to jointly optimize geometry and appearance. It demonstrates strong reconstruction and novel-view synthesis on driving datasets without requiring 3D templates or tracklets, and enables scene editing like object removal and composition. The approach offers robust generalization to casual data and provides a practical framework for urban scene understanding and simulation where 3D supervision is scarce.

Abstract

Dynamic scene rendering and reconstruction play a crucial role in computer vision and augmented reality. Recent methods based on 3D Gaussian Splatting (3DGS), have enabled accurate modeling of dynamic urban scenes, but for urban scenes they require both camera and LiDAR data, ground-truth 3D segmentations and motion data in the form of tracklets or pre-defined object templates such as SMPL. In this work, we explore whether a combination of 2D object agnostic priors in the form of depth and point tracking coupled with a signed distance function (SDF) representation for dynamic objects can be used to relax some of these requirements. We present a novel approach that integrates Signed Distance Functions (SDFs) with 3D Gaussian Splatting (3DGS) to create a more robust object representation by harnessing the strengths of both methods. Our unified optimization framework enhances the geometric accuracy of 3D Gaussian splatting and improves deformation modeling within the SDF, resulting in a more adaptable and precise representation. We demonstrate that our method achieves state-of-the-art performance in rendering metrics even without LiDAR data on urban scenes. When incorporating LiDAR, our approach improved further in reconstructing and generating novel views across diverse object categories, without ground-truth 3D motion annotation. Additionally, our method enables various scene editing tasks, including scene decomposition, and scene composition.

Figures (9)

• Figure 1: We propose Urban Gaussians via Signed Distance Functions (UGSDF) for dynamic object modeling and rendering in urban scenes. UGSDF maintains a Signed Distance Function (SDF) and 3D Gaussian Splatting (3DGS) representation to model and render a dynamic object. It can be used for object removal, scene decomposition, scene composition (inserted object indicated with green arrow) and other tasks related to simulation in urban scenes.
• Figure 2: Overview UGSDF takes posed images, dynamic object masks, 2D tracking data, and depth maps (optionally LiDAR) as input, and outputs rendered dynamic scene. The initial model is constructed through canonicalization described in \ref{['subsec:init']}. A dynamic 3DGS motion representation described in \ref{['subsec:dyn-repr']}. The object of interest is represented using a combined representation of Signed Distance Functions (SDFs) and 3D Gaussian Splatting (3DGS) as detailed in \ref{['subsec:def-net']} and the joint optimization scheme is described in \ref{['subsec:sdf-3dgs']}. These two representations are jointly learned. The coarse geometry of the Gaussians constrains the ray sampling of the SDF, while SDF queries add detail to the Gaussian representation.
• Figure 3: SDF Deformation Network The network takes as input the observed point $\mathbf{x}_{i,t}$ and outputs an SDF value $S_i(t)$ via the $\varphi_{sdf}$ MLP.
• Figure 4: SDF Guidance for Gaussian Primitive Distribution Densification and Pruning of Gaussians is done by querying the SDF network. Points far away from the estimated SDF zero-level set are pruned, while sparse regions close to the zero-level set are chosen for densification. Dens. denotes densification.
• Figure 5: Ablation Analysis Our ablation analysis confirms the need for dense representations of thin objects like pedestrians and cyclists (rows 2 and 3). Furthermore, removing SDF guidance for Gaussian primitive distribution adversely impacts rendering quality (rows 1 and 3).