FlexDrive: Toward Trajectory Flexibility in Driving Scene Reconstruction and Rendering

TL;DR

FlexDrive tackles out-of-path viewpoint rendering in driving scenes by introducing Inverse View Warping to generate high-quality supervision for views away from the vehicle trajectory, supported by Depth Bootstrapping to produce dense depth maps on the fly. Dynamic objects are constrained with separate Gaussian fields within trainable bounding boxes, and a carefully designed loss combines RGB and depth signals to optimize both in-path and out-of-path views. The method, validated on the Waymo Open Dataset and a CARLA-based benchmark, achieves competitive in-path rendering and notable gains in out-of-path rendering, significantly reducing artifacts and floaters. This work enables more flexible and realistic driving scene synthesis, with potential for completely free camera movement in future work.

Abstract

Driving scene reconstruction and rendering have advanced significantly using the 3D Gaussian Splatting. However, most prior research has focused on the rendering quality along a pre-recorded vehicle path and struggles to generalize to out-of-path viewpoints, which is caused by the lack of high-quality supervision in those out-of-path views. To address this issue, we introduce an Inverse View Warping technique to create compact and high-quality images as supervision for the reconstruction of the out-of-path views, enabling high-quality rendering results for those views. For accurate and robust inverse view warping, a depth bootstrap strategy is proposed to obtain on-the-fly dense depth maps during the optimization process, overcoming the sparsity and incompleteness of LiDAR depth data. Our method achieves superior in-path and out-of-path reconstruction and rendering performance on the widely used Waymo Open dataset. In addition, a simulator-based benchmark is proposed to obtain the out-of-path ground truth and quantitatively evaluate the performance of out-of-path rendering, where our method outperforms previous methods by a significant margin.

Figures (8)

• Figure 1: We simulate a cut-in case in a high-speed scenario, which is a typical functionality in driving simulators. The representative method PVG chen2023periodic fails after the lane change. We provide more video demonstrations in the attached supplementary materials.
• Figure 2: The main framework of FlexDrive, we provide high-quality supervision for out-of-path views through Inverse View Warping technique (IVW). To facilitate IVW, we propose Depth Bootstrapping (DB) to guarantee an accurate and dense depth map.
• Figure 3: Pipeline for Inverse Views Warping. Firstly, we lift every pixel in the in-path view to 3D space and then project them on a randomly sampled out-of-path view. Our target is to render the original in-path view at this newly sampled location, however, occlusion is inevitable due to the change of view. For this reason, we utilize our Occlusion-aware Rasterization. Finally, we rearrange the rendering points in the out-of-path view according to their pixel coordinates at the in-path view to form a regular image.
• Figure 4: Example of rearrangement. (a) is the ground truth in-path view. (b) and (c) are the warped rendering results in out-of-path views (right and left shifted, respectively). Note we ignore those rendered-pixels out of image boundaries in this illustration. However, in practice, we still keep the out-of-boundary pixels and rearrange them. (d) is the rearranged rendering results in out-of-path views, which is exactly the same as the GT image (a).
• Figure 5: The strong linear prior between LiDAR depth and GS-rendered depth.