InstDrive: Instance-Aware 3D Gaussian Splatting for Driving Scenes

TL;DR

InstDrive tackles the lack of explicit 3D instance representations in open driving scenes by extending 3D Gaussian Splatting with per-Gaussian instance features and a two-stage training pipeline. It first learns continuous, view-consistent features through SAM-based 2D contrastive supervision and voxel-based 3D regularization, then encodes these features into discrete instance IDs using a static binary codebook with a pseudo-supervision mechanism. A key contribution is the explicit design of a fixed $2^d$ codebook and majority-vote pseudo labels, enabling end-to-end 3D instance segmentation and real-time interactive editing of 3D Gaussians without ground-truth IDs. Experiments on PandaSet demonstrate improved 2D and 3D instance segmentation quality and showcase practical interactive capabilities for editing driving scenes, underscoring the method's potential for autonomous driving, simulation, and digital twin applications.

Abstract

Reconstructing dynamic driving scenes from dashcam videos has attracted increasing attention due to its significance in autonomous driving and scene understanding. While recent advances have made impressive progress, most methods still unify all background elements into a single representation, hindering both instance-level understanding and flexible scene editing. Some approaches attempt to lift 2D segmentation into 3D space, but often rely on pre-processed instance IDs or complex pipelines to map continuous features to discrete identities. Moreover, these methods are typically designed for indoor scenes with rich viewpoints, making them less applicable to outdoor driving scenarios. In this paper, we present InstDrive, an instance-aware 3D Gaussian Splatting framework tailored for the interactive reconstruction of dynamic driving scene. We use masks generated by SAM as pseudo ground-truth to guide 2D feature learning via contrastive loss and pseudo-supervised objectives. At the 3D level, we introduce regularization to implicitly encode instance identities and enforce consistency through a voxel-based loss. A lightweight static codebook further bridges continuous features and discrete identities without requiring data pre-processing or complex optimization. Quantitative and qualitative experiments demonstrate the effectiveness of InstDrive, and to the best of our knowledge, it is the first framework to achieve 3D instance segmentation in dynamic, open-world driving scenes.More visualizations are available at our project page.

Figures (5)

• Figure 1: Existing reconstruction methods fail to achieve structured 3D reconstruction with instance-level editability in dynamic driving scenes. To address this, we propose InstDrive, which directly supervises training using SAM-processed video frames without requiring instance matching. We employ a shared 2D–3D color map to enable bijection between instance IDs and colors. During real-time rendering, trained Gaussians are assigned full opacity and colored according to their instance IDs. By capturing click events in the pixel space and retrieving the corresponding color, we map it back to the instance ID using the color map and select all Gaussians associated with that ID, enabling real-time, interactive selection and manipulation of 3D Gaussian instances.
• Figure 2: Framework Overview. We extend Gaussian attributes with an instance feature dimension and train the scene using multi-view images and LiDAR points. A contrastive loss and a voting-based pseudo-supervision loss guide 2D feature learning, while a voxel-based consistency loss enforces 3D coherence by aligning nearby Gaussians. Both 2D and 3D features are mapped to discrete instance IDs via a binarized static codebook.
• Figure 3: 2D segmentation results. We present 2D segmentation results across diverse scenes and conditions. The first column shows six-view camera inputs arranged in a 2×3 grid (from top to bottom, left to right): front-left, front, front-right, left, back, and right views. The second, third, and fourth columns display results from OpenGS, our method, and GSGroup, respectively. Due to preprocessing failures when GSGroup switches between multiple camera views, we only include its result under the forward-facing view. As shown in the visualizations, our method produces consistent multi-view segmentations, whereas baseline methods often exhibit distortions and significant noise (best viewed in color and zoomed in).
• Figure 4: Comparison of point-level instance segmentation results. The first row shows the forward-facing camera image of the corresponding scene, followed by results from GSGroup, OpenGS, our method without voxel loss, and the full version of our method. As observed from the visualizations, baseline methods often produce fragmented segmentation with noisy points within the same instance. In contrast, our method yields more coherent results, and the incorporation of voxel loss further enhances 3D consistency, resulting in cleaner and more complete instance masks.
• Figure 5: 3D instance segmentation results. Due to GSGroup’s limited applicability, we compare only OpenGS and our method. The first and second rows show results from OpenGS and our method, respectively. Each region represents an instance, with overlaps indicating instance coupling. We present results across diverse scenes and object scales, including street lamps, traffic signs, buildings, and road surfaces. Our method produces more complete segmentations with less scattered points and artifacts, while OpenGS suffers from noise and instance entanglement.