3DGS-CD: 3D Gaussian Splatting-based Change Detection for Physical Object Rearrangement

TL;DR

3DGS-CD is presented, the first 3D Gaussian Splatting (3DGS)-based method for detecting physical object rearrangements in 3D scenes, achieving up to 14% higher accuracy and three orders of magnitude faster performance compared to the state-of-the-art radiance-field-based change detection method.

Abstract

We present 3DGS-CD, the first 3D Gaussian Splatting (3DGS)-based method for detecting physical object rearrangements in 3D scenes. Our approach estimates 3D object-level changes by comparing two sets of unaligned images taken at different times. Leveraging 3DGS's novel view rendering and EfficientSAM's zero-shot segmentation capabilities, we detect 2D object-level changes, which are then associated and fused across views to estimate 3D change masks and object transformations. Our method can accurately identify changes in cluttered environments using sparse (as few as one) post-change images within as little as 18s. It does not rely on depth input, user instructions, pre-defined object classes, or object models -- An object is recognized simply if it has been re-arranged. Our approach is evaluated on both public and self-collected real-world datasets, achieving up to 14% higher accuracy and three orders of magnitude faster performance compared to the state-of-the-art radiance-field-based change detection method. This significant performance boost enables a broad range of downstream applications, where we highlight three key use cases: object reconstruction, robot workspace reset, and 3DGS model update. Our code and data will be made available at https://github.com/520xyxyzq/3DGS-CD.

Figures (9)

• Figure 1: Our method detects 3D object-level changes from pre- and potentially sparse post-change images of a 3D scene. We first train a 3DGS model on pre-change images (Sec.\ref{['sec:3dgs']}), localize the post-change cameras with respect to this model (Sec.\ref{['sec:loc']}), and render RGB-D images at post-change views for 2D change detection using EfficientSAM xiong2024efficientsam (Sec.\ref{['sec:2d']}). The detected 2D object segments are associated across the post-change views (Sec. \ref{['sec:ass']}) to initialize 3D object templates. These templates are used to classify object change types and query EfficientSAM on pre-change views to obtain additional 2D object segments, which are fused to obtain 3D object segments (Sec. \ref{['sec:seg']}). For moved objects, we leverage image-template feature correspondences to estimate their 6D pose changes (Sec. \ref{['sec:pose_est']}) and refine the estimates by a render-and-compare approach (Sec. \ref{['sec:global']}).
• Figure 2: EfficientSAM embedding for 2D change detection at a post-change view (Sec. \ref{['sec:2d']}). While the 2D object masks may be incomplete or missing due to occlusions or prediction failures, associating and fusing masks from multiple views (Sec. \ref{['sec:ass']}) help recover more complete object templates.
• Figure 3: Qualitative results: Estimated 3D object masks projected on novel views. We show the projected object move-out (previous location) and move-in (new location) masks on the pre-change 3DGS render (left) and the post-change capture (right) from the same novel viewpoint for each scene. Please check out the supplementary video for mask projections on more views.
• Figure 4: Qualitative comparison of our method against SOTA 2D change detection method MTP wang2024mtp on Desk.
• Figure 5: Ablation study on the number of post-change images on Cats. Our method maintains consistent performance across different numbers of post-change images.