WildSeg3D: Segment Any 3D Objects in the Wild from 2D Images

TL;DR

WildSeg3D tackles the challenge of real-time 3D segmentation of arbitrary objects from 2D images without scene-specific training. It combines a feed-forward pointmap-based reconstruction with Dynamic Global Aligning (DGA) to focus on hard-to-match points and a Multi-view Group Mapping (MGM) that uses a precomputed mask cache to integrate multi-view masks in real time. The approach delivers state-of-the-art-like accuracy while achieving about a $40x$ speedup over prior SOTA methods and enables near real-time interactive segmentation on sparse view data. By generalizing across diverse scenes and removing scene-specific training requirements, WildSeg3D offers a practical, scalable solution for live 3D segmentation in the wild, with code to be released publicly.

Abstract

Recent advances in interactive 3D segmentation from 2D images have demonstrated impressive performance. However, current models typically require extensive scene-specific training to accurately reconstruct and segment objects, which limits their applicability in real-time scenarios. In this paper, we introduce WildSeg3D, an efficient approach that enables the segmentation of arbitrary 3D objects across diverse environments using a feed-forward mechanism. A key challenge of this feed-forward approach lies in the accumulation of 3D alignment errors across multiple 2D views, which can lead to inaccurate 3D segmentation results. To address this issue, we propose Dynamic Global Aligning (DGA), a technique that improves the accuracy of global multi-view alignment by focusing on difficult-to-match 3D points across images, using a dynamic adjustment function. Additionally, for real-time interactive segmentation, we introduce Multi-view Group Mapping (MGM), a method that utilizes an object mask cache to integrate multi-view segmentations and respond rapidly to user prompts. WildSeg3D demonstrates robust generalization across arbitrary scenes, thereby eliminating the need for scene-specific training. Specifically, WildSeg3D not only attains the accuracy of state-of-the-art (SOTA) methods but also achieves a $40\times$ speedup compared to existing SOTA models. Our code will be publicly available.

Figures (11)

• Figure 1: Framework of WildSeg3D. WildSeg3D operates in three stages. First, during the pre-processing stage, 2D feature masks are constructed from multi-view images using SAM2, providing offline support for interactive segmentation. Second, in the Dynamic Global Alignment (DGA) stage, a dynamic weight adjustment strategy is applied to achieve global alignment of the pointmaps generated by MASt3r, thereby improving the accuracy of object reconstruction. Finally, in the Multi-view Group Mapping (MGM) stage, multi-view masks for the target are retrieved from the mask cache based on user input, and these masks are transformed into an aligned 3D space in real time, where "P2W" refers to the transformation from pixel coordinates to the aligned world coordinates.
• Figure 2: Visualization on the NVOS dataset. (a)-(d) show the sparse view reconstruction and timing results on horns, trex, orchids, and fortress scenes, including preprocessing and DGA-based scene reconstruction. Target objects for segmentation are marked with red dashed lines in the first column. From left to right: segmentation results and elapsed time from prompt input to segmentation acquisition across models.
• Figure 3: Performance of WildSeg3D on indoor and outdoor scenes. For each scene, the left side shows the sparse views for reconstruction, with the segmentation target indicated by red dashed lines in the first view as prompts.
• Figure 4: Visualization of ablation experiments on DGA.
• Figure 5: Ablation Experiments on NVOS Dataset. Left: visualization of different confidence adjustment functions for DGA. Right: Results of these functions.