DivAS: Interactive 3D Segmentation of NeRFs via Depth-Weighted Voxel Aggregation

TL;DR

DivAS tackles the challenge of interactive, zero-shot 3D segmentation for NeRFs by marrying depth-guided refinement of 2D SAM masks with a fast CUDA-based 3D voxel fusion. It operates without per-scene training, using Fibonacci-sphere anchor views and centroid-view prompts to generate refined masks that are robustly fused into a 3D occupancy grid. The method achieves segmentation quality comparable to optimization-based baselines while delivering 2–2.5× speedups end-to-end and up to an order of magnitude faster when excluding user prompting time. This optimization-free pipeline enables real-time feedback and practical 3D editing, bridging strong 2D foundation models with complex 3D geometry for efficient scene understanding.

Abstract

Existing methods for segmenting Neural Radiance Fields (NeRFs) are often optimization-based, requiring slow per-scene training that sacrifices the zero-shot capabilities of 2D foundation models. We introduce DivAS (Depth-interactive Voxel Aggregation Segmentation), an optimization-free, fully interactive framework that addresses these limitations. Our method operates via a fast GUI-based workflow where 2D SAM masks, generated from user point prompts, are refined using NeRF-derived depth priors to improve geometric accuracy and foreground-background separation. The core of our contribution is a custom CUDA kernel that aggregates these refined multi-view masks into a unified 3D voxel grid in under 200ms, enabling real-time visual feedback. This optimization-free design eliminates the need for per-scene training. Experiments on Mip-NeRF 360° and LLFF show that DivAS achieves segmentation quality comparable to optimization-based methods, while being 2-2.5x faster end-to-end, and up to an order of magnitude faster when excluding user prompting time.

Figures (4)

• Figure 1: Overview of DivAS. (a) Stage 1: Anchor and centroid view generation. The user provides point prompts on global-scale anchor views to select distinct object regions. Each selected point generates a zoomed-in novel centroid view$\mathcal{C}$ by moving the camera toward the point and setting it as the look-at target. (b) Stage 2: Interactive centroid view processing and 3D fusion. For each centroid view $c_j$ in an anchor group $\mathcal{A}_i$, SAM inference runs asynchronously while the user continues interacting with subsequent views. When the cache holds at least three processed masks or the group ends, all cached and current masks are synchronized, refined using inverse-normalized NeRF depth maps, and fused by a CUDA-based $3$D voxel kernel into a probabilistic occupancy grid $\mathcal{S}$. The resulting 3D mask is projected back onto the next centroid view for progressive refinement. (c) CUDA voxel fusion kernel. The kernel aggregates multi-view refined masks by enforcing geometric consistency through density, distance, and depth checks. It separately handles thick-structure voxels via direct projection alignment and thin-structure voxels through $2$D footprint coverage, producing the final $3$D occupancy grid.
• Figure 2: IoU vs cumulative-weight($\tau_{cw}$) target
• Figure 3: Results of different methods on LLFF
• Figure 4: Results of different methods on MIP-$360^\circ$