OpenFusion++: An Open-vocabulary Real-time Scene Understanding System

TL;DR

OpenFusion++ tackles real-time open-vocabulary 3D scene understanding by integrating confidence-guided 3D point sampling, an adaptive semantic cache, and a dual-branch hierarchical query framework that fuses SEEM object semantics with environment-aware CLIP features within a TSDF-based map. The system incrementally builds a structured 3D scene with per-voxel instance associations and multi-view semantic fusion to mitigate semantic drift, enabling precise boundary delineation and robust complex querying. Empirical results across ICL, Replica, ScanNet, and ScanNet++ show consistent gains in semantic accuracy and faster, more reliable query responses compared to the OpenFusion baseline, with notable improvements on fine-grained and nested queries. The work advances practical open-world perception for embodied AI, robotics, and AR by delivering improved boundary fidelity, global semantic consistency, and context-driven retrieval in real-time.

Abstract

Real-time open-vocabulary scene understanding is essential for efficient 3D perception in applications such as vision-language navigation, embodied intelligence, and augmented reality. However, existing methods suffer from imprecise instance segmentation, static semantic updates, and limited handling of complex queries. To address these issues, we present OpenFusion++, a TSDF-based real-time 3D semantic-geometric reconstruction system. Our approach refines 3D point clouds by fusing confidence maps from foundational models, dynamically updates global semantic labels via an adaptive cache based on instance area, and employs a dual-path encoding framework that integrates object attributes with environmental context for precise query responses. Experiments on the ICL, Replica, ScanNet, and ScanNet++ datasets demonstrate that OpenFusion++ significantly outperforms the baseline in both semantic accuracy and query responsiveness.

Figures (9)

• Figure 1: Overview of OpenFusion++ and its three core modules. 1) Real-time TSDF-based geometric reconstruction that dynamically fuses multi-view RGB-D data into sparse voxel blocks. 2) Dual-structure map consisting of a management-geometric map that organizes voxel blocks, and a semantic map that stores instance-level features (SEEM object semantics and CLIP spatial context) via adaptive caching. 3) Hierarchical query architecture that integrates object attributes and environmental features through a two-stage retrieval (coarse filtering and fine-grained matching) to resolve semantic queries.
• Figure 2: Semantic confusion problem. The left side of (a) presents segmentation results, where different colors represent different instances, while the right side displays the corresponding confidence maps. Darker regions indicate lower confidence, with black representing the background class. A zoomed-in view at the bottom highlights local details. (b) demonstrates a semantic confusion issue where the pillow is partially eroded by the sofa.
• Figure 3: Sampling strategies within voxel blocks. The figure compares random sampling and confidence-guided sampling within a single voxel block. Different colored squares represent 3D points from different instances.
• Figure 4: Semantic cache. The figure shows the process of dynamically inserting semantic embeddings. The weights of semantic embeddings are managed by inverting the physical coverage area as a minimum heap.
• Figure 5: Hierarchical retrieval framework. The query in the figure goes through an object extractor to get the semantic embedding of the query object, and the instance features from the image in the same feature space with the incremental mapping process for similarity computation, and the range of candidate instances is controlled by the parameter alpha. Alpha-clip matches the query's overall encoding with the instance environment features and filters the best instances within the candidate range.