Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation

TL;DR

HOV-SG, a hierarchical open-vocabulary 3D scene graph mapping approach for language-grounded robot navigation that surpasses previous baselines in open-vocabulary semantic accuracy on the object, room, and floor level while producing a 75% reduction in representation size compared to dense open-vocabulary maps is presented.

Abstract

Recent open-vocabulary robot mapping methods enrich dense geometric maps with pre-trained visual-language features. While these maps allow for the prediction of point-wise saliency maps when queried for a certain language concept, large-scale environments and abstract queries beyond the object level still pose a considerable hurdle, ultimately limiting language-grounded robotic navigation. In this work, we present HOV-SG, a hierarchical open-vocabulary 3D scene graph mapping approach for language-grounded robot navigation. Leveraging open-vocabulary vision foundation models, we first obtain state-of-the-art open-vocabulary segment-level maps in 3D and subsequently construct a 3D scene graph hierarchy consisting of floor, room, and object concepts, each enriched with open-vocabulary features. Our approach is able to represent multi-story buildings and allows robotic traversal of those using a cross-floor Voronoi graph. HOV-SG is evaluated on three distinct datasets and surpasses previous baselines in open-vocabulary semantic accuracy on the object, room, and floor level while producing a 75% reduction in representation size compared to dense open-vocabulary maps. In order to prove the efficacy and generalization capabilities of HOV-SG, we showcase successful long-horizon language-conditioned robot navigation within real-world multi-storage environments. We provide code and trial video data at http://hovsg.github.io/.

Figures (15)

• Figure 1: HOV-SG enables the construction of accurate open-vocabulary 3D scene graphs for large-scale and multi-story environments and enables robots to navigate in them effectively.
• Figure 2: HOV-SG builds hierarchical open-vocabulary 3D scene graphs of indoor household scenes. We first use SAM to extract object masks per frame while obtaining vision-language features via CLIP. In the next step, we aggregate these features on a point level in the map. Secondly, we segment the full point cloud based on merged 3D masks. To generate more meaningful semantic object features, we employ a DBSCAN-based filtering approach to obtain a majority vote feature for each object. To construct an actionable 3D scene graph, we segment the obtained panoptic map into multiple floors, segment and classify distinct regions using several view embeddings, and identify object names via querying. As a result, HOV-SG allows hierarchical querying and navigation using mobile robots even in complex multi-floor environments.
• Figure 3: Floor and Room Segmentation. Given the point cloud of the whole environment, floor and room nodes are subsequently derived based on geometric heuristics. Floor boundaries are computed by finding peaks of the pixel density along the height direction followed by filtering while room segment masks are extracted using the Watershed algorithm.
• Figure 4: Room embedding computation and room type voting. We enrich each room node with open-vocabulary embeddings by associating the observations with it. Given the segmented room region and the contained camera poses we extract 10 distinct CLIP features that represent the semantic distribution of a room.
• Figure 5: Actionable navigational graph: The creation of the actionable navigational graph involves constructing single-floor and cross-floor navigational graphs: (a) By deducting the set of obstacles from the union of camera poses and the per-floor obtained BEV projection of the floor point cloud, we obtain the navigable area. Within this area we construct a Voronoi diagram as shown right. (b) In order to equip our navigational graph with cross-floor navigation capabilities, we extract the camera positions within regions classified as stairs. This subgraph is connected with the corresponding floor-level Voronoi graphs.