LOST-3DSG: Lightweight Open-Vocabulary 3D Scene Graphs with Semantic Tracking in Dynamic Environments

TL;DR

LOST-3DSG tackles dynamic object tracking under open vocabulary by replacing dense CLIP-based semantics with compact attribute-level representations derived from word2vec and sentence embeddings. The model comprises a Perception Module that builds a 3D scene graph with semantic attributes, a Lost Similarity Function that fuses semantic and appearance cues, and a Scene Update Module that updates the graph across time via exploration and tracking behaviors. The approach achieves accurate object identity maintenance with dramatically reduced memory usage compared to dense CLIP embeddings, validated on a TIAGo robot in real indoor environments. This work provides a scalable, open-vocabulary paradigm for 3D scene understanding that favors efficiency and real-time applicability, while outlining avenues for improved temporal aggregation and integration with higher-level reasoning.

Abstract

Tracking objects that move within dynamic environments is a core challenge in robotics. Recent research has advanced this topic significantly; however, many existing approaches remain inefficient due to their reliance on heavy foundation models. To address this limitation, we propose LOST-3DSG, a lightweight open-vocabulary 3D scene graph designed to track dynamic objects in real-world environments. Our method adopts a semantic approach to entity tracking based on word2vec and sentence embeddings, enabling an open-vocabulary representation while avoiding the necessity of storing dense CLIP visual features. As a result, LOST-3DSG achieves superior performance compared to approaches that rely on high-dimensional visual embeddings. We evaluate our method through qualitative and quantitative experiments conducted in a real 3D environment using a TIAGo robot. The results demonstrate the effectiveness and efficiency of LOST-3DSG in dynamic object tracking. Code and supplementary material are publicly available on the project website at https://lab-rococo-sapienza.github.io/lost-3dsg/.

Figures (3)

• Figure 1: Perception Module. The current RGB frame (1) and the corresponding depth image (2) are processed to build the 3DSG of the scene. From the RGB image, open-vocabulary object labels are extracted using a VLM and then grounded in the image to detect the corresponding object bounding boxes on the camera plane (3). At the same time, the VLM is used to extract object-level semantic attributes, including label, color, material, and a fine-grained description. For each object instance, pixel-level segmentation masks are obtained using an object segmentation model and subsequently reprojected into 3D using depth information (5). In parallel, geometric primitives such as centroids and 2D bounding boxes are computed through geometric reprojection and used to estimate 3D bounding boxes (6).
• Figure 2: Scene Update Module. During the exploration phase, the Persistent 3DSG is incrementally populated using LSF-based disambiguation. Once exploration ends, the system switches to the tracking phase, where objects in the current FOV are matched by semantic similarity and spatial consistency: moved objects are updated, new objects are added, and missing objects are removed or marked as uncertain.
• Figure 3: Execution instance. The agent observes a household environment (1). During the exploration phase, a laptop, a tape roll, a hammer, and a gamepad are added to the 3DSG (2). The hammer is then moved to the brown surface on the left. When the hammer is observed at its new location, the previous instance is marked as uncertain (3), and once the original location is revisited and the object is confirmed absent, the instance is removed from the 3DSG (4).