OnlineSI: Taming Large Language Model for Online 3D Understanding and Grounding

TL;DR

OnlineSI proposes an online framework for 3D scene understanding and grounding from streaming video by maintaining a bounded spatial memory and fusing 3D point clouds with semantic cues. A dedicated point cloud encoder and a semantic encoder are used to generate memory tokens that are reasoned over by a multimodal LLM, producing on-the-fly scene descriptions with object-level detections. To handle online ambiguity, the authors introduce the Fuzzy-$F_1$-Score, enabling fair evaluation under partial observations. Experiments on ScanNet and ScanNet++ show that OnlineSI achieves robust continual improvement over time and outperforms several baselines that either ignore memory or rely solely on per-frame predictions, demonstrating its potential for real-world embodied AI tasks.

Abstract

In recent years, researchers have increasingly been interested in how to enable Multimodal Large Language Models (MLLM) to possess spatial understanding and reasoning capabilities. However, most existing methods overlook the importance of the ability to continuously work in an ever-changing world, and lack the possibility of deployment on embodied systems in real-world environments. In this work, we introduce OnlineSI, a framework that can continuously improve its spatial understanding of its surroundings given a video stream. Our core idea is to maintain a finite spatial memory to retain past observations, ensuring the computation required for each inference does not increase as the input accumulates. We further integrate 3D point cloud information with semantic information, helping MLLM to better locate and identify objects in the scene. To evaluate our method, we introduce the Fuzzy $F_1$-Score to mitigate ambiguity, and test our method on two representative datasets. Experiments demonstrate the effectiveness of our method, paving the way towards real-world embodied systems.

Figures (6)

• Figure 1: OnlineSI is a framework specifically designed for online 3D understanding and object grounding. Taking a video stream as input, OnlineSI performs incremental semantic reconstruction and leverages a global spatial memory to aggregate observations over time. As demonstrated in the temporal progression, this allows the framework to continuously refine its scene understanding, updating the previous detection results (e.g., "Update to fit the table") and incrementally detecting new object instances.
• Figure 2: Overview of OnlineSI. For each frame $\mathbf{I}_t$ in the video stream, we first reconstruct the pointmap $\mathbf{X}'_t$, and predict the semantic label for each point $\mathbf{Y}_t$. Next, we fuse the current pointmap and semantic map into the previous spatial memory $\{\mathbf{P}_{t-1}, \mathbf{S}_{t-1}\}$ for updating. We then use the point cloud encoder and the semantic encoder to obtain point cloud features and semantic features, respectively, and add them together as our spatial memory tokens. Lastly, we feed the spatial memory tokens and the text prompt tokens into the LLM backbone, and generate a scene description, which contains the detection results of objects in the current scene.
• Figure 3: Qualitative Results of the Detections. Here we show examples comparing our method with baselines. We show the progress of scene reconstruction as input frames accumulate, and the detection results at each timestep. We zoom in on key parts of the scene to provide a clearer, more direct comparison.
• Figure 4: Ablation on Coordinate System Selection. We demonstrate the impact of different coordinate systems of the reconstructed point clouds on performance. We report fuzzy $F_1$-scores evaluated on ScanNet++ yeshwanthliu2023scannetpp.
• Figure 5: Examples of Fuzzy $F_1$-Score. For each row we show the model's prediction in the blue boxes. In the second column, we only select objects with high visibility as the evaluation annotation, while in the third column, we include all low-visibility objects. The vanilla $F_1$-score differs significantly between the two cases. In the last column, we present a visualization of the Fuzzy $F_1$-Score. The strict ground-truth (must be detected) and the lenient ground-truth (may be detected) form the lower and upper bounds of the annotation, providing a more reasonable metrics for evaluation.