ROOT: VLM based System for Indoor Scene Understanding and Beyond

TL;DR

ROOT is introduced, a VLM-based system designed to enhance the analysis of indoor scenes and demonstrates that \rootname facilitates indoor scene understanding and proves effective in diverse downstream applications, such as 3D scene generation and embodied AI.

Abstract

Recently, Vision Language Models (VLMs) have experienced significant advancements, yet these models still face challenges in spatial hierarchical reasoning within indoor scenes. In this study, we introduce ROOT, a VLM-based system designed to enhance the analysis of indoor scenes. Specifically, we first develop an iterative object perception algorithm using GPT-4V to detect object entities within indoor scenes. This is followed by employing vision foundation models to acquire additional meta-information about the scene, such as bounding boxes. Building on this foundational data, we propose a specialized VLM, SceneVLM, which is capable of generating spatial hierarchical scene graphs and providing distance information for objects within indoor environments. This information enhances our understanding of the spatial arrangement of indoor scenes. To train our SceneVLM, we collect over 610,000 images from various public indoor datasets and implement a scene data generation pipeline with a semi-automated technique to establish relationships and estimate distances among indoor objects. By utilizing this enriched data, we conduct various training recipes and finish SceneVLM. Our experiments demonstrate that \rootname facilitates indoor scene understanding and proves effective in diverse downstream applications, such as 3D scene generation and embodied AI. The code will be released at \url{https://github.com/harrytea/ROOT}.

Figures (13)

• Figure 1: Root is a system designed to interpret indoor scene images and extract various types of meta-information about the scenes. Utilizing this information, Root can generate hierarchical relationships and spatial distances among indoor objects. This enriched data serves to support various downstream tasks.
• Figure 2: We introduce Root, a system designed for understanding indoor scenes. Initially, the system utilizes an iterative object perception module based on GPT-4V to identify entities within a given image. Subsequently, the indoor scene and objects are parsed using existing vision foundation models to gather meta-information about the scene. Finally, the object information is processed by SceneVLM, resulting in a scene graph that illustrates the spatial hierarchical relationships and distance information. In the scene graph, arrows of different colors denote different relationships.
• Figure 3: Four types of hierarchical relationships as defined. In each sub-figure, the larger object represents the parent object, while the smaller object denotes the child object.
• Figure 4: SceneVQA data generation pipeline. This diagram depicts the semi-automated pipeline used to create GraphVQA data, which includes manual annotation, GPT-4 assisted transformation, and iterative refinement. For DistanceVQA, object distances are computed directly from 3D point cloud data.
• Figure 5: Hierarchical scene graph visualization of our method. Each object is assigned a serial number, with the corresponding visual JSON is shown next to the image. Nodes represent objects, while edges indicate relationships. The numbers 1, 2, and 3 represent the floor, wall, and ceiling, respectively. For brevity, relationships such as "support", "hang", "attach", and "contain" are abbreviated to their initial letters. Object names are omitted from the labels to enhance clarity.