Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Finding 3D Scene Analogies with Multimodal Foundation Models

Junho Kim, Young Min Kim

TL;DR

The paper tackles transferring plans between unseen 3D scenes by discovering 3D scene analogies. It introduces a zero-shot, open-vocabulary approach that fuses a CLIP-based sparse graph with a dense PartField 3D feature field in a coarse-to-fine pipeline, first establishing object-level correspondences via graph matching and then refining with feature-field alignment. Quantitatively, it outperforms baselines on the 3D-FRONT dataset in Chamfer accuracy and enables reliable trajectory and waypoint transfer at scene scale, demonstrating practical utility for planning, teleoperation, and imitation learning. This method offers a practical pipeline for leveraging multimodal foundation models to bridge diverse indoor environments without additional domain-specific training, with potential extensions to inference efficiency and dynamic-object handling.

Abstract

Connecting current observations with prior experiences helps robots adapt and plan in new, unseen 3D environments. Recently, 3D scene analogies have been proposed to connect two 3D scenes, which are smooth maps that align scene regions with common spatial relationships. These maps enable detailed transfer of trajectories or waypoints, potentially supporting demonstration transfer for imitation learning or task plan transfer across scenes. However, existing methods for the task require additional training and fixed object vocabularies. In this work, we propose to use multimodal foundation models for finding 3D scene analogies in a zero-shot, open-vocabulary setting. Central to our approach is a hybrid neural representation of scenes that consists of a sparse graph based on vision-language model features and a feature field derived from 3D shape foundation models. 3D scene analogies are then found in a coarse-to-fine manner, by first aligning the graph and refining the correspondence with feature fields. Our method can establish accurate correspondences between complex scenes, and we showcase applications in trajectory and waypoint transfer.

Finding 3D Scene Analogies with Multimodal Foundation Models

TL;DR

The paper tackles transferring plans between unseen 3D scenes by discovering 3D scene analogies. It introduces a zero-shot, open-vocabulary approach that fuses a CLIP-based sparse graph with a dense PartField 3D feature field in a coarse-to-fine pipeline, first establishing object-level correspondences via graph matching and then refining with feature-field alignment. Quantitatively, it outperforms baselines on the 3D-FRONT dataset in Chamfer accuracy and enables reliable trajectory and waypoint transfer at scene scale, demonstrating practical utility for planning, teleoperation, and imitation learning. This method offers a practical pipeline for leveraging multimodal foundation models to bridge diverse indoor environments without additional domain-specific training, with potential extensions to inference efficiency and dynamic-object handling.

Abstract

Connecting current observations with prior experiences helps robots adapt and plan in new, unseen 3D environments. Recently, 3D scene analogies have been proposed to connect two 3D scenes, which are smooth maps that align scene regions with common spatial relationships. These maps enable detailed transfer of trajectories or waypoints, potentially supporting demonstration transfer for imitation learning or task plan transfer across scenes. However, existing methods for the task require additional training and fixed object vocabularies. In this work, we propose to use multimodal foundation models for finding 3D scene analogies in a zero-shot, open-vocabulary setting. Central to our approach is a hybrid neural representation of scenes that consists of a sparse graph based on vision-language model features and a feature field derived from 3D shape foundation models. 3D scene analogies are then found in a coarse-to-fine manner, by first aligning the graph and refining the correspondence with feature fields. Our method can establish accurate correspondences between complex scenes, and we showcase applications in trajectory and waypoint transfer.
Paper Structure (10 sections, 1 equation, 3 figures, 1 table)

This paper contains 10 sections, 1 equation, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Method
  3. Hybrid Scene Representation
  4. Graph Construction
  5. Feature Field Extraction
  6. Coarse-to-Fine Scene Analogy Estimation
  7. Experiments
  8. Performance Analysis of 3D Scene Analogy Estimation
  9. Applications: Trajectory and Waypoint Transfer
  10. Conclusion and Future Work

Figures (3)

  • Figure 1: Overview of our hybrid scene representation. Our method operates in a coarse-to-fine manner, by first obtaining instance-level associations from graph matching and refining the initial estimate with neural field alignment.
  • Figure 2: Visualizations of estimated 3D scene analogies in 3D-FRONT [4]. We show mapping results for open-space points.
  • Figure 3: Visualizations of trajectory transfer using 3D scene analogies.