Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Extending Semantic Abstraction for Efficient Search of Hidden Objects

Tasha Pais, Nikhilesh Belulkar

TL;DR

The paper tackles efficient retrieval of hidden, partially occluded objects by leveraging semantic abstraction and long-term object-location priors learned from past interactions. It builds a 3D localization framework on top of CLIP-driven relevancy maps, converting 2D cues into 3D point clouds and then fitting a Gaussian Mixture Model via Expectation-Maximization to capture object-location distributions. Using these learned priors, the approach achieves significantly faster first-try localization compared to random search, particularly when priors are skewed, and discusses methods to optimize the clustering process with criteria like the Bayesian Information Criterion. Extensions consider LLM-assisted action planning, multi-view disambiguation, and multi-camera setups to further improve search efficiency in household environments.

Abstract

Semantic Abstraction's key observation is that 2D VLMs' relevancy activations roughly correspond to their confidence of whether and where an object is in the scene. Thus, relevancy maps are treated as "abstract object" representations. We use this framework for learning 3D localization and completion for the exclusive domain of hidden objects, defined as objects that cannot be directly identified by a VLM because they are at least partially occluded. This process of localizing hidden objects is a form of unstructured search that can be performed more efficiently using historical data of where an object is frequently placed. Our model can accurately identify the complete 3D location of a hidden object on the first try significantly faster than a naive random search. These extensions to semantic abstraction hope to provide household robots with the skills necessary to save time and effort when looking for lost objects.

On Extending Semantic Abstraction for Efficient Search of Hidden Objects

TL;DR

The paper tackles efficient retrieval of hidden, partially occluded objects by leveraging semantic abstraction and long-term object-location priors learned from past interactions. It builds a 3D localization framework on top of CLIP-driven relevancy maps, converting 2D cues into 3D point clouds and then fitting a Gaussian Mixture Model via Expectation-Maximization to capture object-location distributions. Using these learned priors, the approach achieves significantly faster first-try localization compared to random search, particularly when priors are skewed, and discusses methods to optimize the clustering process with criteria like the Bayesian Information Criterion. Extensions consider LLM-assisted action planning, multi-view disambiguation, and multi-camera setups to further improve search efficiency in household environments.

Abstract

Semantic Abstraction's key observation is that 2D VLMs' relevancy activations roughly correspond to their confidence of whether and where an object is in the scene. Thus, relevancy maps are treated as "abstract object" representations. We use this framework for learning 3D localization and completion for the exclusive domain of hidden objects, defined as objects that cannot be directly identified by a VLM because they are at least partially occluded. This process of localizing hidden objects is a form of unstructured search that can be performed more efficiently using historical data of where an object is frequently placed. Our model can accurately identify the complete 3D location of a hidden object on the first try significantly faster than a naive random search. These extensions to semantic abstraction hope to provide household robots with the skills necessary to save time and effort when looking for lost objects.
Paper Structure (13 sections, 1 equation, 8 figures)

This paper contains 13 sections, 1 equation, 8 figures.

Table of Contents

  1. Problem Definition
  2. Method
  3. Step One
  4. Step Two
  5. Step Three
  6. Step Four
  7. Tests for Robustness
  8. Localization of hidden objects
  9. Search space optimization results:
  10. Procedure:
  11. Results and measurements:
  12. Extensions
  13. Lessons Learned

Figures (8)

  • Figure 1: The camera view most often used to query Semantic Abstraction
  • Figure 2: Query to localize the location of a fork inside a cabinet
  • Figure 3: EM algorithm for two cluster centers
  • Figure 4: Distribution of 3-d location of tomato in asymmetric case projected onto 2-dimensions
  • Figure 5: First-time search accuracy increases as compared to a brute force baseline
  • ...and 3 more figures