Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LIEREx: Language-Image Embeddings for Robotic Exploration

Felix Igelbrink, Lennart Niecksch, Marian Renz, Martin Günther, Martin Atzmueller

TL;DR

LIEREx advances open-set semantic mapping by fusing Vision-Language Foundation Models with 3D Semantic Scene Graphs to enable language-driven exploration in unknown environments. It introduces a Learned View Quality Estimation (VQE) strategy and a data-generation pipeline based on the Habitat HM3D dataset to efficiently score candidate observation poses without online ray-casting. The framework supports flexible, text-based queries and hierarchical grounding of language concepts in spatial context, bridging open-set perception with robust spatial reasoning. A TIAGo indoor demonstrator is used to validate the system and outline steps toward real-world deployment and evaluation.

Abstract

Semantic maps allow a robot to reason about its surroundings to fulfill tasks such as navigating known environments, finding specific objects, and exploring unmapped areas. Traditional mapping approaches provide accurate geometric representations but are often constrained by pre-designed symbolic vocabularies. The reliance on fixed object classes makes it impractical to handle out-of-distribution knowledge not defined at design time. Recent advances in Vision-Language Foundation Models, such as CLIP, enable open-set mapping, where objects are encoded as high-dimensional embeddings rather than fixed labels. In LIEREx, we integrate these VLFMs with established 3D Semantic Scene Graphs to enable target-directed exploration by an autonomous agent in partially unknown environments.

LIEREx: Language-Image Embeddings for Robotic Exploration

TL;DR

LIEREx advances open-set semantic mapping by fusing Vision-Language Foundation Models with 3D Semantic Scene Graphs to enable language-driven exploration in unknown environments. It introduces a Learned View Quality Estimation (VQE) strategy and a data-generation pipeline based on the Habitat HM3D dataset to efficiently score candidate observation poses without online ray-casting. The framework supports flexible, text-based queries and hierarchical grounding of language concepts in spatial context, bridging open-set perception with robust spatial reasoning. A TIAGo indoor demonstrator is used to validate the system and outline steps toward real-world deployment and evaluation.

Abstract

Semantic maps allow a robot to reason about its surroundings to fulfill tasks such as navigating known environments, finding specific objects, and exploring unmapped areas. Traditional mapping approaches provide accurate geometric representations but are often constrained by pre-designed symbolic vocabularies. The reliance on fixed object classes makes it impractical to handle out-of-distribution knowledge not defined at design time. Recent advances in Vision-Language Foundation Models, such as CLIP, enable open-set mapping, where objects are encoded as high-dimensional embeddings rather than fixed labels. In LIEREx, we integrate these VLFMs with established 3D Semantic Scene Graphs to enable target-directed exploration by an autonomous agent in partially unknown environments.
Paper Structure (8 sections, 4 figures)

This paper contains 8 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Open Set Semantic Mapping
  3. Exploration Planning System
  4. View Quality Estimation
  5. Data Generation
  6. Training
  7. Indoor Demonstrator
  8. Discussion & Outlook

Figures (4)

  • Figure 1: Overview of the lierex pipeline. A textual query retrieves the best matching instances from the , taking into account both the scene context provided by the graph and proximity in the embedding space. Candidate view poses are then sampled from the and scored by the estimation module based on their semantic similarity to the query.
  • Figure 2: Example queries in the VL map. (\ref{['subfig:query_1']}) and (\ref{['subfig:query_2']}) show the top 2 results for the text query chair, including cropped regions of the surroundings. (\ref{['subfig:query_3']}) depicts a query for the higher-order concept kitchenette, which comprises multiple object instances.
  • Figure 3: The self-supervised training pipeline for the View Quality Estimation (VQE) module. The model learns to predict quality scores by comparing CLIP embeddings of rendered views against the query embedding.
  • Figure 4: Localization of the TIAGo robot in a large-scale environment. The bottom image shows the pre-recorded polygonal map of the building (excluding furniture) overlaid with LiDAR points that were not associated with the map geometry during registration.