SEEK: Semantic Reasoning for Object Goal Navigation in Real World Inspection Tasks

TL;DR

This paper introduces a framework that enables robots to use semantic knowledge from prior spatial configurations of the environment and semantic common sense knowledge to search for and navigate toward target objects more efficiently, and proposes a novel probabilistic planning framework to search for the object using relational semantic knowledge.

Abstract

This paper addresses the problem of object-goal navigation in autonomous inspections in real-world environments. Object-goal navigation is crucial to enable effective inspections in various settings, often requiring the robot to identify the target object within a large search space. Current object inspection methods fall short of human efficiency because they typically cannot bootstrap prior and common sense knowledge as humans do. In this paper, we introduce a framework that enables robots to use semantic knowledge from prior spatial configurations of the environment and semantic common sense knowledge. We propose SEEK (Semantic Reasoning for Object Inspection Tasks) that combines semantic prior knowledge with the robot's observations to search for and navigate toward target objects more efficiently. SEEK maintains two representations: a Dynamic Scene Graph (DSG) and a Relational Semantic Network (RSN). The RSN is a compact and practical model that estimates the probability of finding the target object across spatial elements in the DSG. We propose a novel probabilistic planning framework to search for the object using relational semantic knowledge. Our simulation analyses demonstrate that SEEK outperforms the classical planning and Large Language Models (LLMs)-based methods that are examined in this study in terms of efficiency for object-goal inspection tasks. We validated our approach on a physical legged robot in urban environments, showcasing its practicality and effectiveness in real-world inspection scenarios.

Figures (9)

• Figure 1: Autonomous object inspections in urban buildings and construction sites. The robot needs to search and inspect the target object. In this paper, we proposed a method to guide the robot to find the object using relational semantic knowledge.
• Figure 2: SEEK system architecture. Our method receives an inspection query and builds a Dynamic Scene Graph (DSG) from a given floor plan. The Relational Semantic Network (RSN) predicts the probability of finding the target object in every room. The semantic-guided global planner uses this prediction to compute an optimal global search policy. Finally, the local finite state controller executes the global policy and switches to an active semantic search controller when the robot sees the object.
• Figure 3: A visualization of the global search policy. The RSN predicts the probability of finding a sink in every room. The policy computes the best action from each room to locate the nearest sink. The circular arrow represents the local search.
• Figure 4: Robot paths comparison of object-goal navigation with SEEK and SB2G in the Gazebo simulator. Our global planner guides the robot to the kitchen, where a fire extinguisher is usually located.
• Figure 5: The left plot compares how fast the robot searches and navigates to the target object from the same starting location using SEEK and SB2G. The shades on the line plot represent the standard deviation across three runs with the same starting locations. The right box plot reports the SPL metric across seven runs with different starting location.