Hierarchical Path-planning from Speech Instructions with Spatial Concept-based Topometric Semantic Mapping

TL;DR

Hierarchical spatial representations provide mutually understandable instruction forms for both humans and robots, thus enabling language-based navigation and providing a novel integrated probabilistic generative model and fast approximate inferences with interactions among the hierarchy levels.

Abstract

Assisting individuals in their daily activities through autonomous mobile robots, especially for users without specialized knowledge, is crucial. Specifically, the capability of robots to navigate to destinations based on human speech instructions is essential. While robots can take different paths to the same goal, the shortest path is not always the best. A preferred approach is to accommodate waypoint specifications flexibly, planning an improved alternative path, even with detours. Additionally, robots require real-time inference capabilities. This study aimed to realize a hierarchical spatial representation using a topometric semantic map and path planning with speech instructions, including waypoints. This paper presents Spatial Concept-based Topometric Semantic Mapping for Hierarchical Path Planning (SpCoTMHP), integrating place connectivity. This approach offers a novel integrated probabilistic generative model and fast approximate inference across hierarchy levels. A formulation based on control as probabilistic inference theoretically supports the proposed path planning algorithm. We conducted experiments in home environments using the Toyota Human Support Robot on the SIGVerse simulator and in a lab-office environment with the real robot, Albert. Users issued speech commands specifying the waypoint and goal, such as "Go to the bedroom via the corridor." Navigation experiments using speech instructions with a waypoint demonstrated a performance improvement of SpCoTMHP over the baseline hierarchical path planning method with heuristic path costs (HPP-I), in terms of the weighted success rate at which the robot reaches the closest target and passes the correct waypoints, by 0.590. The computation time was significantly accelerated by 7.14 seconds with SpCoTMHP compared to baseline HPP-I in advanced tasks.

Figures (9)

• Figure 1: Overview. Left: Hierarchy of spatial representation with topometric semantic mapping; Right: Path planning from spoken instruction with waypoint and goal specifications.
• Figure 2: Graphical model representation of SpCoTMHP (top) spatial concept learning and path planning phases (bottom). The two phases imply different probabilistic inferences for the same generative model. This has the mathematical advantage that different probabilistic inferences can be applied under the same model assumptions. The integration of several parts as a single model allows inferences to consider various probabilities throughout. The graphical model represents the conditional dependency between random variables. Gray nodes indicate observations or learned parameters as fixed conditional variables. White nodes denote unobserved latent variables to be estimated. Arrows from the global variables to local variables other than $T$ and $E$ are omitted. In the learning phase, multimodal observations are obtained several times. Based on these observables, latent variables are estimated. In the planning phase, the parameters estimated in the learning phase and optimality variables are given. Under these conditions, the distribution of trajectories is estimated. $D_{e}$ was omitted from the graphical model representation.
• Figure 3: Overhead view of the simulator environments (top); and the ideal spatial concepts expressed by SpCoTMHP on the environmental map (bottom), in Experiment I. The colors of the position distributions were randomly set. If $( \psi_{k_1,k_2} + \psi_{k_1,k_2} ) / 2 > 1/K$, the centers $\mu_{k_1}, \mu_{k_2}$ of Gaussian distributions are connected by an edge. This means that the edges are drawn only if the average transition probability from $k_{1}$ to $k_{2}$ and $k_{2}$ to $k_{1}$ is higher than the uniform transition probability.
• Figure 4: Example of path planning in the advanced task. The instruction: " Go to the bedroom via the lavatory." (Experiment I).
• Figure 5: Top (a--d): Result of the spatial concept learning. Bottom (e--h): Result of path planning. The speech instruction is " Go to the break room via the white shelf." The break room was taught in two rooms: the upper right and the upper left. The white shelf is in the second room from the left on the upper side. (Experiment II).