Language-guided Robust Navigation for Mobile Robots in Dynamically-changing Environments

TL;DR

An embodied AI system for human-in-the-loop navigation using a wheeled mobile robot and a means to parse human feedback expressed in natural language into local navigation waypoints and integrate it into a global planning system by leveraging a map of semantic features and an aligned obstacle map are developed.

Abstract

In this paper, we develop an embodied AI system for human-in-the-loop navigation with a wheeled mobile robot. We propose a direct yet effective method of monitoring the robot's current plan to detect changes in the environment that impact the intended trajectory of the robot significantly and then query a human for feedback. We also develop a means to parse human feedback expressed in natural language into local navigation waypoints and integrate it into a global planning system, by leveraging a map of semantic features and an aligned obstacle map. Extensive testing in simulation and physical hardware experiments with a resource-constrained wheeled robot tasked to navigate in a real-world environment validate the efficacy and robustness of our method. This work can support applications like precision agriculture and construction, where persistent monitoring of the environment provides a human with information about the environment state.

Figures (6)

• Figure 1: Overview of the human-in-the-loop navigation approach developed in this work. Given a nominal trajectory based on a prior (metric) map of the environment (along with the corresponding aligned semantic VLMap), total path deviations computed in real time invoke a query to the human once a threshold is exceeded. Then, human feedback is transformed from natural language into waypoints which are merged into the ongoing trajectory to generate the updated one for the robot to follow. This process repeats until a task is completed - herein to reach a specific pose on the map.
• Figure 2: (a) Top-down view of the small house environment rendered in Gazebo and (b) the corresponding RGB map created during our map construction process. Note that while the major features agree, there can still be inexact edges caused by sensor noise.
• Figure 3: Trajectories generated by our method (blue) overlaid to the ground truth (green) in simulation testing. Panels (a)-(c) and (d)-(f) correspond to the small house and factory environments, respectively, for three distinctive routes.
• Figure 4: Trajectories generated by our method (blue) overlaid to the ground truth (green) in the language navigation ablation study (in simulation). Panels (a)-(c) and (d)-(f) correspond to the small house and factory environments, respectively, for three distinctive routes.
• Figure 5: Trajectories generated by our method (blue) overlaid to the ground truth (green) in the real-world experiments, for three distinctive routes.