Speech-Guided Sequential Planning for Autonomous Navigation using Large Language Model Meta AI 3 (Llama3)

TL;DR

A system for speech-guided sequential planning in autonomous navigation, utilizing Llama3 and the Robot Operating System~(ROS), utilizing Llama3 and the Robot Operating System~(ROS).

Abstract

In social robotics, a pivotal focus is enabling robots to engage with humans in a more natural and seamless manner. The emergence of advanced large language models (LLMs) such as Generative Pre-trained Transformers (GPTs) and autoregressive models like Large Language Model Meta AI (Llamas) has driven significant advancements in integrating natural language understanding capabilities into social robots. This paper presents a system for speech-guided sequential planning in autonomous navigation, utilizing Llama3 and the Robot Operating System~(ROS). The proposed system involves using Llama3 to interpret voice commands, extracting essential details through parsing, and decoding these commands into sequential actions for tasks. Such sequential planning is essential in various domains, particularly in the pickup and delivery of an object. Once a sequential navigation task is evaluated, we employ DRL-VO, a learning-based control policy that allows a robot to autonomously navigate through social spaces with static infrastructure and (crowds of) people. We demonstrate the effectiveness of the system in simulation experiment using Turtlebot 2 in ROS1 and Turtlebot 3 in ROS2. We conduct hardware trials using a Clearpath Robotics Jackal UGV, highlighting its potential for real-world deployment in scenarios requiring flexible and interactive robotic behaviors.

Figures (3)

• Figure 1: Overview of the proposed system. The process begins with converting verbal commands into text using the Google Cloud Speech API. The Llama3 model processes this text to extract essential details, such as the pickup location, item, and delivery destination, using regex-based parsing. These parsed commands, along with prior environmental information, are sent to the Task Execution module, where they are translated into a sequence of actions that are then executed.
• Figure 2: (Left) The layout of the Mechanical Engineering department at Temple University, used in the ROS2-Nav2 simulation with Turtlebot3. (Center) The layout of the lobby of the College of Engineering at Temple University, which is used for both simulation experiments with Turtlebot2 using DRL-VO and hardware experiments with the Jackal UGV. (Right) The Jackal UGV.
• Figure 3: Simulation experiment depicting the steps of progress for pickup of item $I$ (envelopes) from $L_{\rm pickup}$ (Mail Room) and its delivery to $L_{\rm delivery}$ (Dames' Office).