Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-MARS: Large Language Model for Behavior Tree Generation and NLP-enhanced Dialogue in Multi-Agent Robot Systems

Artem Lykov, Maria Dronova, Nikolay Naglov, Mikhail Litvinov, Sergei Satsevich, Artem Bazhenov, Vladimir Berman, Aleksei Shcherbak, Dzmitry Tsetserukou

TL;DR

Evaluation confirms the LLM-MARS answers on operators questions exhibit high accuracy, relevance, and informativeness, and holds significant potential to revolutionize logistics, enabling autonomous exploration missions and advancing Industry 5.0.

Abstract

This paper introduces LLM-MARS, first technology that utilizes a Large Language Model based Artificial Intelligence for Multi-Agent Robot Systems. LLM-MARS enables dynamic dialogues between humans and robots, allowing the latter to generate behavior based on operator commands and provide informative answers to questions about their actions. LLM-MARS is built on a transformer-based Large Language Model, fine-tuned from the Falcon 7B model. We employ a multimodal approach using LoRa adapters for different tasks. The first LoRa adapter was developed by fine-tuning the base model on examples of Behavior Trees and their corresponding commands. The second LoRa adapter was developed by fine-tuning on question-answering examples. Practical trials on a multi-agent system of two robots within the Eurobot 2023 game rules demonstrate promising results. The robots achieve an average task execution accuracy of 79.28% in compound commands. With commands containing up to two tasks accuracy exceeded 90%. Evaluation confirms the system's answers on operators questions exhibit high accuracy, relevance, and informativeness. LLM-MARS and similar multi-agent robotic systems hold significant potential to revolutionize logistics, enabling autonomous exploration missions and advancing Industry 5.0.

LLM-MARS: Large Language Model for Behavior Tree Generation and NLP-enhanced Dialogue in Multi-Agent Robot Systems

TL;DR

Evaluation confirms the LLM-MARS answers on operators questions exhibit high accuracy, relevance, and informativeness, and holds significant potential to revolutionize logistics, enabling autonomous exploration missions and advancing Industry 5.0.

Abstract

This paper introduces LLM-MARS, first technology that utilizes a Large Language Model based Artificial Intelligence for Multi-Agent Robot Systems. LLM-MARS enables dynamic dialogues between humans and robots, allowing the latter to generate behavior based on operator commands and provide informative answers to questions about their actions. LLM-MARS is built on a transformer-based Large Language Model, fine-tuned from the Falcon 7B model. We employ a multimodal approach using LoRa adapters for different tasks. The first LoRa adapter was developed by fine-tuning the base model on examples of Behavior Trees and their corresponding commands. The second LoRa adapter was developed by fine-tuning on question-answering examples. Practical trials on a multi-agent system of two robots within the Eurobot 2023 game rules demonstrate promising results. The robots achieve an average task execution accuracy of 79.28% in compound commands. With commands containing up to two tasks accuracy exceeded 90%. Evaluation confirms the system's answers on operators questions exhibit high accuracy, relevance, and informativeness. LLM-MARS and similar multi-agent robotic systems hold significant potential to revolutionize logistics, enabling autonomous exploration missions and advancing Industry 5.0.
Paper Structure (29 sections, 7 figures)

This paper contains 29 sections, 7 figures.

Table of Contents

  1. Introduction
  2. Related Works
  3. Advantages of Behavior Trees Approach
  4. Advantages of Using Behavior Trees in Robotics
  5. Advantages of Using Behavior Trees as Output of LLM
  6. Multimodal LLM to Multiagent Robot Control
  7. Multimodal LLM to Control Eurobot Robots
  8. System Overview
  9. Mechanical Construction Overview
  10. Embedded Systems Overview
  11. High-Level Systems Overview
  12. LLM Training Methods and Recipes
  13. Behavior Tree Generation Module Fine-tuning Process
  14. Question Answering Module Fine-tuning Process
  15. Experimental Evaluation
  16. ...and 14 more sections

Figures (7)

  • Figure 1: Strategy generation process. User defines the tasks, Large Language Model generates a Behavior Tree for robots to autonomously solve the tasks given the environment.
  • Figure 2: 3D CAD models of the robots. (a) Isometric view. (b) Straddling view of the frame and grippers.
  • Figure 3: Block diagram of the robot electronics.
  • Figure 4: System architecture of LLM-MARS.
  • Figure 5: Correct answer distribution for experiment on human ability to recognize model-generated behavior tree.
  • ...and 2 more figures