Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RAI: Flexible Agent Framework for Embodied AI

Kajetan Rachwał, Maciej Majek, Bartłomiej Boczek, Kacper Dąbrowski, Paweł Liberadzki, Adam Dąbrowski, Maria Ganzha

TL;DR

RAI addresses the need for scalable, flexible embodied AI by delivering a modular MAS framework that integrates LLMs, robotic stacks, external APIs, and embodiment data. At its core, RAI defines Agents, Connectors, and Tools, with embodiment supported by RAI_whoami and a Faiss-based vector store enabling Retrieval-Augmented Generation. The authors validate the framework through three deployments: a physical Husarion ROSBot XL with ROS 2 navigation, and two simulations involving a robotic arm and a tractor, to assess control, perception, and human-robot interaction. Results show that RAI can coordinate multiple agents across domains and illustrate current LLM limitations in embodied contexts, guiding future enhancements such as improved knowledge streaming and spatio-temporal data management.

Abstract

With an increase in the capabilities of generative language models, a growing interest in embodied AI has followed. This contribution introduces RAI - a framework for creating embodied Multi Agent Systems for robotics. The proposed framework implements tools for Agents' integration with robotic stacks, Large Language Models, and simulations. It provides out-of-the-box integration with state-of-the-art systems like ROS 2. It also comes with dedicated mechanisms for the embodiment of Agents. These mechanisms have been tested on a physical robot, Husarion ROSBot XL, which was coupled with its digital twin, for rapid prototyping. Furthermore, these mechanisms have been deployed in two simulations: (1) robot arm manipulator and (2) tractor controller. All of these deployments have been evaluated in terms of their control capabilities, effectiveness of embodiment, and perception ability. The proposed framework has been used successfully to build systems with multiple agents. It has demonstrated effectiveness in all the aforementioned tasks. It also enabled identifying and addressing the shortcomings of the generative models used for embodied AI.

RAI: Flexible Agent Framework for Embodied AI

TL;DR

RAI addresses the need for scalable, flexible embodied AI by delivering a modular MAS framework that integrates LLMs, robotic stacks, external APIs, and embodiment data. At its core, RAI defines Agents, Connectors, and Tools, with embodiment supported by RAI_whoami and a Faiss-based vector store enabling Retrieval-Augmented Generation. The authors validate the framework through three deployments: a physical Husarion ROSBot XL with ROS 2 navigation, and two simulations involving a robotic arm and a tractor, to assess control, perception, and human-robot interaction. Results show that RAI can coordinate multiple agents across domains and illustrate current LLM limitations in embodied contexts, guiding future enhancements such as improved knowledge streaming and spatio-temporal data management.

Abstract

With an increase in the capabilities of generative language models, a growing interest in embodied AI has followed. This contribution introduces RAI - a framework for creating embodied Multi Agent Systems for robotics. The proposed framework implements tools for Agents' integration with robotic stacks, Large Language Models, and simulations. It provides out-of-the-box integration with state-of-the-art systems like ROS 2. It also comes with dedicated mechanisms for the embodiment of Agents. These mechanisms have been tested on a physical robot, Husarion ROSBot XL, which was coupled with its digital twin, for rapid prototyping. Furthermore, these mechanisms have been deployed in two simulations: (1) robot arm manipulator and (2) tractor controller. All of these deployments have been evaluated in terms of their control capabilities, effectiveness of embodiment, and perception ability. The proposed framework has been used successfully to build systems with multiple agents. It has demonstrated effectiveness in all the aforementioned tasks. It also enabled identifying and addressing the shortcomings of the generative models used for embodied AI.
Paper Structure (23 sections, 8 figures)

This paper contains 23 sections, 8 figures.

Table of Contents

  1. Introduction
  2. Architecture
  3. Component definitions
  4. Embodiment
  5. Example of system configuration
  6. Flexibility and extensibility
  7. Applications
  8. Navigation and Human-Robot-Integration: Autonomous Mobile Robot
  9. Overview
  10. Configuration
  11. Conclusions
  12. Challenges
  13. Manipulation: Robotic arm
  14. Overview
  15. Experiments
  16. ...and 8 more sections

Figures (8)

  • Figure 1: Simple embodied agent setup
  • Figure 2: Conversational Agent
  • Figure 3: StateBased Agent
  • Figure 4: Physical office setting
  • Figure 5: Household with the digital twin
  • ...and 3 more figures