LLM2Swarm: Robot Swarms that Responsively Reason, Plan, and Collaborate through LLMs

TL;DR

This work proposes to address the aforementioned challenges by integrating LLMs with robot swarms and show the potential in proofs of concept (showcases), and explores two approaches.

Abstract

Robot swarms are composed of many simple robots that communicate and collaborate to fulfill complex tasks. Robot controllers usually need to be specified by experts on a case-by-case basis via programming code. This process is time-consuming, prone to errors, and unable to take into account all situations that may be encountered during deployment. On the other hand, recent Large Language Models (LLMs) have demonstrated reasoning and planning capabilities, introduced new ways to interact with and program machines, and incorporate both domain-specific and commonsense knowledge. Hence, we propose to address the aforementioned challenges by integrating LLMs with robot swarms and show the potential in proofs of concept (showcases). For this integration, we explore two approaches. The first approach is 'indirect integration,' where LLMs are used to synthesize and validate the robot controllers. This approach may reduce development time and human error before deployment. Moreover, during deployment, it could be used for on-the-fly creation of new robot behaviors. The second approach is 'direct integration,' where each robot locally executes a separate LLM instance during deployment for robot-robot collaboration and human-swarm interaction. These local LLM instances enable each robot to reason, plan, and collaborate using natural language, as demonstrated in our showcases where the robots are able to detect a variety of anomalies, without prior information about the nature of these anomalies. To enable further research on our mainly conceptual contribution, we release the software and videos for our LLM2Swarm system: https://github.com/Pold87/LLM2Swarm.

Figures (5)

• Figure 1: Overview of LLM2Swarm -- LLM-enabled robot swarms. LLM2Swarm involves four key system components: humans, LLMs, controllers, and platforms. Before mission start: a human designer uses both manual design and LLM2Swarm's controller synthesis module (which prompts a powerful LLM) to generate a robot controller. This controller is executed in simulation, and uses one lightweight LLM per robot to simulate on-device execution of LLM2Swarm's direct integration module. After mission start: a human operator interacts with the real robots' lightweight on-device LLMs to instruct the swarm and to receive information about the swarm state. These LLMs also interact with other robots' controllers in order to reason, plan, and collaborate. In addition, the lightweight LLMs can synthesize new robot controllers on-the-fly during the mission.
• Figure 2: Flow of LLM2Swarm's controller synthesis module. Using LLM2Swarm's controller synthesis module, a user begins by specifying the desired controller in natural language. This specification, together with controller examples, is used as part of an LLM prompt to synthesize a robot controller. The synthesized controller draft is then executed directly in the ARGoS simulator. If ARGoS detects any syntax errors, the errors are reported back to the LLM, with the request to resolve them. If no syntax errors are found, the user can proceed to validate the controller logic: if the robot behavior is not as expected, the user provides information about both the robots' expected behavior and actual behavior to the LLM with the request to improve the controller. Once the controller logic is validated, the user can ask the LLM to check for any security vulnerabilities. After all validation steps are completed, the final controller, specified in programming code, is ready for deployment.
• Figure 3: System interactions for LLM2Swarm's direct integration module. Using LLM2Swarm's direct integration module, a robot's controller is composed of two parts: a classical controller and an on-device LLM. As in traditional approaches, the classical controller manages the robot's actuators and sensors. However, unlike traditional approaches, the controller also creates prompts for the on-device LLM and uses the responses to guide the robot's actions. This on-device LLM also enables a robot to interact with other robots or a human operator by using natural language. As a result, such LLM-enabled robots can reason, plan, and collaborate thanks to the capabilities of their on-device LLMs.
• Figure 4: Showcase --- Robot-robot collaboration (anomaly detection). In this showcase, the robots' task is to determine whether the environment contains more crops (blue floor) or more weeds (red floor). To do so, the robots use LLM2Swarm's direct integration module. The robots are given the prompt (as shown in the upper part of the figure) and use it to generate the responses (i.e., inter-robot messages), as shown in the lower part, across four sub-showcases. The text below the images are LLM-generated responses of a select robot, obtained by executing LLM2Swarm. The system was able to successfully diagnose the studied anomalies without prior information about the exact nature of the anomalies.
• Figure 5: Showcase --- Human-swarm interaction. This showcase demonstrates how LLM2Swarm enables a human operator to retrieve information about the current state of the swarm and to send instructions to the swarm. The upper part of the figure shows the prompt sent by the human operator to the robots, the lower part shows the corresponding responses. In the Inform showcase (left), a selected robot generates a concise summary of the swarm's current activities and intermediate results. In the Instruct showcase (right), the robots move to the specified target that they extracted from the natural language input provided by the human operator.