RACAS: Controlling Diverse Robots With a Single Agentic System

TL;DR

RACAS (Robot-Agnostic Control via Agentic Systems), a cooperative agentic architecture in which three LLM/VLM-based modules (Monitors, a Controller, and a Memory Curator) communicate exclusively through natural language to provide closed-loop robot control.

Abstract

Many robotic platforms expose an API through which external software can command their actuators and read their sensors. However, transitioning from these low-level interfaces to high-level autonomous behaviour requires a complicated pipeline, whose components demand distinct areas of expertise. Existing approaches to bridging this gap either require retraining for every new embodiment or have only been validated across structurally similar platforms. We introduce RACAS (Robot-Agnostic Control via Agentic Systems), a cooperative agentic architecture in which three LLM/VLM-based modules (Monitors, a Controller, and a Memory Curator) communicate exclusively through natural language to provide closed-loop robot control. RACAS requires only a natural language description of the robot, a definition of available actions, and a task specification; no source code, model weights, or reward functions need to be modified to move between platforms. We evaluate RACAS on several tasks using a wheeled ground robot, a recently published novel multi-jointed robotic limb, and an underwater vehicle. RACAS consistently solved all assigned tasks across these radically different platforms, demonstrating the potential of agentic AI to substantially reduce the barrier to prototyping robotic solutions.

Figures (6)

• Figure 1: The basic design of the system tries to minimize the extent of changes needed to apply it to different robots. The majority of the complexity of adapting to different robot platforms is offloaded to large AI models augmented with a dynamic structured memory system.
• Figure 2: The three real-world robotic platforms the system from \ref{['fig:system_diagram']} was applied to.
• Figure 3: An example of RACAS's reasoning process during one of the Dingo (real-world) runs.
• Figure 4: Snapshots of the various setups experiments were conducted in. In all cases, the target was never visible from the starting position.
• Figure 5: Distribution of steps needed to complete the tasks in each of the settings. Note that random control policies were artificially limited to 25 steps.