ZeroCAP: Zero-Shot Multi-Robot Context Aware Pattern Formation via Large Language Models

TL;DR

ZeroCAP presents a zero-shot framework that couples large language models with vision-based perception to translate natural language instructions into context-aware multi-robot patterns around objects in images. By decoupling spatial reasoning from perception and representing object geometry as an edge-vertex graph, ZeroCAP enables precise deployment coordinates computed by an LLM, with segmentation and shape description handled by VLMs and specialized vision tools. Experimental results in real-world and simulated settings show ZeroCAP outperforms baselines by effectively handling context-driven tasks such as surrounding, infilling, and caging, with ablations highlighting the importance of LangSAM segmentation and edge-based shape descriptors. While currently limited to 2D and static formations, the approach promises extensions to 3D and dynamic environments for flexible, intuitive multi-robot coordination in surveillance, logistics, and related domains.

Abstract

Incorporating language comprehension into robotic operations unlocks significant advancements in robotics, but also presents distinct challenges, particularly in executing spatially oriented tasks like pattern formation. This paper introduces ZeroCAP, a novel system that integrates large language models with multi-robot systems for zero-shot context aware pattern formation. Grounded in the principles of language-conditioned robotics, ZeroCAP leverages the interpretative power of language models to translate natural language instructions into actionable robotic configurations. This approach combines the synergy of vision-language models, cutting-edge segmentation techniques and shape descriptors, enabling the realization of complex, context-driven pattern formations in the realm of multi robot coordination. Through extensive experiments, we demonstrate the systems proficiency in executing complex context aware pattern formations across a spectrum of tasks, from surrounding and caging objects to infilling regions. This not only validates the system's capability to interpret and implement intricate context-driven tasks but also underscores its adaptability and effectiveness across varied environments and scenarios. The experimental videos and additional information about this work can be found at https://sites.google.com/view/zerocap/home.

Figures (6)

• Figure 1: A conceptual image illustrating the proposed Zero-Shot multi-robot Context Aware Pattern (ZeroCAP) formation system, beginning with an initial environment where a car is parked incorrectly. Upon receiving the command from human operator to "Surround the incorrectly parked car at the corners!", the system identifies the target vehicle and autonomously positions the robot cones at strategic corner locations, showcasing the successful execution of a zero-shot context aware pattern formation task.
• Figure 2: An overview of the ZeroCAP system. It traces the workflow from the initial natural language instruction and input image of the environment, to the final deployment of robots, illustrating the sequence of processing stages—including context identification using Vision Language Model (VLM), object segmentation, shape description, and Large Language Model (LLM) coordination for precise robot placement in the environment. Three key stages are highlighted and explained in Section \ref{['sec:methodology']}.
• Figure 3: An illustration of the VLM processing a natural language instruction to identify the object of interest and generate a pattern formation instruction, which guides the deployment of robots in a specified arrangement around the object. The object is not explicitly mentioned in the instruction and must be reasoned by the VLM within the context of image of the environment.
• Figure 4: Illustrations of five real-world tasks performed by the ZeroCAP system, showcasing different pattern formations: (a) and (b) illustrate general pattern formations; (c) and (d) depict infill pattern formations; (e) demonstrates a caging task. Each task is executed based on natural language instructions provided at the top of each subfigure. Blue boundaries around the robots indicate their initial positions, while green boundaries represent their final positions. Please zoom in on the images for details.
• Figure 5: Two simulation scenarios demonstrating ZeroCAP's ability to execute context-aware pattern formations. Sim1 (left) depicts a caging task in a hidden object setup, where the object of interest is not directly specified and must be inferred. Sim2 (right) illustrates an infill task in a multi-object setup. Please zoom in on the images for details.