CoPeD-Advancing Multi-Robot Collaborative Perception: A Comprehensive Dataset in Real-World Environments

TL;DR

CoPeD addresses the lack of real-world multi-robot collaborative perception data by providing a heterogeneous air-ground dataset with multiple modalities and synchronized pose data. It introduces a comprehensive hardware setup, indoor and outdoor sequences, calibration pipelines, and zero-shot annotation pipelines to support a range of tasks from depth/semantic estimation to sensor fusion. The dataset enables robust cross-robot collaboration studies and goes beyond BEV-focused scenarios by leveraging air-ground viewpoints. This resource is positioned to accelerate research in robust, real-world multi-robot perception and autonomy.

Abstract

In the past decade, although single-robot perception has made significant advancements, the exploration of multi-robot collaborative perception remains largely unexplored. This involves fusing compressed, intermittent, limited, heterogeneous, and asynchronous environmental information across multiple robots to enhance overall perception, despite challenges like sensor noise, occlusions, and sensor failures. One major hurdle has been the lack of real-world datasets. This paper presents a pioneering and comprehensive real-world multi-robot collaborative perception dataset to boost research in this area. Our dataset leverages the untapped potential of air-ground robot collaboration featuring distinct spatial viewpoints, complementary robot mobilities, coverage ranges, and sensor modalities. It features raw sensor inputs, pose estimation, and optional high-level perception annotation, thus accommodating diverse research interests. Compared to existing datasets predominantly designed for Simultaneous Localization and Mapping (SLAM), our setup ensures a diverse range and adequate overlap of sensor views to facilitate the study of multi-robot collaborative perception algorithms. We demonstrate the value of this dataset qualitatively through multiple collaborative perception tasks. We believe this work will unlock the potential research of high-level scene understanding through multi-modal collaborative perception in multi-robot settings.

Figures (8)

• Figure 1: Multi-robot collaborative data collection. The top row shows the infrared and RGB images from the aerial robots, whereas the bottom row shows the GPS and LiDAR data from the ground robots. The sensor data present several overlapping spatial areas. Red boxes show sample objects identified in multiple heterogeneous sensors' data streams.
• Figure 2: Layout of the ground robots and aerial robots.
• Figure 3: Indoor-NYUARPL environment. Pointcloud data (top row) is captured by the ground robots, and the color images (bottom row) are captured by the aerial robots. Each column shows one subgroup of one aerial robot and one ground robot exploring different parts of the environment.
• Figure 4: Outdoor-FOREST environment. Pointcloud data (top row) is captured by the ground robots, and the color images (bottom row) are captured by the aerial robots. This snapshot captures the scenarios when two groups of robot subteams encounter each other.
• Figure 5: Detected AprilTag mounted on the ground robots captured from one aerial robot's downward-facing camera.