M3CAD: Towards Generic Cooperative Autonomous Driving Benchmark

TL;DR

M3CAD presents a comprehensive, multi-vehicle, multi-task benchmark for cooperative autonomous driving built in CARLA with UE5, delivering 204 sequences, 30k frames, and 267k annotations across perception, mapping, forecasting, occupancy, and planning. It emphasizes global ground truth to enable accurate cooperative supervision and introduces BEV-based end-to-end cooperation (E2EC) to fuse cross-vehicle information for improved planning. Through experiments with UniAD and the proposed framework, the paper demonstrates notable gains in perception and planning when leveraging cooperative data, including improved recall and reduced planning error, while also providing extensive ablations on the importance of global GT and data modalities. The dataset includes an OPV2V* extension for multi-task evaluation and detailed data and map conversion tools, aiming to accelerate research and real-world deployment of robust cooperative autonomous driving systems.

Abstract

We introduce M$^3$CAD, a novel benchmark designed to advance research in generic cooperative autonomous driving. M$^3$CAD comprises 204 sequences with 30k frames, spanning a diverse range of cooperative driving scenarios. Each sequence includes multiple vehicles and sensing modalities, e.g., LiDAR point clouds, RGB images, and GPS/IMU, supporting a variety of autonomous driving tasks, including object detection and tracking, mapping, motion forecasting, occupancy prediction, and path planning. This rich multimodal setup enables M$^3$CAD to support both single-vehicle and multi-vehicle autonomous driving research, significantly broadening the scope of research in the field. To our knowledge, M$^3$CAD is the most comprehensive benchmark specifically tailored for cooperative multi-task autonomous driving research. We evaluate the state-of-the-art end-to-end solution on M$^3$CAD to establish baseline performance. To foster cooperative autonomous driving research, we also propose E2EC, a simple yet effective framework for cooperative driving solution that leverages inter-vehicle shared information for improved path planning. We release M$^3$CAD, along with our baseline models and evaluation results, to support the development of robust cooperative autonomous driving systems. All resources will be made publicly available on https://github.com/zhumorui/M3CAD

Figures (14)

• Figure 1: Illustrations of various autonomous driving tasks using the M$^3$CAD dataset. (a), (b), (d), and (e) show sample images captured from the vehicle's four cameras. (a) Demonstrates the path planning (PP) results, where the ego vehicle's predicted trajectory is represented by a dotted line. (b) Shows object tracking (OT) and motion forecasting (MF) results where dotted lines represent predicted trajectories of other vehicles. (c) Presents object detection (OD) results in 3D space. (f) Depicts mapping (MP) and occupancy prediction (OCC) results.
• Figure 3: Multi-view images captured by vehicles are processed using a BEV encoder to extract BEV features. The ego vehicle then selects a sender vehicle, which shares its local BEV feature. Before processing the received features, the shared BEV is transformed into the ego vehicle's perspective. This transformed feature is then fused with the ego's feature. The final fused BEV is used to support various autonomous driving tasks. Importantly, each of the dashed modules in the system, e.g., sender selection, BEV encoder, BEV fusion, tracking, mapping, motion prediction, occupancy estimation, and planning, can be implemented using different methods or solutions.
• Figure 4: Comparison of local GT vs. global GT for the tracking task on OPV2V.
• Figure 5: Sensor setup for each CAV in M$^3$CAD. Each vehicle is equipped with four cameras (FRONT, LEFT, RIGHT, BACK), all with a $110^\circ$ field of view and an image resolution of $800\times600$ pixels, with lens flare and motion blur disabled. A 64-beam LiDAR, mounted at a height of $1.9\,\mathrm{m}$, operates at $10\,\mathrm{Hz}$ with a $100\,\mathrm{m}$ detection range. GPS/IMU measurements are sampled at $10\,\mathrm{Hz}$, providing global position with an accuracy of approximately $20\,\mathrm{mm}$ and vehicle heading with an accuracy of approximately $2^\circ$.
• Figure 6: This set of images shows different weather conditions captured in CARLA, from the simplest (Sunny, at the top) to the most complex (Night + Rainy, at the bottom). The row labels indicate the weather conditions, and the column labels represent the camera view.