Multiagent Multitraversal Multimodal Self-Driving: Open MARS Dataset

TL;DR

Open MARS introduces a real-world, multimodal autonomous driving dataset that jointly addresses multiagent collaboration, repeated traversals, and surround-view sensors. It provides two subsets for concurrent multiagent scenes and retrospective traversals, and benchmarks perception tasks including visual place recognition and neural reconstruction using multiple baselines (e.g., NetVLAD, MixVPR, CoVPR, EmerNeRF, PVG, iNGP, RobustNeRF, SegNeRF). Evaluation employs $Recall@K$ with a spatial threshold of $S=20$ meters for VPR and standard image quality metrics $PSNR$, $SSIM$, and $LPIPS$ for reconstruction, highlighting strengths and trade-offs across methods in both single- and multi-traversal settings. The results reveal that multiagent collaboration enhances place recognition and that multitraversal data enable novel neural reconstruction and simulation opportunities, establishing Open MARS as a versatile benchmark for collaborative perception, memory-augmented learning, and unsupervised scene understanding in autonomous driving.

Abstract

Large-scale datasets have fueled recent advancements in AI-based autonomous vehicle research. However, these datasets are usually collected from a single vehicle's one-time pass of a certain location, lacking multiagent interactions or repeated traversals of the same place. Such information could lead to transformative enhancements in autonomous vehicles' perception, prediction, and planning capabilities. To bridge this gap, in collaboration with the self-driving company May Mobility, we present the MARS dataset which unifies scenarios that enable MultiAgent, multitraveRSal, and multimodal autonomous vehicle research. More specifically, MARS is collected with a fleet of autonomous vehicles driving within a certain geographical area. Each vehicle has its own route and different vehicles may appear at nearby locations. Each vehicle is equipped with a LiDAR and surround-view RGB cameras. We curate two subsets in MARS: one facilitates collaborative driving with multiple vehicles simultaneously present at the same location, and the other enables memory retrospection through asynchronous traversals of the same location by multiple vehicles. We conduct experiments in place recognition and neural reconstruction. More importantly, MARS introduces new research opportunities and challenges such as multitraversal 3D reconstruction, multiagent perception, and unsupervised object discovery. Our data and codes can be found at https://ai4ce.github.io/MARS/.

Figures (9)

• Figure 1: Overview of MARS.(a) Within a geographical area, we operate four autonomous vehicles, displaying their GPS trajectories from a single day using different colors. (b) Vehicles occasionally come close together (visualized via distinct-colored point clouds), supporting research in multiagent systems. (c) We collect sensory data from repeated traversals of the same location under varying conditions, for learning and perception research with retrospective memory. (d) The dataset includes surround-view RGB images and LiDAR point clouds for cross-modal perception and learning. Note that our data is obtained from May Mobility : https://maymobility.com/
• Figure 2: Sensor setup of the vehicle platform for data collection.
• Figure 3: Multiagent subset statistics.
• Figure 4: Multitraversal subset statistics.
• Figure 5: Number of traversals and frames at each location.