SubT-MRS Dataset: Pushing SLAM Towards All-weather Environments

TL;DR

SubT-MRS tackles the persistent gap in SLAM benchmarks by introducing a five-year, multimodal, multi-robot dataset that encompasses over 500 hours and 100 km of trajectories across all-weather environments. It combines LiDAR, fisheye and depth cameras, thermal imaging, and IMU data from aerial, legged, and wheeled robots, under 30+ degraded conditions, to rigorously test accuracy and robustness. A novel velocity-based robustness metric, Rp and Rr, based on the area under the F1 curve, complements traditional ATE measures and better reflects safety-critical performance in robotics. The dataset is accompanied by ground-truth maps/trajectories and a public SLAM challenge, revealing current methods' limitations in real-world degraded scenarios and providing a standardized benchmark to push towards more reliable SLAM in all-weather operation.

Abstract

Simultaneous localization and mapping (SLAM) is a fundamental task for numerous applications such as autonomous navigation and exploration. Despite many SLAM datasets have been released, current SLAM solutions still struggle to have sustained and resilient performance. One major issue is the absence of high-quality datasets including diverse all-weather conditions and a reliable metric for assessing robustness. This limitation significantly restricts the scalability and generalizability of SLAM technologies, impacting their development, validation, and deployment. To address this problem, we present SubT-MRS, an extremely challenging real-world dataset designed to push SLAM towards all-weather environments to pursue the most robust SLAM performance. It contains multi-degraded environments including over 30 diverse scenes such as structureless corridors, varying lighting conditions, and perceptual obscurants like smoke and dust; multimodal sensors such as LiDAR, fisheye camera, IMU, and thermal camera; and multiple locomotions like aerial, legged, and wheeled robots. We develop accuracy and robustness evaluation tracks for SLAM and introduced novel robustness metrics. Comprehensive studies are performed, revealing new observations, challenges, and opportunities for future research.

Figures (18)

• Figure 1: Dense reconstruction from the SubT-MRS dataset, achieved through collaboration with diverse robots equipped with multimodal sensors. Colors represent different challenging environments (tunnels, caves, urban, confined spaces) captured by various robot types (aerial, legged, wheeled). The bottom section showcases a gallery with diverse visual, LiDAR, and mixed degradations.
• Figure 2: An overview of the sensor pack used in SubT-MRS dataset. It is equipped with a Xavier processing unit with hardware time synchronization for multimodal sensors including LiDAR, fisheye cameras, thermal cameras, depth cameras (option), and an IMU.
• Figure 3: The SubT-MRS datasets were collected across diverse seasons, capturing environments with perceptual challenges such as poor illumination, darkness, and water puddles, where visual sensors falter. They also include geometrically complex areas like long featureless corridors and steep multi-floor structures, challenging LiDAR systems with potential drift. Moreover, these datasets cover conditions with airborne obscurants like dust, fog, snow, and smoke in tough environments, including caves, deserts, long tunnels, and off-road areas.
• Figure 4: The SubT-MRS dataset facilitates the generation of high-precision ground truth maps and trajectories. Figure (a) shows the ground truth trajectory in multi-floor settings. Figure (b) displays ground truth maps for indoor and outdoor areas, encompassing long corridors, multi-floor structures, and open spaces. Figure (c) features photo-realistic scans based on our ground truth maps in cave environments.
• Figure 5: From left to right, it shows robustness metric $R_p$ and $R_r$ for LiDAR and visual sequences respectively. Note: This is a summary of results for all sequences, with weights based on the trajectory length. The area under the curve (AUC) represents the robustness ($R_p, R_r$). The x-axis shows velocity thresholds for classifying estimated velocities as inliers and the y-axis is F-1 score.