Customizable Perturbation Synthesis for Robust SLAM Benchmarking

TL;DR

This work tackles the problem of evaluating SLAM robustness in unstructured environments by introducing a customizable perturbation synthesis pipeline and a comprehensive perturbation taxonomy. It combines trajectory perturbations, sensor corruptions, and multi-sensor misalignment within a programmable render-and-synthesize framework to produce perturbed SLAM data, culminating in the Robust-SLAM benchmark. Robust-SLAM comprises 1,000 sequences (nearly 2 million RGBD frames) derived from Replica scenes, enabling isolated and diverse perturbation testing across classical and neural multi-modal SLAM models. The findings reveal that current state-of-the-art SLAM systems are vulnerable to sensor and motion disturbances, underscoring the need for robust design and evaluation tools, and the work provides an open-source platform for ongoing benchmarking and development.

Abstract

Robustness is a crucial factor for the successful deployment of robots in unstructured environments, particularly in the domain of Simultaneous Localization and Mapping (SLAM). Simulation-based benchmarks have emerged as a highly scalable approach for robustness evaluation compared to real-world data collection. However, crafting a challenging and controllable noisy world with diverse perturbations remains relatively under-explored. To this end, we propose a novel, customizable pipeline for noisy data synthesis, aimed at assessing the resilience of multi-modal SLAM models against various perturbations. This pipeline incorporates customizable hardware setups, software components, and perturbed environments. In particular, we introduce comprehensive perturbation taxonomy along with a perturbation composition toolbox, allowing the transformation of clean simulations into challenging noisy environments. Utilizing the pipeline, we instantiate the Robust-SLAM benchmark, which includes diverse perturbation types, to evaluate the risk tolerance of existing advanced multi-modal SLAM models. Our extensive analysis uncovers the susceptibilities of existing SLAM models to real-world disturbance, despite their demonstrated accuracy in standard benchmarks. Our perturbation synthesis toolbox, SLAM robustness evaluation pipeline, and Robust-SLAM benchmark will be made publicly available at https://github.com/Xiaohao-Xu/SLAM-under-Perturbation/.

Figures (33)

• Figure 1: Overview of the noisy data synthesis pipeline for SLAM evaluation under perturbation. (a) Given the customizable robot system and global trajectory, (b) the local trajectory of each sensor can be generated via the physics engine. (c) Subsequently, the trajectory perturbation composer introduces deviations to simulate locomotion perturbations. (d) Following this, the render combines sensor configurations, perturbed local trajectories, and 3D scene models to generate sensor streams. (e) Finally, the sensor perturbation composer introduces corruptions to the clean sensor streams, (f) resulting in perturbed data for SLAM robustness benchmarking.
• Figure 2: Taxonomy of perturbations for SLAM. Given (a) the original (clean) sensor stream and (b) trajectory, our noisy data synthesis pipeline enables the simulation of (c) sensor corruption (for both RGB images and depth maps), (d) trajectory perturbations, and (e) multi-sensor misalignment for the multi-modal input setting.
• Figure 3: Taxonomy of sensor-level perturbations for RGB imaging. We consider 16 common image corruption types hendrycks2019robustness from 4 main categories of perturbations for SLAM robustness evaluation: 1) noise-based distortions: Gaussian noise, shot noise, impulse noise, and speckle noise; 2) blur-based effects: defocus blur, glass blur, motion blur, and Gaussian blur; 3) environmental interferences: snow effect, frost effect, fog effect, and spatter effect; 4) post-processing manipulations: brightness, contrast, pixelate, and JPEG compression.
• Figure 4: Taxonomy of sensor-level perturbations for depth imaging. We introduce four perturbations to mimic the perception noises (i.e., random noises and missing) and the limited perception field of depth sensors.
• Figure 5: Taxonomy of trajectory deviation and faster motion perturbation. We present the synthesized ground-truth trajectories (in black) and the estimated trajectories (in blue) obtained using the real-time SLAM model CO-SLAM coslam. For clarity, we visualize the projected trajectory on the horizontal x-y plane derived from the 3D trajectory, which shows that slight trajectory deviations can have a significant impact on the trajectory estimation performance, measured by ATE.