Enhancing Lidar-based Object Detection in Adverse Weather using Offset Sequences in Time

TL;DR

This work tackles lidar object detection under adverse weather by exploiting temporal information from lidar sequences. It introduces a temporal offset augmentation strategy and a PBOD-based architecture to fuse multi-frame data, and it evaluates 10 sequence-processing architectures across the DENSE, CADC, and nuScenes datasets. The results show that random frame skipping generally improves robustness, with FC* achieving a notable gain on Dense (mAP at $IoU=0.5$ of $0.590$) and convLSTM variants delivering strong gains on CADC and nuScenes, albeit with higher latency. The study demonstrates that temporal sequence processing can significantly enhance lidar perception in adverse conditions, and suggests future work on optimal sequence length and Transformer-based temporal fusion to further improve performance and efficiency.

Abstract

Automated vehicles require an accurate perception of their surroundings for safe and efficient driving. Lidar-based object detection is a widely used method for environment perception, but its performance is significantly affected by adverse weather conditions such as rain and fog. In this work, we investigate various strategies for enhancing the robustness of lidar-based object detection by processing sequential data samples generated by lidar sensors. Our approaches leverage temporal information to improve a lidar object detection model, without the need for additional filtering or pre-processing steps. We compare $10$ different neural network architectures that process point cloud sequences including a novel augmentation strategy introducing a temporal offset between frames of a sequence during training and evaluate the effectiveness of all strategies on lidar point clouds under adverse weather conditions through experiments. Our research provides a comprehensive study of effective methods for mitigating the effects of adverse weather on the reliability of lidar-based object detection using sequential data that are evaluated using public datasets such as nuScenes, Dense, and the Canadian Adverse Driving Conditions Dataset. Our findings demonstrate that our novel method, involving temporal offset augmentation through randomized frame skipping in sequences, enhances object detection accuracy compared to both the baseline model (Pillar-based Object Detection) and no augmentation.

Figures (6)

• Figure 1: Noise in a point cloud; red markers illustrate noisy points, and green markers show points belonging to objects. Shades of blue indicate reflection intensity. Point Cloud from Canadian Adverse Driving Conditions dataset cadc.
• Figure 2: We compare $10$ different neural network architectures using temporal data sequences and used all architectures to train models on $3$ datasets. The models are evaluated and compared to a baseline model based on Pillar-based Object Detection PBOD on validation splits of these datasets.
• Figure 3: IC - We concatenate temporally succeeding point clouds in the input stage to fuse the information about the current driving scene contained in both point clouds, i.e. concatenating points from two sequential point clouds.
• Figure 4: FC - We fuse sequential samples later in the network after features are extracted by feature concatenation.
• Figure 5: LSTM - We fuse sequential samples later in the network by building a memory with extracted features using long short-term memory cells.