AmodalSynthDrive: A Synthetic Amodal Perception Dataset for Autonomous Driving

TL;DR

AmodalSynthDrive addresses the critical need for amodal perception in autonomous driving by providing a large-scale synthetic dataset with multi-view camera data, LiDAR, and odometry for 150 driving sequences and over 1M object annotations. It introduces a novel amodal depth estimation task alongside established amodal perception benchmarks (APS, AIS, ASS) and demonstrates baselines to highlight the challenges and potential of amodal reasoning in occluded scenes. The dataset supports diverse weather and lighting, multi-modal cues, and BEV information, enabling robust evaluation and transfer learning to real-world data. This work paves the way for improved occlusion handling in driving systems and offers public benchmarking for future research in amodal scene understanding.

Abstract

Unlike humans, who can effortlessly estimate the entirety of objects even when partially occluded, modern computer vision algorithms still find this aspect extremely challenging. Leveraging this amodal perception for autonomous driving remains largely untapped due to the lack of suitable datasets. The curation of these datasets is primarily hindered by significant annotation costs and mitigating annotator subjectivity in accurately labeling occluded regions. To address these limitations, we introduce AmodalSynthDrive, a synthetic multi-task multi-modal amodal perception dataset. The dataset provides multi-view camera images, 3D bounding boxes, LiDAR data, and odometry for 150 driving sequences with over 1M object annotations in diverse traffic, weather, and lighting conditions. AmodalSynthDrive supports multiple amodal scene understanding tasks including the introduced amodal depth estimation for enhanced spatial understanding. We evaluate several baselines for each of these tasks to illustrate the challenges and set up public benchmarking servers. The dataset is available at http://amodalsynthdrive.cs.uni-freiburg.de.

Figures (7)

• Figure 1: Overview of AmodalSynthDrive consisting of multi-view camera image sequences, 3D bounding boxes, LiDAR data, and odometry. It provides annotations for amodal panoptic segmentation, amodal instance segmentation, amodal semantic segmentation, and amodal depth estimation. It also provides supplementary visual cues such as optical flow and bird's-eye-view maps.
• Figure 2: Illustration of different amodal perception tasks in AmodalSynthDrive. For optimal viewing, we recommend magnifying the figure by four times.
• Figure 3: Illustration of amodal depth estimation ground truth, disassembling the scene according to relative occlusion order. Objects are color-coded by level: red for level 0 (excluding amorphous regions for simplicity), yellow for level 1, blue for level 2, and green for level 3.
• Figure 4: Dataset statistics of AmodalSynthDrive.
• Figure 5: Architectural topology of the amodal depth estimation baseline: ADB-DeepLab, Amodal-DORN and AD-Deeplab. Please note that the boxes enclosed in colored dashes in each of the architecture correspond to the expanded version of the same colored boxes depicted on the bottom.