Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PLT-D3: A High-fidelity Dynamic Driving Simulation Dataset for Stereo Depth and Scene Flow

Joshua Tokarsky, Ibrahim Abdulhafiz, Satya Ayyalasomayajula, Mostafa Mohsen, Navya G. Rao, Adam Forbes

TL;DR

PLT-D3 tackles the scarcity of diverse, weather-robust training resources for camera-based autonomous driving perception. It provides a high-fidelity, UE5-rendered stereo dataset with ground-truth depth, disparity, optical flow, and scene flow across dynamic lighting and weather. The dataset comprises 10 driving scenarios with multiple scenes and 21 synchronized samples per scene, enabling robust benchmarking and fine-tuning of perception models. Benchmark results show that state-of-the-art models trained on KITTI struggle to generalize to PLT-D3’s challenging conditions, highlighting the need for dataset-specific adaptation. Overall, PLT-D3 offers a valuable platform for developing and evaluating camera-centric perception under realistic adversarial conditions, enhancing robustness for real-world deployment.

Abstract

Autonomous driving has experienced remarkable progress, bolstered by innovations in computational hardware and sophisticated deep learning methodologies. The foundation of these advancements rests on the availability and quality of datasets, which are crucial for the development and refinement of dependable and versatile autonomous driving algorithms. While numerous datasets have been developed to support the evolution of autonomous driving perception technologies, few offer the diversity required to thoroughly test and enhance system robustness under varied weather conditions. Many public datasets lack the comprehensive coverage of challenging weather scenarios and detailed, high-resolution data, which are critical for training and validating advanced autonomous-driving perception models. In this paper, we introduce PLT-D3; a Dynamic-weather Driving Dataset, designed specifically to enhance autonomous driving systems' adaptability to diverse weather conditions. PLT-D3 provides high-fidelity stereo depth and scene flow ground truth data generated using Unreal Engine 5. In particular, this dataset includes synchronized high-resolution stereo image sequences that replicate a wide array of dynamic weather scenarios including rain, snow, fog, and diverse lighting conditions, offering an unprecedented level of realism in simulation-based testing. The primary aim of PLT-D3 is to address the scarcity of comprehensive training and testing resources that can simulate real-world weather variations. Benchmarks have been established for several critical autonomous driving tasks using PLT-D3, such as depth estimation, optical flow and scene-flow to measure and enhance the performance of state-of-the-art models.

PLT-D3: A High-fidelity Dynamic Driving Simulation Dataset for Stereo Depth and Scene Flow

TL;DR

PLT-D3 tackles the scarcity of diverse, weather-robust training resources for camera-based autonomous driving perception. It provides a high-fidelity, UE5-rendered stereo dataset with ground-truth depth, disparity, optical flow, and scene flow across dynamic lighting and weather. The dataset comprises 10 driving scenarios with multiple scenes and 21 synchronized samples per scene, enabling robust benchmarking and fine-tuning of perception models. Benchmark results show that state-of-the-art models trained on KITTI struggle to generalize to PLT-D3’s challenging conditions, highlighting the need for dataset-specific adaptation. Overall, PLT-D3 offers a valuable platform for developing and evaluating camera-centric perception under realistic adversarial conditions, enhancing robustness for real-world deployment.

Abstract

Autonomous driving has experienced remarkable progress, bolstered by innovations in computational hardware and sophisticated deep learning methodologies. The foundation of these advancements rests on the availability and quality of datasets, which are crucial for the development and refinement of dependable and versatile autonomous driving algorithms. While numerous datasets have been developed to support the evolution of autonomous driving perception technologies, few offer the diversity required to thoroughly test and enhance system robustness under varied weather conditions. Many public datasets lack the comprehensive coverage of challenging weather scenarios and detailed, high-resolution data, which are critical for training and validating advanced autonomous-driving perception models. In this paper, we introduce PLT-D3; a Dynamic-weather Driving Dataset, designed specifically to enhance autonomous driving systems' adaptability to diverse weather conditions. PLT-D3 provides high-fidelity stereo depth and scene flow ground truth data generated using Unreal Engine 5. In particular, this dataset includes synchronized high-resolution stereo image sequences that replicate a wide array of dynamic weather scenarios including rain, snow, fog, and diverse lighting conditions, offering an unprecedented level of realism in simulation-based testing. The primary aim of PLT-D3 is to address the scarcity of comprehensive training and testing resources that can simulate real-world weather variations. Benchmarks have been established for several critical autonomous driving tasks using PLT-D3, such as depth estimation, optical flow and scene-flow to measure and enhance the performance of state-of-the-art models.
Paper Structure (20 sections, 2 figures, 4 tables)

This paper contains 20 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Driving Datasets
  4. Scene Flow Generation
  5. Driving Simulators
  6. Image Augmentation
  7. The PLT-D3 Dataset
  8. Driving Scenarios
  9. Dynamic Lighting
  10. Dynamic Weather
  11. Simulation Engine
  12. Dataset Structure
  13. Limitations
  14. Benchmarks
  15. Evaluation Metrics
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Sample RGB images from PLT-D3. From top-to-bottom (left-to-right): (sun & rain), (snow & fog), (cloudy & sunny), (dust storm & rain) and (snow & partly cloudy).
  • Figure 2: Ground truth visualization from one sample. From left-to-right then top-to-bottom: left RGB, right RGB, optical flow, left depth, right depth, and delta disparity.