Advancing Off-Road Autonomous Driving: The Large-Scale ORAD-3D Dataset and Comprehensive Benchmarks

TL;DR

ORAD-3D tackles data scarcity in off-road autonomous driving by providing the largest diverse dataset with multi-sensor data across terrains, weather, and lighting, together with a comprehensive benchmark suite. It defines five benchmarks—2D free-space detection, 3D occupancy prediction, rough GPS-guided path planning, VLM-based driving, and off-road world modeling—each with dedicated label-generation pipelines and evaluation strategies. Results demonstrate the value of LiDAR-vision fusion for 3D occupancy, the potential of VLM-driven planning, and the promise of diffusion-based world modeling for diverse future scenarios, while highlighting ongoing challenges in unstructured environments. By releasing both data and benchmarks publicly, ORAD-3D aims to accelerate robust, generalizable off-road autonomy research.

Abstract

A major bottleneck in off-road autonomous driving research lies in the scarcity of large-scale, high-quality datasets and benchmarks. To bridge this gap, we present ORAD-3D, which, to the best of our knowledge, is the largest dataset specifically curated for off-road autonomous driving. ORAD-3D covers a wide spectrum of terrains, including woodlands, farmlands, grasslands, riversides, gravel roads, cement roads, and rural areas, while capturing diverse environmental variations across weather conditions (sunny, rainy, foggy, and snowy) and illumination levels (bright daylight, daytime, twilight, and nighttime). Building upon this dataset, we establish a comprehensive suite of benchmark evaluations spanning five fundamental tasks: 2D free-space detection, 3D occupancy prediction, rough GPS-guided path planning, vision-language model-driven autonomous driving, and world model for off-road environments. Together, the dataset and benchmarks provide a unified and robust resource for advancing perception and planning in challenging off-road scenarios. The dataset and code will be made publicly available at https://github.com/chaytonmin/ORAD-3D.

Figures (17)

• Figure 1: On-road environments are characterized by well-structured infrastructure—such as lane markings, traffic signs, and clearly delineated roadways—that provide consistent and reliable navigational cues. In stark contrast, off-road environments lack paved surfaces and structured guidance, instead featuring irregular, heterogeneous, and often ambiguous terrain. This absence of formal infrastructure greatly amplifies the complexity of perception and navigation tasks.
• Figure 2: ORAD-3D dataset contains a variety of off-road scenes.
• Figure 3: Different weather conditions are considered in ORAD-3D.
• Figure 4: Different light conditions are considered in ORAD-3D.
• Figure 5: Different road types are considered in ORAD-3D.