DrivIng: A Large-Scale Multimodal Driving Dataset with Full Digital Twin Integration

TL;DR

DrivIng tackles the lack of long, continuous, geo-referenced driving data with a CARLA-based digital twin by presenting a large-scale, multimodal dataset spanning ~$18{,}000\ \mathrm{m}$ of real-world driving across urban, suburban, and highway scenes. It provides 6 cameras, 1 LiDAR, and RTK-based localization, with $10\ \mathrm{Hz}$ annotations totaling ~1.2 million 3D boxes across 12 classes, plus a geo-referenced digital twin enabling exact real-to-sim replay and interactive simulations. The work includes nuScenes-format conversion and MMDetection3D-based baselines (PETR and CenterPoint) evaluated across Day, Dusk, and Night, demonstrating clear modality advantages and degradation under low illumination. The authors release the dataset, digital twin, HD map, and codebase to support reproducible benchmarking, sim-to-real experiments, and future research in robust perception, multi-agent collaboration, and scenario testing.

Abstract

Perception is a cornerstone of autonomous driving, enabling vehicles to understand their surroundings and make safe, reliable decisions. Developing robust perception algorithms requires large-scale, high-quality datasets that cover diverse driving conditions and support thorough evaluation. Existing datasets often lack a high-fidelity digital twin, limiting systematic testing, edge-case simulation, sensor modification, and sim-to-real evaluations. To address this gap, we present DrivIng, a large-scale multimodal dataset with a complete geo-referenced digital twin of a ~18 km route spanning urban, suburban, and highway segments. Our dataset provides continuous recordings from six RGB cameras, one LiDAR, and high-precision ADMA-based localization, captured across day, dusk, and night. All sequences are annotated at 10 Hz with 3D bounding boxes and track IDs across 12 classes, yielding ~1.2 million annotated instances. Alongside the benefits of a digital twin, DrivIng enables a 1-to-1 transfer of real traffic into simulation, preserving agent interactions while enabling realistic and flexible scenario testing. To support reproducible research and robust validation, we benchmark DrivIng with state-of-the-art perception models and publicly release the dataset, digital twin, HD map, and codebase.

Figures (7)

• Figure 1: This visualization illustrates the core features of DrivIng and its digital twin. The left panel shows a real-world satellite view of the track and its fully geo-referenced digital twin, aligned with a location marker indicating the vehicle’s position. The right panel presents the synchronized sensor suite, including six camera views and a LiDAR frame. The top row displays real-world images, while the bottom row shows the corresponding CARLA simulation with all real-world objects precisely mapped. All images and the LiDAR frame include class-colored 3D bounding boxes for clear object distinction. Satellite image © Esri, i-cubed, USDA, USGS, AEX, GeoEye, Getmapping, Aerogrid, IGN, IGP, UPR-EGP, and the GIS User Community.
• Figure 2: Full sensor setup and coordinate frame of the vehicle.
• Figure 3: Comparison of real-world Dusk and Night illumination.
• Figure 4: Distribution of all 12 object classes in the dataset, measured by the number of annotated 3D bounding boxes. Cars are the most frequently annotated class, whereas Animals and OtherPedestrian appear least often. Overall, the relative distributions of object classes are consistent across the three recorded sequences.
• Figure 5: Visualization (a) illustrates the number of annotated 3D bounding boxes per frame across all sequences. Among the different daytimes, Day contains the highest average number of objects per frame, while Night contains the fewest, yet all sequences include numerous frames with more than 50 objects. Visualization (b) presents the distribution of object orientations relative to the ego vehicle, showing that DrivIng includes a substantial number of objects observed from non-typical traffic angles. Visualization (c) shows the distance distribution of all annotations, with most objects located within 100m, while still including a substantial number of objects at longer ranges beyond 100m.