TartanDrive 2.0: More Modalities and Better Infrastructure to Further Self-Supervised Learning Research in Off-Road Driving Tasks

TL;DR

TartanDrive 2.0 tackles the scarcity of large-scale off-road data by delivering seven hours of multi-modal sensor data, including three LiDARs, at speeds up to 15 m/s, accompanied by open-source collection, processing, and querying tools. The dataset emphasizes self-supervised learning across perception, planning, and control, enabling tasks such as cross-modal supervision, map completion, ground height estimation, learning from demonstration, and aggressive maneuvering. Its infrastructure supports reconfiguration and collaboration, reducing barriers to entry and enabling growth toward multi-site, multi-vehicle datasets. This work advances off-road robotics research by providing rich, modular data streams and scalable tools that facilitate robust representation learning and model generalization in unstructured terrain.

Abstract

We present TartanDrive 2.0, a large-scale off-road driving dataset for self-supervised learning tasks. In 2021 we released TartanDrive 1.0, which is one of the largest datasets for off-road terrain. As a follow-up to our original dataset, we collected seven hours of data at speeds of up to 15m/s with the addition of three new LiDAR sensors alongside the original camera, inertial, GPS, and proprioceptive sensors. We also release the tools we use for collecting, processing, and querying the data, including our metadata system designed to further the utility of our data. Custom infrastructure allows end users to reconfigure the data to cater to their own platforms. These tools and infrastructure alongside the dataset are useful for a variety of tasks in the field of off-road autonomy and, by releasing them, we encourage collaborative data aggregation. These resources lower the barrier to entry to utilizing large-scale datasets, thereby helping facilitate the advancement of robotics in areas such as self-supervised learning, multi-modal perception, inverse reinforcement learning, and representation learning. The dataset is available at https://github.com/castacks/tartan drive 2.0.

Figures (11)

• Figure 1: We provide a new off-road driving dataset for self-supervised learning tasks. Multi-modal data is collected as the robot is driven through several types of terrain that present challenging scenarios to perception, planning, and control algorithms.
• Figure 2: The ATV used for data collection (left); The primary sensor payload on the vehicle (right)
• Figure 3: The 3D scan of the ATV. The rear half of the scan is cropped in the left picture to increase clarity of the front payload in the image.
• Figure 4: The data collection site has changed significantly since our last dataset released in 2021. For example, some older dirt paths (top, shown in red) are now covered in tall grass (bottom).
• Figure 5: Example coverage provided by our LiDAR sensors. Purple and yellow points come from the two Velodynes, and cyan points from the Livox.