One Thousand and One Hours: Self-driving Motion Prediction Dataset

TL;DR

The paper addresses the need for large, open datasets to advance motion forecasting and planning in self-driving systems. It releases a comprehensive dataset comprising 1,118 hours across 170,000 scenes along a fixed route, with a rich HD semantic map and aerial imagery, plus the L5Kit toolkit and baseline benchmarks for forecasting and planning. The study demonstrates that increasing data scale yields tangible improvements in both forecasting accuracy and planning performance, underscoring data as a key driver for ML-based SDV capabilities. By providing open access to a high-detail, route-focused dataset, the work aims to democratize development and accelerate progress toward robust, scalable autonomous driving solutions.

Abstract

Motivated by the impact of large-scale datasets on ML systems we present the largest self-driving dataset for motion prediction to date, containing over 1,000 hours of data. This was collected by a fleet of 20 autonomous vehicles along a fixed route in Palo Alto, California, over a four-month period. It consists of 170,000 scenes, where each scene is 25 seconds long and captures the perception output of the self-driving system, which encodes the precise positions and motions of nearby vehicles, cyclists, and pedestrians over time. On top of this, the dataset contains a high-definition semantic map with 15,242 labelled elements and a high-definition aerial view over the area. We show that using a dataset of this size dramatically improves performance for key self-driving problems. Combined with the provided software kit, this collection forms the largest and most detailed dataset to date for the development of self-driving machine learning tasks, such as motion forecasting, motion planning and simulation. The full dataset is available at http://level5.lyft.com/.

Figures (8)

• Figure 1: An overview of the released dataset for motion modelling, consisting of 1,118 hours of recorded self-driving perception data on a route spanning 6.8 miles between the train station and the office (red). The examples on the bottom-left show released scenes on top of the high-definition semantic map that capture road geometries and the aerial view of the area.
• Figure 2: An example of a state-of-the-art self-driving pipeline. First, the raw LiDAR and camera data are processed to detect the positions of nearby objects around the vehicle. Then, their motion is predicted to allow the SDV to plan a safe collision-free trajectory. The released dataset enables the modelling of a motion prediction component.
• Figure 3: The self-driving vehicle configuration used to collect the data. Raw data from LiDARs and cameras were processed by a perception system to generate the dataset, capturing the poses and motion of nearby vehicles.
• Figure 4: Examples from the scenes in the dataset, projected over a BEV of the rasterised semantic map. The self-driving vehicle is shown in red, other traffic participants in yellow, and lane colours denotes driving direction. The dataset contains 170,000 such sequences, each 25 seconds long with sensor data at 10Hz.
• Figure 5: Elements of the provided HD semantic map (left) and overhead aerial map surrounding the route (right). We provide 15,242 human annotations including 8,505 individual lane segments. The aerial map covers 74 km$^2$ at a resolution of 6 cm per pixel.