Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BETTY Dataset: A Multi-modal Dataset for Full-Stack Autonomy

Micah Nye, Ayoub Raji, Andrew Saba, Eidan Erlich, Robert Exley, Aragya Goyal, Alexander Matros, Ritesh Misra, Matthew Sivaprakasam, Marko Bertogna, Deva Ramanan, Sebastian Scherer

TL;DR

BETTY addresses the need for a comprehensive full-stack autonomous racing dataset by providing large-scale, multi-modal data that spans perception, state estimation, dynamics modeling, motion forecasting, and SLAM/GNSS. The dataset combines exteroceptive and proprioceptive sensors with autonomy stack outputs, extensive metadata, ground-truth labels, and calibration tooling, collected on three ARVs across six diverse environments. Key contributions include raw GNSS observables at high speeds, ground-truth opponent states, auto-labeling pipelines, and ROS2/MCAP data formats with KITTI-compatible annotations, enabling end-to-end and modular evaluation. This resource enables robust development and benchmarking of multi-task autonomous racing algorithms and highlights remaining challenges in state estimation, dynamics, and perception fusion under latency constraints.

Abstract

We present the BETTY dataset, a large-scale, multi-modal dataset collected on several autonomous racing vehicles, targeting supervised and self-supervised state estimation, dynamics modeling, motion forecasting, perception, and more. Existing large-scale datasets, especially autonomous vehicle datasets, focus primarily on supervised perception, planning, and motion forecasting tasks. Our work enables multi-modal, data-driven methods by including all sensor inputs and the outputs from the software stack, along with semantic metadata and ground truth information. The dataset encompasses 4 years of data, currently comprising over 13 hours and 32TB, collected on autonomous racing vehicle platforms. This data spans 6 diverse racing environments, including high-speed oval courses, for single and multi-agent algorithm evaluation in feature-sparse scenarios, as well as high-speed road courses with high longitudinal and lateral accelerations and tight, GPS-denied environments. It captures highly dynamic states, such as 63 m/s crashes, loss of tire traction, and operation at the limit of stability. By offering a large breadth of cross-modal and dynamic data, the BETTY dataset enables the training and testing of full autonomy stack pipelines, pushing the performance of all algorithms to the limits. The current dataset is available at https://pitt-mit-iac.github.io/betty-dataset/.

BETTY Dataset: A Multi-modal Dataset for Full-Stack Autonomy

TL;DR

BETTY addresses the need for a comprehensive full-stack autonomous racing dataset by providing large-scale, multi-modal data that spans perception, state estimation, dynamics modeling, motion forecasting, and SLAM/GNSS. The dataset combines exteroceptive and proprioceptive sensors with autonomy stack outputs, extensive metadata, ground-truth labels, and calibration tooling, collected on three ARVs across six diverse environments. Key contributions include raw GNSS observables at high speeds, ground-truth opponent states, auto-labeling pipelines, and ROS2/MCAP data formats with KITTI-compatible annotations, enabling end-to-end and modular evaluation. This resource enables robust development and benchmarking of multi-task autonomous racing algorithms and highlights remaining challenges in state estimation, dynamics, and perception fusion under latency constraints.

Abstract

We present the BETTY dataset, a large-scale, multi-modal dataset collected on several autonomous racing vehicles, targeting supervised and self-supervised state estimation, dynamics modeling, motion forecasting, perception, and more. Existing large-scale datasets, especially autonomous vehicle datasets, focus primarily on supervised perception, planning, and motion forecasting tasks. Our work enables multi-modal, data-driven methods by including all sensor inputs and the outputs from the software stack, along with semantic metadata and ground truth information. The dataset encompasses 4 years of data, currently comprising over 13 hours and 32TB, collected on autonomous racing vehicle platforms. This data spans 6 diverse racing environments, including high-speed oval courses, for single and multi-agent algorithm evaluation in feature-sparse scenarios, as well as high-speed road courses with high longitudinal and lateral accelerations and tight, GPS-denied environments. It captures highly dynamic states, such as 63 m/s crashes, loss of tire traction, and operation at the limit of stability. By offering a large breadth of cross-modal and dynamic data, the BETTY dataset enables the training and testing of full autonomy stack pipelines, pushing the performance of all algorithms to the limits. The current dataset is available at https://pitt-mit-iac.github.io/betty-dataset/.
Paper Structure (26 sections, 5 figures, 2 tables)

This paper contains 26 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Autonomous Vehicle Perception Datasets
  4. Autonomous Vehicle Planning Datasets
  5. Off-road Datasets
  6. SLAM and GNSS Datasets
  7. Dynamics Datasets
  8. The Dataset
  9. The Platform
  10. Data Collection
  11. Modalities
  12. Perception
  13. Autonomy & State Estimation
  14. Annotations
  15. Metadata
  16. ...and 11 more sections

Figures (5)

  • Figure 1: We provide exteroceptive sensors (camera, LiDAR, and radar), proprioceptive sensors (tire temperature sensor, slip angle sensor, and more), autonomy data (planned trajectories, actions, and more), and semantic metadata. The BETTY dataset is a tribute to Betty, our autonomous racing vehicle.
  • Figure 2: AV-21 sensors and their respective locations.
  • Figure 3: The BETTY dataset was collected in six environments. Each track presents unique characteristics, which present in LiDAR & camera samples. For example, the oval courses have four large, banked turns. Autodromo Nazionale Monza (Monza) has more turns and high longitudinal accelerations. Goodwood Festival of Speed has tight corridors for navigation.
  • Figure 4: Auto-labelling pipeline. Grounding DINO is used as an initial guess for the camera labeling. For LiDAR corrections, we utilize hand-labels to build a bank of templates that are used to refine an initial guess provided by GPS.
  • Figure 5: G-G plot of the top 10 most dynamic runs from BETTY and RACECAR. BETTY extends RACECAR's acceleration distribution with additional high braking deceleration and negative lateral acceleration.