Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

nuCarla: A nuScenes-Style Bird's-Eye View Perception Dataset for CARLA Simulation

Zhijie Qiao, Zhong Cao, Henry X. Liu

TL;DR

nuCarla addresses the lack of standardized, large-scale BEV perception data for closed-loop E2E autonomous driving by delivering a nuScenes-style BEV dataset built inside the CARLA simulator. It aligns with nuScenes formatting, includes 1,000 scenarios across diverse maps and weather, and provides six active classes with ground-truth annotations derived from synchronized segmentation views, all enabling direct transfer of existing BEV backbones. The authors evaluate four state-of-the-art BEV models (BEVFormer, PETR, BEVDet, FastBEV) on nuCarla, release pretrained weights, and upgrade the MMDetection3D framework to ensure compatibility with modern hardware, demonstrating strong validation performance and reasonable generalization to unseen environments. This dataset and accompanying benchmarks facilitate robust, closed-loop E2E development and bridge sim-to-real gaps by supplying reliable BEV representations learned in simulation.

Abstract

End-to-end (E2E) autonomous driving heavily relies on closed-loop simulation, where perception, planning, and control are jointly trained and evaluated in interactive environments. Yet, most existing datasets are collected from the real world under non-interactive conditions, primarily supporting open-loop learning while offering limited value for closed-loop testing. Due to the lack of standardized, large-scale, and thoroughly verified datasets to facilitate learning of meaningful intermediate representations, such as bird's-eye-view (BEV) features, closed-loop E2E models remain far behind even simple rule-based baselines. To address this challenge, we introduce nuCarla, a large-scale, nuScenes-style BEV perception dataset built within the CARLA simulator. nuCarla features (1) full compatibility with the nuScenes format, enabling seamless transfer of real-world perception models; (2) a dataset scale comparable to nuScenes, but with more balanced class distributions; (3) direct usability for closed-loop simulation deployment; and (4) high-performance BEV backbones that achieve state-of-the-art detection results. By providing both data and models as open benchmarks, nuCarla substantially accelerates closed-loop E2E development, paving the way toward reliable and safety-aware research in autonomous driving.

nuCarla: A nuScenes-Style Bird's-Eye View Perception Dataset for CARLA Simulation

TL;DR

nuCarla addresses the lack of standardized, large-scale BEV perception data for closed-loop E2E autonomous driving by delivering a nuScenes-style BEV dataset built inside the CARLA simulator. It aligns with nuScenes formatting, includes 1,000 scenarios across diverse maps and weather, and provides six active classes with ground-truth annotations derived from synchronized segmentation views, all enabling direct transfer of existing BEV backbones. The authors evaluate four state-of-the-art BEV models (BEVFormer, PETR, BEVDet, FastBEV) on nuCarla, release pretrained weights, and upgrade the MMDetection3D framework to ensure compatibility with modern hardware, demonstrating strong validation performance and reasonable generalization to unseen environments. This dataset and accompanying benchmarks facilitate robust, closed-loop E2E development and bridge sim-to-real gaps by supplying reliable BEV representations learned in simulation.

Abstract

End-to-end (E2E) autonomous driving heavily relies on closed-loop simulation, where perception, planning, and control are jointly trained and evaluated in interactive environments. Yet, most existing datasets are collected from the real world under non-interactive conditions, primarily supporting open-loop learning while offering limited value for closed-loop testing. Due to the lack of standardized, large-scale, and thoroughly verified datasets to facilitate learning of meaningful intermediate representations, such as bird's-eye-view (BEV) features, closed-loop E2E models remain far behind even simple rule-based baselines. To address this challenge, we introduce nuCarla, a large-scale, nuScenes-style BEV perception dataset built within the CARLA simulator. nuCarla features (1) full compatibility with the nuScenes format, enabling seamless transfer of real-world perception models; (2) a dataset scale comparable to nuScenes, but with more balanced class distributions; (3) direct usability for closed-loop simulation deployment; and (4) high-performance BEV backbones that achieve state-of-the-art detection results. By providing both data and models as open benchmarks, nuCarla substantially accelerates closed-loop E2E development, paving the way toward reliable and safety-aware research in autonomous driving.

Paper Structure

This paper contains 22 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Open-Loop Trajectory Prediction
  4. Closed-Loop Autonomous Driving
  5. BEV Perception Models
  6. Methodology
  7. Maps Selection
  8. Weather Configuration
  9. Traffic Configuration
  10. Ego and Sensor Configuration
  11. Ground Truth Annotation
  12. Data Generation Limitations
  13. nuScenes Formatting
  14. Experiment
  15. Model Configuration
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Maps of nine CARLA towns with traffic in the nuCarla dataset, shown under diverse weather conditions.
  • Figure 2: Comparison of class-wise distributions of actively traveling participants in nuCarla and nuScenes across six object classes.
  • Figure 3: RGB Camera (top) and corresponding instance segmentation camera (bottom) views in CARLA Town06.
  • Figure 4: BEVFormer model predictions (top row) and ground truth annotations (bottom row) on a Town03 sample, featuring the front, front-left, and front-right camera views. 3D bounding boxes are colored according to the nuScenes colormap: cars (amber orange), trucks (coral red), pedestrians (blue), and cyclists (pink-red).