Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RoScenes: A Large-scale Multi-view 3D Dataset for Roadside Perception

Xiaosu Zhu, Hualian Sheng, Sijia Cai, Bing Deng, Shaopeng Yang, Qiao Liang, Ken Chen, Lianli Gao, Jingkuan Song, Jieping Ye

TL;DR

RoScenes introduces the largest multi-view roadside dataset to advance BEV-based perception in crowded, wide-area traffic. It pairs a novel BEV-to-3D annotation pipeline with a strong implicit BEV baseline, RoBEV, which uses feature-guided position embedding to robustly assign 2D image features to 3D BEV queries across diverse camera layouts. The dataset comprises 14 highway scenes over 64,000 m^2 with 21.13M 3D boxes across 1.30M images, enabling rigorous benchmarking of BEV methods in roadside contexts. Benchmark results show RoBEV achieving state-of-the-art performance on validation data and highlight transferability challenges when generalizing to unseen scene-layouts, underscoring the need for robust cross-scene BEV approaches. The RoScenes dataset and RoBEV baseline are intended to catalyze research in roadside BEV perception and will be released to the community.

Abstract

We introduce RoScenes, the largest multi-view roadside perception dataset, which aims to shed light on the development of vision-centric Bird's Eye View (BEV) approaches for more challenging traffic scenes. The highlights of RoScenes include significantly large perception area, full scene coverage and crowded traffic. More specifically, our dataset achieves surprising 21.13M 3D annotations within 64,000 $m^2$. To relieve the expensive costs of roadside 3D labeling, we present a novel BEV-to-3D joint annotation pipeline to efficiently collect such a large volume of data. After that, we organize a comprehensive study for current BEV methods on RoScenes in terms of effectiveness and efficiency. Tested methods suffer from the vast perception area and variation of sensor layout across scenes, resulting in performance levels falling below expectations. To this end, we propose RoBEV that incorporates feature-guided position embedding for effective 2D-3D feature assignment. With its help, our method outperforms state-of-the-art by a large margin without extra computational overhead on validation set. Our dataset and devkit will be made available at https://github.com/xiaosu-zhu/RoScenes.

RoScenes: A Large-scale Multi-view 3D Dataset for Roadside Perception

TL;DR

RoScenes introduces the largest multi-view roadside dataset to advance BEV-based perception in crowded, wide-area traffic. It pairs a novel BEV-to-3D annotation pipeline with a strong implicit BEV baseline, RoBEV, which uses feature-guided position embedding to robustly assign 2D image features to 3D BEV queries across diverse camera layouts. The dataset comprises 14 highway scenes over 64,000 m^2 with 21.13M 3D boxes across 1.30M images, enabling rigorous benchmarking of BEV methods in roadside contexts. Benchmark results show RoBEV achieving state-of-the-art performance on validation data and highlight transferability challenges when generalizing to unseen scene-layouts, underscoring the need for robust cross-scene BEV approaches. The RoScenes dataset and RoBEV baseline are intended to catalyze research in roadside BEV perception and will be released to the community.

Abstract

We introduce RoScenes, the largest multi-view roadside perception dataset, which aims to shed light on the development of vision-centric Bird's Eye View (BEV) approaches for more challenging traffic scenes. The highlights of RoScenes include significantly large perception area, full scene coverage and crowded traffic. More specifically, our dataset achieves surprising 21.13M 3D annotations within 64,000 . To relieve the expensive costs of roadside 3D labeling, we present a novel BEV-to-3D joint annotation pipeline to efficiently collect such a large volume of data. After that, we organize a comprehensive study for current BEV methods on RoScenes in terms of effectiveness and efficiency. Tested methods suffer from the vast perception area and variation of sensor layout across scenes, resulting in performance levels falling below expectations. To this end, we propose RoBEV that incorporates feature-guided position embedding for effective 2D-3D feature assignment. With its help, our method outperforms state-of-the-art by a large margin without extra computational overhead on validation set. Our dataset and devkit will be made available at https://github.com/xiaosu-zhu/RoScenes.
Paper Structure (22 sections, 5 equations, 21 figures, 12 tables)

This paper contains 22 sections, 5 equations, 21 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. The RoScenes Dataset
  4. Scene Specification
  5. BEV-to-3D Joint Annotation
  6. Statistics and Analysis
  7. RoBEV for RoScenes
  8. Benchmark
  9. Result on Validation Set
  10. Ablation Study on Validation Set
  11. Result on Zero-Shot Test Set
  12. Conclusion
  13. Additional Dataset Analysis
  14. Scene Samples
  15. Data Collection
  16. ...and 7 more sections

Figures (21)

  • Figure 1: Demonstration of our RoScenes dataset. The annotated truck is difficult to recognize in A, B, C, E, F, G, but is clear in D.
  • Figure 2: Performance (NDS), training time and inference model size comparison among two types of methods.
  • Figure 3: Overall data collection and annotation pipeline. We propose BEV-to-3D joint annotation for efficiency.
  • Figure 4: Summary of all 3D annotations. 1st: Pie chart of different vehicle types. 2nd: Histogram of box amount per scene sample. 3rd: Velocity statistics of different vehicles. 4th: Size (width$\times$height) statistics of different vehicles.
  • Figure 5: Camera statistics in terms of occlusion, focal length, pitch, mounting height and road coverage. Monocular/multi-view occlusions are grouped by vehicle types. Camera parameters are grouped by camera types. (green: far-range, purple: near-range)
  • ...and 16 more figures