Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BEV-Seg: Bird's Eye View Semantic Segmentation Using Geometry and Semantic Point Cloud

Mong H. Ng, Kaahan Radia, Jianfei Chen, Dequan Wang, Ionel Gog, Joseph E. Gonzalez

TL;DR

The paper tackles bird's-eye-view semantic segmentation from side-view RGB images by introducing a two-stage, depth-aware perception pipeline that first converts per-view depth and semantics into a semantic point cloud and then parses an incomplete BEV into full BEV semantics. A common intermediate representation enables effective transfer learning across domains by abstracting geometry while stage 1 remains adaptable to target domains. The approach achieves state-of-the-art results, demonstrating large mIoU gains over prior methods and robust transfer when fine-tuning only the first stage, aided by the new BEVSEG-Carla dataset. This work enhances monocular BEV understanding for autonomous driving and offers practical benefits for cross-domain deployment and planning systems.

Abstract

Bird's-eye-view (BEV) is a powerful and widely adopted representation for road scenes that captures surrounding objects and their spatial locations, along with overall context in the scene. In this work, we focus on bird's eye semantic segmentation, a task that predicts pixel-wise semantic segmentation in BEV from side RGB images. This task is made possible by simulators such as Carla, which allow for cheap data collection, arbitrary camera placements, and supervision in ways otherwise not possible in the real world. There are two main challenges to this task: the view transformation from side view to bird's eye view, as well as transfer learning to unseen domains. Existing work transforms between views through fully connected layers and transfer learns via GANs. This suffers from a lack of depth reasoning and performance degradation across domains. Our novel 2-staged perception pipeline explicitly predicts pixel depths and combines them with pixel semantics in an efficient manner, allowing the model to leverage depth information to infer objects' spatial locations in the BEV. In addition, we transfer learning by abstracting high-level geometric features and predicting an intermediate representation that is common across different domains. We publish a new dataset called BEVSEG-Carla and show that our approach improves state-of-the-art by 24% mIoU and performs well when transferred to a new domain.

BEV-Seg: Bird's Eye View Semantic Segmentation Using Geometry and Semantic Point Cloud

TL;DR

The paper tackles bird's-eye-view semantic segmentation from side-view RGB images by introducing a two-stage, depth-aware perception pipeline that first converts per-view depth and semantics into a semantic point cloud and then parses an incomplete BEV into full BEV semantics. A common intermediate representation enables effective transfer learning across domains by abstracting geometry while stage 1 remains adaptable to target domains. The approach achieves state-of-the-art results, demonstrating large mIoU gains over prior methods and robust transfer when fine-tuning only the first stage, aided by the new BEVSEG-Carla dataset. This work enhances monocular BEV understanding for autonomous driving and offers practical benefits for cross-domain deployment and planning systems.

Abstract

Bird's-eye-view (BEV) is a powerful and widely adopted representation for road scenes that captures surrounding objects and their spatial locations, along with overall context in the scene. In this work, we focus on bird's eye semantic segmentation, a task that predicts pixel-wise semantic segmentation in BEV from side RGB images. This task is made possible by simulators such as Carla, which allow for cheap data collection, arbitrary camera placements, and supervision in ways otherwise not possible in the real world. There are two main challenges to this task: the view transformation from side view to bird's eye view, as well as transfer learning to unseen domains. Existing work transforms between views through fully connected layers and transfer learns via GANs. This suffers from a lack of depth reasoning and performance degradation across domains. Our novel 2-staged perception pipeline explicitly predicts pixel depths and combines them with pixel semantics in an efficient manner, allowing the model to leverage depth information to infer objects' spatial locations in the BEV. In addition, we transfer learning by abstracting high-level geometric features and predicting an intermediate representation that is common across different domains. We publish a new dataset called BEVSEG-Carla and show that our approach improves state-of-the-art by 24% mIoU and performs well when transferred to a new domain.

Paper Structure

This paper contains 17 sections, 1 equation, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Bird's-Eye View Representations
  4. Image-Derived Depth
  5. Transfer Learning
  6. Methodology
  7. A Two-Stage Perception Pipeline
  8. Stage 1: RGB to Intermediate Representation
  9. Stage 2: Intermediate Representation to Bird's-Eye Semantic Segmentation
  10. Advantages of Explicit Depth Reasoning and Using Geometry to Transform Between Views
  11. Transfer Learning Via a Common Intermediate Representation
  12. Pipeline and Network Configuration
  13. Experiment Results
  14. Benchmarks on Source Domain
  15. Benchmarks on Target Domain
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Example of RGB, BEV dataset pair.
  • Figure 2: The proposed BEV-Seg pipeline.
  • Figure 3: Qualitative Results on the BEVSEG-Carla dataset.