Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Spatiotemporal Approach to Tri-Perspective Representation for 3D Semantic Occupancy Prediction

Sathira Silva, Savindu Bhashitha Wannigama, Gihan Jayatilaka, Muhammad Haris Khan, Roshan Ragel

TL;DR

This work addresses vision-based 3D semantic occupancy prediction by introducing S2TPVFormer, a spatiotemporal TPV encoder that integrates temporal cues with tri-perspective view features through a Temporal Cross-View Hybrid Attention mechanism and Virtual View Transformation. The unified spatiotemporal fusion enables cross-time and cross-view interaction, yielding temporally coherent 3D SOP and improving mean IoU by +4.1% over the TPVFormer baseline on the nuScenes dataset. The approach demonstrates strong performance on SOP and competitive LiDAR segmentation results, highlighting the potential of temporal information to bridge gaps between vision-based and LiDAR-based 3D perception. The work also provides thorough ablations and supplementary analyses, pointing to future directions in long-range temporal fusion and denser semantic supervision to further enhance 3D scene understanding in autonomous systems.

Abstract

Holistic understanding and reasoning in 3D scenes are crucial for the success of autonomous driving systems. The evolution of 3D semantic occupancy prediction as a pretraining task for autonomous driving and robotic applications captures finer 3D details compared to traditional 3D detection methods. Vision-based 3D semantic occupancy prediction is increasingly overlooked in favor of LiDAR-based approaches, which have shown superior performance in recent years. However, we present compelling evidence that there is still potential for enhancing vision-based methods. Existing approaches predominantly focus on spatial cues such as tri-perspective view (TPV) embeddings, often overlooking temporal cues. This study introduces S2TPVFormer, a spatiotemporal transformer architecture designed to predict temporally coherent 3D semantic occupancy. By introducing temporal cues through a novel Temporal Cross-View Hybrid Attention mechanism (TCVHA), we generate Spatiotemporal TPV (S2TPV) embeddings that enhance the prior process. Experimental evaluations on the nuScenes dataset demonstrate a significant +4.1% of absolute gain in mean Intersection over Union (mIoU) for 3D semantic occupancy compared to baseline TPVFormer, validating the effectiveness of S2TPVFormer in advancing 3D scene perception.

A Spatiotemporal Approach to Tri-Perspective Representation for 3D Semantic Occupancy Prediction

TL;DR

This work addresses vision-based 3D semantic occupancy prediction by introducing S2TPVFormer, a spatiotemporal TPV encoder that integrates temporal cues with tri-perspective view features through a Temporal Cross-View Hybrid Attention mechanism and Virtual View Transformation. The unified spatiotemporal fusion enables cross-time and cross-view interaction, yielding temporally coherent 3D SOP and improving mean IoU by +4.1% over the TPVFormer baseline on the nuScenes dataset. The approach demonstrates strong performance on SOP and competitive LiDAR segmentation results, highlighting the potential of temporal information to bridge gaps between vision-based and LiDAR-based 3D perception. The work also provides thorough ablations and supplementary analyses, pointing to future directions in long-range temporal fusion and denser semantic supervision to further enhance 3D scene understanding in autonomous systems.

Abstract

Holistic understanding and reasoning in 3D scenes are crucial for the success of autonomous driving systems. The evolution of 3D semantic occupancy prediction as a pretraining task for autonomous driving and robotic applications captures finer 3D details compared to traditional 3D detection methods. Vision-based 3D semantic occupancy prediction is increasingly overlooked in favor of LiDAR-based approaches, which have shown superior performance in recent years. However, we present compelling evidence that there is still potential for enhancing vision-based methods. Existing approaches predominantly focus on spatial cues such as tri-perspective view (TPV) embeddings, often overlooking temporal cues. This study introduces S2TPVFormer, a spatiotemporal transformer architecture designed to predict temporally coherent 3D semantic occupancy. By introducing temporal cues through a novel Temporal Cross-View Hybrid Attention mechanism (TCVHA), we generate Spatiotemporal TPV (S2TPV) embeddings that enhance the prior process. Experimental evaluations on the nuScenes dataset demonstrate a significant +4.1% of absolute gain in mean Intersection over Union (mIoU) for 3D semantic occupancy compared to baseline TPVFormer, validating the effectiveness of S2TPVFormer in advancing 3D scene perception.
Paper Structure (32 sections, 4 equations, 7 figures, 7 tables)

This paper contains 32 sections, 4 equations, 7 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Latent 3D Scene Representations:
  4. 2D-3D View Transformation:
  5. Temporal Reasoning:
  6. Vision-centric 3D Semantic Occupancy Prediction:
  7. Methodology
  8. Overall Architecture
  9. Image Backbone
  10. S2TPV Encoder
  11. Unified Spatiotemporal Fusion:
  12. Temporal Cross-View Hybrid Attention:
  13. Experimental Setup and Implementation
  14. Datasets and Evaluation Metrics
  15. Training for 3D Semantic Occupancy Prediction and LiDAR Segmentation
  16. ...and 17 more sections

Figures (7)

  • Figure 1: Comparison of BEV, TPV, and Voxel latent vector fields used to represent 3D scenes.
  • Figure 2: The 3D SOP pipeline for the proposed S2TPVFormer architecture. The S2TPVFormer encoder layers consist of four main components: (a) Three learnable grid-shaped parameters to learn spatiotemporal queries, (b) Self-Attention module, (c) Spatial Fusion (VVT + SCA) Module, and (d) Temporal Cross-View Hybrid Attention (TCVHA) Module. Both (c) and (d) are encapsulated as the Unified Spatiotemporal Fusion Module in the block diagram.
  • Figure 3: Qualitative results on nuScenes validation set. TPVFormer's triperspective predictions are visualized on the left side, and S2TPVFormer's predictions are on the right side.
  • Figure 4: Potential of long-range temporal fusion.
  • Figure 5: This figure presents the confusion matrix of the S2TPVFormer-U (base) model's predictions. It is important to note that this confusion matrix corresponds to the same predictions analyzed in our paper, where we detail the per-class IoUs and the mean IoU for 3D Semantic Occupancy Prediction (SOP) on the nuScenes validation dataset.
  • ...and 2 more figures