Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Manboformer: Learning Gaussian Representations via Spatial-temporal Attention Mechanism

Ziyue Zhao, Qining Qi, Jianfa Ma

TL;DR

This paper tackles 3D semantic occupancy prediction for autonomous driving in a vision-only setting, addressing the high memory cost of dense voxel grids. It builds on GaussianFormer, representing scenes with sparse 3D Gaussians and achieves substantial memory savings around $17.8\%$–$24.8\%$, while acknowledging a performance gap. To boost accuracy, it introduces ManboFormer by incorporating a spatial-temporal self-attention mechanism inspired by BEVFormer, enabling temporal fusion of Gaussian features. Experiments on NuScenes reveal convergence challenges and outline directions for improvement, including richer historical context, better reduction of Gaussian overlap through GMMs, and leveraging multi-sensor initialization.

Abstract

Compared with voxel-based grid prediction, in the field of 3D semantic occupation prediction for autonomous driving, GaussianFormer proposed using 3D Gaussian to describe scenes with sparse 3D semantic Gaussian based on objects is another scheme with lower memory requirements. Each 3D Gaussian function represents a flexible region of interest and its semantic features, which are iteratively refined by the attention mechanism. In the experiment, it is found that the Gaussian function required by this method is larger than the query resolution of the original dense grid network, resulting in impaired performance. Therefore, we consider optimizing GaussianFormer by using unused temporal information. We learn the Spatial-Temporal Self-attention Mechanism from the previous grid-given occupation network and improve it to GaussianFormer. The experiment was conducted with the NuScenes dataset, and the experiment is currently underway.

Manboformer: Learning Gaussian Representations via Spatial-temporal Attention Mechanism

TL;DR

This paper tackles 3D semantic occupancy prediction for autonomous driving in a vision-only setting, addressing the high memory cost of dense voxel grids. It builds on GaussianFormer, representing scenes with sparse 3D Gaussians and achieves substantial memory savings around –, while acknowledging a performance gap. To boost accuracy, it introduces ManboFormer by incorporating a spatial-temporal self-attention mechanism inspired by BEVFormer, enabling temporal fusion of Gaussian features. Experiments on NuScenes reveal convergence challenges and outline directions for improvement, including richer historical context, better reduction of Gaussian overlap through GMMs, and leveraging multi-sensor initialization.

Abstract

Compared with voxel-based grid prediction, in the field of 3D semantic occupation prediction for autonomous driving, GaussianFormer proposed using 3D Gaussian to describe scenes with sparse 3D semantic Gaussian based on objects is another scheme with lower memory requirements. Each 3D Gaussian function represents a flexible region of interest and its semantic features, which are iteratively refined by the attention mechanism. In the experiment, it is found that the Gaussian function required by this method is larger than the query resolution of the original dense grid network, resulting in impaired performance. Therefore, we consider optimizing GaussianFormer by using unused temporal information. We learn the Spatial-Temporal Self-attention Mechanism from the previous grid-given occupation network and improve it to GaussianFormer. The experiment was conducted with the NuScenes dataset, and the experiment is currently underway.

Paper Structure

This paper contains 17 sections, 6 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. 3D Semantic Occupancy Prediction
  4. 3D Gaussian Representation
  5. Gaussian Representation of 3D Occupancy Prediction
  6. Methodology
  7. GaussianFormer
  8. 3D Semantics Gaussian OCC representation
  9. Deformable Transformation
  10. Temporal Self-Attention
  11. ManboFormer
  12. Experiments
  13. Datasets
  14. Evaluation Metrics
  15. Implementation Details
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Comparisions of the proposed 3D Gaussian representation with exiting grid-based scene representations (figures from TPVFormer huang2023tri). The voxel representation li2023voxformerwei2023surroundocc assigns each voxel in the 3D space with a feature and is redundant due to the sparsity nature of the 3D space. BEV BEVFormer and TPV huang2023tri employ 2D planes to describe 3D space but can only alleviate the redundancy issue. Differently, the proposed object-centric 3D Gaussian representation can adapt to flexible regions of interest yet can still describe the fine-grained structure of the 3D scene due to the strong approximating ability of mixing Gaussians. GaussianFormer
  • Figure 2: Framework of our ManboFormer with Temporal Self-Attention.