MFSeg: Efficient Multi-frame 3D Semantic Segmentation

TL;DR

MFSeg addresses the high computational cost of multi-frame 3D semantic segmentation by performing feature-level aggregation of voxel-based local features, guided by an algebraic regularization that promotes associativity and a homomorphism-like behavior. It introduces a commutative monoid-based feature aggregator and a lightweight MLP-based point decoder, preserving compatibility with state-of-the-art point-based backbones like PTv3. Across nuScenes and Waymo, MFSeg achieves state-of-the-art-like accuracy without test-time augmentation and with substantially lower latency than dense multi-frame baselines, outperforming single-frame methods especially on vulnerable road users. The approach offers practical impact for real-time robotic perception by balancing accuracy and efficiency in multi-frame fusion. All mathematical formulations and learning objectives are designed to maintain the geometric coherence of multi-frame point clouds while avoiding costly upsampling of past-frame features.

Abstract

We propose MFSeg, an efficient multi-frame 3D semantic segmentation framework. By aggregating point cloud sequences at the feature level and regularizing the feature extraction and aggregation process, MFSeg reduces computational overhead while maintaining high accuracy. Moreover, by employing a lightweight MLP-based point decoder, our method eliminates the need to upsample redundant points from past frames. Experiments on the nuScenes and Waymo datasets show that MFSeg outperforms existing methods, demonstrating its effectiveness and efficiency.

Figures (7)

• Figure 1: Relative mIoU improvement between various levels of multi-frame point cloud concatenation and single-frame input. Performance are evaluated using a SparseUNet graham2018spunet on the Waymo dataset sun2020waymo.
• Figure 2: Inference time of PTv3 wu2024ptv3 and MFSeg under a range of point cloud sequence length on the nuScenes dataset caesar2020nuscenes (left) and Waymo dataset sun2020waymo (right). PTv3-ds denotes that the point cloud is downsampled using a voxel-grid with size 0.05 $\mathrm{m}^3$, while PTv3 is evaluated with minimum input downsampling.
• Figure 3: An overview of the proposed MFSeg architecture.
• Figure 4: The feature association process (\ref{['sec:feature-association']}). Previous frame LFE features first undergo ego-motion correction, then are jointly voxelized with the current frame features to establish association. Features that cannot be associated are paired with the zero vector $\mathbf{0}$.
• Figure 5: The proposed training scheme and auxiliary loss for the LFE and the feature aggregator.