Equivariant Spatio-Temporal Self-Supervision for LiDAR Object Detection

TL;DR

This work addresses the challenge of limited labeled data in 3D LiDAR perception by proposing E-SSL3D, a spatio-temporal equivariant self-supervision framework. The method enforces equivariance to global spatial transforms via point-level contrastive learning and augmentation classification, and to temporal changes via a 3D scene flow-based warping objective within a BYOL-style online/target setup. Experiments on KITTI-360, SemanticKITTI, and Waymo demonstrate consistent gains in 3D object detection, particularly in low-data regimes, with ablations clarifying when to favor contrastive versus augmentation-based losses. The approach offers a backbone-agnostic pre-training paradigm that improves downstream detectors like SECOND and VoxelRCNN and highlights the practical impact of incorporating realistic temporal deformations through scene flow.

Abstract

Popular representation learning methods encourage feature invariance under transformations applied at the input. However, in 3D perception tasks like object localization and segmentation, outputs are naturally equivariant to some transformations, such as rotation. Using pre-training loss functions that encourage equivariance of features under certain transformations provides a strong self-supervision signal while also retaining information of geometric relationships between transformed feature representations. This can enable improved performance in downstream tasks that are equivariant to such transformations. In this paper, we propose a spatio-temporal equivariant learning framework by considering both spatial and temporal augmentations jointly. Our experiments show that the best performance arises with a pre-training approach that encourages equivariance to translation, scaling, and flip, rotation and scene flow. For spatial augmentations, we find that depending on the transformation, either a contrastive objective or an equivariance-by-classification objective yields best results. To leverage real-world object deformations and motion, we consider sequential LiDAR scene pairs and develop a novel 3D scene flow-based equivariance objective that leads to improved performance overall. We show our pre-training method for 3D object detection which outperforms existing equivariant and invariant approaches in many settings.

Figures (3)

• Figure 1: On the left, an illustration of invariance (as described in Eq. \ref{['eq:inv']}) vs. equivariance (as described in Eq. \ref{['eq:equi']}) on the right. Invariance of $f$ to group of transformations $G$ means that the output representation does not change with the applied input transformation, whereas equivariance of $f$ to $\mathcal{G}$ means the output representation changes by $T'_g$ for some applied input transformation $T_g$. In this visualization, $T_g$ is 3D rotation. Figure based on devillers2022equimod.
• Figure 2: An overview of E-SSL3D A LiDAR point cloud undergoes spatial and temporal augmentations before being input to the network that consists of a 3D feature extraction backbone $f$, a projector network $m$, a predictor network $q$, and a classifier $s$. ($f$, $m$, $q$) form the online branch and the copies ($f'$, $m'$) form the target network that is only updated through an exponential moving average (EMA) of the weights of the online network.
• Figure 3: Relative 3D mean average precision of the object detector SECOND second pre-trained for equivariance for the random spatial augmentations flip, rotation, translation, and scaling using the contrastive and classification objectives. The baseline network is pre-trained to be equivariant to the "random flip" augmentation using contrastive learning.