TeFF: Tracking-enhanced Forgetting-free Few-shot 3D LiDAR Semantic Segmentation

TL;DR

This paper addresses the limitations of current few-shot semantic segmentation by exploiting the temporal continuity of LiDAR data by employing a tracking model to generate pseudo-ground-truths from a sequence of LiDAR frames, significantly augments the dataset.

Abstract

In autonomous driving, 3D LiDAR plays a crucial role in understanding the vehicle's surroundings. However, the newly emerged, unannotated objects presents few-shot learning problem for semantic segmentation. This paper addresses the limitations of current few-shot semantic segmentation by exploiting the temporal continuity of LiDAR data. Employing a tracking model to generate pseudo-ground-truths from a sequence of LiDAR frames, our method significantly augments the dataset, enhancing the model's ability to learn on novel classes. However, this approach introduces a data imbalance biased to novel data that presents a new challenge of catastrophic forgetting. To mitigate this, we incorporate LoRA, a technique that reduces the number of trainable parameters, thereby preserving the model's performance on base classes while improving its adaptability to novel classes. This work represents a significant step forward in few-shot 3D LiDAR semantic segmentation for autonomous driving. Our code is available at https://github.com/junbao-zhou/Track-no-forgetting.

Figures (3)

• Figure 1: Demonstration of our TeFF . In novel data fine-tuning stage, we firstly track each ground truth with tracking model $\mathbf{Track}(\cdot)$, forwardly ($t+T$) and backwardly ($t-T$). The tracking results serve as pseudo ground truths and are combined with ground truths to supervise the novel model. We use unbias cross entropy $\Tilde{\mathcal{L}}_{CE}$, unbias distillation $\Tilde{\mathcal{L}}_{DS}$ and ${\text{Lov}\acute{\text{a}}\text{sz} }$ softmax loss $\mathcal{L}_{LS}$ to fine-tune the model. We further apply LoRA to novel model, which reduces the trainable parameters, thus achieving the goal of forgetting-free.
• Figure 2: Ablation study of tracking gap on SemanticKITTI validation set. It is tested with $\text{shot}=2$ and the tracking frame number is 20 (10 forward and 10 backward).
• Figure 3: Ablation study of tracking frame numbers on SemanticKITTI validation set. It is tested with $\text{shot}=2$ and the tracking gap is 15.