Towards Class-agnostic Tracking Using Feature Decorrelation in Point Clouds

TL;DR

This paper tackles class-agnostic tracking in 3D LiDAR point clouds, where unseen object categories challenge the generalization of Siamese trackers. It introduces a feature decorrelation module that transforms fused template-search features $Z$ into $Z' = \tau(Z)$, uses Random Fourier Features to estimate pairwise dependence via $\hat{\Lambda}_{Z'_{i},Z'_{j}}$, and learns sample weights $\mathbf{w}$ by minimizing the Frobenius norm, alternating with updates to the tracking network. Evaluations on KITTI and NuScenes show that applying the decorrelation framework to baselines P2B and BAT yields improved unseen-object tracking while maintaining performance on seen categories. The work highlights distribution shift as a key obstacle to generalization and demonstrates that explicit decorrelation of fused features can restore robust class-agnostic matching in point-cloud tracking, with practical implications for autonomous systems.

Abstract

Single object tracking in point clouds has been attracting more and more attention owing to the presence of LiDAR sensors in 3D vision. However, the existing methods based on deep neural networks focus mainly on training different models for different categories, which makes them unable to perform well in real-world applications when encountering classes unseen during the training phase. In this work, we investigate a more challenging task in the LiDAR point clouds, class-agnostic tracking, where a general model is supposed to be learned for any specified targets of both observed and unseen categories. In particular, we first investigate the class-agnostic performances of the state-of-the-art trackers via exposing the unseen categories to them during testing, finding that a key factor for class-agnostic tracking is how to constrain fused features between the template and search region to maintain generalization when the distribution is shifted from observed to unseen classes. Therefore, we propose a feature decorrelation method to address this problem, which eliminates the spurious correlations of the fused features through a set of learned weights and further makes the search region consistent among foreground points and distinctive between foreground and background points. Experiments on the KITTI and NuScenes demonstrate that the proposed method can achieve considerable improvements by benchmarking against the advanced trackers P2B and BAT, especially when tracking unseen objects.

Figures (8)

• Figure 1: Comparisons between class-specific and class-agnostic tracking in 3D point clouds. We show the precision (orange) and success (blue) metrics. Here, P2B and BAT focus on the class-specific tracking which train and test only on cars. Differently, for the class-agnostic tacking, the methods with the suffix "-U" are also tested on cars though they do not observe this category during training phrase. There exists the significant gaps between P2B (BAT) and P2B-U (BAT-U).
• Figure 2: Different categories have large point distribution differences. In light of this, exposing unseen categories to trackers can help test their class-agnostic tracking ability.
• Figure 3: The effect of the fused features $Z$ generated by the information embedding module. We use different colors to indicate the possibility that each point belongs to the tracking target. The lower the probability, the darker the color. The gray lines represent voting results, which reflect the contribution of each point for the final tracked box. (a)-(c) show three cases of spurious correlation, which lead to some undesirable clusters (highlighted by gray circle) and spread-out voting results. (d) is an ideal case where there remains consistency among the foreground points and discrimination between the background (dark) and foreground (yellow) points.
• Figure 4: Pipeline of the proposed method. The current 3D trackers often contain the modules of feature extraction, target information embedding, and proposal generation. The inputs of $P_T$ and $P_S$ are first sent to PointNet++ to extract features. Then, the baseline generates the fused features between the template and search region. Here, P2B uses the target-specific feature augmentation module (TFA), and BAT adopts the Box-aware feature fusion (BAFF). To make the fused features class-agnostic, we propose a feature decorrelation module based on the sample weighting method. Through alternative and iterative optimization, the resulting weights are then applied to the loss function computed in the proposal generation to make the points on the target surface more consistent.
• Figure 5: Illustration of class-specific and class-agnostic tracking. The circles, triangles, pentagons, and diamonds represent cars, cyclists, pedestrians, and vans, respectively. The training and testing sets are marked in blue and orange, respectively. The gray shape indicates it is not observed during training phase. As shown, the class-specific tracking trains and tests the models on the same category, whereas the class-agnostic tracking requires to evaluate the models on both observed and unseen categories.