Backpropagation-free Network for 3D Test-time Adaptation

TL;DR

This work tackles 3D test-time adaptation by introducing BFTT3D, a backpropagation-free framework that adapts at inference without updating parameters or relying on noisy pseudo-labels. A frozen source model collaborates with a target-specific non-parametric adaptation that uses a fixed prototype memory and a shared subspace to compute a target logit, which is then adaptively fused with the source logit based on entropy. The approach leverages subspace learning via Transfer Component Analysis to minimize domain divergence and is validated on ModelNet-40C and ScanObjectNN-C, showing superior robustness to corruptions and distribution shifts while being computationally efficient. The method holds practical significance for real-time 3D perception systems where constant data drift occurs and memory and compute are constrained.

Abstract

Real-world systems often encounter new data over time, which leads to experiencing target domain shifts. Existing Test-Time Adaptation (TTA) methods tend to apply computationally heavy and memory-intensive backpropagation-based approaches to handle this. Here, we propose a novel method that uses a backpropagation-free approach for TTA for the specific case of 3D data. Our model uses a two-stream architecture to maintain knowledge about the source domain as well as complementary target-domain-specific information. The backpropagation-free property of our model helps address the well-known forgetting problem and mitigates the error accumulation issue. The proposed method also eliminates the need for the usually noisy process of pseudo-labeling and reliance on costly self-supervised training. Moreover, our method leverages subspace learning, effectively reducing the distribution variance between the two domains. Furthermore, the source-domain-specific and the target-domain-specific streams are aligned using a novel entropy-based adaptive fusion strategy. Extensive experiments on popular benchmarks demonstrate the effectiveness of our method. The code will be available at \url{https://github.com/abie-e/BFTT3D}.

Figures (3)

• Figure 1: (a) Baseline. When faced with new point cloud samples at test time $t$, most existing methods generate the pseudo-labels and train the source model in a self-supervised manner. (b) Backpropagation-free test-time 3D model (BFTT3D). BFTT3D adopts a backpropagation-free adaptation module to output the target-specific logit, which fuses with the logit from the source model for prediction. Compared to the baseline, BFTT3D does not require any pseudo-labeling process and backpropagation.
• Figure 2: The framework of backpropagation-free test-time 3D model (BFTT3D). In the preparation stage, we first extract general features for source point cloud data $\mathbf{x}_s$ using a non-parametric network and then select a subset of all general features as static prototype memory $\mathcal{M}$. At test time, BFTT3D again adopts the non-parametric network to extract the general feature representation $\mathbf{f}_{t}$ from the given test point cloud sample $\mathbf{x}_t$ of $t$ domain. The feature $\mathbf{f}_{t}$ is then compared with static prototype feature $\mathbf{f}_c \in \mathcal{M}_c$ on a shared subspace to compute the target-specific logit $\mathbf{l}_{bf}$. Finally, the logit $\mathbf{l}_{bf}$ supplements the logit produced by the source model, $\mathbf{l}_{s}$, via an adaptive fusion module based on prediction entropy to output the final logit $\mathbf{l}_{t}$ for prediction. Notably, each module of BFTT3D, including the non-parametric network, subspace learning, and adaptive fusion, does not introduce any parameters that need backpropagation during adaptation.
• Figure 3: Comparisons of mean error between adaptive and optimal ratio. Using the backbone PointNet qi2017pointnet and testing on ModelNet40-C sun2022benchmarking. The blue line represents the optimal $p$ from an exhaustive search, and the green line represents our adaptive ratio $p$.