Test-time Adaptation for Regression by Subspace Alignment

TL;DR

The paper tackles test-time adaptation for regression under covariate shift, where target unlabeled data differ in input distribution from the source. It introduces Significant-subspace Alignment (SSA), a regression-tailored TTA method that detects a representative subspace via PCA and weighs subspace directions by their impact on the output, performing feature alignment within this subspace using a symmetric KL objective. SSA updates only the affine parameters of normalization layers, avoiding architecture changes and preserving source knowledge, and shows consistent improvements across SVHN→MNIST, UTKFace, Biwi Kinect, and California Housing compared to baselines designed for classification. The approach also analyzes subspace properties, demonstrates stability and improved Gaussianity of projected features, and confirms the method’s applicability to multi-task regression and beyond, highlighting its practical impact for real-world domain shifts in regression tasks.

Abstract

This paper investigates test-time adaptation (TTA) for regression, where a regression model pre-trained in a source domain is adapted to an unknown target distribution with unlabeled target data. Although regression is one of the fundamental tasks in machine learning, most of the existing TTA methods have classification-specific designs, which assume that models output class-categorical predictions, whereas regression models typically output only single scalar values. To enable TTA for regression, we adopt a feature alignment approach, which aligns the feature distributions between the source and target domains to mitigate the domain gap. However, we found that naive feature alignment employed in existing TTA methods for classification is ineffective or even worse for regression because the features are distributed in a small subspace and many of the raw feature dimensions have little significance to the output. For an effective feature alignment in TTA for regression, we propose Significant-subspace Alignment (SSA). SSA consists of two components: subspace detection and dimension weighting. Subspace detection finds the feature subspace that is representative and significant to the output. Then, the feature alignment is performed in the subspace during TTA. Meanwhile, dimension weighting raises the importance of the dimensions of the feature subspace that have greater significance to the output. We experimentally show that SSA outperforms various baselines on real-world datasets.

Figures (6)

• Figure 1: Overview of significant-subspace alignment (SSA). A more detailed procedure is listed in \ref{['alg:proposed_method']}.
• Figure 2: Reconstruction error of features reconstructed with the source bases relative to the original target features. Note that Source and Prototype are the same, since Prototype does not update the feature extractor of the model. FR eastwood2022sourcefree and VM are not plotted in (a) because they had huge errors.
• Figure 3: Histograms of three randomly selected target feature dimensions. Left: Original features. Right: Projected features.
• Figure 4: PCA visualizations of source and target features of each dataset and method. The blue and orange dots represent the source and target features, respectively. We also report the optimal transport distance (OTD) between the principal components of the source and target features.
• Figure 5: UMAP mcinnes2018umap-software visualizations of source and target features on each dataset. The blue and orange dots represent the source and target features, respectively.