IMSE: Intrinsic Mixture of Spectral Experts Fine-tuning for Test-Time Adaptation

TL;DR

This paper proposes Intrinsic Mixture of Spectral Experts (IMSE) that leverages the spectral experts inherently embedded in Vision Transformers and introduces Domain-Aware Spectral Code Retrieval, which estimates input distributions to detect domain shifts, and retrieves adapted singular values for rapid adaptation.

Abstract

Test-time adaptation (TTA) has been widely explored to prevent performance degradation when test data differ from the training distribution. However, fully leveraging the rich representations of large pretrained models with minimal parameter updates remains underexplored. In this paper, we propose Intrinsic Mixture of Spectral Experts (IMSE) that leverages the spectral experts inherently embedded in Vision Transformers. We decompose each linear layer via singular value decomposition (SVD) and adapt only the singular values, while keeping the singular vectors fixed. We further identify a key limitation of entropy minimization in TTA: it often induces feature collapse, causing the model to rely on domain-specific features rather than class-discriminative features. To address this, we propose a diversity maximization loss based on expert-input alignment, which encourages diverse utilization of spectral experts during adaptation. In the continual test-time adaptation (CTTA) scenario, beyond preserving pretrained knowledge, it is crucial to retain and reuse knowledge from previously observed domains. We introduce Domain-Aware Spectral Code Retrieval, which estimates input distributions to detect domain shifts, and retrieves adapted singular values for rapid adaptation. Consequently, our method achieves state-of-the-art performance on various distribution-shift benchmarks under the TTA setting. In CTTA and Gradual CTTA, it further improves accuracy by 3.4 percentage points (pp) and 2.4 pp, respectively, while requiring 385 times fewer trainable parameters. Our code is available at https://github.com/baek85/IMSE.

Figures (5)

• Figure 1: IMSE-Retrieval with domain bank. When adapting to a new domain, we select initial singular values based on domain similarity using the domain descriptor, then fine-tune the $\boldsymbol{\sigma}$ components within linear layers. The adapted spectral code $\boldsymbol{S}$ is stored in the Domain Bank, and this process repeats for subsequent domains. Note that domain descriptors are designed to estimate the distribution of test data.
• Figure 2: (a) Comparison of top-$R$% vs. bottom-$R$% singular value selection. (b) Feature diversity across various training methods. (c) Adaptation performance across various training methods. (d) Impact of the threshold $\tau$ on domain shift detection and adaptation performance. CE and TTA losses denote $\mathcal{L}_{\text{ce}}$ and $\mathcal{L}_{\text{entmin}}$, respectively.
• Figure 3: Hyperparameter sensitivity of $\lambda_\text{dm}$.
• Figure 4: (a) Diversity of alignment patterns in 3rd Transformer Block. (b) Diversity of alignment patterns in 6th Transformer Block. (c) Diversity of alignment patterns in 9th Transformer Block.
• Figure 5: Domain distance matrix. Pairwise distance matrix among 15 domain descriptors.