Subspace Alignment for Vision-Language Model Test-time Adaptation

TL;DR

Vision-language models exhibit degraded performance under distribution shifts due to modality gap and visual nuisance. SubTTA introduces a subspace-alignment framework that uses chordal distance on Grassmannians to align the semantic subspaces of vision and language and applies a semantic projection to filter nuisance, enabling more reliable zero-shot guidance for test-time adaptation. Across CIFAR-10-C, CIFAR-100-C, and ImageNet-C with multiple CLIP backbones and TTA objectives, SubTTA achieves state-of-the-art robustness and improves zero-shot prediction quality by tightening cross-modal alignment. This approach offers a practical, geometry-driven pathway to robust VLM deployment in open-world settings.

Abstract

Vision-language models (VLMs), despite their extraordinary zero-shot capabilities, are vulnerable to distribution shifts. Test-time adaptation (TTA) emerges as a predominant strategy to adapt VLMs to unlabeled test data on the fly. However, existing TTA methods heavily rely on zero-shot predictions as pseudo-labels for self-training, which can be unreliable under distribution shifts and misguide adaptation due to two fundamental limitations. First (Modality Gap), distribution shifts induce gaps between visual and textual modalities, making cross-modal relations inaccurate. Second (Visual Nuisance), visual embeddings encode rich but task-irrelevant noise that often overwhelms task-specific semantics under distribution shifts. To address these limitations, we propose SubTTA, which aligns the semantic subspaces of both modalities to enhance zero-shot predictions to better guide the TTA process. To bridge the modality gap, SubTTA extracts the principal subspaces of both modalities and aligns the visual manifold to the textual semantic anchor by minimizing their chordal distance. To eliminate visual nuisance, SubTTA projects the aligned visual features onto the task-specific textual subspace, which filters out task-irrelevant noise by constraining visual embeddings within the valid semantic span, and standard TTA is further performed on the purified space to refine the decision boundaries. Extensive experiments on various benchmarks and VLM architectures demonstrate the effectiveness of SubTTA, yielding an average improvement of 2.24% over state-of-the-art TTA methods.

Figures (10)

• Figure 1: Failure modes of zero-shot prediction. (a) Modality gap: Visual features drift away from the textual manifold. A dog image shifts closer to the bird anchor. (b) Visual Nuisance: Task-irrelevant noise overshadows core semantics. a dog is misclassified as bird due to spurious correlation with the blue sky.
• Figure 2: Principal angles ($\downarrow$). Correct predictions exhibit consistently smaller principal angles than mispredictions. Increased shift level results in larger angles and more mispredictions.
• Figure 3: Semantic concentration ($\uparrow$). Correct predictions exhibit markedly higher semantic concentration than mispredictions. Increased shift level results in lower concentration and more mispredictions.
• Figure 4: Overview of SubTTA. (a) VLM encodes images and textual prompts into a shared raw space. (b) Geometric alignment alleviates modality gap by minimizing the chordal distance. (c) Semantic projection retains task-relevant information, e.g., category, while filtering out irrelevant ones, e.g., background color. (d) Standard TTA is further performed on the aligned subspace.
• Figure 5: TTA performance w/ and w/o SubTTA.