Is Less More? Exploring Token Condensation as Training-free Test-time Adaptation

TL;DR

Is Less More? Isolates training-free test-time adaptation for vision-language models via Token Condensation as Adaptation (TCA). The approach uses a domain-aware reservoir of anchor tokens and cross-head token reduction to selectively prune and merge tokens, supplemented by logits self-correction to align visual and textual embeddings without updating parameters. Empirical results across cross-dataset and CIFAR-100-C demonstrate notable accuracy gains (up to 21.4% over strong baselines) and substantial GFLOP reductions (12.2%–48.9%), with robust performance across CLIP and SigLIP families. The work advances practical, efficient TTA for multimodal models, while offering theoretical intuition via PAC-Bayes-inspired analysis and highlighting generalizability and limitation considerations for real-world deployment.

Abstract

Contrastive Language-Image Pretraining (CLIP) excels at learning generalizable image representations but often falls short in zero-shot inference on certain downstream datasets. Test-time adaptation (TTA) mitigates this issue by adjusting components like normalization layers or context prompts, yet it typically requires large batch sizes and extensive augmentations, leading to high computational costs. This raises a key question: Can VLMs' performance drop in specific test cases be mitigated through efficient, training-free approaches? To explore the solution, we investigate token condensation (TC) techniques, originally designed to enhance vision transformer efficiency by refining token usage during inference. We observe that informative tokens improve visual-text alignment in VLMs like CLIP on unseen datasets. However, existing TC methods often fail to maintain in-distribution performance when reducing tokens, prompting us to ask: How can we transform TC into an effective ``free-lunch'' adaptation strategy for VLMs? To address this, we propose Token Condensation as Adaptation (TCA), a training-free adaptation method that takes a step beyond standard TC. Rather than passively discarding tokens, TCA condenses token representation by introducing reservoir-based domain anchor tokens for information-preserving token reduction and logits correction. TCA achieves up to a 21.4% performance improvement over the strongest baseline on cross-dataset benchmark and the CIFAR-100-Corrupted dataset while reducing GFLOPs by 12.2% to 48.9%, with minimal hyperparameter dependency on both CLIP and SigLIP series.

Figures (8)

• Figure 1: Proposed Token Condensation as Adaptation (TCA). Using CLIP as an example, to adapt visual embeddings to text embeddings during test-time, TCA utilizes a domain-aware token reservoir (DTR) to retain historical <cls> tokens with the lowest uncertainty as domain anchor tokens. These anchor tokens assist in (1) condensing tokens with low attentiveness scores (top-right) and (2) acting as token-level classifiers to refine predictions through logits self-correction, moving visual embeddings $\mathbf{z}_t$ toward text embeddings $\mathbf{t}_c$.
• Figure 2: Empirical studies of token influence and the strategy of caching domain anchor token (i.e., <cls> tokens in CLIP) to improve alignment.
• Figure 3: An overview of domain-aware cross-head token pruning.
• Figure 4: Impact of reservoir size and GFLOPs budgets.
• Figure 5: Accuracy gain of TCA and TDA + EViT/ToME over TDA combined with ATS.