Rethinking Test-Time Training: Tilting The Latent Distribution For Few-Shot Source-Free Adaptation
Tahir Qasim Syed, Behraj Khan
TL;DR
This paper tackles the problem of adapting frozen foundation model components for few-shot classification without access to upstream data or parameter updates. It reframes adaptation as a KL-optimal change of measure over the latent embedding distribution induced by the encoder, implemented via exponential tilting using task-relevant scores derived from a small support set. By reweighting latent representations, predictions under the tilted distribution $P_\lambda$ incorporate downstream task structure without modifying the encoder, classifier, or decision rule, and without gradients. Empirically, tilting yields consistent gains across multiple datasets, backbones, and shot regimes, with transductive tilting further boosting performance and cross-domain generalization demonstrating the robustness of this inference-time distributional correction approach. The work highlights latent distribution reweighting as a viable, training-free alternative to parameter-based adaptation for constrained deployment of foundation models.
Abstract
Often, constraints arise in deployment settings where even lightweight parameter updates e.g. parameter-efficient fine-tuning could induce model shift or tuning instability. We study test-time adaptation of foundation models for few-shot classification under a completely frozen-model regime, where additionally, no upstream data are accessible. We propose arguably the first training-free inference method that adapts predictions to the new task by performing a change of measure over the latent embedding distribution induced by the encoder. Using task-similarity scores derived from a small labeled support set, exponential tilting reweights latent distributions in a KL-optimal manner without modifying model parameters. Empirically, the method consistently competes with parameter-update-based methods across multiple benchmarks and shot regimes, while operating under strictly and universally stronger constraints. These results demonstrate the viability of inference-level distributional correction for test-time adaptation even with a fully-frozen model pipeline.