ProtoDCS: Towards Robust and Efficient Open-Set Test-Time Adaptation for Vision-Language Models

TL;DR

ProtoDCS is proposed, a robust framework for OSTTA that effectively separates csID and csOOD samples, enabling safe and efficient adaptation of VLMs to csID data, and an evidence-driven adaptation strategy utilizing uncertainty-aware loss and efficient prototype-level updates, mitigating overconfidence and reducing computational overhead.

Abstract

Large-scale Vision-Language Models (VLMs) exhibit strong zero-shot recognition, yet their real-world deployment is challenged by distribution shifts. While Test-Time Adaptation (TTA) can mitigate this, existing VLM-based TTA methods operate under a closed-set assumption, failing in open-set scenarios where test streams contain both covariate-shifted in-distribution (csID) and out-of-distribution (csOOD) data. This leads to a critical difficulty: the model must discriminate unknown csOOD samples to avoid interference while simultaneously adapting to known csID classes for accuracy. Current open-set TTA (OSTTA) methods rely on hard thresholds for separation and entropy minimization for adaptation. These strategies are brittle, often misclassifying ambiguous csOOD samples and inducing overconfident predictions, and their parameter-update mechanism is computationally prohibitive for VLMs. To address these limitations, we propose Prototype-based Double-Check Separation (ProtoDCS), a robust framework for OSTTA that effectively separates csID and csOOD samples, enabling safe and efficient adaptation of VLMs to csID data. Our main contributions are: (1) a novel double-check separation mechanism employing probabilistic Gaussian Mixture Model (GMM) verification to replace brittle thresholding; and (2) an evidence-driven adaptation strategy utilizing uncertainty-aware loss and efficient prototype-level updates, mitigating overconfidence and reducing computational overhead. Extensive experiments on CIFAR-10/100-C and Tiny-ImageNet-C demonstrate that ProtoDCS achieves state-of-the-art performance, significantly boosting both known-class accuracy and OOD detection metrics. Code will be available at https://github.com/O-YangF/ProtoDCS.

Figures (7)

• Figure 1: Performance comparison on CIFAR-10-C. Our ProtoDCS simultaneously achieves the highest known-class accuracy (ACC) and the best OOD detection performance (AUROC), significantly outperforming all baseline methods. The top-right region indicates better performance.
• Figure 2: Overview of the proposed ProtoDCS. For an incoming unlabeled sample $x$, ProtoDCS first computes its initial openness score $S_{open}(x)$. The First-Check stage uses this score for an initial separation: confident samples ($S_{open}(x) < \Theta_a$) populate the diversity-aware visual cache to build visual prototypes $\bm{P}_v$, while trustworthy samples ($S_{open}(x) < \Theta_b$) undergo evidence-driven optimization to generate temporary prototypes $\bm{P}'_v, \bm{P}'_t$ for a refined openness assessment. The Final-Verification stage then leverages a GMM on the (re-evaluated) openness scores for all samples to make a probabilistic csID/csOOD decision. Only the sample confirmed as csID through this double-check process contribute to the final prototype evolution, forming a robust closed-loop adaptation system free from csOOD corruption.
• Figure 3: Uncertainty Landscapes in Feature Space. We compare the uncertainty surface constructed by (a) Standard Entropy Minimization and (b) our Evidence-Driven Uncertainty-Aware (EDUA) loss. The background color intensity denotes the uncertainty level derived from the loss function, where Yellow/Light indicates lower uncertainty and Purple/Dark indicates higher uncertainty. Crucially, Entropy Minimization creates a deceptive landscape where csOOD samples fall into low-uncertainty regions. In contrast, ProtoDCS correctly maps the csOOD and boundary regions to high uncertainty (dark background), effectively filtering out risky samples during prototype evolution.
• Figure 4: t-SNE Visualization of the Visual Cache on CIFAR-100-C. This figure illustrates how the cached features evolve over time, comparing the state after 3,000 samples (left) with the state after 30,000 samples (right). As more data is processed, our diversity-aware update mechanism creates increasingly compact and representative class clusters.
• Figure 5: Calibration and Uncertainty Analysis on Tiny-ImageNet-C. This figure provides a tripartite analysis demonstrating the safe adaptation capability of ProtoDCS in open-set TTA. (a) Confidence Distribution: TPT exhibits a skewed, low-confidence distribution, indicating predictive confusion induced by Entropy Minimization (EM). ProtoDCS recovers a balanced, well-calibrated distribution through its evidence-driven uncertainty-aware (EDUA) loss. (b) Reliability Diagram: While C-TPT shows perfect calibration (diagonal alignment), ProtoDCS adopts a deliberately conservative strategy (curve above diagonal), systematically assigning underconfidence on uncertain samples to enhance safety against OOD data. (c) Accuracy-Uncertainty Curve: ProtoDCS achieves the highest AUC (0.320), proving its superior ability to rank samples by risk (high uncertainty for OOD/hard ID), which is crucial for reliable sample filtering during adaptation. Replacing the EDUA loss with standard entropy (ProtoDCS(entropy)) yields a lower AUC (0.298).