COSMIC: Clique-Oriented Semantic Multi-space Integration for Robust CLIP Test-Time Adaptation

TL;DR

COSMIC addresses the brittleness of CLIP-based test-time adaptation under domain shift by combining multi-granular, cross-modal semantics with graph-structured querying. The Dual Semantics Graph (CSS and AFV spaces) enriches cached information with textual and fine-grained visual signals, while the Clique Guided Hyper-class (CGH) identifies robust class communities via maximal cliques to guide adaptive inference. This training-free framework achieves substantial gains on out-of-distribution and cross-domain benchmarks, outperforming prompt-based and cache-based baselines while maintaining reasonable efficiency. The approach highlights the practical value of structured semantic integration for robust zero-shot recognition in open-world settings.

Abstract

Recent vision-language models (VLMs) face significant challenges in test-time adaptation to novel domains. While cache-based methods show promise by leveraging historical information, they struggle with both caching unreliable feature-label pairs and indiscriminately using single-class information during querying, significantly compromising adaptation accuracy. To address these limitations, we propose COSMIC (Clique-Oriented Semantic Multi-space Integration for CLIP), a robust test-time adaptation framework that enhances adaptability through multi-granular, cross-modal semantic caching and graph-based querying mechanisms. Our framework introduces two key innovations: Dual Semantics Graph (DSG) and Clique Guided Hyper-class (CGH). The Dual Semantics Graph constructs complementary semantic spaces by incorporating textual features, coarse-grained CLIP features, and fine-grained DINOv2 features to capture rich semantic relationships. Building upon these dual graphs, the Clique Guided Hyper-class component leverages structured class relationships to enhance prediction robustness through correlated class selection. Extensive experiments demonstrate COSMIC's superior performance across multiple benchmarks, achieving significant improvements over state-of-the-art methods: 15.81% gain on out-of-distribution tasks and 5.33% on cross-domain generation with CLIP RN-50. Code is available at github.com/hf618/COSMIC.

Figures (9)

• Figure 1: (a) In the conventional cache-based method, the cache has only dull information with coarse-grained clip visual features and simple query way via similarity between samples and cached visual class centers. (b) In our COSMIC, the cache has diverse structural information via extra fine-grained DINOv2 visual features and effective query way via similarity between samples and meticulously designed hyper-class centers.
• Figure 2: (a) For various types of dogs, like "shiba inu" and "great Pyrenees" DINOv2 features offer more refined similarity perception than CLIP. (b) Samples in a single Gaussian cluster come from the same class, while the maximal clique cluster can explicitly represent cross-class correlations.
• Figure 3: Overview of COSMIC. To refine cache with cross-modal, multi-granular class features, we construct Dual Semantics Graph with complementary semantics, incorporating both joint modalities and fine-grained visual information. To efficiently query the compatibility of diverse semantics, we propose novel Clique Guided Hyper-class to model different communities in the cache as the test domain evolves, enabling adaptive querying of test samples.
• Figure 4: Visualization of the attention maps from CLIP ViT-B/16 and DINOv2 ViT-B/14, along with their top-5 prediction results.
• Figure 5: Ablation of $\beta_2$, $\beta_3$ with CLIP ViT-B/16 and DINOv2 ViT-S/14.