TOMCAT: Test-time Comprehensive Knowledge Accumulation for Compositional Zero-Shot Learning

TL;DR

This work tackles label-distribution shift in compositional zero-shot learning by introducing TOMCAT, a test-time framework that accumulates unlabeled visual and textual knowledge to update multimodal prototypes. Textual prototypes are kept frozen while visual and textual representations are refined through Knowledge Accumulation Modules (KAMs) and an adaptive update mechanism, aided by a dynamic priority queue of high-confidence images. The approach is strengthened by multimodal collaborative representation learning to align text and vision, and an entropy-based objective guides refinement during testing. Empirical results on UT-Zappos, MIT-States, C-GQA, and Clothing16K show state-of-the-art performance in both closed-world and open-world CZSL, with substantial gains particularly in open-world scenarios. The method offers a practical path to robust, deployment-ready CZSL systems that leverage test-time unlabeled data without excessive latency.

Abstract

Compositional Zero-Shot Learning (CZSL) aims to recognize novel attribute-object compositions based on the knowledge learned from seen ones. Existing methods suffer from performance degradation caused by the distribution shift of label space at test time, which stems from the inclusion of unseen compositions recombined from attributes and objects. To overcome the challenge, we propose a novel approach that accumulates comprehensive knowledge in both textual and visual modalities from unsupervised data to update multimodal prototypes at test time. Building on this, we further design an adaptive update weight to control the degree of prototype adjustment, enabling the model to flexibly adapt to distribution shift during testing. Moreover, a dynamic priority queue is introduced that stores high-confidence images to acquire visual knowledge from historical images for inference. Considering the semantic consistency of multimodal knowledge, we align textual and visual prototypes by multimodal collaborative representation learning. Extensive experiments indicate that our approach achieves state-of-the-art performance on four benchmark datasets under both closed-world and open-world settings. Code will be available at https://github.com/xud-yan/TOMCAT .

Figures (11)

• Figure 1: At test time, existing methods (left) fail to leverage test images for model updates, obtaining wrong prediction distribution caused by label space shift, whereas our TOMCAT (right) accumulates multimodal knowledge from unsupervised data at test time to overcome the challenge.
• Figure 2: The overall architecture of our proposed TOMCAT at test time. The model accumulates multimodal knowledge to update prototypes to overcome the label distribution shift.
• Figure 3: Influence of the number of images in each priority queue $K$ on two datasets.
• Figure 4: Influence of the value of update control factor $\theta$ on two datasets.
• Figure 5: Trend of top-1 classification accuracy with increasing test sample size on three datasets.