OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Cong Hua, Qianqian Xu, Zhiyong Yang, Zitai Wang, Shilong Bao, Qingming Huang
TL;DR
This work targets open-world prompt tuning for Vision-Language Models by proposing a unified evaluation framework that jointly measures base-to-new detection and in-domain/class-level classification. It introduces OpenworldAUC, a pairwise ranking metric that remains robust to varying base/new distributions, and a learning framework called Gated Mixture-of-Prompts (GMoP) that optimizes OpenworldAUC via separate prompts for detection and two classifiers, with a gating mechanism and pseudo partitions. The authors provide a generalization bound and demonstrate state-of-the-art performance across 15 open-world benchmarks, showing improved trade-offs between detection and classification while maintaining efficiency. The approach relies on a principled combination of multiple prompts, a zero-shot new-domain classifier, and AUC-style ranking losses, which together offer practical, scalable open-world recognition and robust cross-domain transfer for CLIP-like models.
Abstract
Prompt tuning adapts Vision-Language Models like CLIP to open-world tasks with minimal training costs. In this direction, one typical paradigm evaluates model performance separately on known classes (i.e., base domain) and unseen classes (i.e., new domain). However, real-world scenarios require models to handle inputs without prior domain knowledge. This practical challenge has spurred the development of open-world prompt tuning, which demands a unified evaluation of two stages: 1) detecting whether an input belongs to the base or new domain (P1), and 2) classifying the sample into its correct class (P2). What's more, as domain distributions are generally unknown, a proper metric should be insensitive to varying base/new sample ratios (P3). However, we find that current metrics, including HM, overall accuracy, and AUROC, fail to satisfy these three properties simultaneously. To bridge this gap, we propose OpenworldAUC, a unified metric that jointly assesses detection and classification through pairwise instance comparisons. To optimize OpenworldAUC effectively, we introduce Gated Mixture-of-Prompts (GMoP), which employs domain-specific prompts and a gating mechanism to dynamically balance detection and classification. Theoretical guarantees ensure generalization of GMoP under practical conditions. Experiments on 15 benchmarks in open-world scenarios show GMoP achieves SOTA performance on OpenworldAUC and other metrics. We release the code at https://github.com/huacong/OpenworldAUC