IOTA: Corrective Knowledge-Guided Prompt Learning via Black-White Box Framework

TL;DR

This work tackles the challenge of adapting large pre-trained models to downstream tasks without disrupting upstream knowledge. It proposes IOTA, a Black-White Box prompt-learning framework that couples a frozen Black Box base learner with a White Box that verbalizes corrective knowledge into true-wrong prompts and selects suitable prompts via a MATCH-based strategy. The method demonstrates consistent improvements over state-of-the-art PET methods across 12 image classification benchmarks in both few-shot and easy-to-hard settings, highlighting the value of corrective knowledge and interpretable guidance. The approach is efficient, with most parameters frozen and only a small prompt-related component trained, and shows strong potential for interpretable, knowledge-guided model adaptation in vision-language systems.

Abstract

Recently, adapting pre-trained models to downstream tasks has attracted increasing interest. Previous Parameter-Efficient-Tuning (PET) methods regard the pre-trained model as an opaque Black Box model, relying purely on data-driven optimization and underutilizing their inherent prior knowledge. This oversight limits the models' potential for effective downstream task adaptation. To address these issues, we propose a novel black-whIte bOx prompT leArning framework (IOTA), which integrates a data-driven Black Box module with a knowledge-driven White Box module for downstream task adaptation. Specifically, the White Box module derives corrective knowledge by contrasting the wrong predictions with the right cognition. This knowledge is verbalized into interpretable human prompts and leveraged through a corrective knowledge-guided prompt selection strategy to guide the Black Box module toward more accurate predictions. By jointly leveraging knowledge- and data-driven learning signals, IOTA achieves effective downstream task adaptation. Experimental results on 12 image classification benchmarks under few-shot and easy-to-hard adaptation settings demonstrate the effectiveness of corrective knowledge and the superiority of our method over state-of-the-art methods.

Figures (7)

• Figure 1: Comparison of existing Black Box methods and our Black-White Box method. (a) Our IOTA uses the knowledge-guided prompt learning mechanism (i.e. White Box module) to correct the wrong prediction of the base learner (i.e. Black Box module). (b) Our IOTA surpasses the SOTA methods on 12 datasets.
• Figure 2: An overview of our IOTA framework. The corrective knowledge is leveraged to design a knowledge-guided prompt learning mechanism (i.e. the White Box module), which provides interpretable guidance to enhance the performance of the base learner (i.e. the Black Box module).
• Figure 3: Comparison with the SOTA methods on 12 datasets under the few-shot downstream task adaptation setting, where the base learner is ViT-B/32. (a) 16-shot. (b) 8-shot.
• Figure 4: Performance comparison under the easy-to-hard downstream task adaptation setting, where the base learner is ViT-B/16. Our method performs best on almost all datasets under three easy-to-hard downstream task adaptation settings, namely (a) 16/stage, (b) 8/stage, and (c) 4/stage.
• Figure 5: Ablation studies of the length and depth of our learnable prompt $\textbf{v}$ on all 12 datasets, where the base learner is ViT-B/16.