Multi-Class Textual-Inversion Secretly Yields a Semantic-Agnostic Classifier

TL;DR

This paper first explores Textual Inversion and reveals that the new concept tokens possess both generation and classification capabilities by regarding each category as a single concept, and proposes Multi-Class textual inversion, which includes a discriminative regularization term for the token updating process.

Abstract

With the advent of large pre-trained vision-language models such as CLIP, prompt learning methods aim to enhance the transferability of the CLIP model. They learn the prompt given few samples from the downstream task given the specific class names as prior knowledge, which we term as semantic-aware classification. However, in many realistic scenarios, we only have access to few samples and knowledge of the class names (e.g., when considering instances of classes). This challenging scenario represents the semantic-agnostic discriminative case. Text-to-Image (T2I) personalization methods aim to adapt T2I models to unseen concepts by learning new tokens and endowing these tokens with the capability of generating the learned concepts. These methods do not require knowledge of class names as a semantic-aware prior. Therefore, in this paper, we first explore Textual Inversion and reveal that the new concept tokens possess both generation and classification capabilities by regarding each category as a single concept. However, learning classifiers from single-concept textual inversion is limited since the learned tokens are suboptimal for the discriminative tasks. To mitigate this issue, we propose Multi-Class textual inversion, which includes a discriminative regularization term for the token updating process. Using this technique, our method MC-TI achieves stronger Semantic-Agnostic Classification while preserving the generation capability of these modifier tokens given only few samples per category. In the experiments, we extensively evaluate MC-TI on 12 datasets covering various scenarios, which demonstrates that MC-TI achieves superior results in terms of both classification and generation outcomes.

Figures (8)

• Figure 1: While both existing Single-Concept Textual Inversion and our proposed Multi-Class Textual Inversion (MC-TI) can generate satisfactory results with few samples per person, the single-concept TI lacks the ability to ensure discrimination performance. This is because it does not constrain token updates in a discriminative manner. (CelebA faces liu2015celeba.)
• Figure 2: Diffusion Classifier li2023diffusion_classifier approach classifies samples by computing the text-conditional likelihoods and requires knowledge of the category names. By comparison, Textual Inversion (TI) textual_inversion and MC-TI only need to learn the concept tokens with few-shot samples (5-shot in the figure examples). MC-TI further strengthen the TI by augmenting with a discriminative regularization term and significantly improve the performance. Custom Diffusion kumari2022customdiffusion is one of the best personalization methods by fine-tuning the UNet, it works poorly in classification.
• Figure 3: Illustration of our MC-TI approach. During the training of token <$s^*_k$> for concept $k$ (cat $k$ for example), we add a discriminative regularization term $\mathcal{L}_{reg}$, which is defined as the cosine cross-entropy from the current training text feature to the image features.
• Figure 4: To visualize the textual prompts features, we took the 5-shot conceptual tokens learned by Textual Inversion and MC-TI, respectively. By applying 27 types of various prompt templates, we visualize the PCA components in 2-D maps for 20 categories out of these six datasets. MC-TI improves the clustering of textual characteristics by enforcing discriminative regularization terms.
• Figure 5: MC-TI is compared with the Textual Inversion (TI), the CLIP-feat baseline, Diffusion Classifier (DiC) and prompt learning methods (CoOp, ProGrad) by computing classification accuracies. We vary the $N$-shot ($N=1,2,4,8,16$) numbers to draw the trend plots.