Decoupling Augmentation Bias in Prompt Learning for Vision-Language Models

TL;DR

This work tackles the generalization gap in prompt learning for vision–language models by integrating image-level augmentations into the prompt optimization process. It introduces AAPL, which uses a delta meta token to capture augmentation-specific attributes and an AdTriplet loss to adversarially decouple these attributes from class semantics, improving base-to-new and domain generalization. Across 11 datasets and multiple evaluation settings, AAPL matches or surpasses strong baselines like CoCoOp and ProGrad, with augmentation profiling and weighted sampling further enhancing robustness on challenging datasets. The proposed approach provides a practical pathway to more attribute-aware, robust prompt learning in vision–language systems, with publicly available code to foster reuse and extension.

Abstract

Recent advances in large-scale vision and language models have led to significant progress in zero-shot learning tasks. Methods such as CoOp and CoCoOp have shown that replacing handcrafted prompts with learnable vectors, known as prompt learning, can result in improved performance. However, these models often struggle to generalize to entirely unseen categories. While traditional zero-shot learning techniques benefit from various data augmentation strategies, prompt learning has primarily focused on text-based modifications, leaving the potential of image-based augmentation largely unexplored. In this work, we explore how image-level augmentations, particularly those that introduce attribute-specific variations, can support and enhance prompt learning. Our analysis examines the interaction between these augmentations and soft prompt frameworks, revealing their potential to improve generalization. We also identify a limitation in existing methods, such as CoCoOp, which do not provide explicit guidance for learning prompts that focus on semantically meaningful visual features. To address this, we propose Adding Attributes to Prompt Learning, AAPL, a novel method that introduces adversarial token embeddings to decouple superficial visual variations introduced by augmentation from class-relevant semantic representations. This decoupling enables the learned prompts to concentrate on visually discriminative features that align with the target categories. We conduct comprehensive experiments on eleven benchmark datasets, and AAPL consistently outperforms existing methods across few-shot, zero-shot, cross-dataset, and domain generalization settings. Our source code is publicly available at: https://github.com/Gahyeonkim09/AAPL

Figures (12)

• Figure 1: Comparison of prompt learning strategies in vision-language models. Zero-shot CLIP employs fixed, hand-crafted hard prompts, while CoOp replaces them with learnable soft prompts. CoCoOp further enhances prompt learning by introducing instance-specific biases through dynamic prompts. The proposed method, AAPL, proposes attribute-specific conditional learnable prompts that decompose image features into class semantics and attributes, injecting attribute-guided bias into the prompt. By leveraging attribute-specific information, AAPL improves adaptability to diverse contexts, leading to improved generalization and performance on unseen tasks.
• Figure 2: Overview of AAPL. Two randomly augmented images are processed through a metanet to generate meta tokens. Delta meta tokens are then computed by subtracting class-wise means and trained with AdTriplet loss to decouple augmentation-induced attributes from class-level semantics. In parallel, learnable prompt tokens are optimized through contrastive learning between image features and class-conditioned text embeddings. The class-relevant information extracted from the meta tokens is integrated into the prompt, allowing the model to account for augmentation-specific variation while preserving semantic consistency.
• Figure 3: The comparison between meta tokens of CoCoOp and meta tokens of CoCoOp with random augmentation for FGVCAircraft dataset.
• Figure 4: t-SNE visualization of the meta token and delta meta token from CoCoOp zhou2022conditional and AAPL on the Caltech101, FGVCAircraft, Food101 datasets. Point colors represent the 14 different augmentations, and 100 validation samples are used for visualization. $(a)$ and $(c)$ show the meta token, while $(b)$ and $(d)$ show the delta meta token.
• Figure 5: Comparison of the number of constraints of the AdTriplet loss. The constraints-2 setting's anchor is just one, e.g., $\Delta\pi^{1B}$, and the constraints-4 setting has two anchors, e.g., $\Delta\pi^{1A}$ and $\Delta\pi^{2B}$.