LiteEmbed: Adapting CLIP to Rare Classes

TL;DR

This work tackles CLIP's limited recognition of rare, emerging, or culturally specific classes by introducing LiteEmbed, a lightweight, few-shot adaptation that optimizes a single class token $e_c$ within CLIP's vocabulary without retraining encoders. It leverages a PCA-based subspace decomposition of CLIP's text space to separate coarse semantic alignment (high-variance directions) from fine-grained discrimination (low-variance directions) via three losses: $L_{ ext{img-align}}$, $L_{ ext{coarse}}$, and $L_{ ext{fine}}$, yielding a training-free, plug-and-play embedding applicable to classification, retrieval, segmentation, and detection. Empirically, LiteEmbed achieves significant gains over prior methods across eight datasets and the NOVA benchmark, improving open-vocabulary tasks and maintaining compositionality for prompts. The approach demonstrates practical impact by enabling robust open-vocabulary recognition for underrepresented or post-2023 concepts without large-scale retraining, pointing to broad applicability in multimodal systems.

Abstract

Large-scale vision-language models such as CLIP achieve strong zero-shot recognition but struggle with classes that are rarely seen during pretraining, including newly emerging entities and culturally specific categories. We introduce LiteEmbed, a lightweight framework for few-shot personalization of CLIP that enables new classes to be added without retraining its encoders. LiteEmbed performs subspace-guided optimization of text embeddings within CLIP's vocabulary, leveraging a PCA-based decomposition that disentangles coarse semantic directions from fine-grained variations. Two complementary objectives, coarse alignment and fine separation, jointly preserve global semantic consistency while enhancing discriminability among visually similar classes. Once optimized, the embeddings are plug-and-play, seamlessly substituting CLIP's original text features across classification, retrieval, segmentation, and detection tasks. Extensive experiments demonstrate substantial gains over prior methods, establishing LiteEmbed as an effective approach for adapting CLIP to underrepresented, rare, or unseen classes.

Figures (11)

• Figure 1: Performance disparity in CLIP’s zero-shot classification.
• Figure 2: Overview of LiteEmbed. Given a target class (e.g., anarsa) and a few reference images, the learnable token $e_c$ replaces the placeholder "*" in the prompt and is optimized while all CLIP encoders remain frozen. Subspace-Guided Optimization balances three objectives: image–text alignment ($\mathcal{L}_{\text{img-align}}$), coarse semantic anchoring ($\mathcal{L}_{\text{coarse}}$), and fine-grained separation ($\mathcal{L}_{\text{fine}}$), promoting discriminative yet semantically consistent embeddings.
• Figure 3: t-SNE visualization of CLIP’s text embeddings for anarsa and malapua. (a) Base CLIP embeddings before any adaptation. (b) Naïve image-only optimization leads to discriminative collapse, bringing the two classes too close. (c) Their nearest neighbors shift from related sweets to visually similar but semantically unrelated snacks, indicating semantic drift. (d) Our Subspace-Guided Optimization restores separation while preserving semantic relationships. Sample class images are shown below.
• Figure 4: PCA Analysis of CLIP's Text Embedding Space.
• Figure 5: Comparison of 4-shot continual adaptation results across sequentially added classes. LiteEmbed in red, ENGINE in orange, CoOp in blue, Continual-CLIP in purple, and MoE-Adapters in green; best viewed in color.