TeD-Loc: Text Distillation for Weakly Supervised Object Localization

TL;DR

TeD-Loc tackles weakly supervised object localization by leveraging CLIP's text embeddings to supervise patch-level visual representations without requiring test-time class labels. It distills text anchors into patch embeddings within a MIL framework, using pseudo-labels from CAMs and a FG/BG patch classifier, while orthogonalizing text embeddings via QR to reduce semantic overlap. The model yields a global embedding aligned with the class anchor and computes localization from patch-level signals, eliminating the need for external classifiers. On CUB and ILSVRC, TeD-Loc achieves state-of-the-art localization metrics with substantially lower computational cost than GenPromp, demonstrating practical efficiency for simultaneous localization and classification in weakly supervised settings.

Abstract

Weakly supervised object localization (WSOL) using classification models trained with only image-class labels remains an important challenge in computer vision. Given their reliance on classification objectives, traditional WSOL methods like class activation mapping focus on the most discriminative object parts, often missing the full spatial extent. In contrast, recent WSOL methods based on vision-language models like CLIP require ground truth classes or external classifiers to produce a localization map, limiting their deployment in downstream tasks. Moreover, methods like GenPromp attempt to address these issues but introduce considerable complexity due to their reliance on conditional denoising processes and intricate prompt learning. This paper introduces Text Distillation for Localization (TeD-Loc), an approach that directly distills knowledge from CLIP text embeddings into the model backbone and produces patch-level localization. Multiple instance learning of these image patches allows for accurate localization and classification using one model without requiring external classifiers. Such integration of textual and visual modalities addresses the longstanding challenge of achieving accurate localization and classification concurrently, as WSOL methods in the literature typically converge at different epochs. Extensive experiments show that leveraging text embeddings and localization cues provides a cost-effective WSOL model. TeD-Loc improves Top-1 LOC accuracy over state-of-the-art models by about 5% on both CUB and ILSVRC datasets, while significantly reducing computational complexity compared to GenPromp.

Figures (5)

• Figure 1: Comparison of our TeD-Loc versus CLIP-ES lin23 methods for extracting localization maps from CLIP. (A) CLIP-ES utilizes Grad-CAM to extract localization maps from CLIP, requiring GT class labels during inference. (B) In contrast, our TeD-Loc model distills knowledge from CLIP text embeddings into the visual encoder during training, allowing it to produce both classification scores and localization maps without requiring class labels during inference.
• Figure 2: Overview of the TeD-Loc method for distilling FG text embeddings into the patch embedding backbone. First, pseudo-labels are extracted to guide the identification of FG and BG patches. By leveraging these FG/BG regions, the model minimizes the similarity of $E_V$ with the relevant text embedding for FG classes, while maximizing dissimilarity with embeddings of other classes. Through a binary FG/BG classifier, TeD-Loc generates localization maps by classifying patches as FG or BG, while generating class probabilities for image classification. This joint task enables the model to produce both accurate localization and classification outputs without explicit bounding box supervision.
• Figure 3: t-SNE visualizations of CLIP text embeddings for ILSVRC imagenet_cvpr09 classes before and after orthogonalization. (Left) Prior to orthogonalization, embeddings of semantically similar classes (e.g., "airplane" and "aircraft") cluster closely together, leading to potential confusion. (Right) After orthogonalization (QR decomposition), the embeddings are more uniformly distributed and orthogonal, reducing overlap.
• Figure 4: Qualitative comparison on the ILSVRC dataset. GenPrompt fails to localize objects in complex scenes, often due to its dependency on external classifiers for computing text embeddings, which can fail if the classifier makes mistakes. This dependence on class labels during inference highlights GenPrompt's vulnerability to localization errors. In contrast, TeD-Loc can localize objects within complex scenes. Here, green bboxes denote GT localization, while red bboxes represent predicted localizations.
• Figure 5: Visualization of localization map defined via (patch, class) embeddings dot product: ${\langle z_p, t_y \rangle}$ where $z_p, \forall p \in \Omega$ is the patch embeddings, and $y$ is the true image class over different variants of CLIP, and our method. TE@PE is the vanilla CLIPradford2021learning where TE is the text embedding, and PE is the patch embedding.