Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TokenCLIP: Token-wise Prompt Learning for Zero-shot Anomaly Detection

Qihang Zhou, Binbin Gao, Guansong Pang, Xin Wang, Jiming Chen, Shibo He

TL;DR

TokenCLIP addresses the limitation of indiscriminate textual alignment in CLIP-based zero-shot anomaly detection by introducing token-level supervision. It achieves this through a multi-head text prompt module that projects a base textual space into multiple orthogonal subspaces and an optimal transport based dynamic alignment that assigns each visual patch token to a semantically relevant subspace combination. The framework includes an end-to-end training objective combining global and base local semantics, a dynamic alignment loss, hinge regularization, and orthogonality constraints to promote subspace specialization. Empirically, TokenCLIP delivers state-of-the-art or competitive results on industrial and medical ZSAD benchmarks, with notable gains in pixel-level segmentation and robust cross-domain transfer, while maintaining modest computational overhead compared to competing methods.

Abstract

Adapting CLIP for anomaly detection on unseen objects has shown strong potential in a zero-shot manner. However, existing methods typically rely on a single textual space to align with visual semantics across diverse objects and domains. The indiscriminate alignment hinders the model from accurately capturing varied anomaly semantics. We propose TokenCLIP, a token-wise adaptation framework that enables dynamic alignment between visual and learnable textual spaces for fine-grained anomaly learning. Rather than mapping all visual tokens to a single, token-agnostic textual space, TokenCLIP aligns each token with a customized textual subspace that represents its visual characteristics. Explicitly assigning a unique learnable textual space to each token is computationally intractable and prone to insufficient optimization. We instead expand the token-agnostic textual space into a set of orthogonal subspaces, and then dynamically assign each token to a subspace combination guided by semantic affinity, which jointly supports customized and efficient token-wise adaptation. To this end, we formulate dynamic alignment as an optimal transport problem, where all visual tokens in an image are transported to textual subspaces based on semantic similarity. The transport constraints of OT ensure sufficient optimization across subspaces and encourage them to focus on different semantics. Solving the problem yields a transport plan that adaptively assigns each token to semantically relevant subspaces. A top-k masking is then applied to sparsify the plan and specialize subspaces for distinct visual regions. Extensive experiments demonstrate the superiority of TokenCLIP.

TokenCLIP: Token-wise Prompt Learning for Zero-shot Anomaly Detection

TL;DR

TokenCLIP addresses the limitation of indiscriminate textual alignment in CLIP-based zero-shot anomaly detection by introducing token-level supervision. It achieves this through a multi-head text prompt module that projects a base textual space into multiple orthogonal subspaces and an optimal transport based dynamic alignment that assigns each visual patch token to a semantically relevant subspace combination. The framework includes an end-to-end training objective combining global and base local semantics, a dynamic alignment loss, hinge regularization, and orthogonality constraints to promote subspace specialization. Empirically, TokenCLIP delivers state-of-the-art or competitive results on industrial and medical ZSAD benchmarks, with notable gains in pixel-level segmentation and robust cross-domain transfer, while maintaining modest computational overhead compared to competing methods.

Abstract

Adapting CLIP for anomaly detection on unseen objects has shown strong potential in a zero-shot manner. However, existing methods typically rely on a single textual space to align with visual semantics across diverse objects and domains. The indiscriminate alignment hinders the model from accurately capturing varied anomaly semantics. We propose TokenCLIP, a token-wise adaptation framework that enables dynamic alignment between visual and learnable textual spaces for fine-grained anomaly learning. Rather than mapping all visual tokens to a single, token-agnostic textual space, TokenCLIP aligns each token with a customized textual subspace that represents its visual characteristics. Explicitly assigning a unique learnable textual space to each token is computationally intractable and prone to insufficient optimization. We instead expand the token-agnostic textual space into a set of orthogonal subspaces, and then dynamically assign each token to a subspace combination guided by semantic affinity, which jointly supports customized and efficient token-wise adaptation. To this end, we formulate dynamic alignment as an optimal transport problem, where all visual tokens in an image are transported to textual subspaces based on semantic similarity. The transport constraints of OT ensure sufficient optimization across subspaces and encourage them to focus on different semantics. Solving the problem yields a transport plan that adaptively assigns each token to semantically relevant subspaces. A top-k masking is then applied to sparsify the plan and specialize subspaces for distinct visual regions. Extensive experiments demonstrate the superiority of TokenCLIP.
Paper Structure (41 sections, 3 theorems, 28 equations, 15 figures, 16 tables)

This paper contains 41 sections, 3 theorems, 28 equations, 15 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Zero-shot anomaly detection
  4. Prompt Learning
  5. Optimal Transport
  6. Methodology
  7. Premiliary
  8. Multi-head text prompt learning
  9. Dynamic Alignment via OT
  10. Training and inference
  11. Training
  12. Inference
  13. Experiments
  14. Dataset Details & Baselines
  15. Implementation details
  16. ...and 26 more sections

Key Result

Theorem 3.1

For $\ell_2$-normalized visual features $v_i$ and textual subspace $o_j$, we have $1 - \langle v_i,o_j\rangle = \tfrac{1}{2}\|v_i - o_j\|^2,$ So the balanced OT objective becomes $\mathcal{L}_{\mathrm{OT}} = \sum_{i=1}^N \sum_{j=1}^Q T_{ij}\,\|v_i - o_j\|^2, T \in \Pi(u,v).$ Let $\mathcal{C}_p$ and Since $\|\mu_p - \mu_q\| > 0$ for any pair of distinct clusters, the inequality implies that mixing

Figures (15)

  • Figure 1: Framework comparison. (a): Previous works rely on a textual space to indiscriminately align the diverse visual patch tokens, which would comprise the accurate alignment for diverse anomaly semantics. textbf(b): TokenCLIP introduces multiple orthogonal textual subspaces to dynamically align each visual patch token according to its visual semantics. This enables token-level textual supervision, resulting in fine-grained and comprehensive anomaly learning.
  • Figure 2: The Framework of TokenCLIP. TokenCLIP uses separate text prompts to learn global and local anomaly semantics. The global text embedding is aligned with the visual class token to detect image-level anomalies. In parallel, a corresponding local text prompt provides indiscriminate alignment with all visual patch tokens to capture base-level anomaly semantics. Building upon this, TokenCLIP employs a multi-head projection to map the base textual space into multiple orthogonal textual subspaces, regularized to encourage semantic diversity. The alignment between these subspaces and the visual patch tokens is then formulated as an OT problem. The resulting transport plan is sparsified as a token-wise assignment of textual subspaces. Finally, we jointly optimize TokenCLIP through end-to-end learning.
  • Figure 3: (a) Visualization of the assigned textual subspaces for patch token; Different color denotes different textual subspaces. (b) Selection frequency of textual subspaces across MVTec AD, where the color corresponds to that in (a). (c) Performance gap between TokenCLIP and TokenCLIP-Van.
  • Figure 3: Analysis of computation overhead.
  • Figure 4: Token-level assignment.
  • ...and 10 more figures

Theorems & Definitions (3)

  • Theorem 3.1: OT penalizes subspace mixture and induces specialization
  • Lemma J.1
  • Lemma J.2