CLIP-Guided Adaptable Self-Supervised Learning for Human-Centric Visual Tasks

TL;DR

CLASP tackles the fragmentation of human-centric visual learning by leveraging CLIP to generate multi-level semantic pseudo-labels (low-level body parts and high-level attributes) from unlabeled data, and by employing a Prompt-Controlled Mixture-of-Experts to adapt feature extraction to task-specific semantic granularity. The framework integrates four losses (DINO-based contrastive, part-level, attribute-level, and MoE load balancing) into a unified pre-training objective, enabling strong generalization across six downstream tasks. Experimental results on LUPerson and standard benchmarks demonstrate state-of-the-art or competitive performance in both semantic and appearance-centric tasks, while reducing inter-task gradient conflicts via PC-MoE. The work highlights the practical impact of combining vision-language pseudo supervision with task-aware modular routing for scalable, unified human-centric representation learning, with implications for surveillance, healthcare, and AR.

Abstract

Human-centric visual analysis plays a pivotal role in diverse applications, including surveillance, healthcare, and human-computer interaction. With the emergence of large-scale unlabeled human image datasets, there is an increasing need for a general unsupervised pre-training model capable of supporting diverse human-centric downstream tasks. To achieve this goal, we propose CLASP (CLIP-guided Adaptable Self-suPervised learning), a novel framework designed for unsupervised pre-training in human-centric visual tasks. CLASP leverages the powerful vision-language model CLIP to generate both low-level (e.g., body parts) and high-level (e.g., attributes) semantic pseudo-labels. These multi-level semantic cues are then integrated into the learned visual representations, enriching their expressiveness and generalizability. Recognizing that different downstream tasks demand varying levels of semantic granularity, CLASP incorporates a Prompt-Controlled Mixture-of-Experts (MoE) module. MoE dynamically adapts feature extraction based on task-specific prompts, mitigating potential feature conflicts and enhancing transferability. Furthermore, CLASP employs a multi-task pre-training strategy, where part- and attribute-level pseudo-labels derived from CLIP guide the representation learning process. Extensive experiments across multiple benchmarks demonstrate that CLASP consistently outperforms existing unsupervised pre-training methods, advancing the field of human-centric visual analysis.

Figures (6)

• Figure 1: Feature representation space for person re-identification, attribute recognition, and human parsing tasks. Each task is trained independently using Swin-Tiny as the backbone, with features extracted via average pooling. The visualization highlights that different tasks demand distinct feature representations, highlighting the challenge of learning a unified representation across diverse human-centric tasks.
• Figure 2: The overall architecture of CLASP framework. CLASP consists of four key components: (1) a teacher branch, which generates feature representations using a Prompt-Controlled Mixture of Experts (PC-MoE) module, updated via EMA; (2) a student branch with an identical structure, which learns by distilling knowledge from the teacher; (3) a CLIP-guided Pseudo Labels Generation Module, responsible for producing pseudo part-level and attribute-level semantic labels; (4) three task-specific heads, each designed to optimize a distinct pre-training objective.
• Figure 3: Illustration of the CLIP-guided Pseudo Label Generation Module. This module aims to generate pseudo semantic labels at multiple granularity, including part-level labels and global attribute-level labels, by leveraging CLIP's powerful image-text alignment capability. $L=4$ is an example of the variable sampled from a predefined set $S=\{2,3,4\}$.
• Figure 4: Precision-Recall curves and confidence scores statistics for CLIP-based part-level and attribute-level pseudo labels. (A) represents the PR(Precision-Recall) curves for CLIP-based parts/attributes pseudo labels on LIP and PA100k datasets. (B) represents the distribution of confidence scores for CLIP-based part pseudo-labels when $S=\{2,3,4\}$ on LUPerson. (C) represents the distribution of confidence scores for CLIP-based attribute pseudo-labels on LUPerson.
• Figure 5: Examples of generating part-level and attribute-level pseudo labels using CLIP. (A) presents two examples where CLIP, combined with by predefined body-part texts, assign semantic labels and corresponding probabilities to various clustered body parts. (B) presents three examples where CLIP, combined with predefined attribute texts, assign multiple attribute labels along with their confidence scores to different individuals. Red dashed boxes highlight ground-truth labels that were misclassified.