Learning 3D Texture-Aware Representations for Parsing Diverse Human Clothing and Body Parts

TL;DR

Spectrum introduces a unified framework for detailed human parsing that jointly handles body parts and clothing at the pixel level while providing instance-level part grouping. It repurposes a diffusion based Image-to-Texture I2Tx model trained on 3D human textures to obtain texture aligned features that improve segmentation of diverse clothing and body parts. The method integrates a semantically grounded parsing head with prompt grounded learning to produce accurate, prompt aligned masks for seen and unseen clothing categories across single and multi-human scenes. Through extensive cross dataset evaluations, Spectrum consistently outperforms fixed label parsers and open vocabulary baselines, especially in unseen clothing scenarios. The results demonstrate that 3D texture based representations offer strong generalization and fine grained parsing capabilities for real world clothing diversity.

Abstract

Existing methods for human parsing into body parts and clothing often use fixed mask categories with broad labels that obscure fine-grained clothing types. Recent open-vocabulary segmentation approaches leverage pretrained text-to-image (T2I) diffusion model features for strong zero-shot transfer, but typically group entire humans into a single person category, failing to distinguish diverse clothing or detailed body parts. To address this, we propose Spectrum, a unified network for part-level pixel parsing (body parts and clothing) and instance-level grouping. While diffusion-based open-vocabulary models generalize well across tasks, their internal representations are not specialized for detailed human parsing. We observe that, unlike diffusion models with broad representations, image-driven 3D texture generators maintain faithful correspondence to input images, enabling stronger representations for parsing diverse clothing and body parts. Spectrum introduces a novel repurposing of an Image-to-Texture (I2Tx) diffusion model (obtained by fine-tuning a T2I model on 3D human texture maps) for improved alignment with body parts and clothing. From an input image, we extract human-part internal features via the I2Tx diffusion model and generate semantically valid masks aligned to diverse clothing categories through prompt-guided grounding. Once trained, Spectrum produces semantic segmentation maps for every visible body part and clothing category, ignoring standalone garments or irrelevant objects, for any number of humans in the scene. We conduct extensive cross-dataset experiments, separately assessing body parts, clothing parts, unseen clothing categories, and full-body masks, and demonstrate that Spectrum consistently outperforms baseline methods in prompt-based segmentation.

Figures (11)

• Figure 1: Human-alphabet parsing in the wild. Internet-video frames show people forming the letters of Spectrum, while the model segments diverse clothing and body parts, handling unseen items such as sarees, across complex poses and styles.
• Figure 2: I2Tx feature extraction. Base Stable Diffusion (SD) weights are merged with 3D-texture LoRA matrices, and a context embedding $\mathbf{C}$ from im2context and CLIP-Vision drives a single forward pass in the I2Tx model. Concatenating encoder, denoiser, and decoder representations yields the texture-aligned feature $f$ for clothing and body-part parsing.
• Figure 3: Training. Frozen I2Tx features from image $x$ feed the pixel decoder $\mathcal{P}$ and transformer, producing $N$ class-agnostic masks $m$ optimized with $\mathcal{L}_{\mathrm{BCE}}$ and $\mathcal{L}_{\mathrm{DICE}}$. Key prompt phrases, embedded by frozen $\mathcal{T}$, are contrastively aligned to $m$ via $\mathcal{L}_{\mathrm{G}}$.
• Figure 4: FPP, BHP, CCP, and COP masks.
• Figure 5: Standard parsers (R2–4); w/o $\mathcal{L}_G$ ablation (R5).