Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VLM-PAR: A Vision Language Model for Pedestrian Attribute Recognition

Abdellah Zakaria Sellam, Salah Eddine Bekhouche, Fadi Dornaika, Cosimo Distante, Abdenour Hadid

TL;DR

VLM-PAR addresses pedestrian attribute recognition under severe class imbalance and domain shifts by combining frozen SigLIP2 vision–language encoders with a lightweight two-stage fusion. The method first aligns image and attribute-prompt embeddings via cosine similarity, then applies per-attribute multi-head cross-attention to refine visual features before classification. It achieves state-of-the-art mean accuracy on PA-100K and notable gains on PETA and Market-1501, with ablation showing that cross-attention drives attribute-specific improvements while preserving generalization and reducing fine-tuning overhead. This work demonstrates the value of integrating large-scale vision–language pretraining with targeted cross-modal refinement for robust, low-cost PAR in real-world deployments.

Abstract

Pedestrian Attribute Recognition (PAR) involves predicting fine-grained attributes such as clothing color, gender, and accessories from pedestrian imagery, yet is hindered by severe class imbalance, intricate attribute co-dependencies, and domain shifts. We introduce VLM-PAR, a modular vision-language framework built on frozen SigLIP 2 multilingual encoders. By first aligning image and prompt embeddings via refining visual features through a compact cross-attention fusion, VLM-PAR achieves significant accuracy improvement on the highly imbalanced PA100K benchmark, setting a new state-of-the-art performance, while also delivering significant gains in mean accuracy across PETA and Market-1501 benchmarks. These results underscore the efficacy of integrating large-scale vision-language pretraining with targeted cross-modal refinement to overcome imbalance and generalization challenges in PAR.

VLM-PAR: A Vision Language Model for Pedestrian Attribute Recognition

TL;DR

VLM-PAR addresses pedestrian attribute recognition under severe class imbalance and domain shifts by combining frozen SigLIP2 vision–language encoders with a lightweight two-stage fusion. The method first aligns image and attribute-prompt embeddings via cosine similarity, then applies per-attribute multi-head cross-attention to refine visual features before classification. It achieves state-of-the-art mean accuracy on PA-100K and notable gains on PETA and Market-1501, with ablation showing that cross-attention drives attribute-specific improvements while preserving generalization and reducing fine-tuning overhead. This work demonstrates the value of integrating large-scale vision–language pretraining with targeted cross-modal refinement for robust, low-cost PAR in real-world deployments.

Abstract

Pedestrian Attribute Recognition (PAR) involves predicting fine-grained attributes such as clothing color, gender, and accessories from pedestrian imagery, yet is hindered by severe class imbalance, intricate attribute co-dependencies, and domain shifts. We introduce VLM-PAR, a modular vision-language framework built on frozen SigLIP 2 multilingual encoders. By first aligning image and prompt embeddings via refining visual features through a compact cross-attention fusion, VLM-PAR achieves significant accuracy improvement on the highly imbalanced PA100K benchmark, setting a new state-of-the-art performance, while also delivering significant gains in mean accuracy across PETA and Market-1501 benchmarks. These results underscore the efficacy of integrating large-scale vision-language pretraining with targeted cross-modal refinement to overcome imbalance and generalization challenges in PAR.
Paper Structure (18 sections, 14 equations, 4 figures, 1 table)

This paper contains 18 sections, 14 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Proposed Method
  4. Background knowledge
  5. Siglip Model
  6. Cross-Attention
  7. VLM-PAR Architecture Overview
  8. SigLIP2 Encoder Components
  9. Vision Encoder (ViT-B/16)
  10. Text Encoder
  11. Cross-Attention Fusion
  12. Multi-Task Classification Heads
  13. Experimental Setup
  14. Datasets
  15. Metrics
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Example of pedestrian attribute annotations showing upper‐body clothing color, lower‐body clothing color, Mask, Shoes color, gender, bag color, and hat presence surprising_media_2021_pixabay.
  • Figure 2: VLM-PAR pipeline: a SigLIP-based image encoder and text encoder process person images and attribute-specific questions, align visual and textual features via multi-head cross-attention, then feed the attended features into task-specific classifiers for multi-attribute prediction.
  • Figure 3: Examples of Pedestrian images from three widely-used attribute datasets: PETA, PA‑100K, and Market‑1501.
  • Figure 4: Ablation study comparing attribute classification accuracy with and without cross-attention.