Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zero-Shot Product Attribute Labeling with Vision-Language Models: A Three-Tier Evaluation Framework

Shubham Shukla, Kunal Sonalkar

TL;DR

The paper tackles the problem of fine-grained fashion attribute prediction by proposing a three-tier diagnostic framework that separately evaluates attribute applicability detection and fine-grained classification, in addition to full-task performance. Using DeepFashion-MultiModal with 18 attributes across 3 categories and an explicit NA class, the authors benchmark nine state-of-the-art VLMs in a purely zero-shot setting against a logistic regression baseline on pretrained Fashion-CLIP embeddings. They demonstrate that zero-shot VLMs substantially outperform the baseline on overall F1 (around 64% vs. 21%), with Tier 3 classification showing strong performance (about 70.8% F1) while Tier 2 NA detection remains the primary bottleneck (approximately 34.1% NA-F1). The framework also reveals cost-performance trade-offs, showing efficient models achieving most flagship performance at a fraction of the cost and with relatively low hallucination rates when properly managed. Overall, the diagnostic three-tier evaluation provides actionable insight for deploying VLMs in fashion retail, guiding when to rely on zero-shot classification versus when to integrate NA-detection or supervised refinements, and informing practical choices about model selection and deployment constraints.

Abstract

Fine-grained attribute prediction is essential for fashion retail applications including catalog enrichment, visual search, and recommendation systems. Vision-Language Models (VLMs) offer zero-shot prediction without task-specific training, yet their systematic evaluation on multi-attribute fashion tasks remains underexplored. A key challenge is that fashion attributes are often conditional. For example, "outer fabric" is undefined when no outer garment is visible. This requires models to detect attribute applicability before attempting classification. We introduce a three-tier evaluation framework that decomposes this challenge: (1) overall task performance across all classes (including NA class: suggesting attribute is not applicable) for all attributes, (2) attribute applicability detection, and (3) fine-grained classification when attributes are determinable. Using DeepFashion-MultiModal, which explicitly defines NA (meaning attribute doesn't exist or is not visible) within attribute label spaces, we benchmark nine VLMs spanning flagship (GPT-5, Gemini 2.5 Pro), efficient (GPT-5 Mini, Gemini 2.5 Flash), and ultra-efficient tiers (GPT-5 Nano, Gemini 2.5 Flash-Lite) against classifiers trained on pretrained Fashion-CLIP embeddings on 5,000 images across 18 attributes. Our findings reveal that: (1) zero-shot VLMs achieve 64.0% macro-F1, a threefold improvement over logistic regression on pretrained Fashion-CLIP embeddings; (2) VLMs excel at fine-grained classification (Tier 3: 70.8% F1) but struggle with applicability detection (Tier 2: 34.1% NA-F1), identifying a key bottleneck; (3) efficient models achieve over 90% of flagship performance at lower cost, offering practical deployment paths. This diagnostic framework enables practitioners to pinpoint whether errors stem from visibility detection or classification, guiding targeted improvements for production systems.

Zero-Shot Product Attribute Labeling with Vision-Language Models: A Three-Tier Evaluation Framework

TL;DR

The paper tackles the problem of fine-grained fashion attribute prediction by proposing a three-tier diagnostic framework that separately evaluates attribute applicability detection and fine-grained classification, in addition to full-task performance. Using DeepFashion-MultiModal with 18 attributes across 3 categories and an explicit NA class, the authors benchmark nine state-of-the-art VLMs in a purely zero-shot setting against a logistic regression baseline on pretrained Fashion-CLIP embeddings. They demonstrate that zero-shot VLMs substantially outperform the baseline on overall F1 (around 64% vs. 21%), with Tier 3 classification showing strong performance (about 70.8% F1) while Tier 2 NA detection remains the primary bottleneck (approximately 34.1% NA-F1). The framework also reveals cost-performance trade-offs, showing efficient models achieving most flagship performance at a fraction of the cost and with relatively low hallucination rates when properly managed. Overall, the diagnostic three-tier evaluation provides actionable insight for deploying VLMs in fashion retail, guiding when to rely on zero-shot classification versus when to integrate NA-detection or supervised refinements, and informing practical choices about model selection and deployment constraints.

Abstract

Fine-grained attribute prediction is essential for fashion retail applications including catalog enrichment, visual search, and recommendation systems. Vision-Language Models (VLMs) offer zero-shot prediction without task-specific training, yet their systematic evaluation on multi-attribute fashion tasks remains underexplored. A key challenge is that fashion attributes are often conditional. For example, "outer fabric" is undefined when no outer garment is visible. This requires models to detect attribute applicability before attempting classification. We introduce a three-tier evaluation framework that decomposes this challenge: (1) overall task performance across all classes (including NA class: suggesting attribute is not applicable) for all attributes, (2) attribute applicability detection, and (3) fine-grained classification when attributes are determinable. Using DeepFashion-MultiModal, which explicitly defines NA (meaning attribute doesn't exist or is not visible) within attribute label spaces, we benchmark nine VLMs spanning flagship (GPT-5, Gemini 2.5 Pro), efficient (GPT-5 Mini, Gemini 2.5 Flash), and ultra-efficient tiers (GPT-5 Nano, Gemini 2.5 Flash-Lite) against classifiers trained on pretrained Fashion-CLIP embeddings on 5,000 images across 18 attributes. Our findings reveal that: (1) zero-shot VLMs achieve 64.0% macro-F1, a threefold improvement over logistic regression on pretrained Fashion-CLIP embeddings; (2) VLMs excel at fine-grained classification (Tier 3: 70.8% F1) but struggle with applicability detection (Tier 2: 34.1% NA-F1), identifying a key bottleneck; (3) efficient models achieve over 90% of flagship performance at lower cost, offering practical deployment paths. This diagnostic framework enables practitioners to pinpoint whether errors stem from visibility detection or classification, guiding targeted improvements for production systems.
Paper Structure (47 sections, 3 equations, 4 figures, 6 tables)

This paper contains 47 sections, 3 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Fashion attribute prediction.
  4. LLMs and VLMs for fashion.
  5. Vision-language models.
  6. Evaluation methodologies and benchmarks.
  7. Foundation models for retail.
  8. Dataset
  9. The NA (Not Applicable) Class.
  10. Three-Tier Evaluation Framework
  11. Problem Formulation
  12. Tier Definitions
  13. Tier 1: Full Task Performance.
  14. Tier 2: Attribute Applicability Detection.
  15. Tier 3: Classification Given Applicability.
  16. ...and 32 more sections

Figures (4)

  • Figure 1: Example test image with ground truth annotations. This outfit has 7 NA labels: lower clothing length not determinable from crop, socks/hat/glasses not visible, waist accessories unclear, and no outer garment.
  • Figure 2: Three-tier evaluation framework. Tier 1 evaluates all samples across all classes. Tier 2 measures binary NA detection. Tier 3 evaluates classification on samples where the attribute is visible (ground truth $\neq$ NA), with NA remaining in the prediction space.
  • Figure 3: Complete zero-shot prompt used for VLM evaluation. Models receive this prompt along with the product image and text description. No in-context examples are provided (purely zero-shot).
  • Figure 4: Cost-performance trade-off. $\bullet$ = Gemini, $\circ$ = Gemini (Think), $\blacksquare$ = GPT. Gemini 2.5 Pro achieves best Tier 1 F1; Flash-Lite offers 22$\times$ cost reduction.