Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language as a Label: Zero-Shot Multimodal Classification of Everyday Postures under Data Scarcity

MingZe Tang, Jubal Chandy Jacob

TL;DR

This study probes zero-shot posture recognition under data scarcity by examining how prompt design influences performance across OpenCLIP, MetaCLIP 2, and SigLip on a 285-image COCO-derived dataset. It finds a counterintuitive trend: the simplest label-oriented prompts yield the best results for high-performing VLMs, while more descriptive prompts can cause 'prompt overfitting' and degrade accuracy; in contrast, a pose-oriented SigLip variant gains from detailed prompts for ambiguous classes. The results position prompt design as a practical inference-time supervision tool and highlight the value of cross-modal alignment, with pose priors offering a robust fallback when language supervision is constrained. The work provides deployment guidelines for data-scarce settings and motivates future work on larger, multilingual datasets and multi-person scenes to extend the findings.

Abstract

Recent Vision-Language Models (VLMs) enable zero-shot classification by aligning images and text in a shared space, a promising approach for data-scarce conditions. However, the influence of prompt design on recognizing visually similar categories, such as human postures, is not well understood. This study investigates how prompt specificity affects the zero-shot classification of sitting, standing, and walking/running on a small, 285-image COCO-derived dataset. A suite of modern VLMs, including OpenCLIP, MetaCLIP 2, and SigLip, were evaluated using a three-tiered prompt design that systematically increases linguistic detail. Our findings reveal a compelling, counter-intuitive trend: for the highest-performing models (MetaCLIP 2 and OpenCLIP), the simplest, most basic prompts consistently achieve the best results. Adding descriptive detail significantly degrades performance for instance, MetaCLIP 2's multi-class accuracy drops from 68.8\% to 55.1\% a phenomenon we term "prompt overfitting". Conversely, the lower-performing SigLip model shows improved classification on ambiguous classes when given more descriptive, body-cue-based prompts.

Language as a Label: Zero-Shot Multimodal Classification of Everyday Postures under Data Scarcity

TL;DR

This study probes zero-shot posture recognition under data scarcity by examining how prompt design influences performance across OpenCLIP, MetaCLIP 2, and SigLip on a 285-image COCO-derived dataset. It finds a counterintuitive trend: the simplest label-oriented prompts yield the best results for high-performing VLMs, while more descriptive prompts can cause 'prompt overfitting' and degrade accuracy; in contrast, a pose-oriented SigLip variant gains from detailed prompts for ambiguous classes. The results position prompt design as a practical inference-time supervision tool and highlight the value of cross-modal alignment, with pose priors offering a robust fallback when language supervision is constrained. The work provides deployment guidelines for data-scarce settings and motivates future work on larger, multilingual datasets and multi-person scenes to extend the findings.

Abstract

Recent Vision-Language Models (VLMs) enable zero-shot classification by aligning images and text in a shared space, a promising approach for data-scarce conditions. However, the influence of prompt design on recognizing visually similar categories, such as human postures, is not well understood. This study investigates how prompt specificity affects the zero-shot classification of sitting, standing, and walking/running on a small, 285-image COCO-derived dataset. A suite of modern VLMs, including OpenCLIP, MetaCLIP 2, and SigLip, were evaluated using a three-tiered prompt design that systematically increases linguistic detail. Our findings reveal a compelling, counter-intuitive trend: for the highest-performing models (MetaCLIP 2 and OpenCLIP), the simplest, most basic prompts consistently achieve the best results. Adding descriptive detail significantly degrades performance for instance, MetaCLIP 2's multi-class accuracy drops from 68.8\% to 55.1\% a phenomenon we term "prompt overfitting". Conversely, the lower-performing SigLip model shows improved classification on ambiguous classes when given more descriptive, body-cue-based prompts.
Paper Structure (37 sections, 5 figures, 2 tables)

This paper contains 37 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Vision Language Models for Zero-Shot Classification
  4. Low-Resource & Low-Compute Image Understanding
  5. Multilinguality and Prompt Semantics for Posture Cues
  6. Data and Methods
  7. Dataset
  8. Models and Experimental Approach
  9. Unimodal Vision Models
  10. Vision Transformer (ViT)
  11. DinoV3
  12. Multimodal Vision-Language Models
  13. OpenCLIP
  14. Meta CLIP 2
  15. SigLip
  16. ...and 22 more sections

Figures (5)

  • Figure 1: Random samples from the curated MS COCO subset for sitting, standing, and walking or running.
  • Figure 2: Raw image sizes in the curated subset.
  • Figure 3: Aspect ratio (width/height) by class.
  • Figure 4: Mean test accuracy on the primary three-class HAR task. Models with semantic (VLM) or structural (Pose) priors demonstrate a clear performance advantage.
  • Figure 5: Grad-CAM visualizations for different phrasings of the concepts “sitting” and “standing”. Increased specificity in phrasing leads to more focused saliency in relevant body regions.