Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Auditing Disability Representation in Vision-Language Models

Srikant Panda, Sourabh Singh Yadav, Palkesh Malviya

TL;DR

This work conducts a systematic audit of disability representation in vision–language models by pairing neutral versus disability-contextualized prompts and evaluating outputs with both linguistic metrics and an LLM-based judge anchored in disability representation norms. The authors show that introducing disability context consistently degrades interpretive fidelity, driving speculative inferences, narrative inflation, and affective framing, with stronger effects for certain race and gender groups. They establish a rigorous NP–DP benchmark using the PAIRS dataset and validate the evaluation with human annotations, demonstrating robust agreement between humans and LLM judges. Finally, the study offers practical mitigations—targeted prompting and Direct Preference Optimization—that substantially reduce interpretation drift and related biases, highlighting interpretive fidelity as a concrete objective for responsible multimodal AI development.

Abstract

Vision-language models (VLMs) are increasingly deployed in socially sensitive applications, yet their behavior with respect to disability remains underexplored. We study disability aware descriptions for person centric images, where models often transition from evidence grounded factual description to interpretation shift including introduction of unsupported inferences beyond observable visual evidence. To systematically analyze this phenomenon, we introduce a benchmark based on paired Neutral Prompts (NP) and Disability-Contextualised Prompts (DP) and evaluate 15 state-of-the-art open- and closed-source VLMs under a zero-shot setting across 9 disability categories. Our evaluation framework treats interpretive fidelity as core objective and combines standard text-based metrics capturing affective degradation through shifts in sentiment, social regard and response length with an LLM-as-judge protocol, validated by annotators with lived experience of disability. We find that introducing disability context consistently degrades interpretive fidelity, inducing interpretation shifts characterised by speculative inference, narrative elaboration, affective degradation and deficit oriented framing. These effects are further amplified along race and gender dimension. Finally, we demonstrate targeted prompting and preference fine-tuning effectively improves interpretive fidelity and reduces substantially interpretation shifts.

Auditing Disability Representation in Vision-Language Models

TL;DR

This work conducts a systematic audit of disability representation in vision–language models by pairing neutral versus disability-contextualized prompts and evaluating outputs with both linguistic metrics and an LLM-based judge anchored in disability representation norms. The authors show that introducing disability context consistently degrades interpretive fidelity, driving speculative inferences, narrative inflation, and affective framing, with stronger effects for certain race and gender groups. They establish a rigorous NP–DP benchmark using the PAIRS dataset and validate the evaluation with human annotations, demonstrating robust agreement between humans and LLM judges. Finally, the study offers practical mitigations—targeted prompting and Direct Preference Optimization—that substantially reduce interpretation drift and related biases, highlighting interpretive fidelity as a concrete objective for responsible multimodal AI development.

Abstract

Vision-language models (VLMs) are increasingly deployed in socially sensitive applications, yet their behavior with respect to disability remains underexplored. We study disability aware descriptions for person centric images, where models often transition from evidence grounded factual description to interpretation shift including introduction of unsupported inferences beyond observable visual evidence. To systematically analyze this phenomenon, we introduce a benchmark based on paired Neutral Prompts (NP) and Disability-Contextualised Prompts (DP) and evaluate 15 state-of-the-art open- and closed-source VLMs under a zero-shot setting across 9 disability categories. Our evaluation framework treats interpretive fidelity as core objective and combines standard text-based metrics capturing affective degradation through shifts in sentiment, social regard and response length with an LLM-as-judge protocol, validated by annotators with lived experience of disability. We find that introducing disability context consistently degrades interpretive fidelity, inducing interpretation shifts characterised by speculative inference, narrative elaboration, affective degradation and deficit oriented framing. These effects are further amplified along race and gender dimension. Finally, we demonstrate targeted prompting and preference fine-tuning effectively improves interpretive fidelity and reduces substantially interpretation shifts.
Paper Structure (66 sections, 1 equation, 4 figures, 13 tables)

This paper contains 66 sections, 1 equation, 4 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Bias in Models
  4. Disability Bias in VLMs
  5. Standards for Disability Representation
  6. Methodology
  7. Disability Bias
  8. Human Assessment.
  9. VLM Behavior
  10. Disability Taxonomy
  11. Paired Prompt Dataset Construction
  12. Evaluation Metrics
  13. Why Disability Context Matters
  14. Experimental Setup
  15. Vision-Language Models Evaluated
  16. ...and 51 more sections

Figures (4)

  • Figure 1: Evaluation pipeline for auditing interpretive disability bias in Vision-Language models (VLMs) using paired Neutral and Disability Contextualized prompts and their corresponding responses.
  • Figure 2: Example image from PAIRS dataset (Category: Occupation, Subcategory: Desk)
  • Figure 3: Demographic divergence from model-wide averages ($\Delta(\text{Group}-\text{All})$) for interpret, stereotype and framing across models
  • Figure 4: Distribution of Assigned Disability Types Across Five Open-Source Models.