Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with VLMs

Kapil Garg, Xinru Tang, Jimin Heo, Dwayne R. Morgan, Darren Gergle, Erik B. Sudderth, Anne Marie Piper

TL;DR

The paper investigates how common image quality issues affect Vision-Language Model–generated product captions for blind and low-vision users, combining a BLV survey with a large, annotated VizWiz-derived dataset. It evaluates four VLMs (GPT-4.1, Gemini 2.5, Llama 3.2, Molmo 72B) and shows robust performance on high-quality images but substantial degradation when images are degraded, with multi-issue distortions severely reducing accuracy. It argues for disability-centered model evaluation—covering data, tasks, and metrics aligned with BLV needs—and offers concrete recommendations on data curation, post-training strategies, and inference-time techniques to improve reliability and safety. The findings underscore the need for open, privacy-preserving models and richer user feedback to make VLMs viable for real-world, life-critical product identification tasks among BLV communities. The work thus provides a practical roadmap for designing, evaluating, and deploying VLM-based captioning tools that better serve disabled users in everyday contexts.

Abstract

Vision-Language Models (VLMs) are increasingly used by blind and low-vision (BLV) people to identify and understand products in their everyday lives, such as food, personal products, and household goods. Despite their prevalence, we lack an empirical understanding of how common image quality issues, like blur and misframing of items, affect the accuracy of VLM-generated captions and whether resulting captions meet BLV people's information needs. Grounded in a survey with 86 BLV people, we systematically evaluate how image quality issues affect captions generated by VLMs. We show that the best model recognizes products in images with no quality issues with 98% accuracy, but drops to 75% accuracy overall when quality issues are present, worsening considerably as issues compound. We discuss the need for model evaluations that center on disabled people's experiences throughout the process and offer concrete recommendations for HCI and ML researchers to make VLMs more reliable for BLV people.

"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with VLMs

TL;DR

The paper investigates how common image quality issues affect Vision-Language Model–generated product captions for blind and low-vision users, combining a BLV survey with a large, annotated VizWiz-derived dataset. It evaluates four VLMs (GPT-4.1, Gemini 2.5, Llama 3.2, Molmo 72B) and shows robust performance on high-quality images but substantial degradation when images are degraded, with multi-issue distortions severely reducing accuracy. It argues for disability-centered model evaluation—covering data, tasks, and metrics aligned with BLV needs—and offers concrete recommendations on data curation, post-training strategies, and inference-time techniques to improve reliability and safety. The findings underscore the need for open, privacy-preserving models and richer user feedback to make VLMs viable for real-world, life-critical product identification tasks among BLV communities. The work thus provides a practical roadmap for designing, evaluating, and deploying VLM-based captioning tools that better serve disabled users in everyday contexts.

Abstract

Vision-Language Models (VLMs) are increasingly used by blind and low-vision (BLV) people to identify and understand products in their everyday lives, such as food, personal products, and household goods. Despite their prevalence, we lack an empirical understanding of how common image quality issues, like blur and misframing of items, affect the accuracy of VLM-generated captions and whether resulting captions meet BLV people's information needs. Grounded in a survey with 86 BLV people, we systematically evaluate how image quality issues affect captions generated by VLMs. We show that the best model recognizes products in images with no quality issues with 98% accuracy, but drops to 75% accuracy overall when quality issues are present, worsening considerably as issues compound. We discuss the need for model evaluations that center on disabled people's experiences throughout the process and offer concrete recommendations for HCI and ML researchers to make VLMs more reliable for BLV people.

Paper Structure

This paper contains 37 sections, 3 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Background
  3. How Blind and Low-Vision People Use VLMs for Image Understanding
  4. Understanding and Addressing Image Quality Issues
  5. BLV People's Perspectives on AI Errors
  6. Study 1: Understanding Preferences and Experiences with AI-based Captioning of Product Images
  7. Method
  8. Analysis
  9. Results
  10. AI captioning is preferred for identifying food, personal products, and items in the home
  11. Photo taking is the longest and most difficult part of captioning
  12. Current tools make it difficult to assess and resolve image quality issues
  13. Captions frequently lack important detail and contain inaccurate information
  14. Final Reflections
  15. Study 2: Evaluating VLM Captions for Product Understanding
  16. ...and 22 more sections

Figures (3)

  • Figure 1: Divergent stacked bar charts show the distribution of reported responses for scenario-based preferences for AI vs human assistance (left) and concern-based preferences for AI vs human assistance (right). The x-axis shows the number of participants indicating each response. Bar labels 5 and under are hidden due to bar size.
  • Figure 2: Divergent stacked bar charts show the distribution of reported responses for perceived impact of image quality issues on caption quality (left) and the ability to assess a quality issue in their image (right). The x-axis shows the number of participants indicating each response. "I am not sure" responses are removed. Bar labels 5 and under are hidden due to bar size.
  • Figure 3: Distribution of perceived frequency of various types of errors in image captions when describing products. The x-axis shows the number of participants indicating each response. Bar labels 5 and under are hidden due to bar size.