Vision-Language Models under Cultural and Inclusive Considerations

TL;DR

This work tackles the gap in culturally inclusive evaluation of vision-language models (VLMs) for visually impaired users by constructing a culture-centric benchmark from a filtered VizWiz dataset and conducting zero-shot evaluations across multiple models. It combines a user study on caption preferences with rigorous automatic (COCO metrics) and human assessments to measure cultural sensitivity and accuracy. The results show that closed-access models (e.g., GPT-4o, Gemini) generally outperform open models, and culture-specific prompts can improve human judgments even when automatic metrics lag, highlighting a misalignment between metrics and user-perceived usefulness. The study underscores the potential of culturally aware VLMs for real-world accessibility while calling for broader evaluation frameworks, more diverse data, and future work that extends to Q&A and richer cultural content to reduce hallucinations and increase reliability in diverse contexts.

Abstract

Large vision-language models (VLMs) can assist visually impaired people by describing images from their daily lives. Current evaluation datasets may not reflect diverse cultural user backgrounds or the situational context of this use case. To address this problem, we create a survey to determine caption preferences and propose a culture-centric evaluation benchmark by filtering VizWiz, an existing dataset with images taken by people who are blind. We then evaluate several VLMs, investigating their reliability as visual assistants in a culturally diverse setting. While our results for state-of-the-art models are promising, we identify challenges such as hallucination and misalignment of automatic evaluation metrics with human judgment. We make our survey, data, code, and model outputs publicly available.

Figures (11)

• Figure 1: Survey results from people with visual impairments rating importance and helpfulness of cultural information in image captions. We use a Likert scale from 1 (not important/helpful) to 5 (very important/helpful).
• Figure 2: Examples of various images from the filtered VizWiz dataset with the original ( * X ) and culture-specific ( * X ) annotations, and generated captions from Gemini-1.5-Pro, GPT-4o, InstructBLIP, and LLaVA-1.6 with default ( * X ) and culture-specific ( * X ) prompting.
• Figure 3: Results of the human evaluation for 100 images and their captions selected at random from the filtered VizWiz dataset. The left plot shows the preference score (participants were asked to rank the captions; lower is better). The right plot shows the accuracy evaluation (participants were asked to assess whether a caption is accurate; higher is better). '_D' and '_C' denote default and culture-specific prompting, respectively.
• Figure 4: Distribution of the factors/indicators that lead the annotators to select a specific image as culture-related and specify the corresponding culture.
• Figure 5: Distribution of the cultural concepts identified in the VizWiz dataset by the annotators.