VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model

TL;DR

VLBiasBench introduces a large-scale, synthetic bias benchmark for Large Vision-Language Models, addressing gaps in prior fairness evaluations by combining nine independent bias categories and two intersectional biases with open-ended and close-ended questions. The benchmark leverages SDXL to generate a high-quality image corpus paired with bias-focused prompts, enabling robust cross-model evaluation across 15 open-source and 2 closed-source LVLMs. It employs a multi-faceted evaluation framework, including VADER-based sentiment analysis, a gender-polarity metric, and a text-/image-dominant close-ended protocol, augmented by CLIP-based filtering and manual quality control. Key findings reveal persistent biases across several models, with notable modality- and context-dependent effects, while results on closed-source models suggest comparatively lower bias in many dimensions. The work provides a scalable, ethical benchmark tool and a foundation for improving fairness in future LVLM deployments.

Abstract

The emergence of Large Vision-Language Models (LVLMs) marks significant strides towards achieving general artificial intelligence. However, these advancements are accompanied by concerns about biased outputs, a challenge that has yet to be thoroughly explored. Existing benchmarks are not sufficiently comprehensive in evaluating biases due to their limited data scale, single questioning format and narrow sources of bias. To address this problem, we introduce VLBiasBench, a comprehensive benchmark designed to evaluate biases in LVLMs. VLBiasBench, features a dataset that covers nine distinct categories of social biases, including age, disability status, gender, nationality, physical appearance, race, religion, profession, social economic status, as well as two intersectional bias categories: race x gender and race x social economic status. To build a large-scale dataset, we use Stable Diffusion XL model to generate 46,848 high-quality images, which are combined with various questions to creat 128,342 samples. These questions are divided into open-ended and close-ended types, ensuring thorough consideration of bias sources and a comprehensive evaluation of LVLM biases from multiple perspectives. We conduct extensive evaluations on 15 open-source models as well as two advanced closed-source models, yielding new insights into the biases present in these models. Our benchmark is available at https://github.com/Xiangkui-Cao/VLBiasBench.

Figures (15)

• Figure 1: Framework of synthetic image generation (top), along with specific examples of evaluations for open-ended and close-ended questions (bottom left and bottom right, respectively).
• Figure 2: Prompt library construction for open-ended question evaluation. (a) combination-based construction. (b) automatical construction.
• Figure 3: Construction of the "Base" and "Scene" datasets. (a) shows an ambiguous sample from the "Base" dataset, where the context is dominated by textual information. (b) presents a disambiguated sample in the "Scene" dataset, where the context is predominantly informed by image content.
• Figure 4: Text-induced dataset construction for close-ended question evaluation. We first extract prompts from the corpus to generate images. Attribute information (e.g. the old and school-aged man) from different groups is paired in both image and text modalities. These pairs are then used to build a text-induced single-image dataset.
• Figure 5: Close-ended evaluation framework. We contrast text-dominant contexts (Base, Text)—simple portraits paired with detailed scenario descriptions—with image-dominant contexts (Scene, Scene Text). Image-dominant contexts incorporate descriptive details into complex, prompt-generated images, leaving simplified text context. By introducing comparative text-induced method regarding different groups, we create augmented datasets (Text, Scene Text) revealing exacerbated model biases.