VHELM: A Holistic Evaluation of Vision Language Models

TL;DR

The HELM framework is extended to VLMs to present the VHELM, a comprehensive, multi-dimensional view of the capabilities of the VLMs across these important factors, designed to be lightweight and automatic so that evaluation runs are cheap and fast.

Abstract

Current benchmarks for assessing vision-language models (VLMs) often focus on their perception or problem-solving capabilities and neglect other critical aspects such as fairness, multilinguality, or toxicity. Furthermore, they differ in their evaluation procedures and the scope of the evaluation, making it difficult to compare models. To address these issues, we extend the HELM framework to VLMs to present the Holistic Evaluation of Vision Language Models (VHELM). VHELM aggregates various datasets to cover one or more of the 9 aspects: visual perception, knowledge, reasoning, bias, fairness, multilinguality, robustness, toxicity, and safety. In doing so, we produce a comprehensive, multi-dimensional view of the capabilities of the VLMs across these important factors. In addition, we standardize the standard inference parameters, methods of prompting, and evaluation metrics to enable fair comparisons across models. Our framework is designed to be lightweight and automatic so that evaluation runs are cheap and fast. Our initial run evaluates 22 VLMs on 21 existing datasets to provide a holistic snapshot of the models. We uncover new key findings, such as the fact that efficiency-focused models (e.g., Claude 3 Haiku or Gemini 1.5 Flash) perform significantly worse than their full models (e.g., Claude 3 Opus or Gemini 1.5 Pro) on the bias benchmark but not when evaluated on the other aspects. For transparency, we release the raw model generations and complete results on our website (https://crfm.stanford.edu/helm/vhelm/v2.0.1). VHELM is intended to be a living benchmark, and we hope to continue adding new datasets and models over time.

Figures (7)

• Figure 1: Holistic Evaluation of Vision Language Models (VHELM) is a benchmark with standardized evaluation procedures and automated metrics. We evaluate 9 important dimensions (aspects) across scenarios to create a comprehensive view of VLMs. The metrics listed are not specific to the examples but are a list of those used across all the scenarios in the aspect.
• Figure 2:
• Figure A1:
• Figure A2: Examples of failures when evaluating the gender and racial bias. The text generations are by Claude 3 Sonnet.
• Figure A3: Perturbations. We perturb the instances of VQAv2 with typos perturbations to assess robustness (left) and African American English (AAE) perturbations (right) to assess fairness.