Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena

Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, Albert Gatt

TL;DR

VALSE introduces a six-piece, zero-shot benchmark to diagnose visio-linguistic grounding of core linguistic phenomena via carefully constructed caption foils. The methodology emphasizes bias mitigation (distributional and plausibility) and automatic plus manual validation to ensure valid, image-consistent foils. Across five V&L models and two unimodal baselines, results show strong grounding on object presence but substantial gaps in reasoning about plurality, counting, spatial relations, actions, and coreference, highlighting the need for improved multimodal linguistic grounding. The benchmark provides a flexible, living framework for measuring progress and guiding future development of pretrained vision-language models with a linguistic grounding lens.

Abstract

We propose VALSE (Vision And Language Structured Evaluation), a novel benchmark designed for testing general-purpose pretrained vision and language (V&L) models for their visio-linguistic grounding capabilities on specific linguistic phenomena. VALSE offers a suite of six tests covering various linguistic constructs. Solving these requires models to ground linguistic phenomena in the visual modality, allowing more fine-grained evaluations than hitherto possible. We build VALSE using methods that support the construction of valid foils, and report results from evaluating five widely-used V&L models. Our experiments suggest that current models have considerable difficulty addressing most phenomena. Hence, we expect VALSE to serve as an important benchmark to measure future progress of pretrained V&L models from a linguistic perspective, complementing the canonical task-centred V&L evaluations.

VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena

TL;DR

VALSE introduces a six-piece, zero-shot benchmark to diagnose visio-linguistic grounding of core linguistic phenomena via carefully constructed caption foils. The methodology emphasizes bias mitigation (distributional and plausibility) and automatic plus manual validation to ensure valid, image-consistent foils. Across five V&L models and two unimodal baselines, results show strong grounding on object presence but substantial gaps in reasoning about plurality, counting, spatial relations, actions, and coreference, highlighting the need for improved multimodal linguistic grounding. The benchmark provides a flexible, living framework for measuring progress and guiding future development of pretrained vision-language models with a linguistic grounding lens.

Abstract

We propose VALSE (Vision And Language Structured Evaluation), a novel benchmark designed for testing general-purpose pretrained vision and language (V&L) models for their visio-linguistic grounding capabilities on specific linguistic phenomena. VALSE offers a suite of six tests covering various linguistic constructs. Solving these requires models to ground linguistic phenomena in the visual modality, allowing more fine-grained evaluations than hitherto possible. We build VALSE using methods that support the construction of valid foils, and report results from evaluating five widely-used V&L models. Our experiments suggest that current models have considerable difficulty addressing most phenomena. Hence, we expect VALSE to serve as an important benchmark to measure future progress of pretrained V&L models from a linguistic perspective, complementing the canonical task-centred V&L evaluations.
Paper Structure (59 sections, 2 equations, 4 figures, 11 tables)

This paper contains 59 sections, 2 equations, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Background and Related work
  3. Benchmarking V&L models
  4. Constructing the VALSE benchmark
  5. Existence
  6. Plurality
  7. Counting
  8. Spatial relations
  9. Actions
  10. Coreference
  11. Reliable construction of valid foils
  12. Mitigating distributional bias
  13. Countering plausibility bias
  14. Filtering foils with NL Inference
  15. Manual evaluation of generated foils
  16. ...and 44 more sections

Figures (4)

  • Figure 1: Normalized pronoun frequencies in the coreference subset.
  • Figure 2: Example of an instance from the validation study. The example is from the counting piece, adversarial instrument (see Section \ref{['subsec:counting']}).
  • Figure 3: Word frequency distributions for captions and foils before and after the manual validation for existence, counting and relations.
  • Figure 4: Word frequency distributions for captions and foils before and after the manual validation for plurality, action replacement and FOIL it. The actant swap instrument is not visualised here: By construction, actant swap cannot suffer from distributional bias since caption and foil contain the same words up to a permutation.