Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Probing Image-Language Transformers for Verb Understanding

Lisa Anne Hendricks, Aida Nematzadeh

TL;DR

The paper introduces SVO-Probes, a verb-focused zero-shot benchmark to probe verb grounding in pretrained image–language transformers. By constructing a large verb set from Conceptual Captions and imSitu and assembling over 11k SVO triplets with roughly 48k image–sentence pairs and controlled negatives, the authors systematically test models’ ability to distinguish subtle verb-centered image–sentence relations. Results show verbs are consistently harder than nouns/objects for multiple architectures, and domain noise in pretraining data can degrade fine-grained distinctions, even when overall task performance is strong. The work highlights a clear gap in current multimodal representations for verb understanding and advocates for quieter pretraining data and verb-centric evaluation to guide future improvements.

Abstract

Multimodal image-language transformers have achieved impressive results on a variety of tasks that rely on fine-tuning (e.g., visual question answering and image retrieval). We are interested in shedding light on the quality of their pretrained representations -- in particular, if these models can distinguish different types of verbs or if they rely solely on nouns in a given sentence. To do so, we collect a dataset of image-sentence pairs (in English) consisting of 421 verbs that are either visual or commonly found in the pretraining data (i.e., the Conceptual Captions dataset). We use this dataset to evaluate pretrained image-language transformers and find that they fail more in situations that require verb understanding compared to other parts of speech. We also investigate what category of verbs are particularly challenging.

Probing Image-Language Transformers for Verb Understanding

TL;DR

The paper introduces SVO-Probes, a verb-focused zero-shot benchmark to probe verb grounding in pretrained image–language transformers. By constructing a large verb set from Conceptual Captions and imSitu and assembling over 11k SVO triplets with roughly 48k image–sentence pairs and controlled negatives, the authors systematically test models’ ability to distinguish subtle verb-centered image–sentence relations. Results show verbs are consistently harder than nouns/objects for multiple architectures, and domain noise in pretraining data can degrade fine-grained distinctions, even when overall task performance is strong. The work highlights a clear gap in current multimodal representations for verb understanding and advocates for quieter pretraining data and verb-centric evaluation to guide future improvements.

Abstract

Multimodal image-language transformers have achieved impressive results on a variety of tasks that rely on fine-tuning (e.g., visual question answering and image retrieval). We are interested in shedding light on the quality of their pretrained representations -- in particular, if these models can distinguish different types of verbs or if they rely solely on nouns in a given sentence. To do so, we collect a dataset of image-sentence pairs (in English) consisting of 421 verbs that are either visual or commonly found in the pretraining data (i.e., the Conceptual Captions dataset). We use this dataset to evaluate pretrained image-language transformers and find that they fail more in situations that require verb understanding compared to other parts of speech. We also investigate what category of verbs are particularly challenging.

Paper Structure

This paper contains 18 sections, 3 figures, 6 tables.

Table of Contents

  1. Evaluating Verb Understanding
  2. Related Work
  3. Datasets.
  4. Task Setup and Dataset Collection
  5. Creating a verb list.
  6. Curating triplets.
  7. Collecting images.
  8. Collecting sentences.
  9. Confirming the negative image.
  10. Experimental Setup and Results
  11. Evaluating Nouns with FOIL
  12. Comparing Models on SVO-Probes
  13. Accuracy and Frequency at Training
  14. Similarity between SVOs.
  15. Similarity to Pretraining Data
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Examples from SVO-Probes. Images on the left and right show positive and negative image examples for each sentence. Below each image is the $\langle$subject, verb, object$\rangle$ triplet corresponding to the image.
  • Figure 2: Accuracy of negative pairs for subject, verb, and object negatives given SVO frequencies in CC.
  • Figure 3: Comparing negative scores on MMT and Lang--MMT for images less or more similar to CC.