Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning

Amita Kamath, Jack Hessel, Khyathi Chandu, Jena D. Hwang, Kai-Wei Chang, Ranjay Krishna

TL;DR

The data underlying the popular VLMs OpenCLIP, LLaVA-1.5 and Molmo are investigated through the lens of theories from pragmatics, and it is found that VLMs perform poorly on types of reasoning suppressed in the training data by reporting bias.

Abstract

The lack of reasoning capabilities in Vision-Language Models (VLMs) has remained at the forefront of research discourse. We posit that this behavior stems from a reporting bias in their training data. That is, how people communicate about visual content by default omits tacit information needed to supervise some types of reasoning; e.g., "at the game today!" is a more likely caption than "a photo of 37 people standing behind a field". We investigate the data underlying the popular VLMs OpenCLIP, LLaVA-1.5 and Molmo through the lens of theories from pragmatics, and find that reporting bias results in insufficient representation of four reasoning skills (spatial, temporal, negation, and counting), despite the corpora being of web-scale, and/or synthetically generated. With a set of curated benchmarks, we demonstrate that: (i) VLMs perform poorly on the aforementioned types of reasoning suppressed in the training data by reporting bias; (ii) contrary to popular belief, scaling data size, model size, and to multiple languages does not result in emergence of these skills by default; but, promisingly, (iii) incorporating annotations specifically collected to obtain tacit information is effective. Our findings highlight the need for more intentional training data curation methods, rather than counting on scale for emergence of reasoning capabilities.

Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning

TL;DR

The data underlying the popular VLMs OpenCLIP, LLaVA-1.5 and Molmo are investigated through the lens of theories from pragmatics, and it is found that VLMs perform poorly on types of reasoning suppressed in the training data by reporting bias.

Abstract

The lack of reasoning capabilities in Vision-Language Models (VLMs) has remained at the forefront of research discourse. We posit that this behavior stems from a reporting bias in their training data. That is, how people communicate about visual content by default omits tacit information needed to supervise some types of reasoning; e.g., "at the game today!" is a more likely caption than "a photo of 37 people standing behind a field". We investigate the data underlying the popular VLMs OpenCLIP, LLaVA-1.5 and Molmo through the lens of theories from pragmatics, and find that reporting bias results in insufficient representation of four reasoning skills (spatial, temporal, negation, and counting), despite the corpora being of web-scale, and/or synthetically generated. With a set of curated benchmarks, we demonstrate that: (i) VLMs perform poorly on the aforementioned types of reasoning suppressed in the training data by reporting bias; (ii) contrary to popular belief, scaling data size, model size, and to multiple languages does not result in emergence of these skills by default; but, promisingly, (iii) incorporating annotations specifically collected to obtain tacit information is effective. Our findings highlight the need for more intentional training data curation methods, rather than counting on scale for emergence of reasoning capabilities.
Paper Structure (52 sections, 4 figures, 4 tables)

This paper contains 52 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Explanatory hypotheses.
  4. Reporting Bias in Vision-Language Reasoning
  5. Theory-based Hypotheses of Omitted Types of Reasoning
  6. People tend to omit spatial and temporal language.
  7. People tend to omit counting.
  8. People tend to omit negations.
  9. Testing Hypotheses in Open-Source Image-Text Corpora
  10. Comparison to higher-frequency concepts.
  11. Comparison to concepts that compose similarly.
  12. Benchmarks
  13. Spatial reasoning.
  14. Counting.
  15. Negations.
  16. ...and 37 more sections

Figures (4)

  • Figure 1: Examples from LAION-2B of data points that contain reasoning-related keywords that do and do not operationalize the reasoning capability itself.
  • Figure 2: Examples from our four benchmarks for contrastive and generative evaluations. The generative evaluation is in MCQ format but for counting, for which a free form output with a given range yielded higher scores.
  • Figure 3: Scaling laws for OpenCLIP models on ImageNet (top left) compared to our benchmarks on spatial, counting, negation and temporal tasks respectively. Note the log-log plots and differing $y$ axes across graphs.
  • Figure 4: Instructions provided for the COCO (top left), LLaVA-1.5 (top right), PixMo (bottom left) and our (bottom right) sets of instructions.