Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Combining Language and Vision with a Multimodal Skip-gram Model

Angeliki Lazaridou, Nghia The Pham, Marco Baroni

TL;DR

The authors extend the SKIP-GRAM framework with a visual grounding component to form MMSkip-gram, proposing two variants that jointly predict linguistic contexts and visual features for a subset of words. The models propagate visual information to the full vocabulary, achieving strong performance on semantic benchmarks and enabling zero-shot image labeling and retrieval, even for words without direct visual evidence. They also show that visual grounding influences abstract words, supporting embodied theories of meaning and suggesting applications in captioning and cognitive simulations. Overall, MMSkip-gram provides a general, scalable approach to learning distributional representations that integrate perceptual information without requiring full multimodal coverage at training time.

Abstract

We extend the SKIP-GRAM model of Mikolov et al. (2013a) by taking visual information into account. Like SKIP-GRAM, our multimodal models (MMSKIP-GRAM) build vector-based word representations by learning to predict linguistic contexts in text corpora. However, for a restricted set of words, the models are also exposed to visual representations of the objects they denote (extracted from natural images), and must predict linguistic and visual features jointly. The MMSKIP-GRAM models achieve good performance on a variety of semantic benchmarks. Moreover, since they propagate visual information to all words, we use them to improve image labeling and retrieval in the zero-shot setup, where the test concepts are never seen during model training. Finally, the MMSKIP-GRAM models discover intriguing visual properties of abstract words, paving the way to realistic implementations of embodied theories of meaning.

Combining Language and Vision with a Multimodal Skip-gram Model

TL;DR

The authors extend the SKIP-GRAM framework with a visual grounding component to form MMSkip-gram, proposing two variants that jointly predict linguistic contexts and visual features for a subset of words. The models propagate visual information to the full vocabulary, achieving strong performance on semantic benchmarks and enabling zero-shot image labeling and retrieval, even for words without direct visual evidence. They also show that visual grounding influences abstract words, supporting embodied theories of meaning and suggesting applications in captioning and cognitive simulations. Overall, MMSkip-gram provides a general, scalable approach to learning distributional representations that integrate perceptual information without requiring full multimodal coverage at training time.

Abstract

We extend the SKIP-GRAM model of Mikolov et al. (2013a) by taking visual information into account. Like SKIP-GRAM, our multimodal models (MMSKIP-GRAM) build vector-based word representations by learning to predict linguistic contexts in text corpora. However, for a restricted set of words, the models are also exposed to visual representations of the objects they denote (extracted from natural images), and must predict linguistic and visual features jointly. The MMSKIP-GRAM models achieve good performance on a variety of semantic benchmarks. Moreover, since they propagate visual information to all words, we use them to improve image labeling and retrieval in the zero-shot setup, where the test concepts are never seen during model training. Finally, the MMSKIP-GRAM models discover intriguing visual properties of abstract words, paving the way to realistic implementations of embodied theories of meaning.

Paper Structure

This paper contains 19 sections, 4 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Skip-gram Architecture
  4. Skip-gram Model
  5. Injecting visual knowledge
  6. Multi-modal Skip-gram Model A
  7. Multi-modal Skip-gram Model B
  8. Experimental Setup
  9. Hyperparameters
  10. Experiments
  11. Approximating human judgments
  12. Benchmarks
  13. Alternative Multimodal Models
  14. Results
  15. Zero-shot image labeling and retrieval
  16. ...and 4 more sections

Figures (3)

  • Figure 1: "Cartoon" of MMSkip-gram-B. Linguistic context vectors are actually associated to classes of words in a tree, not single words. Skip-gram is obtained by ignoring the visual objective, MMSkip-gram-A by fixing $M^{u\rightarrow v}$ to the identity matrix.
  • Figure 2: Examples of nearest visual neighbours of some abstract words: on the left, cases where subjects preferred the neighbour to the random foil; on the right, cases where they did not.
  • Figure 3: (a) Distribution of MMSkip-gram-A vector activation for meat (blue) and hope (red). (b) Spearman $\rho$ between concreteness and various measures on the ?) set.