Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision+X: A Survey on Multimodal Learning in the Light of Data

Ye Zhu, Yu Wu, Nicu Sebe, Yan Yan

TL;DR

This paper analyzes the commonness and uniqueness of each data format mainly ranging from vision, audio, text, and motions, and presents the methodological advancements categorized by the combination of data modalities, such as Vision+Text, with slightly inclined emphasis on the visual data.

Abstract

We are perceiving and communicating with the world in a multisensory manner, where different information sources are sophisticatedly processed and interpreted by separate parts of the human brain to constitute a complex, yet harmonious and unified sensing system. To endow the machines with true intelligence, multimodal machine learning that incorporates data from various sources has become an increasingly popular research area with emerging technical advances in recent years. In this paper, we present a survey on multimodal machine learning from a novel perspective considering not only the purely technical aspects but also the intrinsic nature of different data modalities. We analyze the commonness and uniqueness of each data format mainly ranging from vision, audio, text, and motions, and then present the methodological advancements categorized by the combination of data modalities, such as Vision+Text, with slightly inclined emphasis on the visual data. We investigate the existing literature on multimodal learning from both the representation learning and downstream application levels, and provide an additional comparison in the light of their technical connections with the data nature, e.g., the semantic consistency between image objects and textual descriptions, and the rhythm correspondence between video dance moves and musical beats. We hope that the exploitation of the alignment as well as the existing gap between the intrinsic nature of data modality and the technical designs, will benefit future research studies to better address a specific challenge related to the concrete multimodal task, prompting a unified multimodal machine learning framework closer to a real human intelligence system.

Vision+X: A Survey on Multimodal Learning in the Light of Data

TL;DR

This paper analyzes the commonness and uniqueness of each data format mainly ranging from vision, audio, text, and motions, and presents the methodological advancements categorized by the combination of data modalities, such as Vision+Text, with slightly inclined emphasis on the visual data.

Abstract

We are perceiving and communicating with the world in a multisensory manner, where different information sources are sophisticatedly processed and interpreted by separate parts of the human brain to constitute a complex, yet harmonious and unified sensing system. To endow the machines with true intelligence, multimodal machine learning that incorporates data from various sources has become an increasingly popular research area with emerging technical advances in recent years. In this paper, we present a survey on multimodal machine learning from a novel perspective considering not only the purely technical aspects but also the intrinsic nature of different data modalities. We analyze the commonness and uniqueness of each data format mainly ranging from vision, audio, text, and motions, and then present the methodological advancements categorized by the combination of data modalities, such as Vision+Text, with slightly inclined emphasis on the visual data. We investigate the existing literature on multimodal learning from both the representation learning and downstream application levels, and provide an additional comparison in the light of their technical connections with the data nature, e.g., the semantic consistency between image objects and textual descriptions, and the rhythm correspondence between video dance moves and musical beats. We hope that the exploitation of the alignment as well as the existing gap between the intrinsic nature of data modality and the technical designs, will benefit future research studies to better address a specific challenge related to the concrete multimodal task, prompting a unified multimodal machine learning framework closer to a real human intelligence system.
Paper Structure (30 sections, 5 equations, 2 figures, 2 tables)

This paper contains 30 sections, 5 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Data Analysis
  3. Vision
  4. Audio
  5. Text
  6. Other Modalities
  7. Multimodal Representation Learning
  8. Network Architectures
  9. Supervised Learning
  10. Non-supervised Learning
  11. Trend in Representation Learning
  12. Discriminative Applications
  13. Vision+Audio
  14. Vision+Text
  15. Vision+Audio+Text
  16. ...and 15 more sections

Figures (2)

  • Figure 1: Overall structure of our survey. We first present different data modalities and their characteristics, along with examples of multimodal datasets. We then introduce the representation learning area categorized with learning settings. Next, we mainly divide the application area into the discriminative and the generative directions, with more detailed classifications following the combination of data modalities.
  • Figure 2: Illustrations of different audio data representations. From top to bottom: (a) raw audio data in waveform; (b) audio data in mel-spectrogram; (c) music piece in 1D piano-roll from dong2018pypianoroll, where the horizontal and vertical axis represent the timestamps and the audio pitch, respectively; (d) music piece in MIDI from inbook, where colors represent different instrument types.