Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation

Jing Liu, Xinxin Zhu, Fei Liu, Longteng Guo, Zijia Zhao, Mingzhen Sun, Weining Wang, Hanqing Lu, Shiyu Zhou, Jiajun Zhang, Jinqiao Wang

TL;DR

This paper tackles the lack of unified cross-modal models that jointly handle text, vision, and audio for both understanding and generation. It introduces the Omni-perception Pre-Trainer (OPT), a tri-modal encoder-decoder framework with three single-modal encoders, a cross-modal encoder, and two cross-modal decoders, trained via token-, modality-, and sample-level pretext tasks on Open Images triplets. The framework demonstrates strong performance across downstream tasks including classification, retrieval, VQA, and generation (text and image), and ablations confirm the contribution of each pre-training component. This work significantly advances multi-modal pre-training by enabling simultaneous cross-modal understanding and generation across three core modalities, with potential for broader applications and extension to additional modalities.

Abstract

In this paper, we propose an Omni-perception Pre-Trainer (OPT) for cross-modal understanding and generation, by jointly modeling visual, text and audio resources. OPT is constructed in an encoder-decoder framework, including three single-modal encoders to generate token-based embeddings for each modality, a cross-modal encoder to encode the correlations among the three modalities, and two cross-modal decoders to generate text and image respectively. For the OPT's pre-training, we design a multi-task pretext learning scheme to model multi-modal resources from three different data granularities, \ie, token-, modality-, and sample-level modeling, through which OPT learns to align and translate among different modalities. The pre-training task is carried out on a large amount of image-text-audio triplets from Open Images. Experimental results show that OPT can learn strong image-text-audio multi-modal representations and achieve promising results on a variety of cross-modal understanding and generation tasks.

OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation

TL;DR

This paper tackles the lack of unified cross-modal models that jointly handle text, vision, and audio for both understanding and generation. It introduces the Omni-perception Pre-Trainer (OPT), a tri-modal encoder-decoder framework with three single-modal encoders, a cross-modal encoder, and two cross-modal decoders, trained via token-, modality-, and sample-level pretext tasks on Open Images triplets. The framework demonstrates strong performance across downstream tasks including classification, retrieval, VQA, and generation (text and image), and ablations confirm the contribution of each pre-training component. This work significantly advances multi-modal pre-training by enabling simultaneous cross-modal understanding and generation across three core modalities, with potential for broader applications and extension to additional modalities.

Abstract

In this paper, we propose an Omni-perception Pre-Trainer (OPT) for cross-modal understanding and generation, by jointly modeling visual, text and audio resources. OPT is constructed in an encoder-decoder framework, including three single-modal encoders to generate token-based embeddings for each modality, a cross-modal encoder to encode the correlations among the three modalities, and two cross-modal decoders to generate text and image respectively. For the OPT's pre-training, we design a multi-task pretext learning scheme to model multi-modal resources from three different data granularities, \ie, token-, modality-, and sample-level modeling, through which OPT learns to align and translate among different modalities. The pre-training task is carried out on a large amount of image-text-audio triplets from Open Images. Experimental results show that OPT can learn strong image-text-audio multi-modal representations and achieve promising results on a variety of cross-modal understanding and generation tasks.

Paper Structure

This paper contains 18 sections, 11 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model Architecture
  4. Single-Modal Encoders
  5. Cross-Modal Encoder
  6. Cross-Modal Decoders
  7. Pre-Training Tasks
  8. Token-Level Modeling
  9. Modality-Level Modeling
  10. Sample-Level Modeling
  11. Experiments
  12. Pre-Training Dataset
  13. Implementation Details
  14. Results
  15. Ablation Study
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Model architecture of the proposed OPT, consisting of three single-modal encoders, a cross-modal encoder and two cross-modal decoders. We propose three levels of pre-training tasks: (1) token-level modeling, including masked language modeling (MLM), masked vision modeling (MVM), and masked audio modeling (MAM); (2) modality-level modeling, including denoising text reconstruction and denoising image reconstruction; and (3) sample-level modeling, where "$\surd$" denotes the corresponding modalities are matching. We introduce two masking mechanisms: (1) token-level masking, in order for token-level modeling; and (2) modality-level masking, in order for modality-level modeling and enabling arbitrary number of input modalities.
  • Figure 2: Illustration of Vision Decoder. The vision decoder consists of a Transformer decoder that learns image code and a pre-trained dVAE decoder that generates image.
  • Figure 3: Some results of text-to-image generation and text generation (including Image: image captioning, Audio: audio recognition, and Both: text generation with image+audio). The 1st column shows the ground-truth images and the 2nd column shows the generated images.