Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zipper: A Multi-Tower Decoder Architecture for Fusing Modalities

Vicky Zayats, Peter Chen, Melissa Ferrari, Dirk Padfield

TL;DR

Zipper introduces a modular, decoder-decoder fusion architecture that stitches independently pre-trained unimodal decoders via gated cross-attention to enable multimodal generation while preserving unimodal capabilities. It demonstrates competitive cross-modal text generation (ASR) with a frozen text backbone and substantial improvements in speech generation (TTS) when the speech backbone is unfrozen, using limited cross-modal data. The approach reduces data requirements compared to vocabulary-expansion baselines and shows data-efficient learning by leveraging strong unimodal pretraining, suggesting potential for scaling to more modalities and domains. This modular fusion technique offers a flexible framework for integrating diverse modalities without retraining large multimodal models from scratch.

Abstract

Integrating multiple generative foundation models, especially those trained on different modalities, into something greater than the sum of its parts poses significant challenges. Two key hurdles are the availability of aligned data (concepts that contain similar meaning but is expressed differently in different modalities), and effectively leveraging unimodal representations in cross-domain generative tasks, without compromising their original unimodal capabilities. We propose Zipper, a multi-tower decoder architecture that addresses these concerns by using cross-attention to flexibly compose multimodal generative models from independently pre-trained unimodal decoders. In our experiments fusing speech and text modalities, we show the proposed architecture performs very competitively in scenarios with limited aligned text-speech data. We also showcase the flexibility of our model to selectively maintain unimodal (e.g., text-to-text generation) generation performance by freezing the corresponding modal tower (e.g. text). In cross-modal tasks such as automatic speech recognition (ASR) where the output modality is text, we show that freezing the text backbone results in negligible performance degradation. In cross-modal tasks such as text-to-speech generation (TTS) where the output modality is speech, we show that using a pre-trained speech backbone results in superior performance to the baseline.

Zipper: A Multi-Tower Decoder Architecture for Fusing Modalities

TL;DR

Zipper introduces a modular, decoder-decoder fusion architecture that stitches independently pre-trained unimodal decoders via gated cross-attention to enable multimodal generation while preserving unimodal capabilities. It demonstrates competitive cross-modal text generation (ASR) with a frozen text backbone and substantial improvements in speech generation (TTS) when the speech backbone is unfrozen, using limited cross-modal data. The approach reduces data requirements compared to vocabulary-expansion baselines and shows data-efficient learning by leveraging strong unimodal pretraining, suggesting potential for scaling to more modalities and domains. This modular fusion technique offers a flexible framework for integrating diverse modalities without retraining large multimodal models from scratch.

Abstract

Integrating multiple generative foundation models, especially those trained on different modalities, into something greater than the sum of its parts poses significant challenges. Two key hurdles are the availability of aligned data (concepts that contain similar meaning but is expressed differently in different modalities), and effectively leveraging unimodal representations in cross-domain generative tasks, without compromising their original unimodal capabilities. We propose Zipper, a multi-tower decoder architecture that addresses these concerns by using cross-attention to flexibly compose multimodal generative models from independently pre-trained unimodal decoders. In our experiments fusing speech and text modalities, we show the proposed architecture performs very competitively in scenarios with limited aligned text-speech data. We also showcase the flexibility of our model to selectively maintain unimodal (e.g., text-to-text generation) generation performance by freezing the corresponding modal tower (e.g. text). In cross-modal tasks such as automatic speech recognition (ASR) where the output modality is text, we show that freezing the text backbone results in negligible performance degradation. In cross-modal tasks such as text-to-speech generation (TTS) where the output modality is speech, we show that using a pre-trained speech backbone results in superior performance to the baseline.
Paper Structure (19 sections, 3 equations, 4 figures, 2 tables)

This paper contains 19 sections, 3 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model
  4. Auto-Regressive Masking
  5. Inference
  6. Experiments
  7. Experimental Setup
  8. Data
  9. Baseline
  10. Speech Tokenization and Generation
  11. Evaluation
  12. Significance scores
  13. Hyperparameter Tuning
  14. Automatic Speech Recognition
  15. Text to Speech
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Zipper model with gated cross-attention and projection layers.
  • Figure 2: WER on TTS task from Zipper and Single Decoder models vs. max gold transcript length.
  • Figure 3: WER on ASR task (validation set) as a function of the amount of aligned data.
  • Figure 4: Model design - ablations with respect to input projections and number of cross-attention layers.