Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation

Jeongsoo Choi, Ji-Hoon Kim, Kim Sung-Bin, Tae-Hyun Oh, Joon Son Chung

TL;DR

AlignDiT tackles multimodal-to-speech generation from text, video, and reference audio by casting speech synthesis as a conditional diffusion process. It introduces a Diffusion Transformer with audio-video-text fusion and three conditioning strategies, selecting multimodal cross-attention as the most effective, and augments training with a multi-task objective that combines conditional flow matching and CTC-based alignment. The model benefits from audio-only pretraining and a multimodal classifier-free guidance scheme with modality-specific controls, enabling robust handling of varying input modalities. On large-scale benchmarks, AlignDiT achieves state-of-the-art results in ADR, video-to-speech, and visual forced alignment, while also demonstrating strong generalization to related multimodal tasks and practical applicability for dubbing and virtual avatars.

Abstract

In this paper, we address the task of multimodal-to-speech generation, which aims to synthesize high-quality speech from multiple input modalities: text, video, and reference audio. This task has gained increasing attention due to its wide range of applications, such as film production, dubbing, and virtual avatars. Despite recent progress, existing methods still suffer from limitations in speech intelligibility, audio-video synchronization, speech naturalness, and voice similarity to the reference speaker. To address these challenges, we propose AlignDiT, a multimodal Aligned Diffusion Transformer that generates accurate, synchronized, and natural-sounding speech from aligned multimodal inputs. Built upon the in-context learning capability of the DiT architecture, AlignDiT explores three effective strategies to align multimodal representations. Furthermore, we introduce a novel multimodal classifier-free guidance mechanism that allows the model to adaptively balance information from each modality during speech synthesis. Extensive experiments demonstrate that AlignDiT significantly outperforms existing methods across multiple benchmarks in terms of quality, synchronization, and speaker similarity. Moreover, AlignDiT exhibits strong generalization capability across various multimodal tasks, such as video-to-speech synthesis and visual forced alignment, consistently achieving state-of-the-art performance. The demo page is available at https://mm.kaist.ac.kr/projects/AlignDiT.

AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation

TL;DR

AlignDiT tackles multimodal-to-speech generation from text, video, and reference audio by casting speech synthesis as a conditional diffusion process. It introduces a Diffusion Transformer with audio-video-text fusion and three conditioning strategies, selecting multimodal cross-attention as the most effective, and augments training with a multi-task objective that combines conditional flow matching and CTC-based alignment. The model benefits from audio-only pretraining and a multimodal classifier-free guidance scheme with modality-specific controls, enabling robust handling of varying input modalities. On large-scale benchmarks, AlignDiT achieves state-of-the-art results in ADR, video-to-speech, and visual forced alignment, while also demonstrating strong generalization to related multimodal tasks and practical applicability for dubbing and virtual avatars.

Abstract

In this paper, we address the task of multimodal-to-speech generation, which aims to synthesize high-quality speech from multiple input modalities: text, video, and reference audio. This task has gained increasing attention due to its wide range of applications, such as film production, dubbing, and virtual avatars. Despite recent progress, existing methods still suffer from limitations in speech intelligibility, audio-video synchronization, speech naturalness, and voice similarity to the reference speaker. To address these challenges, we propose AlignDiT, a multimodal Aligned Diffusion Transformer that generates accurate, synchronized, and natural-sounding speech from aligned multimodal inputs. Built upon the in-context learning capability of the DiT architecture, AlignDiT explores three effective strategies to align multimodal representations. Furthermore, we introduce a novel multimodal classifier-free guidance mechanism that allows the model to adaptively balance information from each modality during speech synthesis. Extensive experiments demonstrate that AlignDiT significantly outperforms existing methods across multiple benchmarks in terms of quality, synchronization, and speaker similarity. Moreover, AlignDiT exhibits strong generalization capability across various multimodal tasks, such as video-to-speech synthesis and visual forced alignment, consistently achieving state-of-the-art performance. The demo page is available at https://mm.kaist.ac.kr/projects/AlignDiT.
Paper Structure (21 sections, 11 equations, 2 figures, 7 tables)

This paper contains 21 sections, 11 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multimodal Speech Tasks
  4. Text-to-speech (TTS) Synthesis
  5. AlignDiT
  6. Model Architecture
  7. Multimodal Conditioning
  8. Training Objective
  9. Audio-only Pretraining
  10. Multimodal Classifier-Free Guidance
  11. Experiments
  12. Datasets
  13. Evaluation Metrics
  14. Implementation Details
  15. Baseline Models
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Various methods for conditioning multimodal inputs to DiT blocks: (a) channel-wise concatenation of text, reference speech, and visual features; (b) providing the text as a prefix; and (c) conditioning text inputs through cross-attention. In (a), $\phi$ represents filler tokens.
  • Figure 2: We qualitatively compare mel-spectrogram visualizations of ground-truth speech and synthesized speech from the ADR task using AlignDiT (ours) and existing methods, HPMDubbing and StyleDubber. We also provide results from unimodal inputs, either video-only (VTS) or text-only (TTS). The text below each mel-spectrogram represents time-aligned speech extracted using Whisper-large-v3, with red text indicating incorrectly synthesized word compared to the ground truth.