Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Klear: Unified Multi-Task Audio-Video Joint Generation

Jun Wang, Chunyu Qiang, Yuxin Guo, Yiran Wang, Xijuan Zeng, Chen Zhang, Pengfei Wan

TL;DR

Klear addresses critical challenges in audio–video joint generation by unifying architecture, training strategy, and data curation. It employs a single-tower MM-DiT backbone with Omni-Full Attention and MixD-RoPE to tightly fuse audio, video, and their captions, coupled with a progressive multitask training regime and an automated, densely annotated AV dataset of 81 million samples. Empirical results show Klear surpasses prior open-source methods on AV benchmarks, attains strong unimodal performance, and generalizes well to out-of-distribution scenarios, approaching Veo-3 performance. The work presents a scalable, instruction-following AV generation pathway with broad implications for high-fidelity, semantically and temporally aligned AV synthesis.

Abstract

Audio-video joint generation has progressed rapidly, yet substantial challenges still remain. Non-commercial approaches still suffer audio-visual asynchrony, poor lip-speech alignment, and unimodal degradation, which can be stemmed from weak audio-visual correspondence modeling, limited generalization, and scarce high-quality dense-caption data. To address these issues, we introduce Klear and delve into three axes--model architecture, training strategy, and data curation. Architecturally, we adopt a single-tower design with unified DiT blocks and an Omni-Full Attention mechanism, achieving tight audio-visual alignment and strong scalability. Training-wise, we adopt a progressive multitask regime--random modality masking to joint optimization across tasks, and a multistage curriculum, yielding robust representations, strengthening A-V aligned world knowledge, and preventing unimodal collapse. For datasets, we present the first large-scale audio-video dataset with dense captions, and introduce a novel automated data-construction pipeline which annotates and filters millions of diverse, high-quality, strictly aligned audio-video-caption triplets. Building on this, Klear scales to large datasets, delivering high-fidelity, semantically and temporally aligned, instruction-following generation in both joint and unimodal settings while generalizing robustly to out-of-distribution scenarios. Across tasks, it substantially outperforms prior methods by a large margin and achieves performance comparable to Veo 3, offering a unified, scalable path toward next-generation audio-video synthesis.

Klear: Unified Multi-Task Audio-Video Joint Generation

TL;DR

Klear addresses critical challenges in audio–video joint generation by unifying architecture, training strategy, and data curation. It employs a single-tower MM-DiT backbone with Omni-Full Attention and MixD-RoPE to tightly fuse audio, video, and their captions, coupled with a progressive multitask training regime and an automated, densely annotated AV dataset of 81 million samples. Empirical results show Klear surpasses prior open-source methods on AV benchmarks, attains strong unimodal performance, and generalizes well to out-of-distribution scenarios, approaching Veo-3 performance. The work presents a scalable, instruction-following AV generation pathway with broad implications for high-fidelity, semantically and temporally aligned AV synthesis.

Abstract

Audio-video joint generation has progressed rapidly, yet substantial challenges still remain. Non-commercial approaches still suffer audio-visual asynchrony, poor lip-speech alignment, and unimodal degradation, which can be stemmed from weak audio-visual correspondence modeling, limited generalization, and scarce high-quality dense-caption data. To address these issues, we introduce Klear and delve into three axes--model architecture, training strategy, and data curation. Architecturally, we adopt a single-tower design with unified DiT blocks and an Omni-Full Attention mechanism, achieving tight audio-visual alignment and strong scalability. Training-wise, we adopt a progressive multitask regime--random modality masking to joint optimization across tasks, and a multistage curriculum, yielding robust representations, strengthening A-V aligned world knowledge, and preventing unimodal collapse. For datasets, we present the first large-scale audio-video dataset with dense captions, and introduce a novel automated data-construction pipeline which annotates and filters millions of diverse, high-quality, strictly aligned audio-video-caption triplets. Building on this, Klear scales to large datasets, delivering high-fidelity, semantically and temporally aligned, instruction-following generation in both joint and unimodal settings while generalizing robustly to out-of-distribution scenarios. Across tasks, it substantially outperforms prior methods by a large margin and achieves performance comparable to Veo 3, offering a unified, scalable path toward next-generation audio-video synthesis.
Paper Structure (17 sections, 5 equations, 5 figures, 3 tables)

This paper contains 17 sections, 5 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Klear
  4. Preliminary
  5. Single Tower with Full Attention
  6. Multi-Task Progressive Training Strategy
  7. Dataset Construction
  8. Dataset Filtering
  9. Audio-Guided Data Splitting
  10. Dense Annotation and Integration
  11. Experiments
  12. Experimental Setup
  13. Comparison with Existing Methods
  14. Qualitative Results
  15. Ablation and Analysis
  16. ...and 2 more sections

Figures (5)

  • Figure 1: We propose Klear, a unified audio–video generation framework which delivers high fidelity, strong semantic and temporal alignment, and reliable instruction following in both joint and unimodal settings, with robust OOD generalization. Across tasks (T2AV/TI2AV/TI2V/T2V/T2A), it attains performance comparable to Veo-3 among open-source models.
  • Figure 2: Overview of Klear. The model takes four inputs: video, video-related text, audio-related text, and audio. Each input is individually encoded by respective encoders, then fed into the MM-DiT. The MM-DiT module outputs the latent variables of video and audio, which are then decoded separately into video and audio.
  • Figure 3: Overview of our Dataset Annotation Pipeline.
  • Figure 4: Qualitative evaluation of audio-video joint generation across various aspects.
  • Figure 5: Ablations of different training stages. Metrics include video, audio, TTS, and audio-video consistency, with arrows indicating optimization directions.