LTX-2: Efficient Joint Audio-Visual Foundation Model

TL;DR

LTX-2 addresses the gap in text-to-video systems by jointly generating synchronized audiovisual content. It fuses modality-specific VAEs with an asymmetric dual-stream diffusion transformer connected via bidirectional cross-attention and cross-modality AdaLN, enhanced by multilingual grounding and thinking-token based text conditioning. The model introduces modality-aware classifier-free guidance and a scalable multi-scale, multi-tile inference pipeline, coupled with an efficient stereo audio VAE and HiFi-GAN vocoder. Empirical results show state-of-the-art open-source audiovisual quality, strong lip-sync and Foley realism, and significantly faster inference than comparable open and proprietary systems, making high-fidelity T2AV generation practical and accessible.

Abstract

Recent text-to-video diffusion models can generate compelling video sequences, yet they remain silent -- missing the semantic, emotional, and atmospheric cues that audio provides. We introduce LTX-2, an open-source foundational model capable of generating high-quality, temporally synchronized audiovisual content in a unified manner. LTX-2 consists of an asymmetric dual-stream transformer with a 14B-parameter video stream and a 5B-parameter audio stream, coupled through bidirectional audio-video cross-attention layers with temporal positional embeddings and cross-modality AdaLN for shared timestep conditioning. This architecture enables efficient training and inference of a unified audiovisual model while allocating more capacity for video generation than audio generation. We employ a multilingual text encoder for broader prompt understanding and introduce a modality-aware classifier-free guidance (modality-CFG) mechanism for improved audiovisual alignment and controllability. Beyond generating speech, LTX-2 produces rich, coherent audio tracks that follow the characters, environment, style, and emotion of each scene -- complete with natural background and foley elements. In our evaluations, the model achieves state-of-the-art audiovisual quality and prompt adherence among open-source systems, while delivering results comparable to proprietary models at a fraction of their computational cost and inference time. All model weights and code are publicly released.

Figures (7)

• Figure 1: Overview of the LTX-2 architecture. Raw video and audio signals are encoded into modality-specific latent tokens via causal VAEs, while text is processed through a refined embedding pipeline. A dual-stream diffusion transformer jointly denoises audio and video latents with bidirectional audiovisual cross-attention and text conditioning, producing synchronized audiovisual outputs.
• Figure 2: Proposed architecture. (a) The dual-stream backbone processes video and audio latents in parallel, exchanging information via bidirectional cross-attention layers. (b) Detailed view of the Cross-Attention block, utilizing Temporal 1D RoPE for positional alignment and cross-modality AdaLN for timestep conditioning.
• Figure 3: Visualization of AV cross-attention maps. The maps are averaged across attention heads and the model layers; V2A and A2V maps correspond to the first and last 1/3 of inference steps, respectively. Red vertical lines on the audio waveform mark the timestamps of the displayed frames. The visualization demonstrates the model's ability to spatially track a moving vehicle, dynamically shift attention from one speaker to another and then to both simultaneously, and focus on the lip region during close-up speech.
• Figure 4: Overview of the Text Understanding pipeline. The text prompt is encoded by Gemma3 and refined through the Feature Extractor and Text Connector to condition the modality-specific DiT.
• Figure 5: Multimodal Classifier-Free Guidance with independent text and cross-modal control. The guided prediction is formed by combining the fully conditioned model output (orange) with two guidance directions: a text guidance term scaled by $s_t$ (green) and a cross-modal guidance term scaled by $s_m$ (blue). This supports independent control of textual conditioning and inter-modal alignment during inference.