JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation

TL;DR

JavisGPT tackles the problem of unified sounding-video understanding and generation by introducing an encoder-LLM-decoder architecture with a dedicated SyncFusion module for spatio-temporal AV fusion and synchrony-aware learnable queries that bridge to a pretrained JAV-DiT generator. The model integrates semantic conditioning via learnable queries $Q^c$ and $Q^s$ to produce $\\\hat{\mathbf{c}}$ and $\\hat{\mathbf{s}}$, enabling high-fidelity AV generation guided by diffusion-based decoding. A three-stage training pipeline (MM-PreTrain, AV-FineTune, MM-InstTune) together with the JavisInst-Omni dataset (over 200K GPT-4o-curated dialogues) progressively aligns perception and generation, while ablations and benchmarks demonstrate state-of-the-art performance on JAV benchmarks and strong synchronization in generation. The work advances practical, synchronized multimodal AI with potential applications in interactive media, education, and accessibility, while also noting considerations for misuse and the need for responsible deployment.

Abstract

This paper presents JavisGPT, the first unified multimodal large language model (MLLM) for Joint Audio-Video (JAV) comprehension and generation. JavisGPT adopts a concise encoder-LLM-decoder architecture, featuring a SyncFusion module for spatio-temporal audio-video fusion and synchrony-aware learnable queries to bridge a pretrained JAV-DiT generator. This design enables temporally coherent video-audio understanding and generation from multimodal instructions. We design an effective three-stage training pipeline consisting of multimodal pretraining, audio-video fine-tuning, and large-scale instruction-tuning, to progressively build multimodal comprehension and generation from existing vision-language models. To support this, we further construct JavisInst-Omni, a high-quality instruction dataset with over 200K GPT-4o-curated audio-video-text dialogues that span diverse and multi-level comprehension and generation scenarios. Extensive experiments on JAV comprehension and generation benchmarks show that JavisGPT outperforms existing MLLMs, particularly in complex and temporally synchronized settings.

Figures (14)

• Figure 1: JavisGPT supports multi-level synchronized audio-video (SyncVA) content (i.e., sounding video) comprehension (left), and simultaneously complex instruction-based SyncVA generation, interleaved in-context generation (middle), and multi-turn proactive conversations (right).
• Figure 2: Overall architecture of JavisGPT, which can perceive and produce videos and sounds simultaneously. The input video and audio are tokenized and fed into the SyncFusion module. The resulting synchronized audio-video representation, together with the text tokens and learnable JavisQueries are then passed to the LLM backbone. During decoding, the yielded JavisCond embeddings are used to align the LLM intents to the semantic conditional space of downstream JAV-DiT, enabling high-quality and synchronized sounding-video generation.
• Figure 3: Mechanism of SyncFusion. (1) Left: We align temporally-segmented audio tokens with corresponding video frames by using cross-attention to merge audio clues into visual patches, so as to capture spatiotemporal synchrony explicitly. (2) Right: Each resulting SyncAV token $e_{i,j}^{t} \in \mathbb{R}^{C}$ represents a sounding event occurring within the $i$-th row, $j$-th column visual patch of $t$-th frame.
• Figure 4: Instruction-followed and synchronized audio-video generation. We use learnable queries to gather useful information from all the audio, video, and text inputs, and use a two-layer MLP to map the hidden states from LLM to the conditional space of DiT. The hierarchical semantic and spatiotemporal prior conditions further enhance the synchrony in generated sounding videos. Alignment loss and diffusion loss are integrated to reduce optimization difficulty and training cost.
• Figure 5: (1) Left: We curate a large-scale, diverse, and balanced cross-modal instruction-tuning dataset (JavisInst-Omni) from multiple sources. (2) Right: Within JavisInst-Omni, JavisInst-Und and JavisInst-Gen are specifically developed for multi-level audio-video comprehension and generation with synchrony-awareness. Details are presented in \ref{['app:sec:dataset']}.