Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

C3LLM: Conditional Multimodal Content Generation Using Large Language Models

Zixuan Wang, Qinkai Duan, Yu-Wing Tai, Chi-Keung Tang

TL;DR

This work addresses conditional multimodal generation across video, audio, and text by introducing C3LLM, a unified framework that uses an LLM as a modal bridge and a discrete audio vocabulary. The method combines a hierarchical EnCodec-based audio tokenizer, video embeddings projected into LLM space, and a non-autoregressive refinement stage to produce high-fidelity, semantically aligned outputs. Key contributions include (i) conditioning the LLM on three modalities, (ii) discrete audio tokenization for end-to-end generation, and (iii) integration of autoregressive coarse tokens with NAR refinement, demonstrated on VGGSound and AudioCaps with favorable comparisons to CoDi. The results suggest a versatile, end-to-end approach to multimodal content generation with practical implications for cross-modal synthesis and downstream tasks.

Abstract

We introduce C3LLM (Conditioned-on-Three-Modalities Large Language Models), a novel framework combining three tasks of video-to-audio, audio-to-text, and text-to-audio together. C3LLM adapts the Large Language Model (LLM) structure as a bridge for aligning different modalities, synthesizing the given conditional information, and making multimodal generation in a discrete manner. Our contributions are as follows. First, we adapt a hierarchical structure for audio generation tasks with pre-trained audio codebooks. Specifically, we train the LLM to generate audio semantic tokens from the given conditions, and further use a non-autoregressive transformer to generate different levels of acoustic tokens in layers to better enhance the fidelity of the generated audio. Second, based on the intuition that LLMs were originally designed for discrete tasks with the next-word prediction method, we use the discrete representation for audio generation and compress their semantic meanings into acoustic tokens, similar to adding "acoustic vocabulary" to LLM. Third, our method combines the previous tasks of audio understanding, video-to-audio generation, and text-to-audio generation together into one unified model, providing more versatility in an end-to-end fashion. Our C3LLM achieves improved results through various automated evaluation metrics, providing better semantic alignment compared to previous methods.

C3LLM: Conditional Multimodal Content Generation Using Large Language Models

TL;DR

This work addresses conditional multimodal generation across video, audio, and text by introducing C3LLM, a unified framework that uses an LLM as a modal bridge and a discrete audio vocabulary. The method combines a hierarchical EnCodec-based audio tokenizer, video embeddings projected into LLM space, and a non-autoregressive refinement stage to produce high-fidelity, semantically aligned outputs. Key contributions include (i) conditioning the LLM on three modalities, (ii) discrete audio tokenization for end-to-end generation, and (iii) integration of autoregressive coarse tokens with NAR refinement, demonstrated on VGGSound and AudioCaps with favorable comparisons to CoDi. The results suggest a versatile, end-to-end approach to multimodal content generation with practical implications for cross-modal synthesis and downstream tasks.

Abstract

We introduce C3LLM (Conditioned-on-Three-Modalities Large Language Models), a novel framework combining three tasks of video-to-audio, audio-to-text, and text-to-audio together. C3LLM adapts the Large Language Model (LLM) structure as a bridge for aligning different modalities, synthesizing the given conditional information, and making multimodal generation in a discrete manner. Our contributions are as follows. First, we adapt a hierarchical structure for audio generation tasks with pre-trained audio codebooks. Specifically, we train the LLM to generate audio semantic tokens from the given conditions, and further use a non-autoregressive transformer to generate different levels of acoustic tokens in layers to better enhance the fidelity of the generated audio. Second, based on the intuition that LLMs were originally designed for discrete tasks with the next-word prediction method, we use the discrete representation for audio generation and compress their semantic meanings into acoustic tokens, similar to adding "acoustic vocabulary" to LLM. Third, our method combines the previous tasks of audio understanding, video-to-audio generation, and text-to-audio generation together into one unified model, providing more versatility in an end-to-end fashion. Our C3LLM achieves improved results through various automated evaluation metrics, providing better semantic alignment compared to previous methods.
Paper Structure (17 sections, 2 equations, 4 figures, 7 tables)

This paper contains 17 sections, 2 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Audio Tokenizer and Video Tokenizer
  5. Autoregressive Generation of Coarse Audio Tokens
  6. Non-Autoregressive Transformer for Audio Refinement
  7. Detokenization for High Fidelity Output
  8. Experiments
  9. Training Datasets
  10. Evaluation Metrics
  11. Evaluation and comparison
  12. Ablation Studies
  13. Conclusion and Discussion
  14. Limitation and Future Work
  15. Conclusion
  16. ...and 2 more sections

Figures (4)

  • Figure 1: C3LLM is capable of video-to-audio, audio-to-text, and text-to-audio. Examples of different tasks are illustrated in arrows of different colors.
  • Figure 2: Overview of the video encoding part and the LLM. On the right is the non-autoregressive (NAR) transformer to further extend the coarse acoustic tokens into fine-grained acoustic tokens. We freeze the encoder, train a MLP, and finetune LLM using LoRA. For text condition, we directly input them into LLM after tokenization and also treat them as the condition for NAR transformer decoding.
  • Figure 3: Overview of audio encoding part and the LLM. We use a pre-trained CLAP encoder and an MLP layer to align the audio information with LLM embedding.
  • Figure 4: Comparison with baseline for video-to-audio generation task. CoDi failed to generate semantic-aligned audio and the generation is not clean, often mixed with human speaking or noise. Our method can produce aligned audio with clear synchronization.