Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Audio-Visual Intelligence in Large Foundation Models

You Qin, Kai Liu, Shengqiong Wu, Kai Wang, Shijian Deng, Yapeng Tian, Junbin Xiao, Yazhou Xing, Yinghao Ma, Bobo Li, Roger Zimmermann, Lei Cui, Furu Wei, Jiebo Luo, Hao Fei

Abstract

Audio-Visual Intelligence (AVI) has emerged as a central frontier in artificial intelligence, bridging auditory and visual modalities to enable machines that can perceive, generate, and interact in the multimodal real world. In the era of large foundation models, joint modeling of audio and vision has become increasingly crucial, i.e., not only for understanding but also for controllable generation and reasoning across dynamic, temporally grounded signals. Recent advances, such as Meta MovieGen and Google Veo-3, highlight the growing industrial and academic focus on unified audio-vision architectures that learn from massive multimodal data. However, despite rapid progress, the literature remains fragmented, spanning diverse tasks, inconsistent taxonomies, and heterogeneous evaluation practices that impede systematic comparison and knowledge integration. This survey provides the first comprehensive review of AVI through the lens of large foundation models. We establish a unified taxonomy covering the broad landscape of AVI tasks, ranging from understanding (e.g., speech recognition, sound localization) to generation (e.g., audio-driven video synthesis, video-to-audio) and interaction (e.g., dialogue, embodied, or agentic interfaces). We synthesize methodological foundations, including modality tokenization, cross-modal fusion, autoregressive and diffusion-based generation, large-scale pretraining, instruction alignment, and preference optimization. Furthermore, we curate representative datasets, benchmarks, and evaluation metrics, offering a structured comparison across task families and identifying open challenges in synchronization, spatial reasoning, controllability, and safety. By consolidating this rapidly expanding field into a coherent framework, this survey aims to serve as a foundational reference for future research on large-scale AVI.

Audio-Visual Intelligence in Large Foundation Models

Abstract

Audio-Visual Intelligence (AVI) has emerged as a central frontier in artificial intelligence, bridging auditory and visual modalities to enable machines that can perceive, generate, and interact in the multimodal real world. In the era of large foundation models, joint modeling of audio and vision has become increasingly crucial, i.e., not only for understanding but also for controllable generation and reasoning across dynamic, temporally grounded signals. Recent advances, such as Meta MovieGen and Google Veo-3, highlight the growing industrial and academic focus on unified audio-vision architectures that learn from massive multimodal data. However, despite rapid progress, the literature remains fragmented, spanning diverse tasks, inconsistent taxonomies, and heterogeneous evaluation practices that impede systematic comparison and knowledge integration. This survey provides the first comprehensive review of AVI through the lens of large foundation models. We establish a unified taxonomy covering the broad landscape of AVI tasks, ranging from understanding (e.g., speech recognition, sound localization) to generation (e.g., audio-driven video synthesis, video-to-audio) and interaction (e.g., dialogue, embodied, or agentic interfaces). We synthesize methodological foundations, including modality tokenization, cross-modal fusion, autoregressive and diffusion-based generation, large-scale pretraining, instruction alignment, and preference optimization. Furthermore, we curate representative datasets, benchmarks, and evaluation metrics, offering a structured comparison across task families and identifying open challenges in synchronization, spatial reasoning, controllability, and safety. By consolidating this rapidly expanding field into a coherent framework, this survey aims to serve as a foundational reference for future research on large-scale AVI.

Paper Structure

This paper contains 79 sections, 17 figures, 24 tables.

Table of Contents

  1. Introduction
  2. Contribution Summary
  3. Scope and Organization
  4. Preliminary
  5. The Audio Modality: Data and Representation
  6. The Visual Modality: Data and Representation
  7. Task Taxonomy
  8. Understanding the World: Audio-Visual Perception
  9. Creating the World: Audio-Visual Generation
  10. Interacting with the World: Audio-Visual Unified Perception and Generation
  11. Foundation Techniques
  12. Representation-centric Methods
  13. Audio-Visual Feature Extraction and Representation
  14. Audio-Visual Variational Auto-Encoding and Reconstruction
  15. Audio-Visual Discrete Tokenization
  16. ...and 64 more sections

Figures (17)

  • Figure 1: The evolutionary tree of Audio-Visual Intelligence from 2016 to 2026. The four columns track representative methods across understanding the world (audio-visual perception), creating the world (audio-visual generation), interacting with the world (unified perception and generation), and the converged AVI systems of today.
  • Figure 2: Taxonomy of Audio-Visual Intelligence.
  • Figure 3: The overview of Audio-Visual data representation.
  • Figure 4: Overview of audio-visual intelligence tasks.
  • Figure 5: The overview comparison of different representation learning techniques.
  • ...and 12 more figures