A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming

TL;DR

The paper surveys the integration of Generative AI and large language models (LLMs) across video generation, understanding, and streaming, drawing on over 100 publications to build a comprehensive taxonomy of methods and applications. It highlights GAN/VAEs/autoregressive/diffusion-based generation, LLM-driven video captioning, QA, and retrieval for understanding, and LLM-enabled bandwidth, viewport prediction, and adaptive streaming for delivery. Key contributions include a structured framework linking generation, understanding, and streaming, a synthesis of state-of-the-art models and datasets, and a discussion of major challenges (temporal coherence, computation, data availability) and ethical concerns (misinformation, privacy, bias). The study emphasizes cross-disciplinary impacts on multimedia, networking, and AI, and calls for responsible AI practices, standardized benchmarks, and policy guidance to harness benefits while mitigating risks.

Abstract

This paper offers an insightful examination of how currently top-trending AI technologies, i.e., generative artificial intelligence (Generative AI) and large language models (LLMs), are reshaping the field of video technology, including video generation, understanding, and streaming. It highlights the innovative use of these technologies in producing highly realistic videos, a significant leap in bridging the gap between real-world dynamics and digital creation. The study also delves into the advanced capabilities of LLMs in video understanding, demonstrating their effectiveness in extracting meaningful information from visual content, thereby enhancing our interaction with videos. In the realm of video streaming, the paper discusses how LLMs contribute to more efficient and user-centric streaming experiences, adapting content delivery to individual viewer preferences. This comprehensive review navigates through the current achievements, ongoing challenges, and future possibilities of applying Generative AI and LLMs to video-related tasks, underscoring the immense potential these technologies hold for advancing the field of video technology related to multimedia, networking, and AI communities.

Figures (10)

• Figure 1: Taxonomy of video generation, understanding, and streaming with GAI and LLMs.
• Figure 2: An overview of advanced AI-based video generation technologies.
• Figure 3: An overview of LLMs for video scene understanding tasks.
• Figure 4: Illustration of a typical video transmission system. The scene of interest is captured by multiple cameras and the compressed video is conveyed to servers. The videos are distributed through the backbone network and directly received by mobile users from the corresponding wireless base station.
• Figure 5: Video Generation Applications.