Training and Serving System of Foundation Models: A Comprehensive Survey

TL;DR

This survey analyzes training and serving systems for foundation models, focusing on network, computing, and storage optimizations required to scale to hundreds of billions of parameters. It categorizes state-of-the-art techniques into parallelism (data, tensor, pipeline, expert, hybrid), memory and communication optimizations (checkpointing, mixed precision, ZeRO, memory swapping; overlap and compression), and comprehensive serving strategies (batching, sparsity, resource scheduling, memory management, multi-model inference). The work provides concrete method mappings (e.g., Megatron-LM, GShard, Alpa, DeepSpeed Inference) and discusses challenges such as data throughput, latency, and hardware efficiency, offering actionable guidance for system developers. By synthesizing these approaches, the paper aims to accelerate practical deployment and further innovation in foundation-model systems while addressing energy and security considerations.

Abstract

Foundation models (e.g., ChatGPT, DALL-E, PengCheng Mind, PanGu-$Σ$) have demonstrated extraordinary performance in key technological areas, such as natural language processing and visual recognition, and have become the mainstream trend of artificial general intelligence. This has led more and more major technology giants to dedicate significant human and financial resources to actively develop their foundation model systems, which drives continuous growth of these models' parameters. As a result, the training and serving of these models have posed significant challenges, including substantial computing power, memory consumption, bandwidth demands, etc. Therefore, employing efficient training and serving strategies becomes particularly crucial. Many researchers have actively explored and proposed effective methods. So, a comprehensive survey of them is essential for system developers and researchers. This paper extensively explores the methods employed in training and serving foundation models from various perspectives. It provides a detailed categorization of these state-of-the-art methods, including finer aspects such as network, computing, and storage. Additionally, the paper summarizes the challenges and presents a perspective on the future development direction of foundation model systems. Through comprehensive discussion and analysis, it hopes to provide a solid theoretical basis and practical guidance for future research and applications, promoting continuous innovation and development in foundation model systems.

Figures (4)

• Figure 1: Evolutionary Chart of Model Sizes Over Time.
• Figure 2: The lifecycle of the foundation model system.
• Figure 3: Schematic diagram of parallelization strategies in foundation model systems. Different color blocks indicate different layers in the network.
• Figure 4: Schematic diagram of pipeline parallelization and 1F1B pipeline parallelization.