Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey of Resource-efficient LLM and Multimodal Foundation Models

Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, Qiyang Zhang, Zhenyan Lu, Li Zhang, Shangguang Wang, Yuanchun Li, Yunxin Liu, Xin Jin, Xuanzhe Liu

TL;DR

<3-5 sentence high-level summary> The survey addresses the pressing resource challenges of large foundation models spanning language, vision, and multimodal domains. It systematically catalogs resource-efficient architectures, algorithms, and systems—from efficient attention and MoE to diffusion optimizations, NAS, quantization, and edge-enabled serving. Key contributions include a comprehensive taxonomy, cost analyses, and practical guidance on deployment across cloud and edge environments, highlighting trends like sparse models, PEFT, and on-device inference. The work aims to inspire sustainable scalability by detailing techniques that preserve performance while reducing compute, memory, and energy footprints, with an eye toward real-world deployment and privacy-preserving AI.</3-5 sentence high-level summary>

Abstract

Large foundation models, including large language models (LLMs), vision transformers (ViTs), diffusion, and LLM-based multimodal models, are revolutionizing the entire machine learning lifecycle, from training to deployment. However, the substantial advancements in versatility and performance these models offer come at a significant cost in terms of hardware resources. To support the growth of these large models in a scalable and environmentally sustainable way, there has been a considerable focus on developing resource-efficient strategies. This survey delves into the critical importance of such research, examining both algorithmic and systemic aspects. It offers a comprehensive analysis and valuable insights gleaned from existing literature, encompassing a broad array of topics from cutting-edge model architectures and training/serving algorithms to practical system designs and implementations. The goal of this survey is to provide an overarching understanding of how current approaches are tackling the resource challenges posed by large foundation models and to potentially inspire future breakthroughs in this field.

A Survey of Resource-efficient LLM and Multimodal Foundation Models

TL;DR

<3-5 sentence high-level summary> The survey addresses the pressing resource challenges of large foundation models spanning language, vision, and multimodal domains. It systematically catalogs resource-efficient architectures, algorithms, and systems—from efficient attention and MoE to diffusion optimizations, NAS, quantization, and edge-enabled serving. Key contributions include a comprehensive taxonomy, cost analyses, and practical guidance on deployment across cloud and edge environments, highlighting trends like sparse models, PEFT, and on-device inference. The work aims to inspire sustainable scalability by detailing techniques that preserve performance while reducing compute, memory, and energy footprints, with an eye toward real-world deployment and privacy-preserving AI.</3-5 sentence high-level summary>

Abstract

Large foundation models, including large language models (LLMs), vision transformers (ViTs), diffusion, and LLM-based multimodal models, are revolutionizing the entire machine learning lifecycle, from training to deployment. However, the substantial advancements in versatility and performance these models offer come at a significant cost in terms of hardware resources. To support the growth of these large models in a scalable and environmentally sustainable way, there has been a considerable focus on developing resource-efficient strategies. This survey delves into the critical importance of such research, examining both algorithmic and systemic aspects. It offers a comprehensive analysis and valuable insights gleaned from existing literature, encompassing a broad array of topics from cutting-edge model architectures and training/serving algorithms to practical system designs and implementations. The goal of this survey is to provide an overarching understanding of how current approaches are tackling the resource challenges posed by large foundation models and to potentially inspire future breakthroughs in this field.
Paper Structure (61 sections, 3 equations, 21 figures, 5 tables)

This paper contains 61 sections, 3 equations, 21 figures, 5 tables.

Table of Contents

  1. INTRODUCTION
  2. FOUNDATION MODEL OVERVIEW
  3. Language Foundation Models
  4. Model Architectures
  5. Representative Models and Downstream Tasks
  6. Cost Analysis
  7. Vision Foundation Models
  8. Model Architecture
  9. Representative Models and Downstream Tasks
  10. Cost Analysis
  11. Multimodal Foundation Models
  12. Key Architectures
  13. Representative Models and Downstream Tasks
  14. Cost Analysis
  15. RESOURCE-EFFICIENT ARCHITECTURES
  16. ...and 46 more sections

Figures (21)

  • Figure 1: The electricity consumption comparison between countries and AI. Data source: de2023growing.
  • Figure 2: The organization of this survey.
  • Figure 3: The evolutionary trace of foundation models.
  • Figure 4: Weight proportion.
  • Figure 5: FLOPs proportion.
  • ...and 16 more figures