Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Good things come in small packages: Should we build AI clusters with Lite-GPUs?

Burcu Canakci, Junyi Liu, Xingbo Wu, Nathanaël Cheriere, Paolo Costa, Sergey Legtchenko, Dushyanth Narayanan, Ant Rowstron

TL;DR

AI infrastructure scaling faces fundamental constraints on single-GPU die size, packaging, cooling, power, and cost. The paper proposes Lite-GPUs—tiny, single-die GPUs interconnected with co-packaged optics—to form large, high-bandwidth clusters with smaller blast radii and improved yield and power efficiency. It details system-level opportunities across distributed deployment, fine-grained resource management, workload scheduling, fault tolerance, memory, and networking, and presents a case study showing Lite-GPU clusters can match or exceed current H100-like performance at potentially lower cost for LLM inference. If realized, this approach could redefine data-center AI infrastructure by enabling modular, scalable, and more fault-tolerant deployments with tighter energy budgets.

Abstract

To match the blooming demand of generative AI workloads, GPU designers have so far been trying to pack more and more compute and memory into single complex and expensive packages. However, there is growing uncertainty about the scalability of individual GPUs and thus AI clusters, as state-of-the-art GPUs are already displaying packaging, yield, and cooling limitations. We propose to rethink the design and scaling of AI clusters through efficiently-connected large clusters of Lite-GPUs, GPUs with single, small dies and a fraction of the capabilities of larger GPUs. We think recent advances in co-packaged optics can enable distributing AI workloads onto many Lite-GPUs through high bandwidth and efficient communication. In this paper, we present the key benefits of Lite-GPUs on manufacturing cost, blast radius, yield, and power efficiency; and discuss systems opportunities and challenges around resource, workload, memory, and network management.

Good things come in small packages: Should we build AI clusters with Lite-GPUs?

TL;DR

AI infrastructure scaling faces fundamental constraints on single-GPU die size, packaging, cooling, power, and cost. The paper proposes Lite-GPUs—tiny, single-die GPUs interconnected with co-packaged optics—to form large, high-bandwidth clusters with smaller blast radii and improved yield and power efficiency. It details system-level opportunities across distributed deployment, fine-grained resource management, workload scheduling, fault tolerance, memory, and networking, and presents a case study showing Lite-GPU clusters can match or exceed current H100-like performance at potentially lower cost for LLM inference. If realized, this approach could redefine data-center AI infrastructure by enabling modular, scalable, and more fault-tolerant deployments with tighter energy budgets.

Abstract

To match the blooming demand of generative AI workloads, GPU designers have so far been trying to pack more and more compute and memory into single complex and expensive packages. However, there is growing uncertainty about the scalability of individual GPUs and thus AI clusters, as state-of-the-art GPUs are already displaying packaging, yield, and cooling limitations. We propose to rethink the design and scaling of AI clusters through efficiently-connected large clusters of Lite-GPUs, GPUs with single, small dies and a fraction of the capabilities of larger GPUs. We think recent advances in co-packaged optics can enable distributing AI workloads onto many Lite-GPUs through high bandwidth and efficient communication. In this paper, we present the key benefits of Lite-GPUs on manufacturing cost, blast radius, yield, and power efficiency; and discuss systems opportunities and challenges around resource, workload, memory, and network management.
Paper Structure (15 sections, 3 figures, 1 table)

This paper contains 15 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. The Lite-GPU
  3. Systems opportunities
  4. Scale of distribution
  5. Finer-granularity of resource management
  6. Workload management
  7. Fault-tolerance
  8. Memory management
  9. Network management
  10. Data-center management
  11. Case Study: LLM Inference
  12. Methodology and workload
  13. Results
  14. Related Work
  15. Conclusion

Figures (3)

  • Figure 1: Evolution of GPUs in AI clusters.
  • Figure 2: An example Lite-GPU deployment. Each NVIDIA H100 GPU is replaced with four Lite-GPUs, featuring better hardware yield and higher bandwidth-to-compute.
  • Figure 3: Results of the roofline modeling of H100 and Lite-H100 clusters. Note that Lite-H100 is already expected to be cheaper to manufacture, so we expect comparable performance to suffice.