The Feasibility of Implementing Large-Scale Transformers on Multi-FPGA Platforms

TL;DR

The paper addresses the challenge of running large-scale transformers in data centers on multi-FPGA platforms by building an infrastructure and toolflow (Galapagos with the Galapagos Messaging Interface and Cluster Builder) to describe, map, and deploy models across FPGA clusters. A six-FPGA encoder implementation of I-BERT demonstrates end-to-end feasibility and enables latency/throughput estimation, while a detailed Versal ACAP analysis suggests competitive low-latency performance and potential power advantages over GPUs. Key contributions include enhancing Galapagos for scaling, implementing GMI for scalable inter-FPGA communication, automating multi-cluster deployment, and providing a performance model for I-BERT on Versal devices. The work suggests a viable path toward FPGA-based, cloud-scale transformers, with significant room for optimization and energy-efficiency gains as newer hardware and runtimes are explored.

Abstract

FPGAs are rarely mentioned when discussing the implementation of large machine learning applications, such as Large Language Models (LLMs), in the data center. There has been much evidence showing that single FPGAs can be competitive with GPUs in performance for some computations, especially for low latency, and often much more efficient when power is considered. This suggests that there is merit to exploring the use of multiple FPGAs for large machine learning applications. The challenge with using multiple FPGAs is that there is no commonly-accepted flow for developing and deploying multi-FPGA applications, i.e., there are no tools to describe a large application, map it to multiple FPGAs and then deploy the application on a multi-FPGA platform. In this paper, we explore the feasibility of implementing large transformers using multiple FPGAs by developing a scalable multi-FPGA platform and some tools to map large applications to the platform. We validate our approach by designing an efficient multi-FPGA version of the I-BERT transformer and implement one encoder using six FPGAs as a working proof-of-concept to show that our platform and tools work. Based on our proof-of-concept prototype and the estimations of performance using the latest FPGAs compared to GPUs, we conclude that there can be a place for FPGAs in the world of large machine learning applications. We demonstrate a promising first step that shows that with the right infrastructure and tools it is reasonable to continue to explore the possible benefits of using FPGAs for applications such as LLMs.

Figures (24)

• Figure 1: Galapagos hardware stack
• Figure 2: Galapagos hardware architecture
• Figure 3: BERT architecture
• Figure 4: Output Switch and Router in a Galapagos application
• Figure 5: Galapagos stack with the GMI Layer