DABench-LLM: Standardized and In-Depth Benchmarking of Post-Moore Dataflow AI Accelerators for LLMs

TL;DR

DABench-LLM tackles the gap in standardized, in-depth benchmarking for post-Moore dataflow AI accelerators running LLMs. It introduces a two-tier framework that performs intra-chip profiling and inter-chip scalability analysis, using decoder-block units to systematically expose bottlenecks and optimization opportunities. Validations on Cerebras WSE-2, SambaNova RDU, and Graphcore IPU reveal distinct bottlenecks—ranging from high memory bandwidth bottlenecks to load-balance and partitioning inefficiencies—and provide actionable deployment guidelines. The framework, with open-source code, enables researchers to compare hardware platforms and guide kernel, graph compiler, and deployment optimizations for scalable LLM training on dataflow architectures. This work advances practical hardware selection and optimization for large-scale LLMs in dataflow ecosystems.

Abstract

The exponential growth of large language models has outpaced the capabilities of traditional CPU and GPU architectures due to the slowdown of Moore's Law. Dataflow AI accelerators present a promising alternative; however, there remains a lack of in-depth performance analysis and standardized benchmarking methodologies for LLM training. We introduce DABench-LLM, the first benchmarking framework designed for evaluating LLM workloads on dataflow-based accelerators. By combining intra-chip performance profiling and inter-chip scalability analysis, DABench-LLM enables comprehensive evaluation across key metrics such as resource allocation, load balance, and resource efficiency. The framework helps researchers rapidly gain insights into underlying hardware and system behaviors, and provides guidance for performance optimizations. We validate DABench-LLM on three commodity dataflow accelerators, Cerebras WSE-2, SambaNova RDU, and Graphcore IPU. Our framework reveals performance bottlenecks and provides specific optimization strategies, demonstrating its generality and effectiveness across a diverse range of dataflow-based AI hardware platforms.

Figures (12)

• Figure 1: Architecture of CS-2 System. The left side shows the overall architecture of the CS-2 system, the middle illustrates the structure of its core component—the WSE-2 chip, and the right side presents the architecture of the basic computational unit (PE, Processing Element) that composes the WSE-2.
• Figure 2: Architecture of SN30 system. The left side shows two compute units (RDU1 and RDU2) with tiles connected to memory and host through a memory manager and interconnect. The right side is a zoom-in view of an RDU, showing the internal grid of switches connecting processing units and address generators.
• Figure 3: Architecture of Bow-2000. The top-left panel illustrates the overall architecture of the Bow-2000 system. The right panel presents the internal structure of an IPU, while the bottom-left panel shows the layout of a tile column and the basic structure of a single tile.
• Figure 4: Section Partitioning Strategies under Different Compilation Modes (Example: Decoder Forward Pass)
• Figure 5: Overview of framework. Given a new dataflow-based hardware platform, Tier-1 evaluates its performance when executing LLM workloads on a single chip, while Tier-2 assesses its scalability across multiple chips and investigates optimization strategies for deployment.