From Principles to Practice: A Systematic Study of LLM Serving on Multi-core NPUs

TL;DR

This work tackles the challenge of scalable LLM inference on multi-core NPUs with heterogeneous memory and interconnects. It introduces NpuSim, a hybrid, multi-level simulator that fuses transaction-level memory/interconnect modeling with compute-side performance models to enable efficient hardware design exploration and streaming workloads. Through systematic analysis of tensor partition, core placement, memory hierarchy, and PD disaggregation/fusion, the paper demonstrates up to 1.32x–6.03x speedups over SOTA designs and derives concrete guidance for hardware architectures and LLM serving strategies on multi-core NPUs. The framework and findings offer practical pathways to optimize LLM serving performance across diverse NPU configurations and workloads.

Abstract

With the widespread adoption of Large Language Models (LLMs), the demand for high-performance LLM inference services continues to grow. To meet this demand, a growing number of AI accelerators have been proposed, such as Google TPU, Huawei NPU, Graphcore IPU, and Cerebras WSE, etc. Most of these accelerators adopt multi-core architectures to achieve enhanced scalability, but lack the flexibility of SIMT architectures. Therefore, without careful configuration of the hardware architecture, as well as deliberate design of tensor parallelism and core placement strategies, computational resources may be underutilized, resulting in suboptimal inference performance. To address these challenges, we first present a multi-level simulation framework with both transaction-level and performance-model-based simulation for multi-core NPUs. Using this simulator, we conduct a systematic analysis and further propose the optimal solutions for tensor parallelism strategies, core placement policies, memory management methods, as well as the selection between PD-disaggregation and PD-fusion on multi-core NPUs. We conduct comprehensive experiments on representative LLMs and various NPU configurations. The evaluation results demonstrate that, our solution can achieve 1.32x-6.03x speedup compared to SOTA designs for multi-core NPUs across different hardware configurations. As for LLM serving, our work offers guidance on designing optimal hardware architectures and serving strategies for multi-core NPUs across various LLM workloads.

Figures (14)

• Figure 1: Hardware architecture of multi-core NPUs.
• Figure 2: The overall design of NpuSim: Computing/memory/router sub-systems with tracing and scheduler models.
• Figure 3: Different tensor partition strategies: For the GEMM operator, there exist three partition strategies: (1) 1-D partition along the M and N dimensions, which relies on the AllGather primitive; (2) 1-D partition along the K dimension, based on the AllReduce primitive; and (3) 2-D partition across the M/N and K dimensions, which leverages both AllReduce and AllGather primitives.
• Figure 4: Different core placement strategies: Considering both inter-pipe and intra-pipe communication cost.
• Figure 5: Multi-grained KV cache management for different memory hierarchies in NPUs: We adopt the fine-grained memory management for SRAM in block level, while adopts coarse-grained management for HBM in buffer level. The SRAM memory is also elaborately allocated for KV cache blocks, partial weight, activation and etc.