ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs

TL;DR

ArcLight is a lightweight LLM inference architecture designed from the ground up for many-core CPUs that integrates efficient memory management and thread scheduling, and introduces finely controlled tensor parallelism to mitigate the cross-node memory access wall.

Abstract

Although existing frameworks for large language model (LLM) inference on CPUs are mature, they fail to fully exploit the computation potential of many-core CPU platforms. Many-core CPUs are widely deployed in web servers and high-end networking devices, and are typically organized into multiple NUMA nodes that group cores and memory. Current frameworks largely overlook the substantial overhead of cross-NUMA memory access, limiting inference scalability and intelligence enabling on such platforms. To address this limitation, we build ArcLight, a lightweight LLM inference architecture designed from the ground up for many-core CPUs. ArcLight integrates efficient memory management and thread scheduling, and introduces finely controlled tensor parallelism to mitigate the cross-node memory access wall. Experimental results show that ArcLight significantly surpasses the performance ceiling of mainstream frameworks, achieving up to 46% higher inference throughput. Moreover, ArcLight maintains compatibility with arbitrary CPU devices. ArcLight is publicly available at https://github.com/OpenBMB/ArcLight.

Figures (13)

• Figure 1: For many-core CPU platforms, CPU cores and memory are organized using NUMA nodes. Here shows a platform with 128 cores and 4 NUMA nodes.
• Figure 2: Architecture of ArcLight.
• Figure 3: Comparison of UMA (top) and NUMA (bottom) buffer management strategy. In the UMA strategy, the NUMA nodes are transparent.
• Figure 4: Double-buffering mechanism.
• Figure 5: Multi-view thread organization.