Inference Performance Optimization for Large Language Models on CPUs

TL;DR

The paper tackles the challenge of deploying large language models with high performance in CPU-only environments by proposing an end-to-end optimization suite. It introduces SlimAttention for CPU-friendly attention, an INT8 KV cache with per-head scales and a specialized kernel, and a distributed inference framework using oneAPI CCL. Empirical results on Intel CPUs show meaningful improvements, including up to a 2.85× speedup for Llama2-70B when scaling from a single node to a small cluster, and generally superior CPU performance compared to FlashAttention-based baselines. The work is open-sourced, with plans to broaden CPU support, scale to larger batches, and extend to mixture-of-experts models.

Abstract

Large language models (LLMs) have shown exceptional performance and vast potential across diverse tasks. However, the deployment of LLMs with high performance in low-resource environments has garnered significant attention in the industry. When GPU hardware resources are limited, we can explore alternative options on CPUs. To mitigate the financial burden and alleviate constraints imposed by hardware resources, optimizing inference performance is necessary. In this paper, we introduce an easily deployable inference performance optimization solution aimed at accelerating LLMs on CPUs. In this solution, we implement an effective way to reduce the KV cache size while ensuring precision. We propose a distributed inference optimization approach and implement it based on oneAPI Collective Communications Library. Furthermore, we propose optimization approaches for LLMs on CPU, and conduct tailored optimizations for the most commonly used models. The code is open-sourced at https://github.com/intel/xFasterTransformer.

Figures (6)

• Figure 1: Slim Attention
• Figure 2: Flash Attention
• Figure 3: Example of maintained unique scale.
• Figure 4: Custom kernel workflow.
• Figure 5: Distributed inference based on oneCCL.