Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient LLM Inference on CPUs

Haihao Shen, Hanwen Chang, Bo Dong, Yu Luo, Hengyu Meng

TL;DR

This work tackles CPU-based deployment of large language models by combining an automatic INT4 weight-only quantization flow with a highly optimized CPU-oriented runtime. It pinpoints INT4 as a practical balance between model accuracy and inference efficiency, and develops a CPU tensor library with AVX2/AVX512/AMX support to accelerate LLMs. Empirical results show INT4 accuracy within 1% of FP32 and tokens generated with 20–80 ms latency on 4th-gen Xeon CPUs, outperforming a ggml-based baseline by up to 1.6x. The approach is demonstrated across 3B–20B models (e.g., Llama-2, GPT-NeoX), with memory footprints reduced dramatically in the Appendix, and is released publicly to enable CPU-based, high-performance LLM inference and broader accessibility.

Abstract

Large language models (LLMs) have demonstrated remarkable performance and tremendous potential across a wide range of tasks. However, deploying these models has been challenging due to the astronomical amount of model parameters, which requires a demand for large memory capacity and high memory bandwidth. In this paper, we propose an effective approach that can make the deployment of LLMs more efficiently. We support an automatic INT4 weight-only quantization flow and design a special LLM runtime with highly-optimized kernels to accelerate the LLM inference on CPUs. We demonstrate the general applicability of our approach on popular LLMs including Llama2, Llama, GPT-NeoX, and showcase the extreme inference efficiency on CPUs. The code is publicly available at: https://github.com/intel/intel-extension-for-transformers.

Efficient LLM Inference on CPUs

TL;DR

This work tackles CPU-based deployment of large language models by combining an automatic INT4 weight-only quantization flow with a highly optimized CPU-oriented runtime. It pinpoints INT4 as a practical balance between model accuracy and inference efficiency, and develops a CPU tensor library with AVX2/AVX512/AMX support to accelerate LLMs. Empirical results show INT4 accuracy within 1% of FP32 and tokens generated with 20–80 ms latency on 4th-gen Xeon CPUs, outperforming a ggml-based baseline by up to 1.6x. The approach is demonstrated across 3B–20B models (e.g., Llama-2, GPT-NeoX), with memory footprints reduced dramatically in the Appendix, and is released publicly to enable CPU-based, high-performance LLM inference and broader accessibility.

Abstract

Large language models (LLMs) have demonstrated remarkable performance and tremendous potential across a wide range of tasks. However, deploying these models has been challenging due to the astronomical amount of model parameters, which requires a demand for large memory capacity and high memory bandwidth. In this paper, we propose an effective approach that can make the deployment of LLMs more efficiently. We support an automatic INT4 weight-only quantization flow and design a special LLM runtime with highly-optimized kernels to accelerate the LLM inference on CPUs. We demonstrate the general applicability of our approach on popular LLMs including Llama2, Llama, GPT-NeoX, and showcase the extreme inference efficiency on CPUs. The code is publicly available at: https://github.com/intel/intel-extension-for-transformers.
Paper Structure (13 sections, 3 figures, 4 tables)

This paper contains 13 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. Automatic INT4 Quantization Flow
  4. Efficient LLM Runtime
  5. CPU Tensor Library.
  6. LLM Optimizations.
  7. Results
  8. Experimental Setup
  9. Accuracy
  10. Performance
  11. Discussion
  12. Summary and Future Work
  13. Appendix

Figures (3)

  • Figure 1: The left part is the automatic INT4 quantization flow: given a FP32 model, the flow takes the default INT4 quantization recipes and evaluates the accuracy of INT4 model; the recipe tuning loop is optional, if INT4 model can meet the accuracy target. The right part is a simplified runtime for efficient LLM inference built on top of a CPU tensor library with automatic kernel selector.
  • Figure 2: Key components in LLM runtime: general and LLM specialized.
  • Figure 3: KV cache optimization. Left (a) shows the default KV cache, where new token generation requires memory reallocation for all the tokens (5 in this example); right (b) shows the optimized KV cache with pre-allocated KV memory and only new token updated each time.