Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge

Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, Mao Yang

TL;DR

<3-5 sentence high-level summary> T-MAC addresses the challenge of deploying low-bit LLMs on edge devices by replacing traditional multiplications with bit-wise LUT lookups, enabling native mixed-precision GEMM on CPUs. It introduces a LUT-centric data layout and storage-reduction techniques (mirror consolidation and table quantization) to fit LUTs in on-chip memory and maximize throughput. Through TVM-based code generation and hardware-aware optimizations, T-MAC achieves substantial end-to-end speedups and energy savings over dequantization-based baselines, and even surpasses GPU performance on some edge CPUs. The work demonstrates the practicality of edge-on-device LLM inference with low-bit models and motivates future LUT-based hardware accelerations for NLP workloads.

Abstract

The deployment of Large Language Models (LLMs) on edge devices is increasingly important to enhance on-device intelligence. Weight quantization is crucial for reducing the memory footprint of LLMs on devices. However, low-bit LLMs necessitate mixed precision matrix multiplication (mpGEMM) of low precision weights and high precision activations during inference. Existing systems, lacking native support for mpGEMM, resort to dequantize weights for high precision computation. Such an indirect way can lead to a significant inference overhead. In this paper, we introduce T-MAC, an innovative lookup table(LUT)-based method designed for efficient low-bit LLM (i.e., weight-quantized LLM) inference on CPUs. T-MAC directly supports mpGEMM without dequantization, while simultaneously eliminating multiplications and reducing additions required. Specifically, T-MAC transforms the traditional data-type-centric multiplication to bit-wise table lookup, and enables a unified and scalable mpGEMM solution. Our LUT-based kernels scale linearly to the weight bit-width. Evaluated on low-bit Llama and BitNet models, T-MAC demonstrates up to 4x increase in throughput and 70% reduction in energy consumption compared to llama.cpp. For BitNet-b1.58-3B, T-MAC delivers a token generation throughput of 30 tokens/s with a single core and 71 tokens/s with eight cores on M2-Ultra, and 11 tokens/s on lower-end devices like Raspberry Pi 5, which significantly exceeds the adult average reading speed. T-MAC with LUT-based computing paradigm, paves the way for the practical deployment of low-bit LLMs on resource-constrained edge devices without compromising computational efficiency. The system is open-sourced at https://github.com/microsoft/T-MAC .

T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge

TL;DR

<3-5 sentence high-level summary> T-MAC addresses the challenge of deploying low-bit LLMs on edge devices by replacing traditional multiplications with bit-wise LUT lookups, enabling native mixed-precision GEMM on CPUs. It introduces a LUT-centric data layout and storage-reduction techniques (mirror consolidation and table quantization) to fit LUTs in on-chip memory and maximize throughput. Through TVM-based code generation and hardware-aware optimizations, T-MAC achieves substantial end-to-end speedups and energy savings over dequantization-based baselines, and even surpasses GPU performance on some edge CPUs. The work demonstrates the practicality of edge-on-device LLM inference with low-bit models and motivates future LUT-based hardware accelerations for NLP workloads.

Abstract

The deployment of Large Language Models (LLMs) on edge devices is increasingly important to enhance on-device intelligence. Weight quantization is crucial for reducing the memory footprint of LLMs on devices. However, low-bit LLMs necessitate mixed precision matrix multiplication (mpGEMM) of low precision weights and high precision activations during inference. Existing systems, lacking native support for mpGEMM, resort to dequantize weights for high precision computation. Such an indirect way can lead to a significant inference overhead. In this paper, we introduce T-MAC, an innovative lookup table(LUT)-based method designed for efficient low-bit LLM (i.e., weight-quantized LLM) inference on CPUs. T-MAC directly supports mpGEMM without dequantization, while simultaneously eliminating multiplications and reducing additions required. Specifically, T-MAC transforms the traditional data-type-centric multiplication to bit-wise table lookup, and enables a unified and scalable mpGEMM solution. Our LUT-based kernels scale linearly to the weight bit-width. Evaluated on low-bit Llama and BitNet models, T-MAC demonstrates up to 4x increase in throughput and 70% reduction in energy consumption compared to llama.cpp. For BitNet-b1.58-3B, T-MAC delivers a token generation throughput of 30 tokens/s with a single core and 71 tokens/s with eight cores on M2-Ultra, and 11 tokens/s on lower-end devices like Raspberry Pi 5, which significantly exceeds the adult average reading speed. T-MAC with LUT-based computing paradigm, paves the way for the practical deployment of low-bit LLMs on resource-constrained edge devices without compromising computational efficiency. The system is open-sourced at https://github.com/microsoft/T-MAC .
Paper Structure (49 sections, 3 equations, 11 figures, 7 tables, 1 algorithm)

This paper contains 49 sections, 3 equations, 11 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. LLM on Edge
  4. LLM-on-edge benefits.
  5. LLM-on-edge challenges.
  6. Low-Bit (Weight-Quantized) LLM
  7. Deployment Challenges of Low-Bit LLM
  8. Mixed-precison GEMM/GEMV
  9. Bit-width/precision diversity
  10. LUT-based Computation for Quantized Model
  11. Design
  12. T-MAC Algorithm
  13. An example of LUT-based mpGEMM
  14. Challenges of LUT implementation
  15. LUT-Centric Data Layout
  16. ...and 34 more sections

Figures (11)

  • Figure 1: T-MAC vs general practice for mpGEMM.
  • Figure 2: T-MAC design overview.
  • Figure 3: Data flow of T-MAC vs general practice.
  • Figure 4: Interleave weights for fast unpacking.
  • Figure 5: Reduce LUT storage with mirror consolidation and table quantization. Mirror consolidation halves the table length. Table quantization reduces the table width.
  • ...and 6 more figures