Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

W4A16 Mixed-Precision Matrix Multiplication on Decoupled Architecture: Kernel Design and Memory Bottleneck Analysis for Ascend NPUs

Yuanhong He, Peiyu Niu, Jun Chen, Chenchen Zhang, Chao Yang

TL;DR

The paper tackles efficient deployment of weight-only W4A16 quantization for large language models on Huawei's decoupled Ascend 910 NPU. It introduces a co-designed kernel that performs on-the-fly weight dequantization on vector cores and tiled GEMM on cube cores using a Split-K strategy, enabling practical W4A16 execution. While achieving up to $1.74\times$ speedup over data-parallel baselines, the work identifies memory-transfer overhead as the primary bottleneck, capping overall gains at about $1.48\times$ relative to FP16×FP16. These findings provide a foundation for quantized LLM deployment on domain-specific accelerators and motivate future hardware-software co-design to bypass global memory latency.

Abstract

As Large Language Models (LLMs) scale, weight-only quantization (W4A16: 4-bit weights, 16-bit activations) becomes critical for reducing memory footprint with minimal accuracy loss. However, its efficient deployment on Huawei's Ascend 910 Neural Processing Unit (NPU) is challenging due to limited native mixed-precision support and the accelerator's decoupled compute architecture. To enable quantization on such architecture, we present the first practical W4A16 matrix multiplication kernel tailored for the Ascend 910 NPU. Our design leverages vector cores for on-the-fly INT4-to-FP16 dequantization, cube cores for high-throughput GEMM, and Split-K parallelization to mitigate memory latency. Performance evaluations across diverse matrix shapes and batch sizes show our method outperforms data-parallel approaches when K >> N, a typical scenario in LLM decoding. Specially, our method can achieve a speedup ranging from 1.01x to 1.74x. In addition, our profile reveals the primary bottleneck is not dequantization compution itself, but extra global memory transfer for the weight, making W4A16 only reaching a maximum speedup of 1.48x over native FP16xFP16 matrix multiplication in PyTorch. In the long run, our method lays a solid foundation and provides insightful views for the efficient deployment of quantized large language models on various domain-specific accelerators.

W4A16 Mixed-Precision Matrix Multiplication on Decoupled Architecture: Kernel Design and Memory Bottleneck Analysis for Ascend NPUs

TL;DR

The paper tackles efficient deployment of weight-only W4A16 quantization for large language models on Huawei's decoupled Ascend 910 NPU. It introduces a co-designed kernel that performs on-the-fly weight dequantization on vector cores and tiled GEMM on cube cores using a Split-K strategy, enabling practical W4A16 execution. While achieving up to speedup over data-parallel baselines, the work identifies memory-transfer overhead as the primary bottleneck, capping overall gains at about relative to FP16×FP16. These findings provide a foundation for quantized LLM deployment on domain-specific accelerators and motivate future hardware-software co-design to bypass global memory latency.

Abstract

As Large Language Models (LLMs) scale, weight-only quantization (W4A16: 4-bit weights, 16-bit activations) becomes critical for reducing memory footprint with minimal accuracy loss. However, its efficient deployment on Huawei's Ascend 910 Neural Processing Unit (NPU) is challenging due to limited native mixed-precision support and the accelerator's decoupled compute architecture. To enable quantization on such architecture, we present the first practical W4A16 matrix multiplication kernel tailored for the Ascend 910 NPU. Our design leverages vector cores for on-the-fly INT4-to-FP16 dequantization, cube cores for high-throughput GEMM, and Split-K parallelization to mitigate memory latency. Performance evaluations across diverse matrix shapes and batch sizes show our method outperforms data-parallel approaches when K >> N, a typical scenario in LLM decoding. Specially, our method can achieve a speedup ranging from 1.01x to 1.74x. In addition, our profile reveals the primary bottleneck is not dequantization compution itself, but extra global memory transfer for the weight, making W4A16 only reaching a maximum speedup of 1.48x over native FP16xFP16 matrix multiplication in PyTorch. In the long run, our method lays a solid foundation and provides insightful views for the efficient deployment of quantized large language models on various domain-specific accelerators.
Paper Structure (10 sections, 2 equations, 3 figures, 1 algorithm)

This paper contains 10 sections, 2 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Quantization in Deep Neural Networks
  4. Motivation and Challenges
  5. Ascend 910 NPU Architecture
  6. Method
  7. Experimental Results
  8. Kernel Strategy Benchmarking
  9. Memory Bottleneck of W4A16 Kernel
  10. Conclusion and Future Work

Figures (3)

  • Figure 1: Ascend NPU architecture.
  • Figure 2: Execution time of INT4-weight × FP16-activation matrix multiplication on Ascend 910 NPU for various N×K configurations and batch sizes. The results compare two parallelization strategies: Split-K and Data Parallel.
  • Figure 3: Speedup of Split-K INT4×FP16 mixed-precision kernel over PyTorch' on Ascend 910 NPU for various N×K configurations and batch sizes.