Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ENEC: A Lossless AI Model Compression Method Enabling Fast Inference on Ascend NPUs

Jinwu Yang, Jiaan Wu, Zedong Liu, Xinyang Ma, Hairui Zhao, Yida Gu, Yuanhong Huang, Xingchen Liu, Wenjing Huang, Zheng Wei, Jing Xing, Yili Ma, Qingyi Zhang, Baoyi An, Zhongzhe Hu, Shaoteng Liu, Xia Zhu, Jiaxun Lu, Guangming Tan, Dingwen Tao

Abstract

The rapid scaling of Large Language Models presents significant challenges for their deployment and inference, particularly on resource-constrained specialized AI hardware accelerators such as Huawei's Ascend NPUs, where weight data transfer has become a critical performance bottleneck. While lossless compression can preserve model accuracy and reduce data volume, existing lossless compression algorithms exhibit extremely low throughput when ported to the Ascend NPU architecture. In this paper, we propose ENEC, a novel lossless compression method specifically customized for AI model weights and optimized for Ascend Neural Processing Units. ENEC adopts a block-based fixed-length encoding scheme and incorporates a series of NPU-specific optimizations: bit-width quantization with hierarchical halving bit-packing, vectorized branch-free integer transformation, and dependency-decoupled intra-segment scan for efficient prefix-sum computation. Experimental results demonstrate that ENEC outperforms existing state-of-the-art NPU compressors in both compression ratio and throughput. Compared to leading GPU solutions, ENEC achieves a 3.43X higher throughput than DietGPU and a 1.12X better compression ratio than nvCOMP. By reducing weight transmission overhead, ENEC significantly improves end-to-end inference performance, achieving up to a 6.3X speedup. On Ascend NPUs, ENEC is the first open-source lossless compression algorithm for model weights that achieves performance comparable to state-of-the-art GPU compressors, offering an effective solution for deploying large-scale AI models.

ENEC: A Lossless AI Model Compression Method Enabling Fast Inference on Ascend NPUs

Abstract

The rapid scaling of Large Language Models presents significant challenges for their deployment and inference, particularly on resource-constrained specialized AI hardware accelerators such as Huawei's Ascend NPUs, where weight data transfer has become a critical performance bottleneck. While lossless compression can preserve model accuracy and reduce data volume, existing lossless compression algorithms exhibit extremely low throughput when ported to the Ascend NPU architecture. In this paper, we propose ENEC, a novel lossless compression method specifically customized for AI model weights and optimized for Ascend Neural Processing Units. ENEC adopts a block-based fixed-length encoding scheme and incorporates a series of NPU-specific optimizations: bit-width quantization with hierarchical halving bit-packing, vectorized branch-free integer transformation, and dependency-decoupled intra-segment scan for efficient prefix-sum computation. Experimental results demonstrate that ENEC outperforms existing state-of-the-art NPU compressors in both compression ratio and throughput. Compared to leading GPU solutions, ENEC achieves a 3.43X higher throughput than DietGPU and a 1.12X better compression ratio than nvCOMP. By reducing weight transmission overhead, ENEC significantly improves end-to-end inference performance, achieving up to a 6.3X speedup. On Ascend NPUs, ENEC is the first open-source lossless compression algorithm for model weights that achieves performance comparable to state-of-the-art GPU compressors, offering an effective solution for deploying large-scale AI models.

Paper Structure

This paper contains 47 sections, 4 equations, 13 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Ascend NPUs
  4. Lossless Floating-Point Compressors
  5. LC Framework to Search Optimal Compression Algorithm
  6. Data Analysis of AI Model Weights
  7. Architecture Analysis and Basic Design
  8. Architecture Analysis of Lossless Compression Incompatibility on Ascend NPUs
  9. Basic Design of ENEC
  10. Optimizations of ENEC
  11. Overview of optimized ENEC
  12. Bit-Width Quantization with Hierarchical Halving Bit-Packing
  13. Vectorized Branch-Free Integer Transformation
  14. Intra-Segment Dependency Decoupled Scan
  15. Parameter Tuning
  16. ...and 32 more sections

Figures (13)

  • Figure 1: Time breakdown of Qwen3-32B Inference on NPU 910B2 (left) and throughput of ANS across multiple platforms (right).
  • Figure 2: DaVinci architecture (decoupled AIC and AIV).
  • Figure 3: Linear relationship between exponent values and frequency rankings in model weights. The red circle indicates outliers.
  • Figure 4: Execution challenges and constraints on Ascend NPUs.
  • Figure 5: Overview of optimized ENEC compression design on Ascend NPUs.
  • ...and 8 more figures