Post-Training Quantization of OpenPangu Models for Efficient Deployment on Atlas A2

TL;DR

This work addresses the challenge of deploying large openPangu-Embedded models with multi-step CoT reasoning on resource-constrained Ascend NPUs. It introduces a hardware-aware, unified low-bit inference framework supporting INT8 (W8A8) and W4A8 for Atlas A2, and evaluates three CoT modes (slow_think, auto_think, no_think) on HumanEval and MBPP. The findings show INT8 maintains over 90% of FP16 accuracy while delivering up to 1.5× prefill speedup and substantial memory savings; W4A8 provides further compression with moderate accuracy loss, improved by SmoothQuant and Hadamard rotation. Collectively, the study demonstrates practical, high-fidelity, low-bit deployment of openPangu-Embedded models on Ascend hardware, enabling efficient edge reasoning without retraining.

Abstract

Huawei's openPangu-Embedded-1B and openPangu-Embedded-7B, variants of the openPangu large language model, integrate three distinct Chain-of-Thought (CoT) reasoning paradigms, namely slow_think, auto_think, and no_think. While these CoT modes enhance reasoning capabilities, their generation of extended reasoning traces introduces substantial memory and latency overheads, posing challenges for practical deployment on Ascend NPUs. This paper addresses these computational constraints by leveraging low-bit quantization, which transforms FP16 computations into more efficient integer arithmetic. We introduce a unified low-bit inference framework, supporting INT8 (W8A8) and W4A8 quantization, specifically optimized for openPangu-Embedded models on the Atlas A2. Our comprehensive evaluation, conducted across all three CoT modes on code generation benchmarks (HumanEval and MBPP), demonstrates the efficacy of this approach. INT8 quantization consistently preserves over 90\% of the FP16 baseline accuracy and achieves a 1.5x prefill speedup on the Atlas A2. Furthermore, W4A8 quantization significantly reduces memory consumption, albeit with a moderate trade-off in accuracy. These findings collectively indicate that low-bit quantization effectively facilitates efficient CoT reasoning on Ascend NPUs, maintaining high model fidelity.

Figures (4)

• Figure 1: Channel-wise absolute value distributions under different W4A8 quantization configurations. The baseline exhibits heavy-tailed distribution with large outliers, while both SmoothQuant and Hadamard preprocessing significantly smooth the distribution.
• Figure 2: Average word count on HumanEval and MBPP for FP16 and INT8 openPangu-Embedded-1B/7B under three CoT modes. Quantization has only a limited effect on output length in most cases, while the 7B model consistently produces shorter traces than the 1B model, hinting at more compact reasoning at scale.
• Figure 3: Qualitative comparison of CoT outputs from openPangu-Embedded-1B under FP16 and INT8 quantization. Green and red highlights indicate differences in wording or explanation depth between the two versions.
• Figure 4: Frequency of repetitive generation patterns across reasoning modes and model configurations on HumanEval. Repetitive generation is defined as terminal output segments containing identical phrases repeated until sequence termination. Bars represent the percentage of test samples exhibiting this behavior under FP16 and INT8 precision for 1B and 7B model variants.