FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization

TL;DR

FlattenQuant targets the compute-bound bottlenecks in large language model inference by enabling aggressive per-tensor quantization to INT4 in a substantial fraction of layers, while keeping other layers at 8-bit. By flattening large-value channels and applying channel smoothing, it significantly reduces the maximum tensor values, allowing stable 4-bit quantization with minimal accuracy loss. The method demonstrates up to approximately 2x speedups and 2.3x memory reductions on OPT models with negligible degradation, leveraging GPTQ to mitigate weight-quantization errors. This work highlights a pragmatic path toward faster, memory-efficient LLM deployment in compute-heavy regimes, given hardware support for INT4/GEMM and fused kernels for practical industrial use.

Abstract

Large language models (LLMs) have demonstrated state-of-the-art performance across various tasks. However, the latency of inference and the large GPU memory consumption of LLMs restrict their deployment performance. Recently, there have been some efficient attempts to quantize LLMs, yet inference with large batch size or long sequence still has the issue of being compute-bound. Fine-grained quantization methods have showcased their proficiency in achieving low-bit quantization for LLMs, while requiring FP16 data type for linear layer computations, which is time-consuming when dealing with large batch size or long sequence. In this paper, we introduce a method called FlattenQuant, which significantly reduces the maximum value of the tensor by flattening the large channels in the tensor, to achieve low bit per-tensor quantization with minimal accuracy loss. Our experiments show that FlattenQuant can directly use 4 bits to achieve 48.29% of the linear layer calculation in LLMs, with the remaining layers using 8 bits. The 4-bit matrix multiplication introduced in the FlattenQuant method can effectively address the compute-bound caused by large matrix calculation. Our work achieves up to 2$\times$ speedup and 2.3$\times$ memory reduction for LLMs with negligible loss in accuracy.

Figures (4)

• Figure 1: As the batch size increases, the determining factor for inference latency shifts from being memory-bound to compute-bound. GPTQ prioritizes memory optimization while utilizing FP16 computation, whereas W8A8's 8-bit computation prioritizes computation optimization.
• Figure 2: The concept of FlattenQuant is illustrated in (a). The expansion of certain channels is undertaken to accommodate those with larger values, another tensor repeats corresponding channels by the number of extended channels. Such operation itself does not result in any precision loss. We can observe that the activation's maximum absolute value in FlattenQuant is two thirds of that in SmoothQuant, accompanied by the weight's maximum absolute value being half of that in SmoothQuant. This flatter condition leads to a remarkable enhancement in quantization accuracy. (b) displays the activation input and weight in LLM. After the flatten operation, the tensor becomes flat and easily per-tensor quantized.
• Figure 3: Quantization framework based on FlattenQuant. The data flows through orange arrows, while gray arrows indicate operation configuration dependencies. Activation operations are executed during inference; and weight processing is done during model quantization.
• Figure 4: In compute-bound scenarios that involve large batch sizes and long sequences, our per-tensor method replaced FP16 matrix multiplication with INT4 and INT8 alternatives, yielding a considerable improvement in inference speed. This achievement is a direct result of our targeted efforts to overcome compute-bound challenges.