STaMP: Sequence Transformation and Mixed Precision for Low-Precision Activation Quantization
Marco Federici, Riccardo Del Chiaro, Boris van Breugel, Paul Whatmough, Markus Nagel
TL;DR
STaMP introduces a sequence-dimension activation transform combined with mixed-precision quantization to reduce activation quantization error at very low bitwidths. By concentrating energy into a small set of tokens using transforms such as DCT/DWT (a practical stand-in for the optimal KLT), STaMP enables higher precision for a subset of activations while maintaining a fixed average bit budget. It complements prior feature-transform and weight-quantization techniques, delivering consistent gains across Vision-Language Models and Large Language Models with minimal overhead. The approach is training-free and integrates with existing PTQ pipelines, offering practical deployment benefits for large generative models on resource-constrained hardware.
Abstract
Quantization is the key method for reducing inference latency, power and memory footprint of generative AI models. However, accuracy often degrades sharply when activations are quantized below eight bits. Recent work suggests that invertible linear transformations (e.g. rotations) can aid quantization, by reparameterizing feature channels and weights. In this paper, we propose \textit{Sequence Transformation and Mixed Precision} (STaMP) quantization, a novel strategy that applies linear transformations along the \textit{sequence} dimension to exploit the strong local correlation in language and visual data. By keeping a small number of tokens in each intermediate activation at higher precision, we can maintain model accuracy at lower (average) activations bit-widths. We evaluate STaMP on recent LVM and LLM architectures, demonstrating that it significantly improves low bit width activation quantization and complements established activation and weight quantization methods including recent feature transformations.