HALO: Hardware-aware quantization with low critical-path-delay weights for LLM acceleration

TL;DR

HALO tackles the inefficiency of hardware-agnostic quantization for LLMs by embedding circuit-level timing and energy considerations into a post-training quantization framework. It combines sensitivity-aware weight selection, critical-path-delay aware non-uniform quantization, and adaptive DVFS scheduling to create hardware-aware, Pareto-optimal quantized models with tailored DVFS plans. The approach yields substantial practical gains, including up to 353% speedup and significant energy savings across TPUs and GPUs, while maintaining accuracy close to FP16. This work bridges the gap between model compression and hardware-aware optimization, enabling efficient LLM deployment on existing accelerators and guiding future hardware-aware quantization research.

Abstract

Quantization is critical for efficiently deploying large language models (LLMs). Yet conventional methods remain hardware-agnostic, limited to bit-width constraints, and do not account for intrinsic circuit characteristics such as the timing behaviors and energy profiles of Multiply-Accumulate (MAC) units. This disconnect from circuit-level behavior limits the ability to exploit available timing margins and energy-saving opportunities, reducing the overall efficiency of deployment on modern accelerators. To address these limitations, we propose HALO, a versatile framework for Hardware-Aware Post-Training Quantization (PTQ). Unlike traditional methods, HALO explicitly incorporates detailed hardware characteristics, including critical-path timing and power consumption, into its quantization approach. HALO strategically selects weights with low critical-path-delays enabling higher operational frequencies and dynamic frequency scaling without disrupting the architecture's dataflow. Remarkably, HALO achieves these improvements with only a few dynamic voltage and frequency scaling (DVFS) adjustments, ensuring simplicity and practicality in deployment. Additionally, by reducing switching activity within the MAC units, HALO effectively lowers energy consumption. Evaluations on accelerators such as Tensor Processing Units (TPUs) and Graphics Processing Units (GPUs) demonstrate that HALO significantly enhances inference efficiency, achieving average performance improvements of 270% and energy savings of 51% over baseline quantization methods, all with minimal impact on accuracy.

Figures (13)

• Figure 1: HALO quantization framework, using architectural details to yield Pareto-optimal trade-offs for diverse deployments.
• Figure 2: Impact of MAC unit on systolic array efficiency.
• Figure 3: Delay profiles for two weight values. Arrows indicate the maximum delay for each weight across all activations.
• Figure 4: Achievable frequency (GHz) for 8-bit quantized weight values from -128 to 127. Peaks indicate weights with lower critical-path-delays, allowing for higher operating frequencies.
• Figure 5: Power consumption (in Watts) for 8-bit quantized weight values ranging from -128 to 127, where lower values reflect decreased power usage due to reduced switching activity.