Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly

Toshiaki Koike-Akino, Jing Liu, Ye Wang

Abstract

To tackle the huge computational demand of large foundation models, activation-aware compression techniques without retraining have been introduced. However, since these methods highly rely on calibration data, domain shift issues may arise for unseen downstream tasks. We propose a test-time quantization (TTQ) framework which compresses large models on the fly at inference time to resolve this issue. With an efficient online calibration, instant activation-aware quantization can adapt every prompt regardless of the downstream tasks, yet achieving inference speedup. Several experiments demonstrate that TTQ can improve the quantization performance over state-of-the-art baselines.

TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly

Abstract

To tackle the huge computational demand of large foundation models, activation-aware compression techniques without retraining have been introduced. However, since these methods highly rely on calibration data, domain shift issues may arise for unseen downstream tasks. We propose a test-time quantization (TTQ) framework which compresses large models on the fly at inference time to resolve this issue. With an efficient online calibration, instant activation-aware quantization can adapt every prompt regardless of the downstream tasks, yet achieving inference speedup. Several experiments demonstrate that TTQ can improve the quantization performance over state-of-the-art baselines.
Paper Structure (44 sections, 21 equations, 2 figures, 18 tables)

This paper contains 44 sections, 21 equations, 2 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Test-Time Quantization: TTQ
  3. Groupwise Quantization
  4. AWQ: Activation-Aware Quantization
  5. TTQ: Test-Time Quantization with Online AWQ
  6. TTQ with Low-Rank Decomposition
  7. Experiments
  8. Conclusion
  9. Related Work
  10. Model Compression
  11. Dynamic Quantization
  12. Few-Bit Quantization
  13. Activation/Hardware-Aware Quantization
  14. Test-Time Computing
  15. RTN: Groupwise Round-to-Nearest Quantization
  16. ...and 29 more sections

Figures (2)

  • Figure 1: (a) Offline static quantization (e.g., AWQ/GPTQ) requires calibration data, incurs domain shift risk, and cannot be recalibrated after deployment. (b) Our TTQ is online dynamic quantization, with zero offline calibration, and capable of on-device self-calibration at inference time.
  • Figure 2: Histogram of top-5 hyperparameter selections $(\alpha, \lambda, p)$ for different OPT models with quantization bits of $q\in\{2,3,4,5\}$.