MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Elastic LLMs

TL;DR

This work proposes a novel Mixture-of-Bits quantization framework that adjusts weight precision for elastic LLM inference based on token sensitivity, and proposes the many-in-one recursive residual quantization that can iteratively reconstruct higher-precision weights and the token-aware router to dynamically select the number of residual bit slices.

Abstract

Changing runtime complexity on cloud and edge devices necessitates elastic large language model (LLM) deployment, where an LLM can be inferred with various quantization precisions based on available computational resources. However, it has been observed that the calibration parameters for quantization are typically linked to specific precisions, which presents challenges during elastic-precision calibration and precision switching at runtime. In this work, we attribute the source of varying calibration parameters to the varying token-level sensitivity caused by a precision-dependent outlier migration phenomenon.Motivated by this observation, we propose \texttt{MoBiQuant}, a novel Mixture-of-Bits quantization framework that adjusts weight precision for elastic LLM inference based on token sensitivity. Specifically, we propose the many-in-one recursive residual quantization that can iteratively reconstruct higher-precision weights and the token-aware router to dynamically select the number of residual bit slices. MoBiQuant enables smooth precision switching while improving generalization for the distribution of token outliers. Experimental results demonstrate that MoBiQuant exhibits strong elasticity, enabling it to match the performance of bit-specific calibrated PTQ on LLaMA3-8B without repeated calibration.

Figures (12)

• Figure 1: Precision generalization performance on LLaMA3-8B. Existing PTQ methods poorly generalize under calibration–inference precision mismatch, e.g., calibrating for 3-bit but inferring at 4-bit (green). Incorporating token-aware bit adjustment partially mitigates this degradation (pink). Our elastic MoBiQuant improves generalization achieving performance parity with the 4-bit calibrated model (orange).
• Figure 2: Per-token quantization error distribution at layer 5 in LLaMA3-8B: quantization outliers are highly non-uniform at each bit-width. Tokens well-fitted at 4-bit can become outliers at 3-bit, while 4-bit outliers may not be the primary error source at 3-bit.
• Figure 3: Overview of MoBiQuant. (a) MoBiQuant enables token-adaptive elastic inference for linear blocks in LLMs, consisting of two components: (b) MoBiSlice, which decomposes FP16 weights into bit slices via recursive quantization, providing multiple precisions; and (c) MoBiRoute, which is trained to route each token to the optimal bit slice under a budget, achieving token-adaptive precision adjustment.
• Figure 4: Elastic-inference performance comparison. MoBiQuant outperforms PTQ baselines across multiple unseen target bit-widths on various LLaMA-family models, maintaining smooth and fine-grained precision scaling even at extremely low 2--3-bit precision range.
• Figure 5: Per-token quantization error distribution for MoBiQuant at layer 5 in LLaMA3-8B: MoBiQuant shows largely reduced outlier migration phenomenon and, hence, improves generalization for elastic precision switching.