FAQ: Mitigating Quantization Error via Regenerating Calibration Data with Family-Aware Quantization

TL;DR

FAQ tackles PTQ calibration data quality by regenerating calibration samples from a larger in-family model and refining them with chain-of-thought guidance, inter-sample competition, and normalization. This data-centric approach reshapes activation distributions to be more quantization-friendly, enabling standard symmetric quantizers to perform closer to full-precision baselines. Empirically, FAQ yields up to 28.5% reduction in quantization-induced accuracy loss across diverse models (including MoEs) and PTQ baselines, with strong improvements in language modeling, reasoning, multilingual tasks, and specialized math/code benchmarks. The work introduces a practical, plug-in method that strengthens PTQ without retraining, highlighting the value of model family priors for robust, efficient LLM deployment.

Abstract

Although post-training quantization (PTQ) provides an efficient numerical compression scheme for deploying large language models (LLMs) on resource-constrained devices, the representativeness and universality of calibration data remain a core bottleneck in determining the accuracy of quantization parameters. Traditional PTQ methods typically rely on limited samples, making it difficult to capture the activation distribution during the inference phase, leading to biases in quantization parameters. To address this, we propose \textbf{FAQ} (Family-Aware Quantization), a calibration data regeneration framework that leverages prior knowledge from LLMs of the same family to generate high-fidelity calibration samples. Specifically, FAQ first inputs the original calibration samples into a larger LLM from the same family as the target model, regenerating a series of high-fidelity calibration data using a highly consistent knowledge system. Subsequently, this data, carrying Chain-of-Thought reasoning and conforming to the expected activation distribution, undergoes group competition under expert guidance to select the best samples, which are then re-normalized to enhance the effectiveness of standard PTQ. Experiments on multiple model series, including Qwen3-8B, show that FAQ reduces accuracy loss by up to 28.5\% compared to the baseline with original calibration data, demonstrating its powerful potential and contribution.

Figures (5)

• Figure 1: The impact of calibration data on quantization: (a) Traditional PTQ relies on human-provided calibration data, which may not align well with the model's internal activation patterns, leading to suboptimal quantization. (b) FAQ leverages a larger in-family model to generate a 'model-friendly' calibration set, ensuring better alignment and mitigating quantization errors.
• Figure 2: FAQ-enhanced PTQ calibration. Top: standard PTQ workflow. Bottom: zoom-in of the calibration stage. FAQ queries a larger in-family teacher (“elder-sibling”) model (Qwen3-235B-A22B) to regenerate and normalize the original calibration prompts, yielding refined calibration data. The refined set induces smoother activation statistics in the target model (Qwen3-8B), illustrated by fewer extreme peaks in the activations, and improves PTQ results on both GRMC and SDC. GRMC (General Reasoning and Multilingual Capabilities) is the average score over 12 general downstream tasks; SDC (Specialized Domain Capabilities: Math and Code) is the average accuracy over AIME, MATH-500, and LiveCodeBench.
• Figure 3: Overall performance improvement of FAQ across multiple quantization methods and benchmark suites on the Qwen3-8B model. X: average accuracy on general tasks; Y: performance on math/coding tasks. Each point is a quantization method (color = base algorithm, e.g., green for AWQ-INT4; gray vs bold label = baseline vs FAQ). The consistent up-right shift shows FAQ is a plug-and-play enhancement.
• Figure 4: Visualization of activation distributions, demonstrating the outlier suppression effect of FAQ. The subplots (a)-(d) displays the activation value distributions at the input of the self-attention output projection ($O_{proj}$) within the Qwen3-8B model. Each pair compares the activations induced by the baseline calibration data (left) versus our FAQ-generated data (right). The FAQ-generated data consistently produces a smoother activation landscape with substantially suppressed outliers (fewer and shorter red peaks), directly illustrating its effectiveness in creating a more quantization-amenable distribution.
• Figure 5: Why a larger in-family generator yields better calibration for PTQ. We plot the activation landscape of the same quantized target model (Qwen3-8B) under an identical PTQ configuration, measured at the Layer-23 self-attention output projection (O-proj). We vary only the calibration data used for calibration: (a) original seed data, (b) data regenerated by the target model itself (Qwen3-8B; self), and (c) data regenerated by a larger in-family model (Qwen3-235B; FAQ). Compared to (a) and (b), the elder-sibling generator in (c) induces a noticeably smoother landscape with substantially fewer and lower outlier spikes, supporting the choice of a more capable in-family model for calibration regeneration.