Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Calibrating Beyond English: Language Diversity for Better Quantized Multilingual LLM

Everlyn Asiko Chimoto, Mostafa Elhoushi, Bruce A. Bassett

TL;DR

This paper addresses how calibration data language affects post-training quantization of multilingual LLMs. It systematically evaluates eight calibration settings across two PTQ methods (GPTQ and AWQ) on two model families (Llama3.1 8B and Qwen2.5 7B) over 10 languages, revealing that non-English and multilingual calibration sets consistently reduce perplexity compared with English-only baselines, with gains up to $3.52$ perplexity points. The authors find that aligning calibration data with the evaluation language yields the largest per-language improvements, while the quantizer mechanics shape the magnitude of gains (GPTQ being more sensitive to calibration language, AWQ more robust due to activation-based scaling). These findings suggest that calibration data should be linguistically diverse and tailored to the deployment scenario to robustly quantize multilingual LLMs, offering practical deployment guidelines and highlighting areas for future work in broader language coverage and downstream tasks.

Abstract

Quantization is an effective technique for reducing the storage footprint and computational costs of Large Language Models (LLMs), but it often results in performance degradation. Existing post-training quantization methods typically use small, English-only calibration sets; however, their impact on multilingual models remains underexplored. We systematically evaluate eight calibration settings (five single-language and three multilingual mixes) on two quantizers (GPTQ, AWQ) on data from 10 languages. Our findings reveal a consistent trend: non-English and multilingual calibration sets significantly improve perplexity compared to English-only baselines. Specifically, we observe notable average perplexity gains across both quantizers on Llama3.1 8B and Qwen2.5 7B, with multilingual mixes achieving the largest overall reductions of up to 3.52 points in perplexity. Furthermore, our analysis indicates that tailoring calibration sets to the evaluation language yields the largest improvements for individual languages, underscoring the importance of linguistic alignment. We also identify specific failure cases where certain language-quantizer combinations degrade performance, which we trace to differences in activation range distributions across languages. These results highlight that static one-size-fits-all calibration is suboptimal and that tailoring calibration data, both in language and diversity, plays a crucial role in robustly quantizing multilingual LLMs.

Calibrating Beyond English: Language Diversity for Better Quantized Multilingual LLM

TL;DR

This paper addresses how calibration data language affects post-training quantization of multilingual LLMs. It systematically evaluates eight calibration settings across two PTQ methods (GPTQ and AWQ) on two model families (Llama3.1 8B and Qwen2.5 7B) over 10 languages, revealing that non-English and multilingual calibration sets consistently reduce perplexity compared with English-only baselines, with gains up to perplexity points. The authors find that aligning calibration data with the evaluation language yields the largest per-language improvements, while the quantizer mechanics shape the magnitude of gains (GPTQ being more sensitive to calibration language, AWQ more robust due to activation-based scaling). These findings suggest that calibration data should be linguistically diverse and tailored to the deployment scenario to robustly quantize multilingual LLMs, offering practical deployment guidelines and highlighting areas for future work in broader language coverage and downstream tasks.

Abstract

Quantization is an effective technique for reducing the storage footprint and computational costs of Large Language Models (LLMs), but it often results in performance degradation. Existing post-training quantization methods typically use small, English-only calibration sets; however, their impact on multilingual models remains underexplored. We systematically evaluate eight calibration settings (five single-language and three multilingual mixes) on two quantizers (GPTQ, AWQ) on data from 10 languages. Our findings reveal a consistent trend: non-English and multilingual calibration sets significantly improve perplexity compared to English-only baselines. Specifically, we observe notable average perplexity gains across both quantizers on Llama3.1 8B and Qwen2.5 7B, with multilingual mixes achieving the largest overall reductions of up to 3.52 points in perplexity. Furthermore, our analysis indicates that tailoring calibration sets to the evaluation language yields the largest improvements for individual languages, underscoring the importance of linguistic alignment. We also identify specific failure cases where certain language-quantizer combinations degrade performance, which we trace to differences in activation range distributions across languages. These results highlight that static one-size-fits-all calibration is suboptimal and that tailoring calibration data, both in language and diversity, plays a crucial role in robustly quantizing multilingual LLMs.
Paper Structure (23 sections, 3 equations, 9 figures, 14 tables)

This paper contains 23 sections, 3 equations, 9 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Background: Calibration Sets for Quantization
  3. GPTQ
  4. AWQ
  5. Comparison Studies
  6. Experimental Set-up
  7. Models
  8. Quantization Technique
  9. Calibration Data Sources and Preparation
  10. Evaluation Datasets and Languages
  11. Results and Analysis
  12. Conclusion and Future Work
  13. Appendix
  14. Cross-Model Check on Wikipedia (Qwen2.5 7B)
  15. Additional Model Family: BLOOM
  16. ...and 8 more sections

Figures (9)

  • Figure 1: Average perplexity on 10 languages for Llama3.1 8B. Multilingual calibration achieves the lowest perplexity (14.64), illustrating that calibration language affects quantization quality.
  • Figure 2: $\Delta$ Perplexity (higher = better) on the Wikipedia multilingual test set (English, French, Swahili, Xhosa, Chinese, Sesotho, Yoruba, Zulu, Hausa, Igbo), relative to an English-only calibration baseline. Calibration with non-English languages lead to better perplexity than English-only calibration. Multilingual variants(refer to \ref{['sec:data']} for details) provide the largest gains, showing that a linguistically diverse calibration set can outperform an English-only baseline.
  • Figure 3: Comparison of calibration‐set statistics across different languages. (a) The total unique tokens in each calibration set; (b) The average token count per example. Multilingual sets exhibit both larger vocabularies and longer examples, indicating broader coverage of token contexts. This richer distribution correlates with the improved performance shown in \ref{['fig:Delta']}.
  • Figure 4: Heatmaps of maximum activations after AWQ quantization of Llama3.1 8B for (a) the MLP gate projection and (b) the attention query projection in layer 31 (selected for its large quantization errors; see \ref{['sec:appendix']}). Rows denote different calibration language variants. Across languages, the same salient channels dominate, but their peak magnitudes shift—showing that AWQ rescales fixed channels rather than changing which channels matter, which helps explain the modest perplexity deltas.
  • Figure 5: Activation distributions from the unquantized Llama model across different calibration sets. Violin plots compare absolute activations per set: the three multilingual variants (multimix, multi, multi10) exhibit longer upper tails than single-language sets (en, fr, sw, zh, xh), capturing higher-magnitude outliers. This broader coverage suggests that multilingual calibration is better suited to handle unseen extremes at test time.
  • ...and 4 more figures