Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Preserving Fairness and Safety in Quantized LLMs Through Critical Weight Protection

Muhammad Alif Al Hakim, Alfan Farizki Wicaksono, Fajri Koto

TL;DR

This work investigates how model quantization affects fairness and safety in large language models, emphasizing multilingual and dynamic quantization contexts. It introduces Critical Weight Protection, a mixed-precision approach that preserves fairness- and safety-critical weights in FP16 while quantizing the rest, guided by per-parameter FAIRSCORE and SAFESCORE sensitivities whose sum ranks weights for retention. Through extensive experiments across three model families and five quantization methods on multilingual fairness and safety benchmarks, the study shows that quantization generally degrades alignment, with dynamic quantization offering better stability and non-English settings often more vulnerable. A mitigation strategy based on preserving high-sensitivity weights (AWQ with FAIRSCORE/SAFESCORE) demonstrates improved fairness and safety with minimal impact on general utility, outperforming a SNIP-based baseline in several cases. These results provide practical guidance for trustworthy LLM deployment under compression, highlighting the value of sensitivity-driven mixed-precision quantization to balance efficiency with alignment objectives.

Abstract

Quantization is widely adopted to reduce the computational cost of large language models (LLMs); however, its implications for fairness and safety, particularly in dynamic quantization and multilingual contexts, remain underexplored. In this work, we conduct a systematic study of how static and dynamic quantization methods impact fairness and safety across benchmarks measuring intrinsic and extrinsic bias and safety alignment. For fairness, we evaluate English, French, Dutch, Spanish, and Turkish; for safety, we focus on English, Korean, and Arabic. Our findings reveal that quantization consistently degrades fairness and safety, with dynamic methods demonstrating greater stability than static ones. Moreover, fairness degradation varies across languages, while safety deterioration is especially pronounced in non-English settings. To address these risks, we introduce Critical Weight Protection, a novel technique that identifies and preserves fairness- and safety-critical weights during quantization. This approach effectively mitigates bias and safety deterioration without costly retraining or alignment, maintaining trustworthiness while retaining efficiency.

Preserving Fairness and Safety in Quantized LLMs Through Critical Weight Protection

TL;DR

This work investigates how model quantization affects fairness and safety in large language models, emphasizing multilingual and dynamic quantization contexts. It introduces Critical Weight Protection, a mixed-precision approach that preserves fairness- and safety-critical weights in FP16 while quantizing the rest, guided by per-parameter FAIRSCORE and SAFESCORE sensitivities whose sum ranks weights for retention. Through extensive experiments across three model families and five quantization methods on multilingual fairness and safety benchmarks, the study shows that quantization generally degrades alignment, with dynamic quantization offering better stability and non-English settings often more vulnerable. A mitigation strategy based on preserving high-sensitivity weights (AWQ with FAIRSCORE/SAFESCORE) demonstrates improved fairness and safety with minimal impact on general utility, outperforming a SNIP-based baseline in several cases. These results provide practical guidance for trustworthy LLM deployment under compression, highlighting the value of sensitivity-driven mixed-precision quantization to balance efficiency with alignment objectives.

Abstract

Quantization is widely adopted to reduce the computational cost of large language models (LLMs); however, its implications for fairness and safety, particularly in dynamic quantization and multilingual contexts, remain underexplored. In this work, we conduct a systematic study of how static and dynamic quantization methods impact fairness and safety across benchmarks measuring intrinsic and extrinsic bias and safety alignment. For fairness, we evaluate English, French, Dutch, Spanish, and Turkish; for safety, we focus on English, Korean, and Arabic. Our findings reveal that quantization consistently degrades fairness and safety, with dynamic methods demonstrating greater stability than static ones. Moreover, fairness degradation varies across languages, while safety deterioration is especially pronounced in non-English settings. To address these risks, we introduce Critical Weight Protection, a novel technique that identifies and preserves fairness- and safety-critical weights during quantization. This approach effectively mitigates bias and safety deterioration without costly retraining or alignment, maintaining trustworthiness while retaining efficiency.
Paper Structure (42 sections, 7 equations, 6 figures, 21 tables)

This paper contains 42 sections, 7 equations, 6 figures, 21 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Quantization Effects on Fairness
  4. Quantization and Safety Alignment
  5. Ensuring Fairness and Safety
  6. Fairness and Safety-Aware Quantization with Critical Weight Protection
  7. Fairness.
  8. Safety.
  9. Experimental Setup
  10. Fairness Datasets
  11. Safety Datasets
  12. Models
  13. Quantization.
  14. Results and Analysis
  15. Implications of Quantization on Fairness and Safety
  16. ...and 27 more sections

Figures (6)

  • Figure 1: The proposed mitigation pipeline. (Left) We compute a Criticality Score (S) for the model parameter $\theta$. $I(\theta)$ is computed as the average squared gradient of the respective loss over the dataset. The diagram illustrates the generic calculation for a single aspect (Fairness or Safety). In the full pipeline, we perform this process twice to calculate FAIRSCORE and SAFESCORE. The final score used for ranking is the sum: FAIRSCORE + SAFESCORE. (Middle) Parameters are ranked by this final summed score, with a threshold selecting the top-k% most critical ones. (Right) These critical parameters are preserved in FP16, while the remaining parameters are quantized to INT4.
  • Figure 2: Impact of Quantization on Fairness Relative to Full-Precision. This chart displays the aggregated normalized scores for fairness metrics across different quantization methods. Note: Scores should only be used to compare quantization methods within the same model family.
  • Figure 3: Impact of Quantization on Safety Relative to Full-Precision. This chart displays the aggregated differences in safety evaluation scores, highlighting how quantized models compare to their full-precision counterparts stratified across three LLMs.
  • Figure 4: Average changes in stereotype scores across languages in CrowS-Pair. This chart illustrates the mean change in Stereotype Scores (SS) for English (en) and French (fr) relative to the full-precision model, with error bars indicating standard deviation
  • Figure 5: Average changes in bias score across language in MBBQ. This chart illustrates the mean change in Bias Score for English (en), Spanish (es), Dutch (nl), and Turkish (tr) relative to the full-precision model.
  • ...and 1 more figures