Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Trustworthiness with Mixed Precision: Benchmarks, Opportunities, and Challenges

Guanxi Lu, Hao Mark Chen, Zhiqiang Que, Wayne Luk, Hongxiang Fan

TL;DR

This paper explores how quantization affects the trustworthiness of large language models, revealing instability at low bit-widths across quantization methods. It introduces a precision-ensemble voting approach that combines multiple mixed-precision variants to stabilize predictions and improve trustworthiness metrics by up to $5.8\%$, while preserving multi-domain task performance. The work provides a framework for evaluating trustworthiness under compression, demonstrates method- and metric-dependent effects, and outlines opportunities for joint compression, multi-modal extensions, and algorithm-system-hardware co-design to enable safe, efficient deployment of quantized LLMs.

Abstract

Large language models (LLMs) have shown promising performance across various tasks. However, their autoregressive decoding process poses significant challenges for efficient deployment on existing AI hardware. Quantization alleviates memory and compute pressure by compressing weights, activations, and KV caches to low precisions while preserving generation quality. However, existing quantization frameworks typically focus on perplexity or classification accuracy, often omitting critical trustworthiness metrics. This gap introduces risks when applying quantized LLMs to downstream high-stakes domains such as finance and healthcare. In this work, we systematically investigate the impact of quantization on four trustworthiness metrics (adversarial robustness, fairness, machine ethics, and out-of-distribution robustness) and identify the instability across compression ratios and quantization methods. Building on these observations, we develop a novel precision-ensemble voting approach that leverages predictions from mixed-precision variants of the same model and consistently improves performance by up to $5.8\%$ on trustworthiness metrics. Our results highlight the importance of considering trustworthiness when developing model compression techniques and point to research opportunities at the intersection of compression and trustworthiness for safety-critical applications.

Enhancing Trustworthiness with Mixed Precision: Benchmarks, Opportunities, and Challenges

TL;DR

This paper explores how quantization affects the trustworthiness of large language models, revealing instability at low bit-widths across quantization methods. It introduces a precision-ensemble voting approach that combines multiple mixed-precision variants to stabilize predictions and improve trustworthiness metrics by up to , while preserving multi-domain task performance. The work provides a framework for evaluating trustworthiness under compression, demonstrates method- and metric-dependent effects, and outlines opportunities for joint compression, multi-modal extensions, and algorithm-system-hardware co-design to enable safe, efficient deployment of quantized LLMs.

Abstract

Large language models (LLMs) have shown promising performance across various tasks. However, their autoregressive decoding process poses significant challenges for efficient deployment on existing AI hardware. Quantization alleviates memory and compute pressure by compressing weights, activations, and KV caches to low precisions while preserving generation quality. However, existing quantization frameworks typically focus on perplexity or classification accuracy, often omitting critical trustworthiness metrics. This gap introduces risks when applying quantized LLMs to downstream high-stakes domains such as finance and healthcare. In this work, we systematically investigate the impact of quantization on four trustworthiness metrics (adversarial robustness, fairness, machine ethics, and out-of-distribution robustness) and identify the instability across compression ratios and quantization methods. Building on these observations, we develop a novel precision-ensemble voting approach that leverages predictions from mixed-precision variants of the same model and consistently improves performance by up to on trustworthiness metrics. Our results highlight the importance of considering trustworthiness when developing model compression techniques and point to research opportunities at the intersection of compression and trustworthiness for safety-critical applications.

Paper Structure

This paper contains 33 sections, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Low-Precision LLM Inference
  4. Weight-Only Quantization
  5. Weight-Activation Quantization
  6. KV Cache Quantization
  7. Trustworthiness of Low-Precision LLMs
  8. Benchmarking Trustworthiness of Quantized LLMs
  9. Models and Quantization Frameworks
  10. Evaluation Metrics
  11. Adversarial Robustness
  12. Fairness
  13. Machine Ethics
  14. Out-of-Distribution Robustness
  15. Performance on Multi-domain Tasks and Trustworthiness
  16. ...and 18 more sections

Figures (3)

  • Figure 1: Accuracy and refusal rate for multi-domain tasks (MMLU) and trustworthiness metrics. For MMLU, the refusal rate is zero and therefore omitted. On multi-domain tasks, AWQ and GPTQ match the 13B dense baseline at 8-bit and remain close at 4-bit, with a larger degradation at 3-bit. On trustworthiness metrics, both methods are comparable to the dense baseline at 8-bit; at lower precisions (4- and 3-bit), AWQ is more robust than GPTQ.
  • Figure 2: Workflow of precision-ensemble voting. A dense LLM is quantized to multiple precisions; each quantized model generates its own response. After response filtering, the remaining responses are aggregated via unweighted majority voting.
  • Figure 3: The precision-ensemble voting mechanism maintains MMLU accuracy while consistently improving performance on the trustworthiness benchmarks.