Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct

Uygar Kurt

TL;DR

The paper tackles the challenge of selecting an efficient llama.cpp quantization scheme for a modern instruction-tuned model by conducting a unified, model-specific evaluation of 13 GGUF formats on Llama-3.1-8B-Instruct. Using a fixed FP16 baseline and a controlled hardware setup, it measures downstream task performance across GSM8K, HellaSwag, IFEval, MMLU, and TruthfulQA, along with WikiText-2 perplexity and inference throughput. Key findings show that mid-bit quantizations (notably 4–5 bits) often preserve accuracy while delivering substantial size reductions and speedups, whereas 3-bit schemes can cause large degradations on reasoning tasks; a Pareto frontier highlights Q5_0 as a leading default and Q4_K_S as a solid balanced option. The work demonstrates that the specific quantization format materially impacts task-specific behavior beyond what perplexity alone predicts, offering practical guidance for deployment and a template for future, model-wide evaluation. Overall, the study provides actionable recommendations for memory- and compute-constrained deployments, with 5-bit options emerging as particularly robust for a broad mix of tasks, and 3-bit options reserved for the most memory-limited scenarios.

Abstract

Quantization is a practical technique for making large language models easier to deploy by reducing the precision used to store and operate on model weights. This can lower memory use and improve runtime feasibility on constrained hardware, which is especially relevant for users running models locally. Quantization in llama.cpp enables large language models to run on commodity hardware, but available formats are often evaluated inconsistently, making it hard to choose among schemes. We present a unified empirical study of the llama.cpp quantization on a single modern model, Llama-3.1-8B-Instruct (FP16, GGUF), covering 3-8 bit K-quant and legacy formats. We evaluate downstream task performance across standard reasoning, knowledge, instruction-following, and truthfulness benchmarks, and also measure perplexity and CPU throughput (prefill/decoding) alongside model size, compression, and quantization time. Ultimately, this work is a practical guide for choosing a llama.cpp quantization scheme, helping readers make informed, context-aware decisions for their intended use and resource budget.

Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct

TL;DR

The paper tackles the challenge of selecting an efficient llama.cpp quantization scheme for a modern instruction-tuned model by conducting a unified, model-specific evaluation of 13 GGUF formats on Llama-3.1-8B-Instruct. Using a fixed FP16 baseline and a controlled hardware setup, it measures downstream task performance across GSM8K, HellaSwag, IFEval, MMLU, and TruthfulQA, along with WikiText-2 perplexity and inference throughput. Key findings show that mid-bit quantizations (notably 4–5 bits) often preserve accuracy while delivering substantial size reductions and speedups, whereas 3-bit schemes can cause large degradations on reasoning tasks; a Pareto frontier highlights Q5_0 as a leading default and Q4_K_S as a solid balanced option. The work demonstrates that the specific quantization format materially impacts task-specific behavior beyond what perplexity alone predicts, offering practical guidance for deployment and a template for future, model-wide evaluation. Overall, the study provides actionable recommendations for memory- and compute-constrained deployments, with 5-bit options emerging as particularly robust for a broad mix of tasks, and 3-bit options reserved for the most memory-limited scenarios.

Abstract

Quantization is a practical technique for making large language models easier to deploy by reducing the precision used to store and operate on model weights. This can lower memory use and improve runtime feasibility on constrained hardware, which is especially relevant for users running models locally. Quantization in llama.cpp enables large language models to run on commodity hardware, but available formats are often evaluated inconsistently, making it hard to choose among schemes. We present a unified empirical study of the llama.cpp quantization on a single modern model, Llama-3.1-8B-Instruct (FP16, GGUF), covering 3-8 bit K-quant and legacy formats. We evaluate downstream task performance across standard reasoning, knowledge, instruction-following, and truthfulness benchmarks, and also measure perplexity and CPU throughput (prefill/decoding) alongside model size, compression, and quantization time. Ultimately, this work is a practical guide for choosing a llama.cpp quantization scheme, helping readers make informed, context-aware decisions for their intended use and resource budget.
Paper Structure (20 sections, 3 equations, 1 figure, 4 tables)

This paper contains 20 sections, 3 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Post Training Quantization of Large Language Models
  4. llama.cpp and Its Quantization Schemes
  5. Standard integer formats.
  6. K-quant formats.
  7. Existing Quantization Benchmarks
  8. Experimental Setup
  9. Base Model & Environment
  10. Quantization Configurations
  11. Evaluation Tasks & Datasets
  12. Quantization Results and Trade-offs
  13. Overall Accuracy-Compression behavior
  14. Task Sensitivity Under Quantization
  15. Pareto Frontier Analysis: Size Reduction and Performance Tradeoff
  16. ...and 5 more sections

Figures (1)

  • Figure 1: Compression vs benchmark quality loss. AvgLoss(%) $=\frac{\mathrm{Avg}_{\text{F16}}-\mathrm{Avg}}{\mathrm{Avg}_{\text{F16}}}\times 100$.