Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An empirical study of LLaMA3 quantization: from LLMs to MLLMs

Wei Huang, Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, Michele Magno

TL;DR

This study empirically evaluates LLaMA3 under various low-bit quantization schemes across three tracks: PTQ for LLMs, LoRA-FineTuning Quantization, and PTQ for LLaMA3-based MLLMs. It demonstrates that while several PTQ methods can preserve performance down to 4 bits, ultra-low bit widths (notably 2-bit) cause substantial degradation, especially in MLLMs, with 3-bit results often intermediate and highly method-dependent. LoRA-FT quantization on LLaMA3-8B generally fails to fully compensate quantization losses and can even worsen performance, though certain 4-bit configurations (e.g., QLoRA and IR-QLoRA) offer strong results with notable memory savings. Across multi-modal settings (LLaVA-Next-8B), language abilities degrade after vision fine-tuning, and 4-bit quantization remains the most viable option, while 2-bit quantization collapses. The work provides a benchmark, comprehensive methodology, and public resources to guide future low-bit quantization research for LLMs and MLLMs.

Abstract

The LLaMA family, a collection of foundation language models ranging from 7B to 65B parameters, has become one of the most powerful open-source large language models (LLMs) and the popular LLM backbone of multi-modal large language models (MLLMs), widely used in computer vision and natural language understanding tasks. In particular, LLaMA3 models have recently been released and have achieved impressive performance in various domains with super-large scale pre-training on over 15T tokens of data. Given the wide application of low-bit quantization for LLMs in resource-constrained scenarios, we explore LLaMA3's capabilities when quantized to low bit-width. This exploration can potentially provide new insights and challenges for the low-bit quantization of LLaMA3 and other future LLMs, especially in addressing performance degradation issues that suffer in LLM compression. Specifically, we comprehensively evaluate the 10 existing post-training quantization and LoRA fine-tuning (LoRA-FT) methods of LLaMA3 on 1-8 bits and various datasets to reveal the low-bit quantization performance of LLaMA3. To uncover the capabilities of low-bit quantized MLLM, we assessed the performance of the LLaMA3-based LLaVA-Next-8B model under 2-4 ultra-low bits with post-training quantization methods. Our experimental results indicate that LLaMA3 still suffers from non-negligible degradation in linguistic and visual contexts, particularly under ultra-low bit widths. This highlights the significant performance gap at low bit-width that needs to be addressed in future developments. We expect that this empirical study will prove valuable in advancing future models, driving LLMs and MLLMs to achieve higher accuracy at lower bit to enhance practicality. Our project is released on https://github.com/Macaronlin/LLaMA3-Quantization , and quantized models are released at https://huggingface.co/Efficient-ML .

An empirical study of LLaMA3 quantization: from LLMs to MLLMs

TL;DR

This study empirically evaluates LLaMA3 under various low-bit quantization schemes across three tracks: PTQ for LLMs, LoRA-FineTuning Quantization, and PTQ for LLaMA3-based MLLMs. It demonstrates that while several PTQ methods can preserve performance down to 4 bits, ultra-low bit widths (notably 2-bit) cause substantial degradation, especially in MLLMs, with 3-bit results often intermediate and highly method-dependent. LoRA-FT quantization on LLaMA3-8B generally fails to fully compensate quantization losses and can even worsen performance, though certain 4-bit configurations (e.g., QLoRA and IR-QLoRA) offer strong results with notable memory savings. Across multi-modal settings (LLaVA-Next-8B), language abilities degrade after vision fine-tuning, and 4-bit quantization remains the most viable option, while 2-bit quantization collapses. The work provides a benchmark, comprehensive methodology, and public resources to guide future low-bit quantization research for LLMs and MLLMs.

Abstract

The LLaMA family, a collection of foundation language models ranging from 7B to 65B parameters, has become one of the most powerful open-source large language models (LLMs) and the popular LLM backbone of multi-modal large language models (MLLMs), widely used in computer vision and natural language understanding tasks. In particular, LLaMA3 models have recently been released and have achieved impressive performance in various domains with super-large scale pre-training on over 15T tokens of data. Given the wide application of low-bit quantization for LLMs in resource-constrained scenarios, we explore LLaMA3's capabilities when quantized to low bit-width. This exploration can potentially provide new insights and challenges for the low-bit quantization of LLaMA3 and other future LLMs, especially in addressing performance degradation issues that suffer in LLM compression. Specifically, we comprehensively evaluate the 10 existing post-training quantization and LoRA fine-tuning (LoRA-FT) methods of LLaMA3 on 1-8 bits and various datasets to reveal the low-bit quantization performance of LLaMA3. To uncover the capabilities of low-bit quantized MLLM, we assessed the performance of the LLaMA3-based LLaVA-Next-8B model under 2-4 ultra-low bits with post-training quantization methods. Our experimental results indicate that LLaMA3 still suffers from non-negligible degradation in linguistic and visual contexts, particularly under ultra-low bit widths. This highlights the significant performance gap at low bit-width that needs to be addressed in future developments. We expect that this empirical study will prove valuable in advancing future models, driving LLMs and MLLMs to achieve higher accuracy at lower bit to enhance practicality. Our project is released on https://github.com/Macaronlin/LLaMA3-Quantization , and quantized models are released at https://huggingface.co/Efficient-ML .
Paper Structure (10 sections, 2 equations, 6 figures, 12 tables)

This paper contains 10 sections, 2 equations, 6 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Quantization methods
  3. Evaluation datasets
  4. Track1: Post-Training Quantization
  5. PTQ methods
  6. Track2: LoRA-FineTuning Quantization
  7. Quantization Framework
  8. LoRA-FT Methods
  9. Track3: Multi-modal Large Language Model Quantization
  10. Conclusion

Figures (6)

  • Figure 1: The overview of our empirical study
  • Figure 2: The VQA results of LLaVA-Next-8B for different quantization bit widths (1/5)
  • Figure 3: The VQA results of LLaVA-Next-8B for different quantization bit widths (2/5)
  • Figure 4: The VQA results of LLaVA-Next-8B for different quantization bit widths (3/5)
  • Figure 5: The VQA results of LLaVA-Next-8B for different quantization bit widths (4/5)
  • ...and 1 more figures