Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

INT2.1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptation

Yuji Chai, John Gkountouras, Glenn G. Ko, David Brooks, Gu-Yeon Wei

TL;DR

The paper tackles the challenge of memory-intensive fine-tuning of large language models by introducing Extremely Memory-Efficient Finetuning (EMEF) that leverages Low-Rank Adaptation (LoRA) on quantized models to drastically reduce VRAM requirements. It further proposes Low-Rank Error Correction (LREC), a quantization-agnostic framework that uses additional LoRA parameters to mitigate degradation from quantization, enabling fully functional INT2/INT4 LLMs and extending applicability to INT3 and INT8. The authors validate their approach on LLaMA-7B with INT4 and INT2 configurations, reporting substantial memory savings (e.g., ~4.93–5.96 GB VRAM on consumer GPUs) while maintaining competitive downstream performance and enabling instruction-tuning with Alpaca. Qualitative and ablation analyses demonstrate the effectiveness of LREC and EMEF in preserving text quality and stability, despite some limitations in speed and optimization. Overall, the work advances accessible fine-tuning of quantized LLMs on low-resource hardware and lays groundwork for broader adoption of ultra-low-bit strategies with error correction.

Abstract

We introduce a method that dramatically reduces fine-tuning VRAM requirements and rectifies quantization errors in quantized Large Language Models. First, we develop an extremely memory-efficient fine-tuning (EMEF) method for quantized models using Low-Rank Adaptation (LoRA), and drawing upon it, we construct an error-correcting algorithm designed to minimize errors induced by the quantization process. Our method reduces the memory requirements by up to 5.6 times, which enables fine-tuning a 7 billion parameter Large Language Model (LLM) on consumer laptops. At the same time, we propose a Low-Rank Error Correction (LREC) method that exploits the added LoRA layers to ameliorate the gap between the quantized model and its float point counterpart. Our error correction framework leads to a fully functional INT2 quantized LLM with the capacity to generate coherent English text. To the best of our knowledge, this is the first INT2 Large Language Model that has been able to reach such a performance. The overhead of our method is merely a 1.05 times increase in model size, which translates to an effective precision of INT2.1. Also, our method readily generalizes to other quantization standards, such as INT3, INT4, and INT8, restoring their lost performance, which marks a significant milestone in the field of model quantization. The strategies delineated in this paper hold promising implications for the future development and optimization of quantized models, marking a pivotal shift in the landscape of low-resource machine learning computations.

INT2.1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptation

TL;DR

The paper tackles the challenge of memory-intensive fine-tuning of large language models by introducing Extremely Memory-Efficient Finetuning (EMEF) that leverages Low-Rank Adaptation (LoRA) on quantized models to drastically reduce VRAM requirements. It further proposes Low-Rank Error Correction (LREC), a quantization-agnostic framework that uses additional LoRA parameters to mitigate degradation from quantization, enabling fully functional INT2/INT4 LLMs and extending applicability to INT3 and INT8. The authors validate their approach on LLaMA-7B with INT4 and INT2 configurations, reporting substantial memory savings (e.g., ~4.93–5.96 GB VRAM on consumer GPUs) while maintaining competitive downstream performance and enabling instruction-tuning with Alpaca. Qualitative and ablation analyses demonstrate the effectiveness of LREC and EMEF in preserving text quality and stability, despite some limitations in speed and optimization. Overall, the work advances accessible fine-tuning of quantized LLMs on low-resource hardware and lays groundwork for broader adoption of ultra-low-bit strategies with error correction.

Abstract

We introduce a method that dramatically reduces fine-tuning VRAM requirements and rectifies quantization errors in quantized Large Language Models. First, we develop an extremely memory-efficient fine-tuning (EMEF) method for quantized models using Low-Rank Adaptation (LoRA), and drawing upon it, we construct an error-correcting algorithm designed to minimize errors induced by the quantization process. Our method reduces the memory requirements by up to 5.6 times, which enables fine-tuning a 7 billion parameter Large Language Model (LLM) on consumer laptops. At the same time, we propose a Low-Rank Error Correction (LREC) method that exploits the added LoRA layers to ameliorate the gap between the quantized model and its float point counterpart. Our error correction framework leads to a fully functional INT2 quantized LLM with the capacity to generate coherent English text. To the best of our knowledge, this is the first INT2 Large Language Model that has been able to reach such a performance. The overhead of our method is merely a 1.05 times increase in model size, which translates to an effective precision of INT2.1. Also, our method readily generalizes to other quantization standards, such as INT3, INT4, and INT8, restoring their lost performance, which marks a significant milestone in the field of model quantization. The strategies delineated in this paper hold promising implications for the future development and optimization of quantized models, marking a pivotal shift in the landscape of low-resource machine learning computations.
Paper Structure (48 sections, 3 equations, 3 figures, 6 tables)

This paper contains 48 sections, 3 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Large Language Models
  4. Quantization for Deep Networks
  5. Parameter Efficient Fine-tuning
  6. Instruction Fine-tuning
  7. Methodology
  8. Extremely-Memory Efficient Finetuning (EMEF)
  9. Implementation Details
  10. Optimization
  11. Experimental Validation
  12. Experimental Setup
  13. INT4 Extremely Memory-Efficient Finetuning (EMEF)
  14. Memory Utilization and Hardware
  15. Low-Rank Error Correction (LREC)
  16. ...and 33 more sections

Figures (3)

  • Figure 1: Flowchart of the forward and backward path, designed for a quantized model with injected LoRA parameters. In the diagram, the yellow blocks represent matrices with quantized parameters, while the green blocks represent matrices with floating point parameters. MM stands for matrix multiplication.
  • Figure 2: Perplexity comparison of quantized $7$B LLaMA models with and without using our Low-Rank Error Correction method at different parameter precisions ranging from INT2 to INT4.
  • Figure 3: Example responses from three different versions of instruction fine-tuned LLaMA $7$B model. The blue text is the prompt. The gray ellipsis [...] indicates that the response was trimmed to fit this page, but the actual response is longer.