Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models

Michael S. Zhang, Rishi A. Ruia, Arnav Kewalram, Saathvik Dharmapuram, Utkarsh Sharma, Kevin Zhu

TL;DR

<3-5 sentence high-level summary> This paper investigates how quantization precision and replay buffer size interact to affect catastrophic forgetting in continual learning for large language models. Using LLaMA-3.1-8B with LoRA adapters, it compares FP16, INT8, and INT4 across replay buffers during sequential learning on NLU, Math, and Code tasks, revealing that quantization noise can serve as implicit regularization that enhances retention. The key findings show that while FP16 has superior initial task performance, quantized models—especially INT8 and INT4—often surpass FP16 on later tasks with small to moderate replay sizes, and even minimal replay significantly boosts retention. The work provides practical guidelines for deploying compressed models in continual learning and introduces a benchmark framework for studying quantized continual learning in LLMs.

Abstract

Catastrophic forgetting poses a fundamental challenge in continual learning, particularly when models are quantized for deployment efficiency. We systematically investigate the interplay between quantization precision (FP16, INT8, INT4) and replay buffer strategies in large language models, revealing unexpected dynamics. While FP16 achieves superior initial task performance (74.44% on NLU), we observe a striking inversion on subsequent tasks: quantized models outperform FP16 by 8-15% on final task forward accuracy, with INT4 achieving nearly double FP16's performance on Code generation (40% vs 20%). Critically, even minimal replay buffers (0.1%) dramatically improve retention - increasing NLU retention after Math training from 45% to 65% across all precision levels - with INT8 consistently achieving the optimal balance between learning plasticity and knowledge retention. We hypothesize that quantization-induced noise acts as implicit regularization, preventing the overfitting to new task gradients that plagues high-precision models. These findings challenge the conventional wisdom that higher precision is always preferable, suggesting instead that INT8 quantization offers both computational efficiency and superior continual learning dynamics. Our results provide practical guidelines for deploying compressed models in continual learning scenarios: small replay buffers (1-2%) suffice for NLU tasks, while Math and Code benefit from moderate buffers (5-10%), with quantized models requiring less replay than FP16 to achieve comparable retention. Code is available at https://github.com/Festyve/LessIsMore.

When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models

TL;DR

<3-5 sentence high-level summary> This paper investigates how quantization precision and replay buffer size interact to affect catastrophic forgetting in continual learning for large language models. Using LLaMA-3.1-8B with LoRA adapters, it compares FP16, INT8, and INT4 across replay buffers during sequential learning on NLU, Math, and Code tasks, revealing that quantization noise can serve as implicit regularization that enhances retention. The key findings show that while FP16 has superior initial task performance, quantized models—especially INT8 and INT4—often surpass FP16 on later tasks with small to moderate replay sizes, and even minimal replay significantly boosts retention. The work provides practical guidelines for deploying compressed models in continual learning and introduces a benchmark framework for studying quantized continual learning in LLMs.

Abstract

Catastrophic forgetting poses a fundamental challenge in continual learning, particularly when models are quantized for deployment efficiency. We systematically investigate the interplay between quantization precision (FP16, INT8, INT4) and replay buffer strategies in large language models, revealing unexpected dynamics. While FP16 achieves superior initial task performance (74.44% on NLU), we observe a striking inversion on subsequent tasks: quantized models outperform FP16 by 8-15% on final task forward accuracy, with INT4 achieving nearly double FP16's performance on Code generation (40% vs 20%). Critically, even minimal replay buffers (0.1%) dramatically improve retention - increasing NLU retention after Math training from 45% to 65% across all precision levels - with INT8 consistently achieving the optimal balance between learning plasticity and knowledge retention. We hypothesize that quantization-induced noise acts as implicit regularization, preventing the overfitting to new task gradients that plagues high-precision models. These findings challenge the conventional wisdom that higher precision is always preferable, suggesting instead that INT8 quantization offers both computational efficiency and superior continual learning dynamics. Our results provide practical guidelines for deploying compressed models in continual learning scenarios: small replay buffers (1-2%) suffice for NLU tasks, while Math and Code benefit from moderate buffers (5-10%), with quantized models requiring less replay than FP16 to achieve comparable retention. Code is available at https://github.com/Festyve/LessIsMore.

Paper Structure

This paper contains 17 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Methods and Experimental Setup
  3. Model and quantization conditions.
  4. Tasks and datasets.
  5. Continual learning with replay.
  6. Analysis
  7. Baseline degradation.
  8. Amplification under continual learning.
  9. Replay as a stabilizer.
  10. Interpretation.
  11. Related Works
  12. Continual learning.
  13. Replay sampling.
  14. Quantization strategies.
  15. Conclusion and Limitations
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Forward Accuracy. Forward Accuracy. Current task performance across different quantization levels and replay buffer sizes (log scale). NLU performance is unaffected by replay buffers.
  • Figure 2: Backward Accuracy. Retention of previous task performance after training on subsequent tasks. Dashed lines indicate retention metrics. X-axis uses log scale to emphasize critical low-replay region.