Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Models

Juan Pablo Muñoz, Jinjie Yuan, Nilesh Jain

TL;DR

SQFT presents a cohesive, end-to-end framework for low-precision sparse fine-tuning of large pre-trained models, tackling the longstanding challenge of merging adapters with compressed weights without sacrificing sparsity or accuracy. It introduces SparsePEFT and QA-SparsePEFT to enable mergeable PEFT in sparse and quantized regimes, coupled with Neural Low-rank Adapter Search to replace fixed-rank LoRA with elastic, task-adaptive adapters. Through extensive experiments across Llama-3-8B, Mistral-7B-v0.3, and Phi-3-Mini-4K-Instruct on GSM8K, math reasoning, and commonsense tasks, SQFT demonstrates competitive or superior performance to baselines while enabling efficient deployment and inference acceleration. The results highlight the practical impact of mergeable adapters for resource-constrained settings, offering a path toward scalable, efficient adaptation of foundation models with controllable trade-offs between accuracy, speed, and memory.

Abstract

Large pre-trained models (LPMs), such as large language models, have become ubiquitous and are employed in many applications. These models are often adapted to a desired domain or downstream task through a fine-tuning stage. This paper proposes SQFT, an end-to-end solution for low-precision sparse parameter-efficient fine-tuning of LPMs, allowing for effective model manipulation in resource-constrained environments. Additionally, an innovative strategy enables the merging of sparse weights with low-rank adapters without losing sparsity and accuracy, overcoming the limitations of previous approaches. SQFT also addresses the challenge of having quantized weights and adapters with different numerical precisions, enabling merging in the desired numerical format without sacrificing accuracy. Multiple adaptation scenarios, models, and comprehensive sparsity levels demonstrate the effectiveness of SQFT. Models and code are available at https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning.

SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Models

TL;DR

SQFT presents a cohesive, end-to-end framework for low-precision sparse fine-tuning of large pre-trained models, tackling the longstanding challenge of merging adapters with compressed weights without sacrificing sparsity or accuracy. It introduces SparsePEFT and QA-SparsePEFT to enable mergeable PEFT in sparse and quantized regimes, coupled with Neural Low-rank Adapter Search to replace fixed-rank LoRA with elastic, task-adaptive adapters. Through extensive experiments across Llama-3-8B, Mistral-7B-v0.3, and Phi-3-Mini-4K-Instruct on GSM8K, math reasoning, and commonsense tasks, SQFT demonstrates competitive or superior performance to baselines while enabling efficient deployment and inference acceleration. The results highlight the practical impact of mergeable adapters for resource-constrained settings, offering a path toward scalable, efficient adaptation of foundation models with controllable trade-offs between accuracy, speed, and memory.

Abstract

Large pre-trained models (LPMs), such as large language models, have become ubiquitous and are employed in many applications. These models are often adapted to a desired domain or downstream task through a fine-tuning stage. This paper proposes SQFT, an end-to-end solution for low-precision sparse parameter-efficient fine-tuning of LPMs, allowing for effective model manipulation in resource-constrained environments. Additionally, an innovative strategy enables the merging of sparse weights with low-rank adapters without losing sparsity and accuracy, overcoming the limitations of previous approaches. SQFT also addresses the challenge of having quantized weights and adapters with different numerical precisions, enabling merging in the desired numerical format without sacrificing accuracy. Multiple adaptation scenarios, models, and comprehensive sparsity levels demonstrate the effectiveness of SQFT. Models and code are available at https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning.
Paper Structure (30 sections, 4 equations, 5 figures, 10 tables)

This paper contains 30 sections, 4 equations, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Sparsification and Quantization Stage
  4. Fine-tuning with Neural Low-rank Adapter Search (NLS)
  5. SparsePEFT
  6. Quantization-aware SparsePEFT
  7. Model Serving and Inference Acceleration
  8. Experimental Results
  9. Setup
  10. Models
  11. Datasets and Downstream Tasks
  12. Evaluation Settings
  13. Reference Configuration
  14. Main Results
  15. Fine-tuning on GSM8K
  16. ...and 15 more sections

Figures (5)

  • Figure 1: Limitations of existing approaches for fine-tuning sparse and quantized models. Full fine-tuning is expensive. Low-rank adapters (LoRA) for Parameter-efficient Fine-tuning (PEFT) on sparse or quantized models cannot easily merge with the compressed weights due to loss of previously induced sparsity or different numerical precision.
  • Figure 2: SQFT Overview. Several pipeline configurations can be utilized to efficiently fine-tune large models while addressing several limitations of existing approaches.
  • Figure 3: Sparse Parameter-efficient Fine-tuning (SparsePEFT). A binary mask is obtained from the sparsified weights and applied to the adapters, allowing for the later merge without loss of sparsity.
  • Figure 4: The adapter rank distribution of the optimal configurations obtained from the hill-climbing search algorithm (Phi-3-Mini-4K-Instruct with commonsense reasoning).
  • Figure 5: Comparison of various sparsity levels for Llama-3-8B with GSM8K. SQFT achieves similar performance as Shears but with the added benefit of merging adapters with different numerical precision.