Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EDGE-LLM: Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning and Voting

Zhongzhi Yu, Zheng Wang, Yuhan Li, Haoran You, Ruijie Gao, Xiaoya Zhou, Sreenidhi Reedy Bommu, Yang Katie Zhao, Yingyan Celine Lin

TL;DR

Edge-LLM tackles the dual bottleneck of compute and memory in on-device LLM tuning by coupling layer-wise unified compression with adaptive layer tuning and a hardware scheduling layer. The LUC component assigns per-layer quantization and pruning policies informed by layer sensitivity, while adaptive layer tuning uses exit-layer connections and voting to reduce backpropagation depth and memory. A hardware scheduler then maps these irregular patterns to edge accelerators, balancing compute, memory, and I/O costs. Across benchmarks, Edge-LLM delivers substantial speedups and memory reductions with competitive task accuracy, enabling practical privacy-preserving on-device adaptation. This combination of algorithmic and hardware co-design charts a viable path for deploying adaptive LLMs on resource-constrained edge devices.

Abstract

Efficient adaption of large language models (LLMs) on edge devices is essential for applications requiring continuous and privacy-preserving adaptation and inference. However, existing tuning techniques fall short because of the high computation and memory overheads. To this end, we introduce a computation- and memory-efficient LLM tuning framework, called Edge-LLM, to facilitate affordable and effective LLM adaptation on edge devices. Specifically, Edge-LLM features three core components: (1) a layer-wise unified compression (LUC) technique to reduce the computation overhead by generating layer-wise pruning sparsity and quantization bit-width policies, (2) an adaptive layer tuning and voting scheme to reduce the memory overhead by reducing the backpropagation depth, and (3) a complementary hardware scheduling strategy to handle the irregular computation patterns introduced by LUC and adaptive layer tuning, thereby achieving efficient computation and data movements. Extensive experiments demonstrate that Edge-LLM achieves a 2.92x speed up and a 4x memory overhead reduction as compared to vanilla tuning methods with comparable task accuracy. Our code is available at https://github.com/GATECH-EIC/Edge-LLM

EDGE-LLM: Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning and Voting

TL;DR

Edge-LLM tackles the dual bottleneck of compute and memory in on-device LLM tuning by coupling layer-wise unified compression with adaptive layer tuning and a hardware scheduling layer. The LUC component assigns per-layer quantization and pruning policies informed by layer sensitivity, while adaptive layer tuning uses exit-layer connections and voting to reduce backpropagation depth and memory. A hardware scheduler then maps these irregular patterns to edge accelerators, balancing compute, memory, and I/O costs. Across benchmarks, Edge-LLM delivers substantial speedups and memory reductions with competitive task accuracy, enabling practical privacy-preserving on-device adaptation. This combination of algorithmic and hardware co-design charts a viable path for deploying adaptive LLMs on resource-constrained edge devices.

Abstract

Efficient adaption of large language models (LLMs) on edge devices is essential for applications requiring continuous and privacy-preserving adaptation and inference. However, existing tuning techniques fall short because of the high computation and memory overheads. To this end, we introduce a computation- and memory-efficient LLM tuning framework, called Edge-LLM, to facilitate affordable and effective LLM adaptation on edge devices. Specifically, Edge-LLM features three core components: (1) a layer-wise unified compression (LUC) technique to reduce the computation overhead by generating layer-wise pruning sparsity and quantization bit-width policies, (2) an adaptive layer tuning and voting scheme to reduce the memory overhead by reducing the backpropagation depth, and (3) a complementary hardware scheduling strategy to handle the irregular computation patterns introduced by LUC and adaptive layer tuning, thereby achieving efficient computation and data movements. Extensive experiments demonstrate that Edge-LLM achieves a 2.92x speed up and a 4x memory overhead reduction as compared to vanilla tuning methods with comparable task accuracy. Our code is available at https://github.com/GATECH-EIC/Edge-LLM
Paper Structure (19 sections, 3 equations, 4 figures, 2 tables)

This paper contains 19 sections, 3 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Efficient Tuning Techniques
  4. Memory Overhead During Tuning
  5. Opportunities for Efficient LLM Tuning
  6. Edge-LLM Algorithm
  7. Overview
  8. Layer-wise Unified Compression (LUC)
  9. Adaptive Layer Tuning and Voting
  10. Edge-LLM Hardware Scheduling
  11. Overview
  12. Searching Objective
  13. Block Search Space
  14. Cost Models
  15. Evaluation
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Profiling results on the memory footprint when tuning LLaMA-7B with LoRA hu2021lora and QLoRA dettmers2023qlora on the Alpaca taori2023stanford dataset.
  • Figure 2: Comparison between (a) the compressing-then-tuning baseline and (b/c) our proposed Edge-LLM method.
  • Figure 3: Visualization of LLaMA-7B's layer-wise sensitivity to (a) quantization and (b) pruning.
  • Figure 4: The overview of our hardware scheduling.