Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LCSB: Layer-Cyclic Selective Backpropagation for Memory-Efficient On-Device LLM Fine-Tuning

Juneyoung Park, Eunbeen Yoon, Seongwan Kim. Jaeho Lee

TL;DR

This work proposes Layer-Cyclic Selective Backpropagation (LCSB), which computes gradients for only a subset of layers per step, and interprets LCSB as Block Coordinate Descent on the LoRA parameter space, providing theoretical justification for convergence.

Abstract

Memory-efficient backpropagation (MeBP) has enabled first-order fine-tuning of large language models (LLMs) on mobile devices with less than 1GB memory. However, MeBP requires backward computation through all transformer layers at every step, where weight decompression alone accounts for 32--42% of backward time. We propose Layer-Cyclic Selective Backpropagation (LCSB), which computes gradients for only a subset of layers per step. Our key insight is that residual connections guarantee gradient flow through identity paths, while AdamW momentum provides implicit updates for non-selected layers. We interpret LCSB as Block Coordinate Descent on the LoRA parameter space, providing theoretical justification for convergence. LCSB achieves up to 1.40$\times$ speedup with less than 2\% quality degradation across five models and three tasks. Surprisingly, in 4-bit quantized settings, LCSB exhibits superior stability: a 3B model that completely diverges under full backpropagation converges smoothly with LCSB, suggesting an implicit regularization effect from selective gradient computation.

LCSB: Layer-Cyclic Selective Backpropagation for Memory-Efficient On-Device LLM Fine-Tuning

TL;DR

This work proposes Layer-Cyclic Selective Backpropagation (LCSB), which computes gradients for only a subset of layers per step, and interprets LCSB as Block Coordinate Descent on the LoRA parameter space, providing theoretical justification for convergence.

Abstract

Memory-efficient backpropagation (MeBP) has enabled first-order fine-tuning of large language models (LLMs) on mobile devices with less than 1GB memory. However, MeBP requires backward computation through all transformer layers at every step, where weight decompression alone accounts for 32--42% of backward time. We propose Layer-Cyclic Selective Backpropagation (LCSB), which computes gradients for only a subset of layers per step. Our key insight is that residual connections guarantee gradient flow through identity paths, while AdamW momentum provides implicit updates for non-selected layers. We interpret LCSB as Block Coordinate Descent on the LoRA parameter space, providing theoretical justification for convergence. LCSB achieves up to 1.40 speedup with less than 2\% quality degradation across five models and three tasks. Surprisingly, in 4-bit quantized settings, LCSB exhibits superior stability: a 3B model that completely diverges under full backpropagation converges smoothly with LCSB, suggesting an implicit regularization effect from selective gradient computation.
Paper Structure (38 sections, 2 equations, 1 figure, 17 tables, 2 algorithms)

This paper contains 38 sections, 2 equations, 1 figure, 17 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Memory-Efficient LLM Training.
  4. Selective Layer Update Methods.
  5. Zeroth-Order Optimization.
  6. Method
  7. The MeBP Bottleneck
  8. Selective Backward with Residual Gradient Flow
  9. The LCSB Algorithm
  10. Theoretical Justification: Block Coordinate Descent
  11. Experiments
  12. Experimental Setup
  13. Models and Tasks.
  14. Configuration.
  15. Main Results
  16. ...and 23 more sections

Figures (1)

  • Figure 1: Overview of LCSB. Blue layers (e.g., Layer $i$-2, $i$-1, $i$+2) are selected for full backward computation, while gray layers (e.g., Layer $i$, $i$+1) are detached. Detached layers participate in the forward pass normally but gradients bypass their computation graph via the residual identity path during backward. Non-selected layers still receive implicit parameter updates through AdamW optimizer momentum.