Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU

Ruijia Yang, Zeyi Wen

Abstract

Fine-tuning Large Language Models (LLMs) has become essential for domain adaptation, but its memory-intensive property exceeds the capabilities of most GPUs. To address this challenge and democratize LLM fine-tuning, we present SlideFormer, a novel system designed for single-GPU environments. Our innovations are: (1) A lightweight asynchronous engine that treats the GPU as a sliding window and overlaps GPU computation with CPU updates and multi-tier I/O. (2) A highly efficient heterogeneous memory management scheme significantly reduces peak memory usage. (3) Optimized Triton kernels to solve key bottlenecks and integrated advanced I/O. This collaborative design enables fine-tuning of the latest 123B+ models on a single RTX 4090, supporting up to 8x larger batch sizes and 6x larger models. In evaluations, SlideFormer achieves 1.40x to 6.27x higher throughput while roughly halving CPU/GPU memory usage compared to baselines, sustaining >95% peak performance on both NVIDIA and AMD GPUs.

An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU

Abstract

Fine-tuning Large Language Models (LLMs) has become essential for domain adaptation, but its memory-intensive property exceeds the capabilities of most GPUs. To address this challenge and democratize LLM fine-tuning, we present SlideFormer, a novel system designed for single-GPU environments. Our innovations are: (1) A lightweight asynchronous engine that treats the GPU as a sliding window and overlaps GPU computation with CPU updates and multi-tier I/O. (2) A highly efficient heterogeneous memory management scheme significantly reduces peak memory usage. (3) Optimized Triton kernels to solve key bottlenecks and integrated advanced I/O. This collaborative design enables fine-tuning of the latest 123B+ models on a single RTX 4090, supporting up to 8x larger batch sizes and 6x larger models. In evaluations, SlideFormer achieves 1.40x to 6.27x higher throughput while roughly halving CPU/GPU memory usage compared to baselines, sustaining >95% peak performance on both NVIDIA and AMD GPUs.
Paper Structure (17 sections, 1 equation, 13 figures, 1 table)

This paper contains 17 sections, 1 equation, 13 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Memory Challenges in LLM Fine-Tuning
  4. Existing Offloading Techniques
  5. Limitations of Existing Solutions
  6. System Design
  7. The Layer-Sliding Architecture
  8. Efficient Heterogeneous Memory Co-Design
  9. Integrated I/O and Compute Co-Design
  10. Evaluation
  11. Experimental Setup
  12. Throughput Scalability
  13. Heterogeneous Memory Usage
  14. Analysis of NVMe Offloading
  15. Maximum Trainable Model Size
  16. ...and 2 more sections

Figures (13)

  • Figure 1: The widening gap between CPU and GPU memory.
  • Figure 2: Overview of SlideFormer.
  • Figure 3: Backward overlaps with parameter updates.
  • Figure 4: Critical batch size for achieving full backward overlap with updates ($T_{bwd} \geq T_{grad\_d2h} + T_{update}$).
  • Figure 5: Computation-communication overlap during backward propagation in GPU-CPU tier pipeline.
  • ...and 8 more figures