Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NestedFP: High-Performance, Memory-Efficient Dual-Precision Floating Point Support for LLMs

Haeun Lee, Omin Kwon, Yeonhong Park, Jae W. Lee

TL;DR

The paper addresses meeting strict LLM inference SLOs under bursty demand by enabling dynamic, memory-efficient dual-precision serving. It introduces NestedFP, which overlays FP8 onto FP16 by decomposing FP16 weights into two 8-bit parts and reconstructing FP16 on-the-fly with a specialized CUTLASS-based GEMM kernel, achieving FP8 throughput without extra memory overhead. The approach yields FP8 model quality comparable to FP8 baselines and FP16 throughput nearly identical to standard FP16 pipelines, while significantly improving SLO attainment during load spikes. This work provides a practical pathway to balance latency, throughput, and quality in production LLM serving, with code available at the provided GitHub link.

Abstract

Meeting service-level objectives (SLOs) in Large Language Models (LLMs) serving is critical, but managing the high variability in load presents a significant challenge. Recent advancements in FP8 inference, backed by native hardware support, offer a potential solution: executing FP16 models by default, while switching to FP8 models during sudden load surges to achieve higher throughput at the cost of a slight quality degradation. Although this approach facilitates effective SLO management, it introduces additional memory overhead due to storing two versions of the same model. In response, this paper proposes NestedFP, an LLM serving technique that supports both FP16 and FP8 models in a memory efficient manner by overlaying FP8 parameters onto FP16 parameters, allowing both models to share the same FP16 memory footprint. By leveraging a compact data format for the overlay and a specialized GEMM kernel optimized for this format, NestedFP ensures minimal degradation in both model quality and inference throughput across both FP8 and FP16 modes. NestedFP provides a flexible platform for dynamic, SLO-aware precision selection. The code is available at https://github.com/SNU-ARC/NestedFP.

NestedFP: High-Performance, Memory-Efficient Dual-Precision Floating Point Support for LLMs

TL;DR

The paper addresses meeting strict LLM inference SLOs under bursty demand by enabling dynamic, memory-efficient dual-precision serving. It introduces NestedFP, which overlays FP8 onto FP16 by decomposing FP16 weights into two 8-bit parts and reconstructing FP16 on-the-fly with a specialized CUTLASS-based GEMM kernel, achieving FP8 throughput without extra memory overhead. The approach yields FP8 model quality comparable to FP8 baselines and FP16 throughput nearly identical to standard FP16 pipelines, while significantly improving SLO attainment during load spikes. This work provides a practical pathway to balance latency, throughput, and quality in production LLM serving, with code available at the provided GitHub link.

Abstract

Meeting service-level objectives (SLOs) in Large Language Models (LLMs) serving is critical, but managing the high variability in load presents a significant challenge. Recent advancements in FP8 inference, backed by native hardware support, offer a potential solution: executing FP16 models by default, while switching to FP8 models during sudden load surges to achieve higher throughput at the cost of a slight quality degradation. Although this approach facilitates effective SLO management, it introduces additional memory overhead due to storing two versions of the same model. In response, this paper proposes NestedFP, an LLM serving technique that supports both FP16 and FP8 models in a memory efficient manner by overlaying FP8 parameters onto FP16 parameters, allowing both models to share the same FP16 memory footprint. By leveraging a compact data format for the overlay and a specialized GEMM kernel optimized for this format, NestedFP ensures minimal degradation in both model quality and inference throughput across both FP8 and FP16 modes. NestedFP provides a flexible platform for dynamic, SLO-aware precision selection. The code is available at https://github.com/SNU-ARC/NestedFP.

Paper Structure

This paper contains 25 sections, 1 equation, 12 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Floating Point Representation
  4. FP8 Quantization for LLMs
  5. Motivation
  6. Challenge: Dynamic Load in LLM Serving
  7. Our Proposal: Dual-Precision LLM Serving
  8. How to Implement Dual-Precision LLM Serving?
  9. NestedFP
  10. Compact yet Effective Data Format for Dual-Precision LLM
  11. FP16 GEMM Kernel with On-the-fly Reconstruction
  12. Evaluation
  13. FP8 Model Quality
  14. FP16 GEMM Kernel Performance
  15. FP16 Mode End-to-End Inference Performance
  16. ...and 10 more sections

Figures (12)

  • Figure 1:
  • Figure 2:
  • Figure 4: FP16 weight is decomposed into 8-bit upper and lower parts. Both parts are used for FP16 mode while only upper part is used for FP8 mode.
  • Figure 5:
  • Figure 8: Refactoring to NestedFP
  • ...and 7 more figures