Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

F-BFQ: Flexible Block Floating-Point Quantization Accelerator for LLMs

Jude Haris, José Cano

TL;DR

The paper addresses the challenge of running LLM inference on resource-constrained edge devices by leveraging block floating point (BFP) quantization within llama.cpp and introducing a flexible accelerator, F-BFQ, capable of dynamically switching between two BFP variants (Q2_K and Q3_K) for MatMul. The architecture features a Dynamic Super Block Processor (DSBP), a data-parallel vector compute unit, and a driver that orchestrates data loading and operation scheduling via AXI opcodes. Evaluated on an AMD KV260 platform across three LLMs, F-BFQ achieves an average speedup of 1.4x over a NEON-based CPU baseline and reaches up to 5.2 tokens per second (approximately 3.9 words/s). The work demonstrates that hardware-software co-design with variant-flexible BFP acceleration can significantly reduce latency and memory demands for edge LLM inference, with future work aimed at extending support to additional BFP variants and optimizing dataflow to further mitigate bandwidth bottlenecks.

Abstract

Large Language Models (LLMs) have become increasingly prominent for daily tasks, from improving sound-totext translation to generating additional frames for the latest video games. With the help of LLM inference frameworks, such as llama.cpp, which support optimizations such as KV-caching and quantization, it is now easier than ever to deploy LLMs on edge devices. Quantization is fundamental to enable LLMs on resource-constrained edge devices, and llama.cpp utilizes block floating point (BFP) quantization to drastically reduce the bit width of weights and input tensors, the memory footprint, and the computational power required to run LLMs. LLMs are typically quantized with mixed BFP quantization across the model layers to reduce the loss of model accuracy due to quantization. Therefore, to efficiently accelerate across the layers of BFP-quantized LLMs, specialized accelerators need to support different BFP variants without reconfiguration. To address this issue, we propose a Flexible Block FloatingPoint Quantization (F-BFQ) accelerator, which can dynamically switch between two BFP quantization variants and perform matrix multiplication (MatMul) operations. Our initial F-BFQ accelerator design, deployed on the AMD Kria board, reduces inference time by 1.4x on average over the Arm NEON-based CPU execution across three BFP quantized LLMs while achieving 5.2 tokens per second (~3.9 words per second).

F-BFQ: Flexible Block Floating-Point Quantization Accelerator for LLMs

TL;DR

The paper addresses the challenge of running LLM inference on resource-constrained edge devices by leveraging block floating point (BFP) quantization within llama.cpp and introducing a flexible accelerator, F-BFQ, capable of dynamically switching between two BFP variants (Q2_K and Q3_K) for MatMul. The architecture features a Dynamic Super Block Processor (DSBP), a data-parallel vector compute unit, and a driver that orchestrates data loading and operation scheduling via AXI opcodes. Evaluated on an AMD KV260 platform across three LLMs, F-BFQ achieves an average speedup of 1.4x over a NEON-based CPU baseline and reaches up to 5.2 tokens per second (approximately 3.9 words/s). The work demonstrates that hardware-software co-design with variant-flexible BFP acceleration can significantly reduce latency and memory demands for edge LLM inference, with future work aimed at extending support to additional BFP variants and optimizing dataflow to further mitigate bandwidth bottlenecks.

Abstract

Large Language Models (LLMs) have become increasingly prominent for daily tasks, from improving sound-totext translation to generating additional frames for the latest video games. With the help of LLM inference frameworks, such as llama.cpp, which support optimizations such as KV-caching and quantization, it is now easier than ever to deploy LLMs on edge devices. Quantization is fundamental to enable LLMs on resource-constrained edge devices, and llama.cpp utilizes block floating point (BFP) quantization to drastically reduce the bit width of weights and input tensors, the memory footprint, and the computational power required to run LLMs. LLMs are typically quantized with mixed BFP quantization across the model layers to reduce the loss of model accuracy due to quantization. Therefore, to efficiently accelerate across the layers of BFP-quantized LLMs, specialized accelerators need to support different BFP variants without reconfiguration. To address this issue, we propose a Flexible Block FloatingPoint Quantization (F-BFQ) accelerator, which can dynamically switch between two BFP quantization variants and perform matrix multiplication (MatMul) operations. Our initial F-BFQ accelerator design, deployed on the AMD Kria board, reduces inference time by 1.4x on average over the Arm NEON-based CPU execution across three BFP quantized LLMs while achieving 5.2 tokens per second (~3.9 words per second).
Paper Structure (16 sections, 4 figures, 4 tables)

This paper contains 16 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Large Language Models
  4. BFP Quantization in llama.cpp
  5. BFP Acceleration
  6. F-BFQ Accelerator Architecture
  7. Overview
  8. Instruction Decoder
  9. Data Loader
  10. Scheduler
  11. Dynamic Super Block Processor
  12. F-BFQ Driver
  13. Evaluation
  14. Experimental Setup
  15. Results
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Weights quantization distribution percentage breakdown of the three LLMs under study.
  • Figure 2: Q3_K super-block Data Format.
  • Figure 3: Overview of our F-BFQ accelerator design for Q2_K and Q3_K MatMul operations.
  • Figure 4: Detailed view of the Dynamic Super Block Processor.