ITERA-LLM: Boosting Sub-8-Bit Large Language Model Inference via Iterative Tensor Decomposition

TL;DR

ITERA-LLM tackles the challenge of deploying sub-8-bit LLM inference on FPGA by combining post-training quantization with SVD-based iterative tensor decomposition and a sensitivity-based rank allocation strategy. It factorizes weight matrices into low-rank components and applies iterative refinement with quantized factors to mitigate quantization error, while a hardware-aware Design Space Exploration selects Pareto-optimal points mapped to MatMul engines. The framework demonstrates up to a 41.1% per-layer latency reduction at W4A8 with comparable accuracy and up to 4.9% BLEU gains over quantization baselines, validating a practical sub-8-bit compression pathway for latency-sensitive inference on FPGA. These results on OPUS-MT models deployed to a ZCU111 platform illustrate a viable co-design route for efficient LLM deployment in resource-constrained environments.

Abstract

Recent advancements in Large Language Models (LLMs) have demonstrated impressive capabilities as their scale expands to billions of parameters. Deploying these large-scale models on resource-constrained platforms presents significant challenges, with post-training fixed-point quantization often used as a model compression technique. However, quantization-only methods typically lead to significant accuracy degradation in LLMs when precision falls below 8 bits. This paper addresses this challenge through a software-hardware co-design framework, ITERA-LLM, which integrates sub-8-bit quantization with SVD-based iterative low-rank tensor decomposition for error compensation, leading to higher compression ratios and reduced computational complexity. The proposed approach is complemented by a hardware-aware Design Space Exploration (DSE) process that optimizes accuracy, latency, and resource utilization, tailoring the configuration to the specific requirements of the targeted LLM. Our results show that ITERA-LLM achieves linear layer latency reduction of up to 41.1%, compared to quantization-only baseline approach while maintaining similar model accuracy.

Figures (12)

• Figure 1: Post-training quantization results for an OPUS-MT model tiedemann2023democratizing evaluated on a language translation dataset. The results demonstrate the reduction in the BLEU Score (a metric for translation accuracy) as model precision decreases, with the IEEE Single Precision (FP32) baseline serving as a reference. At the extreme quantization of W4A8, the BLEU Score is reduced by 2.22 (5.37% compared to the FP32 baseline).
• Figure 2: The overview of proposed ITERA-LLM framework. This work focuses on the topic of post-training compression and FPGA accelerator co-design for producing Pareto-optimal design points on the accuracy-latency frontier.
• Figure 3: The proposed iterative tensor decomposition algorithm with SVD and quantization in a closed loop
• Figure 4: The sensitivity analysis measures the reduction in BLEU Score when varying the percentage of rank retained in each layer. Each layer's weight matrices are truncated to the specified rank percentage, temporarily replacing the original matrices in the model while keeping other layers unchanged.
• Figure 5: Dataflow and parallelism scheme for a target dense matrix-matrix multiplication layer.