FPQVAR: Floating Point Quantization for Visual Autoregressive Model with FPGA Hardware Co-design

TL;DR

This work tackles the memory and compute bottlenecks of Visual Autoregressive Generation (VAR) by introducing FPQVAR, a post-training floating-point quantization framework with algorithm-hardware co-design. It jointly proposes Dual Format Quantization to handle imbalanced FC2 activations, Group-wise Hadamard Transformation to mitigate time-varying outliers, and a GHT-Aware Learnable Transformation to further reduce quantization error, along with a LUT-based low-bit FP quantizer and a FPGA-based VAR accelerator featuring a two-level pipeline. Empirical results show FPQVAR dramatically improves image quality metrics under low-bit quantization (FID reduction from 10.83 to 3.58 and IS increase to 241.5 on 256×256 at 4-bit) and delivers competitive or near-FP16 performance at 6-bit, plus a throughput of 1.1 image/s on AMD-Xilinx VCK190 with strong energy efficiency gains. The combination of FP-based quantization and hardware co-design enables efficient VAR inference on edge devices, representing a significant step toward practical deployment of high-quality visual autoregressive models.

Abstract

Visual autoregressive (VAR) modeling has marked a paradigm shift in image generation from next-token prediction to next-scale prediction. VAR predicts a set of tokens at each step from coarse to fine scale, leading to better image quality and faster inference speed compared to existing diffusion models. However, the large parameter size and computation cost hinder its deployment on edge devices. To reduce the memory and computation cost, we propose FPQVAR, an efficient post-training floating-point (FP) quantization framework for VAR featuring algorithm and hardware co-design. At the algorithm level, we first identify the challenges of quantizing VAR. To address them, we propose Dual Format Quantization for the highly imbalanced input activation. We further propose Group-wise Hadamard Transformation and GHT-Aware Learnable Transformation to address the time-varying outlier channels. At the hardware level, we design the first low-bit FP quantizer and multiplier with lookup tables on FPGA and propose the first FPGA-based VAR accelerator featuring low-bit FP computation and an elaborate two-level pipeline. Extensive experiments show that compared to the state-of-the-art quantization method, our proposed FPQVAR significantly improves Fréchet Inception Distance (FID) from 10.83 to 3.58, Inception Score (IS) from 175.9 to 241.5 under 4-bit quantization. FPQVAR also significantly improves the performance of 6-bit quantized VAR, bringing it on par with the FP16 model. Our accelerator on AMD-Xilinx VCK190 FPGA achieves a throughput of 1.1 image/s, which is 3.1x higher than the integer-based accelerator. It also demonstrates 3.6x and 2.8x higher energy efficiency compared to the integer-based accelerator and GPU baseline, respectively.

Figures (11)

• Figure 1: VAR model architecture (a,b,c,d), its next-scale prediction (a), and the number of parameters and operations breakdown (e).
• Figure 2: (a) Weight distribution of QKV projection layer in the 9-th block and the corresponding quantization levels for INT4 and FP4-E2M1. (b) Quantization error of the QKV projection weight in each block with different quantization formats.
• Figure 3: Activation distributions of the FC1 and FC2 layer and the corresponding quantization levels. Standard FP quantization and AFPQ zhang2023afpq can not fit in FC2 input activation distribution.
• Figure 4: Outlier channels (marked by red triangles) in VAR vary with the progression of time steps. Note that the number of tokens in the figure is doubled considering Classifier-Free Guidance (CFG) ho2022classifier.
• Figure 5: (a) Quantization error of FC2 input activation with different quantization formats across all blocks. (b) Our proposed method for constructing the calibration dataset in Sec. \ref{['subsec:GHT-Aware Learnable Transformation']}, only one layer is plotted here for clarity.