Co-Designing Binarized Transformer and Hardware Accelerator for Efficient End-to-End Edge Deployment

TL;DR

This work addresses the challenge of deploying Transformer models on resource-constrained edge devices by proposing a holistic co-design of binarized Transformers and hardware accelerators. It introduces BMT, a hardware-friendly binarized Transformer with hardware-aware quantization and WTWS, and BAT, a streaming-processor-mixed accelerator optimized for end-to-end binarized Transformer inference. Through a design-space exploration that jointly tunes algorithmic and hardware parameters, the approach achieves substantial end-to-end gains in throughput and energy efficiency while maintaining accuracy and robustness. The results demonstrate strong potential for practical edge deployment, with large improvements over state-of-the-art accelerators and substantial reductions in energy use compared to CPU/GPU baselines.

Abstract

Transformer models have revolutionized AI tasks, but their large size hinders real-world deployment on resource-constrained and latency-critical edge devices. While binarized Transformers offer a promising solution by significantly reducing model size, existing approaches suffer from algorithm-hardware mismatches with limited co-design exploration, leading to suboptimal performance on edge devices. Hence, we propose a co-design method for efficient end-to-end edge deployment of Transformers from three aspects: algorithm, hardware, and joint optimization. First, we propose BMT, a novel hardware-friendly binarized Transformer with optimized quantization methods and components, and we further enhance its model accuracy by leveraging the weighted ternary weight splitting training technique. Second, we develop a streaming processor mixed binarized Transformer accelerator, namely BAT, which is equipped with specialized units and scheduling pipelines for efficient inference of binarized Transformers. Finally, we co-optimize the algorithm and hardware through a design space exploration approach to achieve a global trade-off between accuracy, latency, and robustness for real-world deployments. Experimental results show our co-design achieves up to 2.14-49.37x throughput gains and 3.72-88.53x better energy efficiency over state-of-the-art Transformer accelerators, enabling efficient end-to-end edge deployment.

Figures (12)

• Figure 1: Overview of encoder-based Transformer structure: (a) vanilla Transformer and (b) quantized Transformer.
• Figure 2: Quantization-dequantization computational flow.
• Figure 3: The illustration for WTWS.
• Figure 4: Hardware overview of our proposed BAT.
• Figure 5: Data access pattern for (a) $\textit{$activation \times weight$}$ and (b) $\textit{$activation \times activation$}$ QMMs. (c) The structure of DPU.